Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: Rows lost between steps (kettle 3.2.1)

  1. #1
    Join Date
    Mar 2007
    Posts
    8

    Default Rows lost between steps (kettle 3.2.1)

    Hi,

    We have been running kettle 2.3.1 during 3 years processing big amounts of data without any problem.

    But recently we have detected some random problems of rows lost between steps (specially in set variables steps: previous step outputs 1 row but the set variables steps indicates that there are 0 input rows. We have also detected the same random behaviour in javascript steps)

    We have upgraded our servers adding more processors, memoy and also we have upgrade the java version (now we are using 1.6.0_12 java version)

    Can be this problems related to this upgrade?

    Any help would be appreciated

    Thanks in advance
    Manel Gimeno
    Last edited by mgimeno; 09-06-2010 at 06:55 AM.

  2. #2
    Join Date
    Mar 2007
    Posts
    8

    Default

    We are using kettle 2.31 not kettle 3.2.1 as I wrote in the title

    In addition we are using 64 bit servers

  3. #3
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    This is most likely the bug you are hitting: http://www.javaforge.com/issue/3692

    The change of JVM and or addition of CPUs probably makes the race condition surface. It was an issue with minimal amounts of rows (like <20) where one step finished faster than the next one could start. A faster CPU could and would trigger it.

    The fix is mentioned in the bug so you could patch BaseStep and compile it all OR get 2.4.0.

    Good luck,
    Matt

  4. #4
    Join Date
    Mar 2007
    Posts
    8

    Default

    Hi Matt,

    Thanks for the answer.

    The behaviour we have detected is exactly the same described in the bug. We will try to apply the patch or upgrade to a new version.

    In relation to upgrade to a new version I have a question:

    In our jobs/tranformations (version 2.3.1) we use intensively the java script step. If we upgrade to 4.0.1 the java script steps will continue working as they are if we use the compatibility mode of the modified java script step or it will be needed to do additional modifications?

    Regards,
    Manel Gimeno

  5. #5
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    You might need some modifications left and right.
    Check the upgrade guide: http://wiki.pentaho.com/display/EAI/...2.5.x+to+3.0.0

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.