Hitachi Vantara Pentaho Community Forums
Results 1 to 7 of 7

Thread: PDI Crashes while running large job

  1. #1

    Default PDI Crashes while running large job

    I have a large nested job which calls other jobs and sometimes when I run it in PDI, the entire PDI interfaces crashes and closes. When looking through the logs, the stacktraces says:

    Code:
    ERROR 09-09 09:21:09,795 - Insert Rows Job - java.lang.ArrayIndexOutOfBoundsException: 10
        at java.util.ArrayList.add(ArrayList.java:352)
        at org.pentaho.di.core.gui.JobTracker.addJobTracker(JobTracker.java:108)
        at org.pentaho.di.job.Job.execute(Job.java:565)
        at org.pentaho.di.job.Job.access$000(Job.java:100)
        at org.pentaho.di.job.Job$1.run(Job.java:642)
        at java.lang.Thread.run(Thread.java:662)
    Unexpected error occurred while launching entry [table7.0]
    10
    
    
        at org.pentaho.di.job.Job$1.run(Job.java:648)
        at java.lang.Thread.run(Thread.java:662)
    Caused by: java.lang.ArrayIndexOutOfBoundsException: 10
        at java.util.ArrayList.add(ArrayList.java:352)
        at org.pentaho.di.core.gui.JobTracker.addJobTracker(JobTracker.java:108)
        at org.pentaho.di.job.Job.execute(Job.java:565)
        at org.pentaho.di.job.Job.access$000(Job.java:100)
        at org.pentaho.di.job.Job$1.run(Job.java:642)
        ... 1 more
    INFO  09-09 09:21:09,820 - parent_job - Starting entry [Send Failed Email]
    ERROR 09-09 09:21:09,828 - parent_job - java.lang.NullPointerException
        at org.pentaho.di.job.entries.mail.JobEntryMail.addBacktracking(JobEntryMail.java:1310)
        at org.pentaho.di.job.entries.mail.JobEntryMail.addBacktracking(JobEntryMail.java:1350)
        at org.pentaho.di.job.entries.mail.JobEntryMail.addBacktracking(JobEntryMail.java:1350)
        at org.pentaho.di.job.entries.mail.JobEntryMail.addBacktracking(JobEntryMail.java:1305)
        at org.pentaho.di.job.entries.mail.JobEntryMail.execute(JobEntryMail.java:998)
        at org.pentaho.di.job.Job.execute(Job.java:528)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:667)
        at org.pentaho.di.job.Job.execute(Job.java:393)
        at org.pentaho.di.job.Job.run(Job.java:313)
    Any guesses about what might be causing the problem? I don't think it's my send failed email step since it works for other errors. It almost feels like kettle is running out of memory, but there isn't an OutOfMemoryException. It also does not reproduce consistently (sometimes my jobs all run without errors and every once in a while, kettle crashes with this error).

  2. #2
    Join Date
    Sep 2013
    Posts
    235

    Default

    What is the version of Kettle you are using because based on stack-trace compare to current code in github:

    Code:
    ERROR 09-09 09:21:09,795 - Insert Rows Job - java.lang.ArrayIndexOutOfBoundsException: 10
        at java.util.ArrayList.add(ArrayList.java:352)
        at org.pentaho.di.core.gui.JobTracker.addJobTracker(JobTracker.java:108)
        at org.pentaho.di.job.Job.execute(Job.java:565)
        at org.pentaho.di.job.Job.access$000(Job.java:100)
    Is not even ArrayList now.

    org.pentaho.di.core.gui.JobTracker.addJobTracker(JobTracker.java:108)

    Is version less then 5.x ?

  3. #3

    Default

    Quote Originally Posted by Dzmitry View Post
    What is the version of Kettle you are using because based on stack-trace compare to current code in github:

    Code:
    ERROR 09-09 09:21:09,795 - Insert Rows Job - java.lang.ArrayIndexOutOfBoundsException: 10
        at java.util.ArrayList.add(ArrayList.java:352)
        at org.pentaho.di.core.gui.JobTracker.addJobTracker(JobTracker.java:108)
        at org.pentaho.di.job.Job.execute(Job.java:565)
        at org.pentaho.di.job.Job.access$000(Job.java:100)
    Is not even ArrayList now.

    org.pentaho.di.core.gui.JobTracker.addJobTracker(JobTracker.java:108)

    Is version less then 5.x ?
    Yes, I'm stuck on version 4.3 of pdi. I did search jira to try to see if this was fixed (to see if I could think of a workaround for it).

  4. #4
    Join Date
    Sep 2013
    Posts
    235

    Default

    Sincerely says even it is a bug in 4.3 I suppose nobody will be happy to dig into old code branches, especially if enterprise support is unavailable.
    This may be specific to this particular jobs/transformations organization - so nobody else meet it. And this is indirectly fixed by some kind of refactoring in new versions.
    I can only suggest to upgrade to next versions and check if this issue is annoying work stopper and constantly reproducible.
    At least on 4.8+ but aware of 5.0
    Another option is attempt to simplify jobs/transfromations nesting.

    Hope someone else have a workaround.
    You may describe how is jobs and transformations is organized so someone can suggest how to simplify it.

  5. #5
    njain111 Guest

    Default

    That's a strange behavior.

    But I am not surprised by it. Kettle UI is unstable and it has given such errors to me before as well.
    But during dev and not during execution.

    I would suggest to execute job/transformation via command line (kitchen,pan) and see if you can replicate it.
    If you can't that means it is some kind of instability in Kettle UI and no related to your job/trans or data.

  6. #6
    Join Date
    Sep 2013
    Posts
    235

    Default

    As far as I see in code for Job and from previous post stack-traces - JobTrackers is used to track job execution status and even they are in 'gui' package when it is thrown from Job() code it may happens only on Job execution phase. (I do check only master branch - sorry don't have 4.3 or old ones in my workspace).

    Even it will run on Kitchen or any other command line/schedules - JobTrackers will be used since it is core functionality.

    So it might be unstable Kettle UI only if job is executed from Spoon perspective. From my point of view that is looks like concurrency issue. But once again - who knows?

    Usually for this kinds of issues transformation or job to reproduce attached to the bug where is this issue can be reproduced regularly is almost 100% guarantee this will be fixed. Otherwise - static code analysis 'why it might happens here in some circumstances' is not efficient.

  7. #7

    Default

    Hi

    If you increase the memory of spoon as well as kitchen.bat then it will work properly.

    Regards,
    Rushikesh

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.