Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: Batch Loading into DB2 - new/modified steps?

  1. #1

    Exclamation Batch Loading into DB2 - new/modified steps?

    We're experiencing an error 968 when posting updates to a fact table located on our DB2 database.


    I researched the issue and found the following post that has a probable solution for batch updating the table with prepared sql statements. Here's the reference with some example code:

    http://forum.java.sun.com/thread.jsp...sageID=3861543

    Looks like a very significant performance boost as well.

    I'd like to adapt this java code into two special purpose steps, one "update/insert" and one "table output" component that can take advantage of this logic for DB2 fact table updates.

    I see that there is a source code stub for steps or I can work off of the current table output and update/insert steps, maybe even inherit and override some methods and properties.

    If the code is portable to other JDBC drivers, then maybe this could be added as a seperate method to the existing steps and toggled off and on in the UI using a checkbox?

    Does anyone have any other guidance or suggestions. Or is anyone who has written steps before interested in helping to adapt this?

    I'm not sure how difficult this will be, but I hope that the post I found above on Sun's java forums is a good start.

    I'm going to checkout the kettle code this weekend and see what I can do, but would certainly appreciate some pointers or assistance from someone who's "been there, done that."
    Sincerly,

    Jon Roberts

  2. #2

    Red face oops? - batch update option in table output step

    No sooner had I posted this than I noticed the following checkbox in the table output step (see attached image).

    I'll research this to see if it's already using the batch update like I think it might be.

    If that's the case, then I'll verify and make sure that we're not really running out of space on our server before I pursue it further.
    Attached Images Attached Images  
    Sincerly,

    Jon Roberts

  3. #3
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    The "Table Output" step is indeed using batch updates as described in the Sun forum as well as commit in "batches" (every x rows)

    And obviously a million rows is peanuts these days :-)

    Matt

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.