Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: make "batch inserts in table output" generate single multi-rows insert statement

  1. #1

    Default make "batch inserts in table output" generate single multi-rows insert statement

    Hello All,

    The option "Use batch update for inserts" in table output step seem to generate a group of insert statements, as opposed to "single multi-row insert statements", as stated in wiki
    "Enable if you want to use batch inserts. This feature groups inserts statements to limit round trips to the database.".

    But this feature seem to have been bottleneck in terms of performance in the columnar db platforms like Amazon Redshift.
    So I would like to explore, if there is anything that could make the table output step generate the single multi-row insert statements.

    Please share your thoughts or let me know, if I am missing something...

    Thanks,
    Sujen

  2. #2
    Join Date
    Apr 2007
    Posts
    2,010

    Default

    Yes, you can do this in mysql by adding the rewriteBatchedStatements true property to the connection.
    As the JDBC driver does this it'll depend on the database you are using.

  3. #3

    Default

    Thanks codek for the reply back.
    Yes, rewriteBatchedStatements works for mysql, but looks like there is nothing such for Redshift(which is based on postgres engine) driver.

    Have been in a real pain, using table output step with redshift. Had to convert all of the write steps, into Redshift bulk load(Using S3 and Copy Command).

    Thanks again for the comment.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.