Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Update step is performing very slow.

  1. #1

    Default Update step is performing very slow.

    Hi,

    I am using Kettle 'pdi-ce-4.4.0-stable'.

    I am reading data from mongo db, mssql server and in-memory H2 database.
    After computing each minute level data, I am trying to update in-memory H2 database.
    I have attached the KTR file.

    For each luminaire, I need to update 24*60 ie 1440 records in in-memory H2 database.
    For this I am using update step. It is taking about 9-10 minutes to complete Update step. It means 25 seconds for each update. This is too much.
    In H2 db, there are 1,440,000 records in total.

    I am not sure which is causing performance issue.

    http://www.h2database.com/html/performance.html says 406 ms for update.

    I have tried giving number of copies to 10, connection pools to 10, Auto Commit true. But still same result. I have tried giving MVCC=TRUE for H2 database. Still same result.

    Worst part is I have 6000 luminaires. So it will take 10mins * 6000 = 41 Days... This is not at all acceptable.

    Please help me to solve this performance issue.

    -Prakash
    Attached Files Attached Files

  2. #2
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    What happens to your performance if you change the update to a Serialize to file, and then have another tranformation that deserializes from file and does the update step?

    I'm betting that the Update step is not where the bottle-neck is, but possibly one of the other steps (like "read old fact data") and that if you time the two transformations, the Update is quick, but the work used to get there is taking a long time.
    **THIS IS A SIGNATURE - IT GETS POSTED ON (ALMOST) EVERY POST**
    I'm no expert.
    Take my comments at your own risk.

    PDI user since PDI 3.1
    PDI on Windows 7 & Linux

    Please keep in mind (and this may not apply to this thread):
    No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.
    Signature Updated: 2014-06-30

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.