Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: Performance issue while processing 100 ML records from DB to File

  1. #1
    Join Date
    Sep 2015
    Posts
    12

    Default Performance issue while processing 100 ML records from DB to File

    I am using PDI 6.0 Community version. I have very simple transformation which read data from Oracle DB view (Table Input) and insert it to File (Text File output). It is taking more than 5 hours to transfer 100 ML records. So average is 1 ML/3 min. Is this the normal time or there is any scope of improvement? I also tried the Lazy conversion and Fast Dump options. However not helping much.

  2. #2
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    I don't have relevant speed rates for your case, but if you remove Text-File-Output you can measure your retrieval speed.
    And if you substitute Table-Input with Generate Rows you can measure your output speed, too.
    Fast Data Dump avoids padding and other formatting chores, thus can help to save significant amounts of disk space. Not sure about the exact impact on speed rates, though.
    So long, and thanks for all the fish.

  3. #3
    Join Date
    Aug 2015
    Posts
    313

    Default

    Hi,

    Database to Text File output is best approach to load huge data, i am also using this approach. have you tried with Oracle bulk loader ?.

    i have observed another steps (Serialize to file and De-Serialize from file) steps these are also excellent but content will be stored in binary format, this is the only issue with these steps. you can,'t verify if jobs fails, why it was failed..

    Hope you will get responses from experts i am also waiting for this.

    Thank you

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.