Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: How to improve performance of copying tables

  1. #1
    Join Date
    Jun 2010
    Posts
    11

    Default How to improve performance of copying tables

    Hi,
    I need to move huge Oracle tables into Vertica, over TB of data.
    I tested with small size Oracle schema, which moved about 9000 tables, totaling to 25G or so in size. It took about 5.5 hours. Speed was about 25k-30k/sec.
    I allocated 10g to jvm. I have Vertica commit size set to 10 million rows. (almost all tables were under 10 million rows in size).
    My job is very simple. Loops through a list of tables it reads from a file, selects * from ${tablename} -> inserts into Vertica. There is no transformation on data.

    What can I do to improve performance? Bump up jvm in spoon.sh/kitchen.sh? (side question: should I expect to see same performance when running a job from spoon as from kitchen?)
    If someone has a doc on this, that would be great (I searched forum and found 1 thread that had 2 links to tuning check list, but links didn't work).

    Many thanks!
    Marjana

  2. #2
    Join Date
    Jun 2010
    Posts
    11

    Default running in parallel did not help

    Just wanted to add that I tried running 2 jobs at the same time, but it cut the size of rows/second in half.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.