Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: bad performance while loading data into MySQL myisam table using PDI 4.2.1, 4.2.0 and

  1. #1
    Join Date
    Sep 2011
    Posts
    3

    Default bad performance while loading data into MySQL myisam table using PDI 4.2.1, 4.2.0 and

    We encounter very heavy performance issue while using PDI 4.2 to load data from MyISAM engine mysql tables then INNODB engine mysql tables.

    I just do a simple test to load 5000 rows from a myisam table to both myisam destination and innodb destination, the performance gap is huge. while using myisam, i can only handle 17rows/second, while handling innodb, that is 1000rows/second,

    I am using the latest 5.18 jconnector now, but i tried 5.17 and 5.08 also, same result.

    But inside mysql, load 5000 rows into myisam is only 0.02 second, comparing 5000 rows into innodb's 0.17 seconds, it is even faster ?


    Do I really miss something during my configure ?
    I can not find something useful in advance options in my PDI db connection

    Any help will be welcomed, thanks a lot.

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Both numbers (17 & 1000) are horribly low for MySQL. Perhaps there's a network (latency/speed) issue?

    Did you try the bulk loader?

  3. #3
    Join Date
    Sep 2011
    Posts
    3

    Default

    no network issues, all happen on one test server now.

    we need an "Insert/Update" step to load into tables. can that work with Bulk load ?

    Let me try Bulkload first...


    I have a test of Bulk loader, it works great, 16777 records/second,
    But I need a Insert/Update step while loading records into my table, in innodb, it is hundreds records per second, in myisam, it is still 17 records/ second, I made sure autocommit is set to zero.
    Last edited by housy.hou; 12-01-2011 at 03:33 PM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.