Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: PDI 5 vs PDI 4.3.0 performance

  1. #1

    Default PDI 5 vs PDI 4.3.0 performance

    Has anyone else experienced speed issues with PDI 5?

    I'm running a update function which updates a single column on a postgres table (the lookup and the column itself are indexed).

    Running the ktr on 4.3.0 I can achieve updates speeds in excess of 1,500 rows per second, whilst in PDI 5, I get just 54 rows per second from the same ktr and the same table.

    Both versions have been given 10 gig of memory to run, and I'm running both on the same machine Mac Mini running OS X 10.9.

    Am I missing anything?

  2. #2
    Join Date
    Apr 2008
    Posts
    1,771

    Default

    I would check:
    1. options in connection
    2. driver version

    Are those identical in both PDI versions?
    -- Mick --

  3. #3
    Join Date
    Sep 2011
    Posts
    1

    Default

    I installed PDI 5.0.1 (in a new repository) and had my update function go down from 1,300 records/sec on 4.2 to 250 records/sec on 5.0.1 using an oracle database.

    I am using java 1.7.15 and have tried several JDBC drivers for oracle as well as the one that comes with 5.0.1. All with the same results. Tried using connection pooling also.

    I also noticed that 5.0.1 uses much more memory and takes over a windows box running XP SP3 with 4GB of memory,and 5.0.1 hogs up to 95% of the cpu.

  4. #4
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Well, if any of you have a simple reproduction case, please make sure to file it in JIRA so we can have a look at it.
    Thanks in advance!

  5. #5
    Join Date
    Dec 2012
    Posts
    8

    Default

    I created a simple test case comparing 4.4.0.2 to 5.0.1 focusing on the DB Lookup step. The issue is visible in both PostgreSQL and SQL Server, though it has a much larger impact on PostgreSQL. I didn't see any specific changes made to the DB Lookup step that would cause a performance change.

    http://jira.pentaho.com/browse/PDI-11363

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.