Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: preformance degradation of Get data from XML

  1. #1
    Join Date
    Jun 2009
    Posts
    2

    Unhappy performance degradation of Get data from XML

    I've seen this behaviour previously in 3.2 and it's the same in 4.0. The transform reading data from multiple XML files rapidly becomes slower with the increased number of files.

    The number of rows read per second is inversly dependent on the number of files being proccessed - the step reads the twice number of files two times slower.

    If I double the number of records in each file and run on half number of files I get better performance per file, but the inverse dependency on the number of files remains.

    This looks very strange as imho the processing of each file should be done by itself alone and then the step goes to the next file.

    I've prepared a test case and attach for those interested: the transform and two sets of source datafiles (20 and 40 records per file) in subdirs.
    Copy the files to the transformation folder, run, see results.

    My results are shown below: on the X axis is total number of records in files, on the Y axis the rows per sec. Blue dots are for 2-record files and red dots for 40-record files.


    Is this a bug or is it by design? How can we work around this?

    Regards,
    Maxim

    Updated: sorry, i have a problem with uploading of attachments here. So I've put the file to rapidshare. Here is the link: http://rapidshare.com/files/411152494/testcase.zip
    Last edited by maxvar; 08-05-2010 at 05:15 AM. Reason: spelling errors...

  2. #2

    Cool

    Hi Maxim,

    this a bug already fixed :

    http://jira.pentaho.com/browse/PDI-4212

    Take care

    Samatar
    Samatar

  3. #3
    Join Date
    Jun 2009
    Posts
    2

    Default

    Quote Originally Posted by shassan2 View Post
    this a bug already fixed...
    many thanks! works like a charm now =)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.