Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: PDI 2.5.1 big preview (performance?) issue : no activity

  1. #1
    Join Date
    Mar 2007
    Posts
    216

    Exclamation PDI 2.5.1 big preview (performance?) issue : no activity

    Hi,

    Configuration : Windows XP Pro SP2, Processor 2,02 Ghz, RAM 1 GB, PDI Spoon v2.5.1.GA

    In an "Input Xbase File" step, I try to preview 50 000 lines of a 1,05 GB data source file in DBF format which is on an external USB hard drive.
    Processor goes 100%,
    "Progress Information" complete in less than 1 minute
    Empty table is displayed after 1 minute. Processor at 100% until minute 4.
    My problem : the table is empty, the cursor is in standby mode, but there is no activity (neither hdd leds, neither processor use).
    If I had reduced Spoon window it could have shown a blank screen, with only windows title. How can I change that ?

    a+, =)
    -=Clément=-

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    xbase 1Gb, 50.000 lines preview on a USB stuck lol

    A first hunch would be that the steps tries to read the file until the end, and you probably don't see any lights flicker because of some caching.

    So what is actually your problem? If you want to extract 1Gb of data from Oracle it will also take you some time. For Xbase it probably doesn't matter whether it finds data or not, it will just read the file as in the "good old mainframe days".

    Regards,
    Sven
    Last edited by sboden; 09-28-2007 at 09:37 AM.

  3. #3
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Actually Sven,

    It's a performance problem in PreviewRowsDialog. I fixed this in 3.0.0-RC1.
    The problem is this: the table allocates 50.000 rows, this goes very fast.
    After that, it looks up the TableItem for all the 50.000 items and populates the values.
    However, that leads to 50.000 indexOf() statements internally and I bet it's not indexed.
    I bet as such it's doing 50.000 x 25.000 compares on average: a lot of wasted time.

    Fine for small stuff, not great for larger tables. (fix in revision 5188 for 2.5.2)

    Matt

  4. #4
    Join Date
    May 2006
    Posts
    4,882

    Default

    Cool

    Sven

  5. #5
    Join Date
    Mar 2007
    Posts
    216

    Post

    Hi,

    Quote Originally Posted by sboden View Post
    Cool

    Sven
    I can confirm : that's cool (as ever)
    Even for those who (like myself) bite off more thanthey can chew, there are solutions.
    Good week end for all.

    a+,=)
    -=Clément=-

    PS : french guy including mistakes in his posts.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.