Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: a folding@home like cpu consumption policy

  1. #1
    Join Date
    Mar 2007
    Posts
    216

    Default a folding@home like cpu consumption policy

    Hi,

    I am using Kettle Spoon with a Text File Input step.
    My problem is that the file is large (415 MB, 1 800 000 rows, 64 columns).
    So, when I launch a Kettle transformation, Spoon takes 100% cpu time for himself and I have no other PC to continue my job while Spoon is working.
    That's why It would be great if it was possible to have a cpu consumption policy like folding@home's one.
    In folding@home the cpu is always used at 100% but I think that their process has a so poor priority that you can do anything you want without being slow down.

    By the way, I read that v3.0 will have a faster Text File Input step.
    I hope it will be released not to far from now.

    What do you think about it ?

    (Spoon v2.5.0 running on Windows XP Pro SP2)

    a+, =)
    -=Clement=-


    PS : I'm a french guy , I apogilize for mistakes

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    Hi,

    I am using Kettle Spoon with a Text File Input step.
    My problem is that the file is large (415 MB, 1 800 000 rows, 64 columns).
    So, when I launch a Kettle transformation, Spoon takes 100% cpu time for himself and I have no other PC to continue my job while Spoon is working.
    That's why It would be great if it was possible to have a cpu consumption policy like folding@home's one.
    In folding@home the cpu is always used at 100% but I think that their process has a so poor priority that you can do anything you want without being slow down.
    Assumption has always been that you do the big jobs on a dedicated computer I think. Not that you run it in the background while doing other stuff. If you want you can raise a CRQ for priority of Kettle. In ETL you usually want the thing to execute as fast as possible, and not to be nice to other applications.

    By the way, I read that v3.0 will have a faster Text File Input step.
    I hope it will be released not to far from now.
    v3.0 will be faster in general, but Matt built also some special cases in for text input to make it faster+

    You can try v3.0 (it's in one of the sticky posts at the top)... but it's only M1, not complete yet and not as stable as we want it to be. Normally we'd have an M2, RC1 and then final version.... somewhere between august and end of year.

    Regards,
    Sven

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.