Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Maximum number of "steps" per transformation

  1. #1

    Default Maximum number of "steps" per transformation

    Good Morning.

    I am finally completing all development for my POC with Pentaho Data Integration. Right now I am doing most of my step in 3 transformations, and I would like some help.

    Is there a maximum number of steps (joiners, javascripts, db lkps, etc) I should respect on a transformation?

    I am asking this because my main transformation is kind big right now (15 steps) and will have almost 40 when I complete doing the small lookups on it (this is my fact load). Right now I am using a stand alone server, with no parallel option on using version 2.5 on a linux server, and I am getting a 2500 r/s what is really good, considering all the transformations I am doing.

    Thks for all the help,

    Reinaldo A Vasconcellos
    Brazil

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    lol... I normally used a maximum of 4 "steps" in DataStage job

    In kettle... verification will take a lot longer, it will use more memory. So sometimes it's better to split up big transformations into smaller ones... personally I would not go over 30 a 40 steps but I've seen transformations of upto 100 a 150 steps.

    Regards,
    Sven

  3. #3

    Default

    This fact table of mine has a pk formed on 17 columns, and I need to do at least 17 lkps on then to translate legacy code into the dimensions keys.

    Do you think I should do it on a single transformation ? If I do this in a step, I will have to write only once, and I will save lots of I/O (otherwise I would need to write the same table 17 times just to go changing the date, not to mention the amount of disk space I will need for this).

    I can break this lookup part from the main transformation (it only has some filters and a javascript on it + the lookups).

    I will post this tranformation so that you can give me a few tips on it (I am still learning the best practices on Pentaho Datas Integration)....

    Thks for the help

    Reinaldo A Vasconcellos


    PS.: I only realyzed that I could create functions on Java Script at the very end and I did not rebuild my Javascript using it because of time. I will redo my javascripts as soon as I got some spare time (as the performance is doing ok, with 2500 r/s).
    Attached Files Attached Files

  4. #4
    Join Date
    May 2006
    Posts
    4,882

    Default

    It's all hard to say... it all depends.

    Regards,
    Sven

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.