Hitachi Vantara Pentaho Community Forums
Results 1 to 9 of 9

Thread: Eliminate Jobs

  1. #1

    Default Eliminate Jobs

    Perhaps a bit controversial, but I propose the elimination of the separation between Jobs and Transformations.

    Basically all the Job steps would be added to Transformations, and all the steps would have "hop types". Some hops would be row-based (like tranformations), others would be flow-based (like jobs). This would help me in an area I regard as one of the more confusing in PDI- interaction between steps in a Job- things like passing files, arguments, rows, etc.

    Something similar to the talenD way of doing things.

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    If I'm not mistaken, everything runs in a single thread in Talend. In Kettle each step runs in a separate thread.
    Hence the clear need for separation.

    You could make an engine that mixes the 2 concepts, but it would be a challenge to make it stick in a GUI. I'm not sure the end-result would be any more transparent.

    That being said, I am looking into ways to make the passing of the results more transparent and explicit. I'm also open to suggestions in that department.

    Matt

  3. #3

    Default

    That sounds reasonable.

    Another thing I would like is for more of the types of steps to be available in both jobs and transformations, but with the clear distinction of one being row-based and one not.

  4. #4
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Like what?

  5. #5

    Default

    Maybe like copy files..
    Today, we had one job entry that do the job...and i planned to add copy files function
    in Java script Mod step...

    I think that JS Mod step can deal with many ''job entry'' functions like :
    - copy files, create file, create foder, delete file, ...

    Rgds

    Samatar

  6. #6

    Default

    another example is XSLT or DTD/XSD validation process...
    sometimes, we need to transform XML stream "à la volée" and thus porting XSLT job entry
    at step level is very helpfull: we did it today in 2.5.2.

    Samatar

  7. #7
    Join Date
    May 2006
    Posts
    4,882

    Default

    Personally the distinction of jobs and transformations has grown on me, I kind of like it . Maybe some entries/steps could exists in 2 forms as step and as job entry, I wouldn't mind that.

    Mxing both would be confusing, and probably not that easy to code, or visualize.

    Regards,
    Sven
    Last edited by sboden; 12-06-2007 at 06:09 PM.

  8. #8
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Same here, if you have to or someone feels a need for it, you can factor out the part that actually performs the work from the job entry and you can create the step. That still wouldn't change the fact that steps run in parallel and job entries are driven by a sequential backtracking algorithm.

    With that I mean that if you for example copy files in a step, say one by one based on some input, you are doing other actions at the same time in parallel. There is no order of execution to be found. I think you will see in a lot of cases that the transformations that do that would end up using one or two steps and nothing after the step performing the action. (copy)

    Matt

  9. #9

    Default

    For me it makes sens how PDI deals with jobs and trans and i am sure that it make the process
    clear when you make the difference between the :
    - Flow : jobs
    - rows transformations

    In our company, it happens that for some project, someone work on transformations and another one on flow process : i really like the separation because it help us :

    - first to build transformations, test it,..
    - and then "package" with jobs

    cOncerning the copy files (i created a JS Mod function at work),we used with getfilesname step
    in a carefull way...but of course it should be possible to use job entry :-)


    Rgds

    Samatar

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.