Hitachi Vantara Pentaho Community Forums
Results 1 to 11 of 11

Thread: "Block this step until steps finish" doesn't work

  1. #1
    Join Date
    May 2013
    Posts
    11

    Default "Block this step until steps finish" doesn't work

    Hi everyone,

    below shows a simple transformation with only reading and writing to tables. But this first line must be run completely till finish, then the second at last the third line, but when I run this whole transformation all the "block" steps don't work at all. Actually I get the transformation running like described below:
    This first line would be run and simultaneously the second line and the third line transformations are also run as if these two steps wouldn't exist.
    Here I got big big confusion, Why these "block" steps don't block the flow?
    I use kettle 4.4.0

    Thanks in advance!
    Attached Images Attached Images  

  2. #2
    Join Date
    Apr 2008
    Posts
    1,771

    Default

    The step "wait" block the flow of the data, not the connection/locking of tables in database.
    In your example, the best approach would be to create 3 transformations and place them inside one job.

    Steps within a transformation are not sequential, they are all initiated at the same time!
    Only transformations within Jobs are "truly" sequential.
    -- Mick --

  3. #3
    Join Date
    May 2013
    Posts
    11

    Default

    I did't know that.Thanks for your reply.

  4. #4
    Join Date
    Oct 2010
    Posts
    369

    Default

    add a hop between T.I 6 and wait 16 and bwtween liqv_exp and wait_for_above to make it sequential

  5. #5

    Default

    In transformation all process are working parallel and in your above image,you are not join hope between 16 to wait16 and liqv_exp to wait .In above image

    first you add hope in between them then it work properly.

    Other solution is to create three transformation t1,t2,t3 .In T1 Transformation-: you add first row process ,another T2 transformation: you add second process and T3 transformation:- you add third process then it work properly. And add three transformation in one job .

    Rushikesh

  6. #6
    Join Date
    Feb 2011
    Posts
    840

    Default

    I think I can make this a little brighter...

    Transformations in PDI don't have a "logical sequence", but a "stream sequence". Data starts from somewhere, then goes on following hops.
    If you need something sequential, you're going to need jobs, instead.

    So, applying to your example, I'd create a job, put that "delete duplicates" SQL code on the job right after the "Start step", and then follow on with the transformations - one for each row, no need for the "Block".

    I have one example where that block is quite useful:
    Name:  2013-07-10_14h16_17.jpg
Views: 4323
Size:  33.1 KB

    Here, the Block PEPR and Block POPR are used to make the stream wait until previous Excel Writer output steps finish - it's a nice touch for having several inputs writing to a single XLS/X file without breaking up to several transformations. The block will only allow data stream from input to go on for the Excel Writer once the previous Excel Writer has finished with the file.

    Hope this helps
    Join us on IRC! =)

    Twitter / Google+ / Timezone: BRT-BRST
    BI Server & PDI 5.4 / MS SQL 2012 / Learning CDE & CTools
    Windows 8 64-bit / Java 7 (jdk1.8.0_75)

    Quote Originally Posted by gutlez
    PLEASE NOTE: No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.

    I'm no expert.Take my comments at your own risk.

  7. #7
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    So far, I've only found one use-case for the Block Until Steps finish step.

    I would do Joao's example as distinct transforms, together in a job - it adds to the readability.

    However, if you are reading from and writing to the same Excel file in one transform (eg. Reading data from a spreadsheet, merging with with data from a CSV, and writing to another tab), then the Block Until Steps Finish step is needed.
    **THIS IS A SIGNATURE - IT GETS POSTED ON (ALMOST) EVERY POST**
    I'm no expert.
    Take my comments at your own risk.

    PDI user since PDI 3.1
    PDI on Windows 7 & Linux

    Please keep in mind (and this may not apply to this thread):
    No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.
    Signature Updated: 2014-06-30

  8. #8
    Join Date
    Feb 2011
    Posts
    840

    Default

    in my case, I'm reading from 4-5 different files, writing to one final file, in different sheets/tabs.
    I even thought about sepparating them in different transformations on a job, but thought that'd add up too many files =p
    Join us on IRC! =)

    Twitter / Google+ / Timezone: BRT-BRST
    BI Server & PDI 5.4 / MS SQL 2012 / Learning CDE & CTools
    Windows 8 64-bit / Java 7 (jdk1.8.0_75)

    Quote Originally Posted by gutlez
    PLEASE NOTE: No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.

    I'm no expert.Take my comments at your own risk.

  9. #9
    Join Date
    Apr 2008
    Posts
    1,771

    Default

    Hi there!
    I have used the BLOCK step in the same scenario.
    The reason why I did not use 2 Jobs is that I wanted to use variables for the entire Transformation.
    This transformation was inside a Job to get some variables which was inside another job to get another set of (looping) variables.
    -- Mick --

  10. #10

    Default

    Quote Originally Posted by Mick_data View Post
    The step "wait" block the flow of the data, not the connection/locking of tables in database.
    In your example, the best approach would be to create 3 transformations and place them inside one job.

    Steps within a transformation are not sequential, they are all initiated at the same time!
    Only transformations within Jobs are "truly" sequential.
    Even when I use ONE job, that call several transformations (each transformation calls a Sub transformation passing parameters)...the parameters drive which TAB of the xlsx the data goes in. This still runs into locking issues on the file

  11. #11
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    As has been mentioned Transformations (including Sub Transformations) run completely in parallel. If you have two steps trying to write to the same file, it will cause the issue.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.