Hitachi Vantara Pentaho Community Forums
Results 1 to 11 of 11

Thread: 'Block this step until steps finish' doesn't work

  1. #1
    Join Date
    May 2011
    Posts
    15

    Exclamation 'Block this step until steps finish' doesn't work

    hi,

    I am facing an issue with PDI, the step 'Block this step until steps finish' deosn't work as expected.

    My transformation has two flows

    flow-1) reads a file and writes to a table
    steps : (1) Text file input (2) Table output

    flow-2) it waits for flow-1 to complete. Once flow-1 is complete it updates the tables.
    steps: (1) Block this step until steps finish (2) Execute SQL script

    However, the update doesn't happend after table loading is completed. Possibly, the blocking step doesn't wait for flow-1 to complete.

    Please see the attached zip file for the PDI transformation and input file.

    table used:
    Table "public.test_ex1"
    Column | Type | Modifiers
    --------+---------------+-----------
    id | integer |
    name | character(20) |
    Attached Files Attached Files

  2. #2
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    Use a job.

    When you want step completion flow control, you want a job.

  3. #3
    Join Date
    Aug 2011
    Posts
    1

    Question

    So is this a bug? I also have tried to use the blocking steps and have ended up using jobs instead (as suggested). This has caused simple transformations to become needlessly complicated; because they now have to be wrapped in a job with either more job steps or an additional transformation. At best, I have twice as many files to manage; sometimes 3x.

  4. #4
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Not really, it's simply a case of folks not understanding the "execute for each row" feature in the "Execute SQL" step.

  5. #5
    Join Date
    Jul 2011
    Posts
    1

    Default

    Thanks Matt. I am aware of the "execute for each row" solution when using the Execute SQL step, but what can I do if I am using two Insert/Update steps with a blocking step in between to manage the dependencies? So in my case the flow is 1) Text file input, 2) Insert/Update parent table, 3) Blocking Step, 4) Insert/Update child table.

  6. #6
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    For parent-child tables it's easier to make the transformation "database transactional" (transformation settings, Misc tab).

  7. #7
    Join Date
    Jul 2009
    Posts
    476

    Default

    Matt,

    Does "Make the transformation database transactional" override all of the commit settings in the various steps, i.e. the transformation issues only one commit on each connection when it completes successfully, or a rollback if the transformation fails? (I'm thinking about insert or update steps that are configured to commit every so often, such as every 100 rows. Are these step-specific settings overriden when the entire transformation is database-transactional?)

    Does it work well with Postgres? Including Insert, Update, Execute SQL Script, and all other steps that use database connections?

    I didn't know about this setting, and it might be useful for us in certain situations.

    Rob
    Last edited by robj; 11-22-2011 at 11:08 AM. Reason: Clarify what I mean by "override"

  8. #8
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Yes. Should work on Postgres, not with Execute SQL Script (Dynamic SQL, what can I tell you), rest should be fine.

  9. #9

    Default

    Did you find any resolution for your problem? I am having the exact same problem.
    My understanding is all plugins in same transformation get started at parallelly. Then, why they have provided plugin "block this step until steps finish" for transformation? This doesn't make any sense to me.

  10. #10

    Default

    You're correct. Every step copy is executed in parallel in the transformation.

    The "Block this step until steps finish" step is generally a good indicator that a transformation should have been designed differently, but there are always that 0.0001% of cases where an exception to the rule is justified for that specific case.

  11. #11
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    Quote Originally Posted by samanl@hotmail.com View Post
    Then, why they have provided plugin "block this step until steps finish" for transformation? This doesn't make any sense to me.
    Because some file types don't like you reading and writing to the same file at the same time. That's the only use case that I have found for it.

    Excel Document:
    (Tab)Parent
    (Tab)Child

    If you are reading from the tab Parent to get keys for validating records that are going to be added to tab Child, the "Block" step is needed.

    But that's the only one that I've found so far...

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.