Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Blocking Step not working?

  1. #1
    Join Date
    Aug 2012
    Posts
    3

    Default Blocking Step not working?

    I'm new to Kettle and ETL tools. I have a flow where I'm writing some data to a temporary Postgres table 1000 rows at a time, then I have a "Blocking Step", then I have a query that pulls data out of that temp table to put it in another table. The query step after the blocking step only pulls a portion of the rows that were inserted, which I guess is the last set of rows that were inserted. I see the "pass all rows" checkbox under the Blocking Step, but that doesn't seem to have any effect.

    Does anyone know what i'm doing wrong, or if there's a bug? Am I misunderstanding what the Blocking Step is supposed to do?

    I attached a screenshot of my flow with the part I'm talking about here circled in red.

    Thanks in advance.
    Attached Images Attached Images  

  2. #2
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Actually you have a sequence of transformations.
    So why not create a job and put the three transformations in there?
    So long, and thanks for all the fish.

  3. #3
    Join Date
    Aug 2012
    Posts
    3

    Default

    Excellent advice. I didn't understand the relation between transformations and jobs.

    For anyone that makes the error I made:

    Split your overall process up into separate transforms. In this example, the first transform is to extract data from Active Directory and store it in a CSV by running a batch process. The second is to pull the data from that CSV and put it into a temporary table. The third is to pull data from the temporary table and enter it into a final table.

    Then create a job that runs each of these transforms in a row.

  4. #4
    Join Date
    Aug 2012
    Posts
    3

    Default

    Excellent advice. Thank you. I didn't understand the relation between jobs and transforms.

    For those who make the same mistake I did:

    Split your overall objective into separate transforms. In this case, my transforms are 1) Extract data from Active Directory and store in a CSV file (using a batch process to do this). 2) Pull data from the new CSV file and insert it into a temporary table. 3) Query the temporary table, massage the data, and insert it into a finalized table.

    Put each of your transforms into a separate Transform file, then create a new Job that executes each Transform in sequence. There is a step called "Transform" in the Job Design panel.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.