Hitachi Vantara Pentaho Community Forums
Results 1 to 6 of 6

Thread: Set CSV Input process one by one row

  1. #1
    Join Date
    Mar 2015
    Posts
    5

    Default Set CSV Input process one by one row

    How to set PDI process data using CSV input row one by one (next row if previous row finish processing)

    iam using PDI 5.3 CE
    OS windows 8 LE

  2. #2
    Join Date
    May 2016
    Posts
    280

    Default

    That's not how PDI works, why do you need to work with only one row at a time? If you give us some more details, we may come up with some ideas.
    Something it might work for you is working with different transformations inside a job, the first transformation just reads the CSV and puts all the rows in the CSV in the output. Then in the job you set up that the next transformation receives the output of the previous as input and process the rows one by one.
    Not efficient if you have to process a lot of rows, but if the CSV is not big, it's feasible.
    Regards
    OS: Ubuntu 16.04 64 bits
    Java: Openjdk 1.8.0_131
    Pentaho 6.1 CE

  3. #3
    Join Date
    Aug 2016
    Posts
    289

    Default

    You can easily handle one row at a time by having a job with two transformations A and B. Transformation A can read all the CSV rows and push them to result rows. Double-click on transformation B, go to advance and tick "Execute for every input row?". Then give the fields in Parameters tab under "Stream column name". Voila! Handle each row individually in B. This is useful for example when processing files and each file must be finished from start to finish before the next.

  4. #4

    Default

    @Sparkles,

    1st transformation reads all the records but then how to make sure that when 2nd transformation picking the results from the previous transformation, to which csv input file does it belongs to?
    Regards,
    Dileep
    Mail ID

  5. #5
    Join Date
    May 2016
    Posts
    280

    Default

    Given your questions, Dileep, you really need this: https://www.packtpub.com/big-data-an...tegration-8-ce
    OS: Ubuntu 16.04 64 bits
    Java: Openjdk 1.8.0_131
    Pentaho 6.1 CE

  6. #6
    Join Date
    Aug 2016
    Posts
    289

    Default

    Quote Originally Posted by saidlp13 View Post
    @Sparkles,

    1st transformation reads all the records but then how to make sure that when 2nd transformation picking the results from the previous transformation, to which csv input file does it belongs to?
    Simply add the file name to all rows found in the file? Pass it to result rows and you can pick up each row one at a time in the 2nd transformation and also know which file it came from.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.