Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: How to Dynamically specify the rows to read from a file

  1. #1
    Join Date
    Feb 2013
    Posts
    15

    Default How to Dynamically specify the rows to read from a file

    Hi

    I have a huge file with some date columns. I have to select the rows for some particular dates ( lets say last 2 days )

    Any ideas to read only the data for the specific dates passed (without reading the unmatched rows in file ) as an argument to the transformation

    I have seen "filter string" in text file input step . But we can specify only a hardcoded text there I guess .Not able to give a variable or argument there..

    Any thoughts ?
    Thanks & Regards,
    Siby

  2. #2
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Use parameters and a Get Variables step to provide an absolute date range to your transformation.
    Use Get System Info and Calculator to cope with relative date ranges.
    Eventually you will need a Filter Rows step to select the rows that fall into the specified interval.
    So long, and thanks for all the fish.

  3. #3
    Join Date
    Feb 2013
    Posts
    15

    Default

    Hi Marabu

    When you say " use a filter step"- that means all the rows from the file needs to be read - right ?

    I have a file with more than 20M records and I might need to pull couple of days data (lets say 10K) records - I am looking for a solution where I can pull the records without reading all the 20M rows
    Thanks & Regards,
    Siby

  4. #4
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    With a couple of assumptions you could minimize the amount of text to be scanned e.g. by implementing binary search, but usually you have to read the whole text file, even though you only are interested in specific lines.
    Reading a local textfile sequentially shouldn't give you any headaches.
    I suggest you do the filtering on the remote server, if your network is the bottleneck.
    So long, and thanks for all the fish.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.