Hitachi Vantara Pentaho Community Forums
Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: Filtering lines between an header and a footer in a multirecord file

  1. #1
    Join Date
    Apr 2016
    Posts
    7

    Default Filtering lines between an header and a footer in a multirecord file

    Hi,

    I need help on this topic, please.
    I have a multirecord text file that has different sections. I would split every section and send it in an output file step or something similar.
    The file has a similar structure:

    01 .. data header 1..
    02 .. data detail block1..
    02 .. data detail block1..
    99 .. data footer 1..
    01 .. data header 2..
    02 .. data detail block2..
    02 .. data detail block2..
    02 .. data detail block2..
    02 .. data detail block2..
    99 .. data footer 2..

    Headers and footers have the same beginning characters (01 and 99) and all details have always the same beginning characters (02) but fields and data have different sructures

    How could split the different blocks in different file output?

    Thank you in advance!
    Fernando

  2. #2

    Default

    you have sample rows step, you can try using that. Also you must try doing it with job having multiple transformations to achieve this. I remember I worked on the same scenario 2 years back.

    I was able to achieve with above mentioned notes. Let me know if you have any concerns.
    Regards,
    Dileep
    Mail ID

  3. #3
    Join Date
    Aug 2016
    Posts
    289

    Default

    It would be fairly easy to achieve with a jave code step and one target step per header also. Maybe you could add the header to all the non-header rows and do a switch/case afterwards.

  4. #4
    Join Date
    Apr 2016
    Posts
    7

    Default

    Hi Dileep,

    thank you for your answer. Sample rows step doesn't work, because I don't know exact number of lines in each group. File is generated dinamically.
    Maybe through a Java code is possible to fetch every line and so isolating group of lines between ripetitive header and footer.
    Have you a better idea? It's very strange there isn't a way in steps that accomplish this issue...
    Regards
    Fernando

  5. #5
    Join Date
    Apr 2016
    Posts
    7

    Default

    Hi Sparkles,
    thank you for your answer. I think the same about Java code and swith/case step....but I wanted to avoid it because performance. :-) Files are very big, but I'm be wrong. Anyway I'll try. If you have a better idea please don't hesitate to say me.
    Thanks!
    Fernando

  6. #6
    Join Date
    Apr 2008
    Posts
    4,690

    Default

    If you look through the posts from Marabu (A wonder poster!) he has an example that shows how to increment a counter for each line starting with 01, and then use the running total step to generate a group counter, which can then allow you to process your data by data block.

  7. #7
    Join Date
    Apr 2016
    Posts
    7

    Default

    Quote Originally Posted by gutlez View Post
    If you look through the posts from Marabu (A wonder poster!) he has an example that shows how to increment a counter for each line starting with 01, and then use the running total step to generate a group counter, which can then allow you to process your data by data block.
    Hi Gutlez,
    i am sorry but i was not able to find it. Could give me title or keyword please?
    Fernando

  8. #8
    Join Date
    Apr 2016
    Posts
    7

    Default

    Hi all,

    I made a little step in the solution...I was able to add a sequence to the rows list, and trough Filter step I was able to detect the lines of the different groups.
    So now I have a structure of this kind:

    01 .. data header 1.. 1
    02 .. data detail block1.. 2
    02 .. data detail block1.. 3
    99 .. data footer 1.. 4
    01 .. data header 2.. 5
    02 .. data detail block2.. 6
    02 .. data detail block2.. 7
    02 .. data detail block2.. 8
    02 .. data detail block2.. 9
    99 .. data footer 2.. 10

    And I know that group1 details go from line 2 to line 3 and
    group2 details go from line 6 to line 9, but i don't know how to pass dinamically parameters at the same file text input to filter 1st and 2nd block.
    Could someone help me, please?
    Thank you in advance!
    Fernando


  9. #9
    Join Date
    May 2016
    Posts
    280

    Default

    Take a look a this blog post:
    http://type-exit.org/adventures-with...ues-in-kettle/
    If you can figure out a way to implement something similar to the third method (the script-free solution) possibly it would be the better performance solution.
    Regards
    OS: Ubuntu 16.04 64 bits
    Java: Openjdk 1.8.0_131
    Pentaho 6.1 CE

  10. #10
    Join Date
    Apr 2008
    Posts
    4,690

    Default

    Quote Originally Posted by Iridedoro View Post
    Hi Gutlez,
    i am sorry but i was not able to find it. Could give me title or keyword please?
    Fernando
    This helps:
    https://forums.pentaho.com/threads/1...xed-text-file/

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.