Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: text file output with different column (header, detail & footer)

  1. #1
    Join Date
    Oct 2012
    Posts
    11

    Default text file output with different column (header, detail & footer)

    Hi All,

    How to generate file (output text) from mongodb query ?

    output:
    ====
    0|22-MAY-2015 17:28:42|2
    1|24-APR-2015|1442011|22-MAY-2015
    1|11-MAY-2015|1442128|22-MAY-2015
    2|EOF

    explanation:
    from the output above, there are 4 line/row.
    - 1st line:

    • 1st column: '0' indicate header
    • 2nd column: time stamp
    • 3rd column: row count of mongodb query

    - 2nd & 3rd line:
    • 1st column: '1' indicate details
    • 2nd column: field from mongodb row
    • 3rd column: field from mongodb row
    • 4th column: field from mongodb row

    - last line
    • 1st column: '2' indicate footer
    • 2nd column: static string


    your help would be very appreciated.

    Regards,
    dmaf

  2. #2
    Join Date
    May 2014
    Posts
    358

    Default

    In one stream, prepare the header row. In second, load the query from Mongo. In the third, prepare the EOF line (probably as a constant?).
    Then use the Append streams step twice to append the streams in the correct order (header, data, footer) and dump the rows to a text file.

  3. #3
    Join Date
    Oct 2012
    Posts
    11

    Default

    Quote Originally Posted by Lukfi View Post
    In one stream, prepare the header row. In second, load the query from Mongo. In the third, prepare the EOF line (probably as a constant?).
    Then use the Append streams step twice to append the streams in the correct order (header, data, footer) and dump the rows to a text file.
    Hi Lukfi,

    FYI, Append streams step wiki said "..the input data coming from both steps has to be identical..".
    in my output file (header, detail and footer) are different field & type.

    I have resolved with this:
    Name:  resolved pdi.jpg
Views: 44
Size:  20.5 KB
    Thanks for your response.

  4. #4
    Join Date
    May 2014
    Posts
    358

    Default

    Yes, they have to be identical, but I thought that would not be a problem since it would always be either one string already delimited or the same number of fields. Having three file outputs works too, but your solution will fail. The Group By has to read all rows from the MongoDB input, but MongoDB input also has to send rows to the other branches, which will not start processing until the first Group By works through all data. Once you have more than 10 000 rows from the DB, the buffers will fill up, the blocked branches won't be accepting any new rows and the Group By won't get any more because the DB input cannot send more rows out.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.