Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: Text File Output Header Appending

  1. #1

    Default Text File Output Header Appending

    I'm trying to append the text output file but seems like header is missing, so what I did for this,

    1. Checked if the file exits
    2. If it doesn't exist Icreated and initial CSV file
    3. If it existed then read that CSV into a Serialize file
    4. And then appended the read file and the transformed file


    But the issue is I cannot have a select file after reading the file as my columns are dynamic some times it has 2 and somtimes it has more than 10, so basically it varies. Now, while appending i"m getting an error stating
    Code:
    We detected rows with varying number of fields, this is not allowed in a transformation.  The first row contained 0 fields, another one contained 6 :
    Please help me on how to achieve this.

  2. #2
    Join Date
    Oct 2014
    Posts
    15

    Default

    There are many solutions to your problem:
    You don't need to read the file, can't you use the content -> append function in the "Text file output" to continue to write to the file if it already exists otherwise it create it?

    Otherwise the easiest solution is to always add empty columns in the file to the maximum fields, if that is known.
    You can also concat the fields to a row before join the streams. Then the number of fields is always one.
    Otherwise you can use the "File Metadata" or "Metadata structure of stream" to determine the number of fields in both streams and then add the number of fields missing.
    Both streams need to have the same number of fields to join them.

  3. #3
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    There's an issue with Text-File-Output when option "Append" is used.
    While that issue was closed as fixed for version 5.1 I can't say it really is.
    So long, and thanks for all the fish.

  4. #4

    Default

    Quote Originally Posted by marabu View Post
    There's an issue with Text-File-Output when option "Append" is used.
    While that issue was closed as fixed for version 5.1 I can't say it really is.
    Any other alternate way to append the rows by the way the columns are dynamic for each file, so, technically i can't fix that. Anyways let me once also check this out in v5.1.

  5. #5

    Default

    Quote Originally Posted by marabu View Post
    There's an issue with Text-File-Output when option "Append" is used.
    While that issue was closed as fixed for version 5.1 I can't say it really is.
    So, what I figured out with "text file output" is that - It has 2 behaviours, if header option is checked then it will insert the header whereas if you also check the append then it won't. So typically you need to have 2 out's one with "header" checked when creating the file for the first time and another with checking "append" and unchecking the "header", by this, we can append rows with the header. For me, it look's like the documentation is not clear on this.

    Thanks for all your suggestions.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.