Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Text file output generates a huge file

  1. #1

    Default Text file output generates a huge file

    Hi

    I've generated a very simple transformation to obtain a CSV file from a database table. The table is 6 GB and usually the generated CSV using SQL takes 400 MB.
    Using Pentaho Kettle the file filled all my disk space 46 GB !!!!!

    is that a bug?

  2. #2
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Check your output formatting: Content options Fast-Data-Dump and Right-Pad-Fields of Text-File-Output step come to mind.
    So long, and thanks for all the fish.

  3. #3

    Default

    Hi marabu

    Here my settings:
    encoding UF-8
    Right pad fields: checked
    Fast Data dump: checked

    Still generating a bigger than table size file. it makes no sense inflating 6 GB table to 8 GB csv file when exporting from the very same database via SQL command i get a 400 mb file. maybe a bug?

    Thanks for your quick response
    Last edited by Joselitux; 12-17-2015 at 05:52 AM.

  4. #4
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Why would you want to right-pad in a CSV file?
    That only makes sense when producing a fixed format output, so disable padding.
    So long, and thanks for all the fish.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.