Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: ็How to read LARGE CSV file?

  1. #1
    Join Date
    Oct 2017
    Posts
    1

    Default ็How to read LARGE CSV file?

    Hi, I have a problem I create transformations work with CSV File Input, and tested it with a small file (~ 100 records).
    But when I use file with 1.7 million rows (with 13 colums ~ 170 mb), it didn't work at all.
    Program doesn't crash or stopped working but it didn't work any more.

    When I observed working progress it's seem like each steps have its stop after it read 30,000-40,000 rows.

    Do you have any suggestions or work around with this problem?

    Thank you in advance.

  2. #2
    Join Date
    Sep 2011
    Posts
    152

    Default

    There should not be any problem with CSV Input step as i have tried it for aroound 20Million reocrds.

    What are the other steps are there in your transformation ?

  3. #3
    Join Date
    Mar 2017
    Posts
    4

    Default i have the same problem

    Hello, i have the same problem and i think is the CSV STEP, because i use the preview to chek around 20.000 rows and it dont show, their limit is around 13000, so i don´t know where is the problem.

  4. #4
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Put your CSV-File-Input step in an otherwise empty transformation to see if the problem really is with this step.
    If there's no external problem (cpu, memory, disk space, ...), there's a good chance one of the other steps in your design is blocking the dataflow.
    It's possible to produce a deadlock if your graph contains a mesh closed by a Merge-Join, after all.
    In that case just watch outgoing row numbers of non-leaf steps in Spoon step metrics when the transformation comes to a halt.
    Compare row numbers with your configured transformation rowset size and you should see what's wrong with your design.
    So long, and thanks for all the fish.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.