Hitachi Vantara Pentaho Community Forums
Results 1 to 7 of 7

Thread: Help with solution to read in 100s of csv files and store in DB2 Database for bootstr

  1. #1
    Join Date
    Jun 2013
    Posts
    21

    Default Help with solution to read in 100s of csv files and store in DB2 Database for bootstr

    Hello,

    Does anyone have any examples (or can point me to them) of using pentaho to read in many csv data files and store the contents of the files into a DB ?

    It would be great if we can use this tool for said activity

    Thanks in advance

  2. #2
    Join Date
    Nov 2008
    Posts
    271

    Default

    Do all the files have the same metadata structure, i.e. same fields in the same order?
    Andrea Torre
    twitter: @andtorg

    join the community on ##pentaho - a freenode irc channel

  3. #3
    Join Date
    Jun 2013
    Posts
    21

    Default

    Yes they do. And similiar file names where only date and version changes

  4. #4
    Join Date
    Apr 2008
    Posts
    4,683

    Default

    Someone posted a set of Jobs and transforms a while ago (maybe a year?!) that took a series of files and loaded them to a DB:
    1) tablename.ddl
    2) tablename.column
    3) tablename.csv

    The job makes sure that all 3 files exist, then creates the table based on the DDL, and reads the CSV file based on the CSV definition in .column. That CSV is then written to the database table. The column file configures the CSV step in a Transform using metadata injection, so it needs to know all the format setups, etc.
    Last edited by gutlez; 05-14-2014 at 02:39 PM.

  5. #5
    Join Date
    Apr 2008
    Posts
    4,683

    Default

    Quote Originally Posted by kmr View Post
    Yes they do. And similiar file names where only date and version changes
    That makes my previous reply pointless...
    You should be able to find enough samples in the sample directory to get you started on the steps you need to employ.

  6. #6
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Clear the first hurdle by using Text File Input instead of CSV Input.
    You can process all the files by specifying a regular expression for the filenames that way.
    So long, and thanks for all the fish.

  7. #7
    Join Date
    Jun 2013
    Posts
    21

    Default

    OK thanks. I will give that a go. I had been looking a the CSV input and it as not working as expected.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.