Hitachi Vantara Pentaho Community Forums
Results 1 to 6 of 6

Thread: Reading and Executing csv files

  1. #1

    Default Reading and Executing csv files

    I have a transformatiion that reads a csv file and loads the data into a database table.

    When I process the transformation, no data is being loaded.

    I have enabled debugging but I don't see any errors.

    When I preview the Text File input step,

    This is what I get
    Name:  FileInputStep.jpg
Views: 23
Size:  8.5 KB.

    Can some some please help me out here. I am getting frustrated but I cannot give up.

    I have attached the necessary files to help me come up to a solution.

    Thanks,

    Ron
    Attached Files Attached Files

  2. #2
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    From your logs:
    2014/04/02 16:44:03 - Read 711.0 - Signaling 'output done' to 1 output rowsets.
    2014/04/02 16:44:03 - Read 711.0 - Finished processing (I=3306, O=0, R=0, W=0, U=1, E=0)
    2014/04/02 16:44:03 - Filter rows.0 - Signaling 'output done' to 2 output rowsets.
    2014/04/02 16:44:03 - Filter rows.0 - Finished processing (I=0, O=0, R=0, W=0, U=0, E=0)
    2014/04/02 16:44:03 - Trim Operation.0 - Signaling 'output done' to 1 output rowsets.
    2014/04/02 16:44:03 - Dummy (do nothing).0 - Signaling 'output done' to 0 output rowsets.
    2014/04/02 16:44:03 - Trim Operation.0 - Finished processing (I=0, O=0, R=0, W=0, U=0, E=0)
    2014/04/02 16:44:03 - Dummy (do nothing).0 - Finished processing (I=0, O=0, R=0, W=0, U=0, E=0)
    So EVERYTHING is being filtered out in your text input step.
    (PSST... Look for **** NO SRP711 REPORT DATA ****)

    You're also doing your filter in a REALLY strange way: If Receiving_Branch = 'Spacer' then Dummy Else <process> ?!
    Last edited by gutlez; 04-02-2014 at 05:14 PM.
    **THIS IS A SIGNATURE - IT GETS POSTED ON (ALMOST) EVERY POST**
    I'm no expert.
    Take my comments at your own risk.

    PDI user since PDI 3.1
    PDI on Windows 7 & Linux

    Please keep in mind (and this may not apply to this thread):
    No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.
    Signature Updated: 2014-06-30

  3. #3

    Default

    I saw that, but when I removed the filter and executed the transformation. Nothing is still being inserted into the database table.

    Attached is the log snippet.Name:  Logs.jpg
Views: 21
Size:  30.7 KB

    I really donot what is happening when I try to preview the Read 711 step,

    something strange is happening.

    Thanks,

    Ron

  4. #4
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    The fact that the line reads:
    2014/04/02 16:44:03 - Read 711.0 - Finished processing (I=3306, O=0, R=0, W=0, U=1, E=0)
    still tells me that NOTHING is making it out of your Text File Input - this means that something is happening with the Filter line.
    It is pulling 3305 lines from the Text File(s), but NONE of them are acceptable to move onwards.

    I=Input: Items brought into the stream from outside
    O=Output: Items put somewhere outside of the stream
    R= Read: Items picked up from the stream from a previous step
    W=Written: Items put into the stream for another step to use
    U= Updated: Items changed in this step (Indicates number of files read in the Text File Input)
    E= Errors: Problems with data in this step

    ---
    Please do not post pictures for logs - they are harder to read, and cannot be copied / pasted to bring more information to your attention.
    Last edited by gutlez; 04-02-2014 at 05:41 PM.
    **THIS IS A SIGNATURE - IT GETS POSTED ON (ALMOST) EVERY POST**
    I'm no expert.
    Take my comments at your own risk.

    PDI user since PDI 3.1
    PDI on Windows 7 & Linux

    Please keep in mind (and this may not apply to this thread):
    No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.
    Signature Updated: 2014-06-30

  5. #5
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    Quote Originally Posted by rejalu@sharperimageonline View Post
    I saw that, but when I removed the filter and executed the transformation. Nothing is still being inserted into the database table.
    Another thing to do when troubleshooting is to make sure your systems help you, not work against you:
    1. Turn off "Ignore Errors" in your Text File Input
    2. Turn off "Skip Error Rows" in your Text File Input
    **THIS IS A SIGNATURE - IT GETS POSTED ON (ALMOST) EVERY POST**
    I'm no expert.
    Take my comments at your own risk.

    PDI user since PDI 3.1
    PDI on Windows 7 & Linux

    Please keep in mind (and this may not apply to this thread):
    No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.
    Signature Updated: 2014-06-30

  6. #6

    Default

    Quote Originally Posted by gutlez View Post
    Another thing to do when troubleshooting is to make sure your systems help you, not work against you:
    1. Turn off "Ignore Errors" in your Text File Input
    2. Turn off "Skip Error Rows" in your Text File Input

    I removed the filter in Text File input step and I did turn off the Ignore Errors and Skip Error Rows in the same Text File Input step and I had some errors to debug, I was able to fix my the errors and transformations runs.

    2014/04/02 19:34:30 - LandingLocal - Connection to database closed!
    2014/04/02 19:34:30 - Save to Landing 2.0 - Finished processing (I=0, O=3304, R=3304, W=3304, U=0, E=0)
    2014/04/02 19:34:30 - Spoon - The transformation has finished!!

    Thank you for all the necessary information you provided and pointing me in the right direction.

    Thanks alot for your help,

    Thanks and Regards,

    Ron

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.