Hitachi Vantara Pentaho Community Forums
Results 1 to 7 of 7

Thread: Issue with setting up error handling

  1. #1
    Join Date
    Feb 2017
    Posts
    7

    Default Issue with setting up error handling

    I fed bad data into csv input step to make it fail and then to capture error in the error logging step. Attached is my ktr and test file.
    When parsing fails, the bad data doesn;t flow to the error handling path.

    Any help is appreciated. Thanks.

    Quote Originally Posted by pdi View Post
    Thank you very much gutlez. I followed your instructions and the transformation was successful. Here is a screen capture of my steps. The TFI - error handling step shows the rows that were causing the transformation to fail.
    However, the Text file output - Error handling 2 step does not contain any error logs. Yet, without it, the transformation fails. I am confused by this phenomenon. If possible, could you please shed some light on this for me.
    Again I appreciate your guidance.
    Attached Images Attached Images  
    Attached Files Attached Files
    Last edited by jasonjo; 02-27-2018 at 05:34 PM.

  2. #2
    Join Date
    Feb 2017
    Posts
    7

    Default How to handle parsing errors in the csv data file.

    I tried a lot but unable to find a solution to capture the bad data to a log while reading a csv file. Attached is the ktr, test bad data and error screenshot.
    Attached Images Attached Images  
    Attached Files Attached Files

  3. #3
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    Please take the time to *READ* your logs.
    The error is not happening with the Text File Input step, it's happening on the Write to Log.

  4. #4
    Join Date
    Feb 2017
    Posts
    7

    Default

    I understand what you mean, I saw that too but the log doesn't have the field data type specified. If you open the ktr file I attached, you will see that.
    Also, I am wondering why the record would even goto "Write to log- TRUE" path as it has parsing error should be goto "Write to log- FALSE" path (error path).
    Last edited by jasonjo; 02-28-2018 at 01:23 PM.

  5. #5
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    Probably because you have Lazy conversion turned on.
    That delays converting data between types as long as it can.

    Since Write To Log needs a string, it takes the incoming data, tries to convert it to an number (as per the FTI), and then convert it to a string. The convert to number fails, which then causes the Write to Log to fail.

  6. #6
    Join Date
    Feb 2017
    Posts
    7

    Default

    Unchecking the Lazy Conversion worked. Thanks gutlez. I somehow totally forgot about this option.

    However, I am doing a similar operation using a GZIP CSV file input step; I see that "Define error handling" is not available in this step. is it true ? Also, per this Pentaho JIRA ticket https://jira.pentaho.com/browse/PDI-14324 , looks like true. Any idea ?

    I am trying to load a gzip file which has millions of records on a daily basis. So, my choice is a GZIP CSV file input because it is light weight, faster and enables parallel execution. I also need a error handling feature to log errors to a database table and load the good records.

  7. #7
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    No idea.
    I haven't used the gzip csv step before.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.