Hitachi Vantara Pentaho Community Forums
Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: Carte treats transformation different than Spoon

  1. #1

    Default Carte treats transformation different than Spoon

    Hi,

    I have noticed something interesting: when processing my Apache Tomcat log, Carte treats the transformation other than Spoon.
    In Spoon, the transformation works fine and the Select step splits the data into the dummy and table output steps correctly.

    But when I am executing the transformation via Carte, all my data gets redirceted to the Dummy step after the Select step (I am using the same input file to test it). For some reason, Carte seams to treat/interpret the date value other than Spoon.

    I am executing Spoon on the same machine as Carte and both are working with the same repository.

    Has anybody an idea, what the reason for this behavior could be?

    Bobse

    tr_parse_log.ktrName:  2014-05-12 10_56_40-Select _ Rename values.jpg
Views: 75
Size:  19.4 KB
    PDI 7
    Postgres
    Windows 7

  2. #2

    Default

    Apparently your date value is unparseable, causing the error hop to be taken, but it's hard to guess why. Can you replace the "Dummy" step with "Text file output" perhaps so that you can capture the actual date values that failed, and see any error messages when running under Carte? Those should provide a valuable clue.

  3. #3

    Default

    Actually, it IS parseable when executed locally in Spoon. Only when executed in Carte, it does not work. I have tried your idea with the text output and here is one of the lines that get are redirected to that step:

    line;result;RemoteHost;RemoteLogicalName;RemoteUsername;DateTime;Request;StatusCode;BytesSent
    "10.61.4.5 - - [12/May/2014:00:00:01 +0200] ""GET /birt/run?__report=\available.rptdesign "" 200 14854";Y;10.61.4.5;-;-;12/May/2014:00:00:01 +0200;GET /birt/run?__report=\available.rptdesign ;200;14854
    "10.61.4.4 - - [12/May/2014:00:00:03 +0200] ""GET /birt/run?__report=\available.rptdesign "" 200 14854";Y;10.61.4.4;-;-;12/May/2014:00:00:03 +0200;GET /birt/run?__report=\available.rptdesign ;200;14854

    To be honest, I do not see the reason why it should not work. :/
    I have also attached a sample file of the Tomcat log, maybe that helps.

    Bobse

    localhost_access_log.2014-05-12.txt
    PDI 7
    Postgres
    Windows 7

  4. #4

    Default

    BTW, I have just tried to execute it via Kitchen and produced the same error.
    PDI 7
    Postgres
    Windows 7

  5. #5

    Default

    Thanks for the log example. I tried a few variations of this on my local machine and could not get it to fail under pan.bat or carte.bat. However I am running PDI 4.4 and have not yet installed version 5, so it's possible this is a version 5 idiosyncrasy.

    A few general suggestions that may or may not help:

    1) I changed the date format to "dd/MMM/yyyy:HH:mm:ss Z". This caused the step to parse and use the correct time zone from the Tomcat logs. I was guessing that the date conversion might fail because the date string contained additional characters (the time zone offset), but Java doesn't seem to care. Regardless it seems more correct with the updated format.

    2) In your step error handling, try setting the Error descriptions fieldname and/or Error codes fieldname. This should yield additional clues as to why the rows were rejected.

    3) Double-check that kitchen/carte are running under the same JVM as Spoon. Most likely they are, but it's worth checking because different Java versions can sometimes introduce subtle differences in date/time behavior.

    Sorry I couldn't solve your problem. If I find time later today I'll see if I can download 5.x and give it a try.

  6. #6
    Join Date
    Apr 2008
    Posts
    4,690

    Default

    Quote Originally Posted by jsturm View Post
    1) I changed the date format to "dd/MMM/yyyy:HH:mm:ss Z". This caused the step to parse and use the correct time zone from the Tomcat logs. I was guessing that the date conversion might fail because the date string contained additional characters (the time zone offset), but Java doesn't seem to care. Regardless it seems more correct with the updated format.
    This may have fixed it, depending on the date and time that was kicking out, as there are dates that exist in one timezone, and not in others.
    Also, Bobse may be running with different locale settings between Spoon and Kitchen / Carte. This may allow the import of some dates that cannot be imported in the other execution path. (eg. 01/May/2014:00:00:00 +0000 vs 01/Mai/2014:00:00:00 +0000 )

    Just throwing some options into the mix....

  7. #7

    Default

    Hi,

    1) Tried to add the Z, no difference.
    2) Done, here the result:

    Code:
    line;result;RemoteHost;RemoteLogicalName;RemoteUsername;DateTime;Request;StatusCode;BytesSent;error_no;error_desc;error_fields;error_codes
    "10.61.4.5 - - [13/May/2014:00:00:02 +0200] ""GET /birt/run?__report=\available.rptdesign "" 200 14854";Y;10.61.4.5;-;-;13/May/2014:00:00:02 +0200;GET /birt/run?__report=\available.rptdesign ;200;14854;1;"
    DateTime String : couldn't convert string [13/May/2014:00:00:02 +0200] to a date using format [dd/MMM/yyyy:HH:mm:ss Z] on offset location 3
    Unparseable date: ""13/May/2014:00:00:02 +0200""
    ";;SELECT001
    Where exactly is the 'offset location 3'?

    3) How exactly do I check that?

    Thanks for your input!
    PDI 7
    Postgres
    Windows 7

  8. #8

    Default

    Quote Originally Posted by gutlez View Post
    with different locale settings between Spoon and Kitchen / Carte.
    Ok, but to counter that I have set explicitly the Date locale flag to en_US since that is the formatting of the Tomcat log.
    PDI 7
    Postgres
    Windows 7

  9. #9
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    The first character is at offset 0, so May isn't recognized as a valid abbreviated month.
    I would make a POC, but hey, you're the one hooked on version 5
    So long, and thanks for all the fish.

  10. #10

    Default

    Thanks for pointing me in that direction
    Found the 'bug': it was actually the Date local flag: if you set it to 'en' and not 'en_US', it works like charme
    PDI 7
    Postgres
    Windows 7

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.