Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: Variable Data Types (and a logging question)

  1. #1
    Join Date
    Mar 2007
    Posts
    20

    Default Variable Data Types (and a logging question)

    A little about what I am trying to do:
    Process all files (one at a time) that sit on a particular folder and load them into an ODS.
    All this is working but now I want to log the execution.

    I started by using the logging provided out of the box.
    But I found that I needed more information. This provides information about the execution of jobs and trans but I want to re-use some of these jobs and trans so when I execute a batch, I need to know the client name and the file names that were processed.

    So I started building my own logging with a Job table and a batch table. A batch is an execution of a job (I also have a parent_batch concept but I will ignore that for this discussion.) I have a generic reusable "Log Batch Start" step which uses variables as input so I can reuse it. I set the generated batch id to a variable.

    I want to log the records input, processed and errors count. so I threw a "Group By" step off of my "Input File" step to get the Records Input and similarly on a Table Output (processed into the desired table) and Table Output (errored records into the error table using the error handling from the 'processed' step). I then set these to variables.

    Then when I want to log the end time, processed count, insert count and error count, I "Get" these variables and Update the batch record using the batch Id that I set to a variable previously.

    The problem I am having is with String to Integer conversion. It is throwing an error saying it cant do this conversion. I have tried to set these counts (say error count) to an integer in the "Get Variable" step, in a java script step and in a select values step and none have worked.

    I then set these values using a Group By count. I output them to a text file to see what they look like and it looks like " 000000001". I am trying to trim them, etc but can't seem to get it to run without failing.

    Any thoughts on what I am doing wrong or what I can do to get past this?

    Additionally, I was thinking that I could use the counts that are being written through the pentaho logging but I am not quite sure how to get at them. I am guessing that there are some internal variables that I can get at (either the batch_id or the counts themselves.) so if someone has a way that I can use those, that would be great as well.

    thanks, paul

  2. #2
    Join Date
    Mar 2007
    Posts
    20

    Default

    OK - I figured this out. I am not so smart sometimes.

    Turns out when you set a variable you don't specify the ${var} format - just var.
    I know this and just spaced out.

    That being said, if someone has a tip on using the data that is gathered in the Penthao logging framework, I would appreciate it, but I can probably research that a little harder at some point on my own as well.
    But otherwise I am back in business at least.

  3. #3
    Join Date
    Jun 2011
    Posts
    1

    Default

    You are a life saver! I have been struggling with this all day. Once I read your post realized I had done the same thing when I set my variables.

    In case someone else is having this same problem, this is what it looks like!

    It seemed like I was setting the variables properly. The log showed :
    2011/06/22 09:18:42 - Set Variables.0 - Set variable ${MAX_CROSS_REF_D_PARTY_NUM} to value [3150497]
    2011/06/22 09:18:42 - Set Variables.0 - Set variable ${MAX_PARTY_DIM_KEY} to value [3150497]

    But when I tried to get them I got the following errors:
    2011/06/22 09:20:02 - Get Max Party Key and Party Num Parameter Values.0 - ERROR (version 4.1.2.1-GA, build 14760 from 2011-01-26 13.31.23 by buildguy) : Unexpected conversion error while converting value [MAX_PARTY_DIM_KEY String(28)] to an Integer
    2011/06/22 09:20:02 - Get Max Party Key and Party Num Parameter Values.0 - ERROR (version 4.1.2.1-GA, build 14760 from 2011-01-26 13.31.23 by buildguy) :
    2011/06/22 09:20:02 - Get Max Party Key and Party Num Parameter Values.0 - ERROR (version 4.1.2.1-GA, build 14760 from 2011-01-26 13.31.23 by buildguy) : MAX_PARTY_DIM_KEY String(28) : couldn't convert String to Integer
    2011/06/22 09:20:02 - Get Max Party Key and Party Num Parameter Values.0 - ERROR (version 4.1.2.1-GA, build 14760 from 2011-01-26 13.31.23 by buildguy) : Unparseable number: "${MAX_PARTY_DIM_KEY}"
    Last edited by CLMORRIS; 06-22-2011 at 11:36 AM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.