Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Logging Best Practice and "Job ID"

  1. #1
    Join Date
    Sep 2014
    Posts
    175

    Default Logging Best Practice and "Job ID"

    Hey there ---

    I'm re-imagining my logging procedures. Trying to use more modular/ standard templates.
    I've already done this in SSIS. Pentaho PDI has its uses too of course.

    One thing I noticed is useful is a universal logging table -- not logging in separate tables for separate ETL.

    One thing SSIS has that Pentaho PDI doesn't is a 'generated' unique job ID (Numeric) that doesn't change with name changes.

    Pentaho doesn't have unique Job IDs, right? The only built-in Internal Variable for Job really is "Job Name". Names are changeable and can break referential links.

    I'm wondering if anyone here has logging best practices. And before its mentioned, many 'out of the box' logging -- for Pentaho or SSIS -- is not very good. It's good if you're short on time and quite lazy, and then you probably will never look at it or spend way too much time digging throug hit.

    I guess the answer is to create a variable "Job ID" for each job. Thing is, this pretty manual (to make it unique). So when the name changes in the job, the ID is persisted like in much technology paradigms. I know there are 'batch IDs' but that's different.

    Again, to really make this concrete, you could check a log table and say "when's the last time the Load_Call ran succesfully, and how long did it take?". If this is based on name alone --- names don't change frequently if at all for jobs, but they could change. Right?

  2. #2
    Join Date
    Sep 2014
    Posts
    175

    Default

    Although SSIS is better out of the box by generating a random alphanumeric "ID" for jobs (that is preserved with name changes, which can be a danger potentially but I think is nice for logging).

    Looks like for Pentaho I'll just have to get creative when persisting a "job reference name/ ID" in a master logging table. Individual logging tables are just a mess.

    One thing I noticed when redoing some of my Pentaho logging is that how each Transformation, you can specify which step exactly you want to log "Lines Written" or "Lines Output" which is great and simplifies things. In SSIS it's not possible to count exactly what was written --- unless you just query (using a log id) what was just written after the fact which is a bit redundant. I've seen some other consultants "row count" stuff in SSIS right before the writing. But uh ... it didn't successfully write yet! Hacky work around. Pentaho is ahead in this respect. But behind with the referential integrity thing.

  3. #3
    Join Date
    Aug 2016
    Posts
    288

    Default

    Using a log table is new to me. The built-in log system is way too verbose and messy for my taste. Logging is critical to my use, I need to know exactly what has been done and when. I also need to know if and when something goes wrong. Historically I have seen problems with the database connection, some times it doesn't work. How would this affect logging to table? I'd imagine the log would be attempted to be written, but failed due to the connection doesn't work at that particular moment. In that case, the log would be silent? I much prefere making writing custom log lines to file. Then I can define exactly what is written and it isn't depending on a db-connection that may not always be reliable.

  4. #4
    Join Date
    May 2016
    Posts
    275

    Default

    @cool_runnings You are not specifying it, but I suppose you are using PDI logging tables in your own database? There's this old note about solving some locking issues when using the log tables in the database that I haven't tried. It seems it doesn't solve it completely, but I have made a note of trying it because I also sometimes get the locking error messages:
    https://wiki.pentaho.com/display/EAI...current+access
    OS: Ubuntu 16.04 64 bits
    Java: Openjdk 1.8.0_131
    Pentaho 6.1 CE

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.