Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: Writing kettle logs (log4j) in Mongo DB

Hybrid View

Previous Post Previous Post   Next Post Next Post
  1. #1
    Join Date
    Feb 2012
    Posts
    26

    Default Writing kettle logs (log4j) in Mongo DB

    Hi,

    is there any way to write PDI (kettle) logs of any job/transformation in mongo db directly rather than writing it in to log file.

    Thanks,
    Sushil

  2. #2
    Join Date
    Nov 2009
    Posts
    688

    Default

    In the Tranformation or Job properties you can adjust logging options and also specify a database / tables to write the logging

  3. #3
    Join Date
    Feb 2012
    Posts
    26

    Default

    Hi johanhammink,

    Thanks for your quick response. I am interested to write logs which is generated by PDI (Kettle).

    for e.g.

    2019/07/16 10:34:51 - Add constants.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
    2019/07/16 10:34:51 - Set field value.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
    2019/07/16 10:34:51 - Select values.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
    2019/07/16 10:34:51 - Microsoft Excel writer.0 - Finished processing (I=0, O=2, R=1, W=1, U=0, E=0)
    2019/07/16 10:34:51 - Get count.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
    2019/07/16 10:34:51 - audit_log_success - Starting entry [set_count]
    2019/07/16 10:34:51 - Set eventCode.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
    2019/07/16 10:34:51 - Filter rows 2.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)

    Regards,
    Sushil

  4. #4
    Join Date
    May 2016
    Posts
    282

    Default

    I think MongoDB is not one of the officially supported databases, so you'll probably have to make adjustments to make it work: https://www.helicaltech.com/audit-lo...tegration-pdi/
    https://www.youtube.com/watch?v=r5fY7C4uCq4
    https://help.pentaho.com/Documentati...nce_monitoring
    Pentaho documentation mentions Pentaho Server, you don't need it to set up a database to get log information, just set up the database and tables and edit EVERY transformation and job you want to monitor. I just have a prototype transformation and job that I use with the Save as option each time I create a new one, so that information is already set in the prototype.
    Matt Casters has a docker image uploaded to the public repository where the logging and lineage information is in a Neo4j database, I have yet to test it, but it seems promising: https://hub.docker.com/r/mattcasters/webspoon-neo4j/
    Regards
    OS: Ubuntu 16.04 64 bits
    Java: Openjdk 1.8.0_131
    Pentaho 6.1 CE

  5. #5
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    Since PDI implements log4J, you shouldbe able to simply configure it into the configuration files.

    This SO page might help:
    https://stackoverflow.com/questions/...to-use-mongodb


    It looks (from my simple digging) that you want to modify the one in: ${PDI_INSTALL_DIRECTORY}/classes/log4j.xml
    Last edited by gutlez; 07-24-2019 at 03:01 PM.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.