Writing kettle logs (log4j) in Mongo DB

[Sushilgyl]

Hi,

is there any way to write PDI (kettle) logs of any job/transformation in mongo db directly rather than writing it in to log file.

Thanks,
Sushil

[johanhammink]

In the Tranformation or Job properties you can adjust logging options and also specify a database / tables to write the logging

[Sushilgyl]

Hi johanhammink,

Thanks for your quick response. I am interested to write logs which is generated by PDI (Kettle).

for e.g.

2019/07/16 10:34:51 - Add constants.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2019/07/16 10:34:51 - Set field value.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2019/07/16 10:34:51 - Select values.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2019/07/16 10:34:51 - Microsoft Excel writer.0 - Finished processing (I=0, O=2, R=1, W=1, U=0, E=0)
2019/07/16 10:34:51 - Get count.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2019/07/16 10:34:51 - audit_log_success - Starting entry [set_count]
2019/07/16 10:34:51 - Set eventCode.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2019/07/16 10:34:51 - Filter rows 2.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)

Regards,
Sushil

[Ana GH]

I think MongoDB is not one of the officially supported databases, so you'll probably have to make adjustments to make it work: https://www.helicaltech.com/audit-lo...tegration-pdi/
https://www.youtube.com/watch?v=r5fY7C4uCq4
https://help.pentaho.com/Documentati...nce_monitoring
Pentaho documentation mentions Pentaho Server, you don't need it to set up a database to get log information, just set up the database and tables and edit EVERY transformation and job you want to monitor. I just have a prototype transformation and job that I use with the Save as option each time I create a new one, so that information is already set in the prototype.
Matt Casters has a docker image uploaded to the public repository where the logging and lineage information is in a Neo4j database, I have yet to test it, but it seems promising: https://hub.docker.com/r/mattcasters/webspoon-neo4j/
Regards

[gutlez]

Since PDI implements log4J, you shouldbe able to simply configure it into the configuration files.

This SO page might help:
https://stackoverflow.com/questions/...to-use-mongodb


It looks (from my simple digging) that you want to modify the one in: ${PDI_INSTALL_DIRECTORY}/classes/log4j.xml