Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: [Beginner] - Is it possible to extract data from raw log files?

  1. #1
    Join Date
    Nov 2013

    Default [Beginner] - Is it possible to extract data from raw log files?

    Hi everyone,

    I am working on a project proposal for a credit union that deals with extracting statistical data (transaction volume) from different payment processing applications and then using Pentaho to load the data in to a target database. My question is:

    1. Is it possible for PDI (kettle) to extract from raw logs? (the log contains ISO 8583 messages, unfortunately I cannot provide a sample as it is confidential info). They do have some kind of format but the positioning of each message is not fixed, so it might be difficult to use cut/grep/awk, etc. (LOG --> ETL --> Target DB)

    2. ISO messages can be parsed so I'm assuming it would easier to parse the logs using some kind of script, the output would be a formatted file (pipe-delimited, or comma separated). Than would be easier for Pentaho to consume? (LOG --> formatted log file through parsing --> ETL --> Target DB)

    Sorry for these basic questions, any suggestions/ideas would be appreciated. Thanks very much. -- David

  2. #2
    Join Date
    Jun 2012


    I would try to use the parser made by Vikrant Labde in a User Defined Java Class.
    So long, and thanks for all the fish.

  3. #3
    Join Date
    Mar 2014



    I have to analyze a log file of a portal and would like to use as a tool Pentaho. The file has the extension. Log
    The format of the log is to Apache and I should extract the information contained in it.
    The information I want to know are:
    - Content displayed more
    - User agent used to make requests etc.

    Basically I should be doing the business intelligence analysis on this log file. Can you help me or give me some suggestions to parse this file?

    Thank you in advance for your reply


Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.