Hitachi Vantara Pentaho Community Forums
Results 1 to 8 of 8

Thread: Parameters to Transformation

  1. #1

    Default Parameters to Transformation

    Hi,

    I have a transformation file which accepts a csv. This csv is formatted and passed into a Apache Kafka Node.

    I need the ability to change the "Filename" of the csv step and the "Topic Name" of the Kafka Producer on the fly.( Please find attached transformation file.)
    I'm using the following terminal command(Ubuntu) to run the transformation. It would be perfect if I'm able to add/change these two parameters within the command. Command is:
    /home/Pentaho/Pentaho/data-integration/pan.sh -file='/home/Pentaho/Pentaho/CSV.ktr' '2014-01-01 00:00:00' '2014-01-01 00:00:00' -Level=Basic > //tmp/tr_get_command_line_arguments.err.log

    Please let me know if this can be done. If not, I'm open to suggestions.

    Thank All
    Attached Files Attached Files

  2. #2
    Join Date
    Apr 2008
    Posts
    4,687

    Default

    If you declare two named parameters (eg. FilePath and Topic), and default them to /home/fraxses/druid-services-0.6.73/AIindexing/TwoLineInput1.csv and Topic040614 respectively, you can include the parameters in your CSV Input and your Kafka Producer.

    Your commandline would become:
    Code:
    /home/Pentaho/Pentaho/data-integration/pan.sh  -file='/home/Pentaho/Pentaho/CSV.ktr' -param:FilePath=/home/fraxses/druid-services-0.6.73/AIindexing/TwoLineInput1.csv -param:Topic=Topic040614 -Level=Basic > //tmp/tr_get_command_line_arguments.err.log
    **THIS IS A SIGNATURE - IT GETS POSTED ON (ALMOST) EVERY POST**
    I'm no expert.
    Take my comments at your own risk.

    PDI user since PDI 3.1
    PDI on Windows 7 & Linux

    Please keep in mind (and this may not apply to this thread):
    No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.
    Signature Updated: 2014-06-30

  3. #3
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    I can't open your transformation, as long as I don't have the Kafka Producer installed, but no problem: You can specify parameters by name using the Pan command line options. You must declare your parameters in the transformation settings, though (Ctrl-T). You can specify variables everywhere you encounter the small dollar icon attached to the top right corner of an edit control.
    So long, and thanks for all the fish.

  4. #4

    Default

    OK, I have it working for the FileName, but not for the "Kafka Producer". (It works when I specify a Topic Name for it as normal but not a parameter)

    I believe I have the command line correct now. Thank you gutlez!
    Is this maybe a bug or am I just not doing it correctly.
    Please see screenshots and attachment.
    Name:  TransParam.jpg
Views: 60
Size:  22.5 KB
    Name:  KafkaProducer.jpg
Views: 62
Size:  27.3 KB
    Attached Files Attached Files
    Last edited by Lunendl; 06-06-2014 at 06:26 AM.

  5. #5
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Quote Originally Posted by Lunendl View Post
    It works when I specify a Topic Name for it as normal but not a parameter
    The author indicated variable substitution for the kafka topic in the UI, but forgot to perform the substitution.
    I had to add the bold line to the init method of class KafkaProducerStep to make it work:

    Code:
    public boolean init(StepMetaInterface smi, StepDataInterface sdi) {
        super.init(smi, sdi);
    
    
        KafkaProducerMeta meta = (KafkaProducerMeta) smi;
        KafkaProducerData data = (KafkaProducerData) sdi;
    
    
        meta.setTopic(environmentSubstitute(meta.getTopic()));
        Properties properties = meta.getKafkaProperties();
    So long, and thanks for all the fish.

  6. #6

    Default

    Thank you SOOOOOO much marabu!!! You good!

    I'm sitting with one more issue however. When I run the transformation, and I've added or taken a column out of the csv, it either fails(column taken away) or does not read the extra column(s). I've looked around on the internet on how to get around this problem, but I could only find the following: http://www.ibridge.be/?p=273

    This seems to be very complicated. Do you think there is any other way we can do that?
    Please find attached Transformation I've made that retrieves the column names, but I dont know how to assign it to the CSV file input - "Fields" and the Json output - "Fields"
    Attached Files Attached Files
    Last edited by Lunendl; 06-09-2014 at 09:21 AM.

  7. #7

    Question

    I am also able to create an XML file now.
    I still can't get my head around actually using the fields in the XML file as the columns field names("Get Fields") within the "CSV file input".

    Please find attached Transformation process for CSV file into Kafka.
    Attached Files Attached Files

  8. #8
    Join Date
    Jun 2007
    Posts
    103

    Default

    Hi - I don't know if it's just me, but I can't detach the KTRs that are in this thread. Clicking on them opens a blank window while right-clicking prompts me to save the GIF.

    I'm looking for any help with sending messages into Kafka.

    Thanks
    - Russell

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.