Hitachi Vantara Pentaho Community Forums
Results 1 to 8 of 8

Thread: Sending data to Apache Kafka Producer

  1. #1

    Default Sending data to Apache Kafka Producer

    Hi guys,

    I'm new with the product and would like to know how to send data from a JSON (or other) file into a Kafka Producer.

    Thanks

  2. #2
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    There are a couple of Input steps you might find useful, e.g. JSON Input.
    If your producer API is compatible with Java 1.4 (Janino restriction, no generics) you can send messages from a User Defined Java Class.
    So long, and thanks for all the fish.

  3. #3

    Default

    Thank you very much. Although I'm not sure if I'm grasping the JSON Output / Input module. In essence, I have an CSV file that need to go into Kafka.
    Thus far, I can do the following.
    CSV File contains something like:
    ccy | code
    JPY | HR00
    NZ | HN00

    I can transfer the above data into a JSON file that contains:
    {"JsonBlockName":[{"ccy":"JPY","code":"HR00"},{"ccy":"NZ","code":"HN00"}]}

    My two nodes are CSV file input and Json output. ANY bit of help or ideas will be much appreciated. Name:  csv_to_kafka.jpg
Views: 629
Size:  28.8 KB

  4. #4
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Sorry, can't read your screenshot - didn't you check yourself before posting?
    So long, and thanks for all the fish.

  5. #5

    Default

    Quote Originally Posted by marabu View Post
    Sorry, can't read your screenshot - didn't you check yourself before posting?

    Hi, sorry. Thought the post would have enabled people to open the original size format. Lesson learnt.

    Currently I'm getting an error on the Kafka Producer step. "Input field 'OutputValue' is not binary!"
    Name:  1and2_CSV.jpg
Views: 626
Size:  27.6 KB
    Name:  3_JSON_Output.jpg
Views: 587
Size:  20.5 KB
    Name:  4_JSON_Input_File.jpg
Views: 592
Size:  19.9 KBName:  5_JSON_Input_Additionl_output_fields.jpg
Views: 605
Size:  27.0 KB
    Name:  6_Apache_Kafka_Producer.jpg
Views: 647
Size:  29.7 KB

  6. #6

    Default

    Is there anyone that can help me with this. Please

  7. #7

    Default

    Managed to load the data into kafka by using the following format and setting the "Header" as a binary value (Contributing already!):
    Header
    {"ccy":"JPY","code":"HR00"}
    {"ccy":"NZ","code":"HN00"}

    I still sit with a small data manipulation issue when I'm transforming the CSV file to JSON Output. The closest I can get to this format is:
    {"":[{"ccy":"JPY","code":"HR00"},{"ccy":"NZ","code":"HN00"}]}

  8. #8
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Set the block factor to 1 and extract the row.
    Attached Files Attached Files
    So long, and thanks for all the fish.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.