Hitachi Vantara Pentaho Community Forums
Results 1 to 9 of 9

Thread: Why the trasformation doesn't write on the table?

  1. #1
    Join Date
    Feb 2011
    Posts
    4

    Default Why the trasformation doesn't write on the table?

    Hi all,
    I'm new with pentaho Data Integration (kettle) and I have a problem with my trasformation,
    afterwards the steps:

    1) I generate a row with two values: "system_name" and "URL" ;
    2) I save the value of "system_name" into a variable (SYSTEM_NAME) ;
    3) I pass the URL to the HTTP Client connector ;
    4) I download the data from the HTTP Server and I write these on a txt file ;
    5) I try to read from the txt file ;
    6) I try to write the data, just read from the txt file, to a mysql table ;
    7) I try to update the mysql table using the variable previously memorized .

    Running the trasformation it seems that the link between step 4) and 5) dosen't exist...why?

    Thank for your support

    Gabriele
    Attached Files Attached Files

  2. #2
    Join Date
    Feb 2011
    Posts
    4

    Question Between the connectors "Text file input" and "Text file output"

    In a trasformation, to link the two connectors "Text file input" and "Text file output", is necessary to use a particulat collector (job) ?

    In the same trasformation I would like to read a txt file after its generation (I download a web content, I store the content in the txt file and than I try to read this file with the connector "Text file input"..all in the same trasformation...but it's seams impossibile!! )

    Thanks for support

    Gabriele

  3. #3
    Join Date
    Sep 2007
    Posts
    834

    Default

    Hi, Gabriele.
    There are a couple of issues in your transformation:
    - you shouldn't use a variable in the same transformation that sets it.
    - there is no need to save to a file and read the file in order to parse the results of the service. You can parse the results right after the http step.
    - How does the result looks like? is it xml? is it json? Depending on the format, you have to parse the results in a different way. It's not just putting the name of the fields in a Text file input step.

    I suggest you to focus in parsing the output of the http step. Once it works, you can proceed with the rest of the tasks.
    good luck,

  4. #4
    Join Date
    Mar 2008
    Posts
    140

    Default

    Gabriele,

    Welcome to PDI!

    There are a few things you are going to hit with the transformation you have attached. They pretty much all stem from the fact that all steps in a transformation execute in parallel. (Job entries execute in series)

    1) You cannot set a variable and use a variable in the same transformation
    2) Your output file is open for writing (and is actively being written to) while the text input step is trying to read from it

    What I would recommend is creating a job and seeing if the [File management->HTTP] job entry is sufficient to fetch the file for you. Then use a [General->Transformation] job entry to read the file, insert into the database and perform your update.

    I hope this helps.

    -Curtis

  5. #5
    Join Date
    Mar 2008
    Posts
    140

    Default

    Sorry Maria,

    I didn't mean to duplicate your efforts.

    -Curtis

  6. #6
    Join Date
    Sep 2007
    Posts
    834

    Default

    Quote Originally Posted by cboyden View Post
    Sorry Maria,

    I didn't mean to duplicate your efforts.

    -Curtis
    no problem. our posts complement each other,

  7. #7
    Join Date
    Feb 2011
    Posts
    4

    Default

    Hi Maria, Curtis

    first of all, thanks for your support!!

    Like you suggest, I tried to parse the output of the HTTP connector but without success. You can find in the new attached file a txt file with the output I would like to obtain at the end of the transformation and the xml file I tied to parse..any suggestion?? :s
    I use the "Set Variables" connector because at the end of the transformation I need to update a mysql table with a "system_name"; this transformation is the second transformation I use in a JOB:

    JOB STRUCTURE:
    1 - Start
    2 - Delete File
    3 - first transformation (I generate a list file of system_name/URL (see the attached file))
    4 - second transformation (I download the performance data for each server, I try to store the data in the db table and then I associate the system_name stored in the variable with any line of the table (using the connector "Execute SQL script") - THIS TRANSFORMATION "Execute for every input row").
     
    What do you think? J
     
    Thanks!!
     
    Gabriele
    Attached Files Attached Files

  8. #8
    Join Date
    Feb 2011
    Posts
    4

    Default

    Sorry, I forgot to attach the XML file

    Gabriele
    Attached Files Attached Files

  9. #9
    Join Date
    Sep 2007
    Posts
    834

    Default

    Gabriele,
    here you have a ktr that parses your xml file. I guess it must be a simpler way, but this is what I have so far. It works.
    With regard to the variables, try pass them as named parameters to the transformation. We can discuss it later, if you need. You are really courageous being a Kettle beginner and trying to do all this.
    Attached Files Attached Files

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.