Hitachi Vantara Pentaho Community Forums
Page 1 of 2 12 LastLast
Results 1 to 10 of 15

Thread: Getting double (2x) rows

  1. #1
    Join Date
    Apr 2007
    Posts
    21

    Default Getting double (2x) rows

    Hi,

    trying to build a simple job with some transformation - t1, t2.

    t1 generate rows, pass it to "copy rows to result".
    t2 starts with "get rows from result".
    But the I get duplicates (2x) the rows in t2?

    Any idea what could be wrong.

    Thanks
    Attached Files Attached Files

  2. #2
    Join Date
    Nov 2008
    Posts
    777

    Default

    Can you post t1 and t2? It's kind of hard to tell what's going on without them.
    pdi-ce-4.4.0-stable
    Java 1.7 (64 bit)
    MySQL 5.6 (64 bit)
    Windows 7 (64 bit)

  3. #3
    Join Date
    Apr 2007
    Posts
    21

    Default

    Hi,

    Attached.

    I have set each transformation to output the rows to a text file. the file file shows there are 2x rows.

    Thanks
    Attached Files Attached Files

  4. #4
    Join Date
    Feb 2008
    Posts
    107

    Default

    At first I thought it was because you had an output file step in t1 (writes header plus data = 2 rows) followed by the transformation entry in testjob for t2 which had "Execute for every input row" ticked. But I removed these and found the same result.

    I noticed that if you check the "include time in filename" and specify the format as yyyyMMddHHmmssSSS, then you can see that there are actually two files being written. Since in your original the filename will be the same both times and you have "append" ticked, the output is written twice to the same file. I'm still stumped as to why this would be happening.

    Also weird is that the headers come after the data in the output.

    Finally, for safety I'd go into the .ktr files and delete the connection info to your database before posting.

    Hoping someone else can shed some light on this.

    Cheers,
    Paul

  5. #5
    Join Date
    Nov 2008
    Posts
    777

    Default

    For some reason, t2 is actually running twice (according to the job execution log) and I can't figure out why. I fiddled with all the settings I could find and couldn't get it to run only once. It could be a bug. Perhaps one of the pro's could have a look?

    Paul - Note that 'Footer' is also ticked. That is why the "headers" come after the data in the output. It's also interesting to note that even with 'Header' ticked the headers don't also appear before the data unless 'Append' is unticked. Then both headers and footers appear properly. This is perhaps a planned behavior of 'Append' although I would think that headers would at least be written when the file is created but that doesn't seem to be the case.
    Last edited by darrell.nelson; 09-16-2011 at 12:09 PM.
    pdi-ce-4.4.0-stable
    Java 1.7 (64 bit)
    MySQL 5.6 (64 bit)
    Windows 7 (64 bit)

  6. #6

    Default

    I added a success step to your job and though it stops t2 from appearing in the execution log twice, I still get the following final output:

    868686867.0;test
    testrun_id;testrun_lotid
    DONE 868686867.0;test
    testrun_id;testrun_lotid
    DONE

    This is a really simple job/transformation setup so I'm going to fiddle around with it a little more when I return from my meeting and see if I can get it straight for you.

  7. #7

    Default

    Actually, problem fixed.

    On your T2 text file output you were appending the data from the T1 transformation to the T2 output transformation. When removing 'Append' option on the second text file output i get the following output:

    testrun_id;testrun_lotid
    868686867.0;test
    testrun_id;testrun_lotid
    DONE

  8. #8
    Join Date
    Nov 2008
    Posts
    777

    Default

    Quote Originally Posted by ccorliss View Post
    I added a success step to your job and though it stops t2 from appearing in the execution log twice ...
    I tried that and it didn't seem to stop t2 from running twice for me.
    Last edited by darrell.nelson; 09-16-2011 at 11:43 AM.
    pdi-ce-4.4.0-stable
    Java 1.7 (64 bit)
    MySQL 5.6 (64 bit)
    Windows 7 (64 bit)

  9. #9
    Join Date
    Nov 2008
    Posts
    777

    Default

    Quote Originally Posted by ccorliss View Post
    Actually, problem fixed.

    On your T2 text file output you were appending the data from the T1 transformation to the T2 output transformation. When removing 'Append' option on the second text file output
    As far as I can tell, removing 'Append' on the second text file output only masks the problem. When t2 runs the second time it just overwrites what was written when t2 ran the first time. Paul's suggestion of adding the milliseconds to the filename confirms this. Two files are created regardless of how 'Append' is set.
    Last edited by darrell.nelson; 09-16-2011 at 12:00 PM.
    pdi-ce-4.4.0-stable
    Java 1.7 (64 bit)
    MySQL 5.6 (64 bit)
    Windows 7 (64 bit)

  10. #10

    Default

    Ah yeah, I was in a hurry and didn't realize I had room to scroll down in my log, just realized it's still duplicating (not only the T2 step but also the success step). I'm going to keep messing around with it and see what I can come up with.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.