Hitachi Vantara Pentaho Community Forums
Results 1 to 15 of 15

Thread: Getting double (2x) rows

  1. #1
    Join Date
    Apr 2007
    Posts
    21

    Default Getting double (2x) rows

    Hi,

    trying to build a simple job with some transformation - t1, t2.

    t1 generate rows, pass it to "copy rows to result".
    t2 starts with "get rows from result".
    But the I get duplicates (2x) the rows in t2?

    Any idea what could be wrong.

    Thanks
    Attached Files Attached Files

  2. #2
    Join Date
    Nov 2008
    Posts
    777

    Default

    Can you post t1 and t2? It's kind of hard to tell what's going on without them.
    pdi-ce-4.4.0-stable
    Java 1.7 (64 bit)
    MySQL 5.6 (64 bit)
    Windows 7 (64 bit)

  3. #3
    Join Date
    Apr 2007
    Posts
    21

    Default

    Hi,

    Attached.

    I have set each transformation to output the rows to a text file. the file file shows there are 2x rows.

    Thanks
    Attached Files Attached Files

  4. #4
    Join Date
    Feb 2008
    Posts
    107

    Default

    At first I thought it was because you had an output file step in t1 (writes header plus data = 2 rows) followed by the transformation entry in testjob for t2 which had "Execute for every input row" ticked. But I removed these and found the same result.

    I noticed that if you check the "include time in filename" and specify the format as yyyyMMddHHmmssSSS, then you can see that there are actually two files being written. Since in your original the filename will be the same both times and you have "append" ticked, the output is written twice to the same file. I'm still stumped as to why this would be happening.

    Also weird is that the headers come after the data in the output.

    Finally, for safety I'd go into the .ktr files and delete the connection info to your database before posting.

    Hoping someone else can shed some light on this.

    Cheers,
    Paul

  5. #5
    Join Date
    Nov 2008
    Posts
    777

    Default

    For some reason, t2 is actually running twice (according to the job execution log) and I can't figure out why. I fiddled with all the settings I could find and couldn't get it to run only once. It could be a bug. Perhaps one of the pro's could have a look?

    Paul - Note that 'Footer' is also ticked. That is why the "headers" come after the data in the output. It's also interesting to note that even with 'Header' ticked the headers don't also appear before the data unless 'Append' is unticked. Then both headers and footers appear properly. This is perhaps a planned behavior of 'Append' although I would think that headers would at least be written when the file is created but that doesn't seem to be the case.
    Last edited by darrell.nelson; 09-16-2011 at 12:09 PM.
    pdi-ce-4.4.0-stable
    Java 1.7 (64 bit)
    MySQL 5.6 (64 bit)
    Windows 7 (64 bit)

  6. #6

    Default

    I added a success step to your job and though it stops t2 from appearing in the execution log twice, I still get the following final output:

    868686867.0;test
    testrun_id;testrun_lotid
    DONE 868686867.0;test
    testrun_id;testrun_lotid
    DONE

    This is a really simple job/transformation setup so I'm going to fiddle around with it a little more when I return from my meeting and see if I can get it straight for you.

  7. #7

    Default

    Actually, problem fixed.

    On your T2 text file output you were appending the data from the T1 transformation to the T2 output transformation. When removing 'Append' option on the second text file output i get the following output:

    testrun_id;testrun_lotid
    868686867.0;test
    testrun_id;testrun_lotid
    DONE

  8. #8
    Join Date
    Nov 2008
    Posts
    777

    Default

    Quote Originally Posted by ccorliss View Post
    I added a success step to your job and though it stops t2 from appearing in the execution log twice ...
    I tried that and it didn't seem to stop t2 from running twice for me.
    Last edited by darrell.nelson; 09-16-2011 at 11:43 AM.
    pdi-ce-4.4.0-stable
    Java 1.7 (64 bit)
    MySQL 5.6 (64 bit)
    Windows 7 (64 bit)

  9. #9
    Join Date
    Nov 2008
    Posts
    777

    Default

    Quote Originally Posted by ccorliss View Post
    Actually, problem fixed.

    On your T2 text file output you were appending the data from the T1 transformation to the T2 output transformation. When removing 'Append' option on the second text file output
    As far as I can tell, removing 'Append' on the second text file output only masks the problem. When t2 runs the second time it just overwrites what was written when t2 ran the first time. Paul's suggestion of adding the milliseconds to the filename confirms this. Two files are created regardless of how 'Append' is set.
    Last edited by darrell.nelson; 09-16-2011 at 12:00 PM.
    pdi-ce-4.4.0-stable
    Java 1.7 (64 bit)
    MySQL 5.6 (64 bit)
    Windows 7 (64 bit)

  10. #10

    Default

    Ah yeah, I was in a hurry and didn't realize I had room to scroll down in my log, just realized it's still duplicating (not only the T2 step but also the success step). I'm going to keep messing around with it and see what I can come up with.

  11. #11

    Default

    I recreated the job and transformations exactly as you had them in the attached files, and everything processed successfully (did not process T2 twice) and all appears to be fine.

    I've attached the job and transformations so you can compare.

    Obviously you'll need to change file locations and such for the transformations and output files, but that shouldn't cause any problems.
    Attached Files Attached Files

  12. #12
    Join Date
    Nov 2008
    Posts
    777

    Default

    OMG! In the original job, t1 is actually connected to t2 twice! Delete one of the hops and everything will work as expected.
    Last edited by darrell.nelson; 09-17-2011 at 03:32 PM.
    pdi-ce-4.4.0-stable
    Java 1.7 (64 bit)
    MySQL 5.6 (64 bit)
    Windows 7 (64 bit)

  13. #13
    Join Date
    Apr 2007
    Posts
    21

    Default

    Apologize for my silly mistake!

    But I could not tell from the visual t1 connected to t2 twice.

    Thanks a lot to all!

  14. #14
    Join Date
    Nov 2008
    Posts
    777

    Default

    Quote Originally Posted by nicklv View Post
    But I could not tell from the visual t1 connected to t2 twice.
    I know. The two hops were on top of each other so there was no way to see it.

    When ccorliss rebuilt the job and it worked correctly I suspected there was something "hidden" fouling things up. I opened the XML file and saw the duplicate t1/t2 hop definition. Ding!
    Last edited by darrell.nelson; 09-16-2011 at 12:49 PM.
    pdi-ce-4.4.0-stable
    Java 1.7 (64 bit)
    MySQL 5.6 (64 bit)
    Windows 7 (64 bit)

  15. #15
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Yeah, we'll have to fix that bug ASAP.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.