Hitachi Vantara Pentaho Community Forums
Results 1 to 16 of 16

Thread: Transformation preview works, run gets stuck?

  1. #1
    Join Date
    Feb 2011
    Posts
    840

    Default Transformation preview works, run gets stuck?

    Sooooooooo... I have a simple transformation that reads XLS, denormalizes and normalizes data. When I preview the result, everything shows up fine - denormalize gets 2904 reads 2222 writes, normalize gets 2222 reads and 79992 writes. But when I try to run, denormalize finishes with 2904 read 2222 written, normalize reads 834 writes 30002 and stops. No errors, no crashes, it justs says it's running but nothing happens. I've left it "running" for almost an hour now, and it's going nowhere. Ideas?
    Join us on IRC! =)

    Twitter / Google+ / Timezone: BRT-BRST
    BI Server & PDI 5.4 / MS SQL 2012 / Learning CDE & CTools
    Windows 8 64-bit / Java 7 (jdk1.8.0_75)

    Quote Originally Posted by gutlez
    PLEASE NOTE: No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.

    I'm no expert.Take my comments at your own risk.

  2. #2
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    No clues in the log?
    So long, and thanks for all the fish.

  3. #3
    Join Date
    Feb 2011
    Posts
    840

    Default

    not one. I went as far as using Rowlevel log, and it doesn't show anything wrong. It's like Normalizer got tired and decided to take a break.
    Join us on IRC! =)

    Twitter / Google+ / Timezone: BRT-BRST
    BI Server & PDI 5.4 / MS SQL 2012 / Learning CDE & CTools
    Windows 8 64-bit / Java 7 (jdk1.8.0_75)

    Quote Originally Posted by gutlez
    PLEASE NOTE: No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.

    I'm no expert.Take my comments at your own risk.

  4. #4
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    I'm clueless, then
    So long, and thanks for all the fish.

  5. #5
    Join Date
    Dec 2012
    Posts
    8

    Default

    Does the flow break apart and come back together? A screen shot might help.

  6. #6
    Join Date
    Feb 2011
    Posts
    840

    Default

    Nothing breaks, there are no errors, no abnormal lines on the log.

    Excel input reads a total of 2904 rows, Row Demorn reads them all and writes 2222, Row Norm reads 2222 and writes 79992 on preview, but reads only 834 and writes only 30002. It keeps status "Running", but reads nothing and writes nothing.

    It's been running for 1 hour and 45 minutes as of now, Speed (r/s) 5.

    I'm currently unable to provide screenshot due to network restrictions =(

    Last lines on log are:
    Code:
    2013/05/21 12:55:00 - Microsoft Excel Input.0 - Get sheet #1.0
    2013/05/21 12:55:00 - Microsoft Excel Input.0 - Out of index error: move to next sheet!
    2013/05/21 12:55:01 - Microsoft Excel Input.0 - No more files to be processed! (2 files done)
    2013/05/21 12:55:01 - Microsoft Excel Input.0 - Signaling 'output done' to 1 output rowsets.
    2013/05/21 12:55:01 - Microsoft Excel Input.0 - Finished processing (I=2904, O=0, R=0, W=2904, U=0, E=0)
    Join us on IRC! =)

    Twitter / Google+ / Timezone: BRT-BRST
    BI Server & PDI 5.4 / MS SQL 2012 / Learning CDE & CTools
    Windows 8 64-bit / Java 7 (jdk1.8.0_75)

    Quote Originally Posted by gutlez
    PLEASE NOTE: No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.

    I'm no expert.Take my comments at your own risk.

  7. #7
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    I have a sinking feeling that we're not getting told a key detail again :P

    Can you post your KTR file?
    **THIS IS A SIGNATURE - IT GETS POSTED ON (ALMOST) EVERY POST**
    I'm no expert.
    Take my comments at your own risk.

    PDI user since PDI 3.1
    PDI on Windows 7 & Linux

    Please keep in mind (and this may not apply to this thread):
    No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.
    Signature Updated: 2014-06-30

  8. #8
    Join Date
    Feb 2011
    Posts
    840

    Default

    I have no idea what key detail you could be talking about =p

    I'm trying to sample a KTR and source file to share... hang on.

    ==

    couldn't sample the source, but I've kinda pinpointed the problem... there's more to the transformation, but since I had already other transformations like this one that never had any problems, I discarded that part as being the problem.

    After the norm step, I add a constant and set field value to yyyyMM string (variable, set on run). From the set field value, stream gets copied in two: one goes to a block until step is done and insert/update "values", the other goes to sort rows step and insert/update "accounts". Block is set to wait for the accounts.

    If I disable the hops going for the block and for the values insert, everything finishes in about 8 seconds. I can't run the insert/update values without having the insert/update accounts first because of a DB constraint set, that there can only be accounts on "values" table that already are on "accounts" table.

    What did I do wrong here, gutlez? =p
    Last edited by joao.ciocca; 05-21-2013 at 02:13 PM.
    Join us on IRC! =)

    Twitter / Google+ / Timezone: BRT-BRST
    BI Server & PDI 5.4 / MS SQL 2012 / Learning CDE & CTools
    Windows 8 64-bit / Java 7 (jdk1.8.0_75)

    Quote Originally Posted by gutlez
    PLEASE NOTE: No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.

    I'm no expert.Take my comments at your own risk.

  9. #9
    Join Date
    Feb 2011
    Posts
    840

    Default

    tried a couple different builds for insert/update account number to table1, and only after that insert/update account value to table2, but no deal. Every one of them has the side effect of making Row norm freeze.

    =edit=

    I guess the only build I didn't try before was account number -> account value, one output after the other... and it worked -_- still no idea what happened with that other scenario...
    Last edited by joao.ciocca; 05-21-2013 at 02:54 PM.
    Join us on IRC! =)

    Twitter / Google+ / Timezone: BRT-BRST
    BI Server & PDI 5.4 / MS SQL 2012 / Learning CDE & CTools
    Windows 8 64-bit / Java 7 (jdk1.8.0_75)

    Quote Originally Posted by gutlez
    PLEASE NOTE: No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.

    I'm no expert.Take my comments at your own risk.

  10. #10
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    Quote Originally Posted by joao.ciocca View Post
    After the norm step, I add a constant and set field value to yyyyMM string (variable, set on run). From the set field value, stream gets copied in two: one goes to a block until step is done and insert/update "values", the other goes to sort rows step and insert/update "accounts". Block is set to wait for the accounts.
    My guess: Block step was preventing other rows from exiting the set field, and Sort Rows was waiting for more rows from set field.

    It's a HIDDEN form of Split & Rejoin....
    Last edited by gutlez; 05-21-2013 at 04:04 PM.
    **THIS IS A SIGNATURE - IT GETS POSTED ON (ALMOST) EVERY POST**
    I'm no expert.
    Take my comments at your own risk.

    PDI user since PDI 3.1
    PDI on Windows 7 & Linux

    Please keep in mind (and this may not apply to this thread):
    No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.
    Signature Updated: 2014-06-30

  11. #11
    Join Date
    Mar 2008
    Posts
    140

    Default

    When you do a preview you typically limit the rows to be previewed. I think the default is 1000. If you get back more than 1000 the transformation will keep running even after you close the preview window. You can hit the stop button to stop it. Try setting your preview rows to a number greater than you will actually get back and it should stop on its own.

    I think there is a way to get the 'next x rows' which is why the transformation keeps running, but I don't remember how to do it.

    Hopefully this is your problem.

    - Curtis

    -- EDIT --
    I dyslexified the thread title. My response does not apply.
    Last edited by cboyden; 05-22-2013 at 11:50 AM.

  12. #12
    Join Date
    Feb 2011
    Posts
    840

    Default

    How could the block step be preventing rows from exiting the set field? O_o

    if that was the case, okay, makes sense. if one further step can prevent the previous of sending rows to another on a split stream...

    'cause then, set field could prevent add constant, and add constant could prevent norm step...

    but still doesn't explain why this doesn't happen on preview, only on the execution...

    I'm still looking for a better way to do this, since one Insert/Update after another seems to have a HUGE performance issue. Other transformations with the same set-up and around the same volume of rows usually take less than a minute to finish, this one (with output->output) takes almost 10 minutes to run.
    Join us on IRC! =)

    Twitter / Google+ / Timezone: BRT-BRST
    BI Server & PDI 5.4 / MS SQL 2012 / Learning CDE & CTools
    Windows 8 64-bit / Java 7 (jdk1.8.0_75)

    Quote Originally Posted by gutlez
    PLEASE NOTE: No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.

    I'm no expert.Take my comments at your own risk.

  13. #13
    Join Date
    Feb 2011
    Posts
    840

    Default

    Quote Originally Posted by cboyden View Post
    When you do a preview you typically limit the rows to be previewed. I think the default is 1000. If you get back more than 1000 the transformation will keep running even after you close the preview window. You can hit the stop button to stop it. Try setting your preview rows to a number greater than you will actually get back and it should stop on its own.
    yeah, I thought of that when I previewed for the first time that transformation, when I noticed I wasn't getting a full preview. I upped preview to 100,000 rows. Hence how I got to know that there were 79,992 writes
    Join us on IRC! =)

    Twitter / Google+ / Timezone: BRT-BRST
    BI Server & PDI 5.4 / MS SQL 2012 / Learning CDE & CTools
    Windows 8 64-bit / Java 7 (jdk1.8.0_75)

    Quote Originally Posted by gutlez
    PLEASE NOTE: No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.

    I'm no expert.Take my comments at your own risk.

  14. #14
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    Quote Originally Posted by joao.ciocca View Post
    How could the block step be preventing rows from exiting the set field? O_o
    Remember the Set Size? That's how...

    So assume for a minute that you have a couple of steps:
    Code:
    TFI --+--- Wait for SomeOtherStep --- Sucess
          ---- Sort Rows --- SomeOther Step
    So... Sort Rows needs all the rows before it can continue, but "Wait For" will only accept up to it's input buffer (lets say 1000 rows). The other end of the "Wait For" input buffer is the TFI output buffer. When the TFI output buffer gets full, it stops reading rows from the file. However, Sort Rows won't say it's done, since it doesn't have all the rows. Therefore, "Wait For" won't continue, since Sort Rows isn't finished. BANG. Deadlock, and only 1000 rows have been read.
    **THIS IS A SIGNATURE - IT GETS POSTED ON (ALMOST) EVERY POST**
    I'm no expert.
    Take my comments at your own risk.

    PDI user since PDI 3.1
    PDI on Windows 7 & Linux

    Please keep in mind (and this may not apply to this thread):
    No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.
    Signature Updated: 2014-06-30

  15. #15
    Join Date
    Feb 2011
    Posts
    840

    Default

    so, then, having no way to change Set Size on Block, I simply can't do that way. Got it now. Thanks! =)
    Join us on IRC! =)

    Twitter / Google+ / Timezone: BRT-BRST
    BI Server & PDI 5.4 / MS SQL 2012 / Learning CDE & CTools
    Windows 8 64-bit / Java 7 (jdk1.8.0_75)

    Quote Originally Posted by gutlez
    PLEASE NOTE: No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.

    I'm no expert.Take my comments at your own risk.

  16. #16
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    The Set Size that I was referring to was the buffer size... I can't remember what the correct name for it is off the top of my head, and IT is in the process of updating my workstation, so I have no PDI.
    **THIS IS A SIGNATURE - IT GETS POSTED ON (ALMOST) EVERY POST**
    I'm no expert.
    Take my comments at your own risk.

    PDI user since PDI 3.1
    PDI on Windows 7 & Linux

    Please keep in mind (and this may not apply to this thread):
    No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.
    Signature Updated: 2014-06-30

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.