Hitachi Vantara Pentaho Community Forums
Results 1 to 9 of 9

Thread: tableOutput step hangs

  1. #1
    Join Date
    Feb 2011
    Posts
    152

    Default tableOutput step hangs

    I am testing out PDI 5.0 and have a transformation that reads a table from a postgresql database, manipulates some of the fields (data cleansing), and then writes them back to the same postgresql database. The problem though, is that the table output step hangs. I can process the first 94 rows, but as soon as I try 95, it hangs with no error message.

    No, the problem is not data on that 95th row. I can do that row by itself, and it works. It seems like some kind of buffering issue because when my batch size is set to 100, it hangs, but if I say batch size = 50, it works for all rows. The table has 340 columns in it. This same transformation worked fine in version 4.3 / 4.4. Has anyone else experienced this kind of issue? Is there some kind of limit as to how many columns or row size the tableOutput step can handle?

  2. #2
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    the same postgresql database, or the same table?

    If it's the same table, you could be running into DB level deadlock.
    **THIS IS A SIGNATURE - IT GETS POSTED ON (ALMOST) EVERY POST**
    I'm no expert.
    Take my comments at your own risk.

    PDI user since PDI 3.1
    PDI on Windows 7 & Linux

    Please keep in mind (and this may not apply to this thread):
    No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.
    Signature Updated: 2014-06-30

  3. #3
    Join Date
    Feb 2011
    Posts
    152

    Default

    It is a different table in the same database, so there is no deadlock issue. The 2 tables are even on different schemas.

  4. #4
    Join Date
    Jul 2009
    Posts
    476

    Default

    In Postgres, check pg_stat_activity to see if any sessions have waiting = true. If the 95th row in your transformation lands on the same key as one of the earlier rows, and your transformation has some kind of parallel workers, then you might have a blocking/deadlock problem.

    EDIT: I see that gutlez had the same idea. Even if the source and destination tables are different, you could have a deadlock on the destination if your cleansed rows can collide there.
    Last edited by robj; 11-25-2013 at 04:37 PM. Reason: add information after others made same point

  5. #5
    Join Date
    Feb 2011
    Posts
    152

    Default

    Deadlock is not the problem. I am reading from one copy of the table, and writing to an empty copy of the table on another schema. Before this transformation runs, I have a job that drops / creates the destination table, so it is empty when this transformation is running. Also, I have no parallel steps, 'set number copies to start = 1' for all steps, so I am reading 1 row and writing 1 row at time in the empty destination table.

    Also, there are no other users connected to this database as it is a local installation, and I am the only one using it.
    Last edited by dnrickner; 11-25-2013 at 04:48 PM.

  6. #6
    Join Date
    Feb 2011
    Posts
    152

    Default

    I have 200 tables that I am doing this process for. Only 2 experience this problem. The other tables that work have anywhere from 10 to 900,000 rows in them and have their batch size set to 5000, and I have set number of copies to 4 on the table output step. The only tables I experience this issue with are ones with high column counts. The other table that shows this problem has 181 columns in it. All others work fine.

  7. #7

    Default

    Hi dnrickner,
    I think I have exactly the same problem, did you finally manage to solve it?
    Regards

  8. #8
    Join Date
    Feb 2011
    Posts
    152

    Default

    No. I never did. I just re-worked the design of my transformations / jobs so it worked differently.
    Quote Originally Posted by elkarel View Post
    Hi dnrickner,
    I think I have exactly the same problem, did you finally manage to solve it?
    Regards

  9. #9

    Default

    Ok thanks! Quite annoying, I'll try to file a bug.
    Regards

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.