Hitachi Vantara Pentaho Community Forums
Results 1 to 9 of 9

Thread: Limited Table Input

  1. #1

    Default Limited Table Input

    Please help,

    The Case is i've got a external table input transformation which use postgreSQL,
    it works fine until Row 300.000.
    yes, it's stop at row 300.000.
    ...
    the log said that the connection is closed.

    so the question is, is there a way around it?
    Like is it possible to make the input stop at 299.999 and continue again after a minute or something?
    or
    is it possible to use job to use 2 transformation, the first one to grab 299.999 row and second one for another 299.999 ?

    thank you

  2. #2
    Join Date
    Aug 2016
    Posts
    290

    Default

    If you have an ID column, you could use that to split the select.

    Use a loop by having job A output the ID limits in result rows, then for job B go to "Advanced" and use the "Execute for every input row" checkbox. Then pass the ID limits in Parameters under "Stream column name".

  3. #3
    Join Date
    Nov 2013
    Posts
    382

    Default

    You should better look for a limit on your PostgreSQL options. We use Table Input step with a PostgreSQL with some milion records without major problems.

  4. #4

    Default

    Quote Originally Posted by Sparkles View Post
    If you have an ID column, you could use that to split the select.

    Use a loop by having job A output the ID limits in result rows, then for job B go to "Advanced" and use the "Execute for every input row" checkbox. Then pass the ID limits in Parameters under "Stream column name".
    ID Column? is it akin of rownum?

    what if there's no id column?

    i used to use date of entry but one of the table don't have one

  5. #5

    Default

    Quote Originally Posted by DepButi View Post
    You should better look for a limit on your PostgreSQL options. We use Table Input step with a PostgreSQL with some milion records without major problems.
    i'm sorry but, where could i look for my PostgreSQL options?

  6. #6
    Join Date
    Nov 2013
    Posts
    382

    Default

    SQL: show all

    No ofense intended, but if you don't know what an ID column is, neither where to look for PostgreSQL parameters, better ask a database manager/admin to look at your database.

  7. #7

    Default

    Quote Originally Posted by DepButi View Post
    SQL: show all

    No ofense intended, but if you don't know what an ID column is, neither where to look for PostgreSQL parameters, better ask a database manager/admin to look at your database.
    Yes, thank you for answering
    no offense accepted
    i'm new to database

    i'm asking the question abaout id column because at the database i'm working on the id column is not just generated number, but akin of random alphanumeric.
    and i'm working with this case in more than five table so i'm looking for an answer which i could use as a standard on how i would use it to process all.

    a little confirmation please, about the PostgreSQL parameters do you mean i should change the postgresql.conf?
    well, the thing is i'm working on external database which mean i don't have an access to change the postgresql.conf.

  8. #8
    Join Date
    Nov 2013
    Posts
    382

    Default

    If you get a consistent error after 300,000 rows read it means there is somewhere some parameter that limits the input rows. Most probably on PostgreSQL side ... but this is something you should check with your system people.

    I'm not aware of any kettle parameter closing the connection after a specific number of rows.

  9. #9
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    As always I recommend installing a standalone Java JDBC client like SQuirreL to eliminate problems that are not related to Kettle in the first place.
    So long, and thanks for all the fish.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.