Hitachi Vantara Pentaho Community Forums
Results 1 to 6 of 6

Thread: missing records in table input step

  1. #1
    Join Date
    Apr 2008
    Posts
    29

    Default missing records in table input step

    hello, everyone;

    we are using Pentaho version 3.0.3. build 569.
    we have experienced the following issue:
    In one of our transformations we have a table input step that uses a connection to the SQL server 2005 (instance). Connection works fine. If you run this transformation, it finishes, and no errors are reported. However 3 last records available in the source system table are not transfered to the target at all, and the very last record have data missing (the fields are set to null/blanks even though in the source datatbase tese fields are set to actual non-null/non-blank values).
    When copying the SQL statement from the table input step to the Microsoft SQL Server management Studio and running it from the query window, it returns 199 records; when using a preview in Pentaho the same statement results in just 196 records and in the very last record displayed in preview window, some data is missing (the preview limit is set to default (which is 1000).
    Does anyone know what might be causing this issue?
    Unfortunately, we cannot alter the data set of the underlying database to test what might be causing this issue and whether it is data related. We have a smaller data set in anothther database, and the same transformations runs fine (there are only 77 records in that database in the table), and the same transformations runs fine pulling all 77 records.

    Thank you for all of your help

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Perhaps you forgot to commit a transaction here or there?

  3. #3
    Join Date
    Apr 2008
    Posts
    29

    Default

    Thanks, Matt;
    I have a commit size set on the output step which updates the target table. But I do not understand how not committing a transaction in the table output step (which in my case, I know, is not so) will impact the number of the records read in the input step (the records are not transferred to the target because they are not read from the source). The problem is that the input step does not read all the records available in the table, skipping the last three (see my original message).

    Thanks again.

  4. #4
    Join Date
    Sep 2008
    Posts
    1

    Default Posting a question...

    Hi,

    Can anyone say how to post a question? On the left panel i see "New posts" but not able to post a new question.

    Thanks,
    krp

  5. #5
    Join Date
    Apr 2008
    Posts
    29

    Default

    Never mind, and thank you again.

    I think I have figured it out. I have three transformations (stored outside of the repository) which have the same name and which use the same-named connection to the source. I have three dbs, named differently, which are specified in each of the connections of these three transformations. If I load one transformation and run it, and then I close it and load another one with the same name, but which uses a different database specified in the connection, Pentaho uses the connection of the previous transformation. I need to close and restart the Pentaho in order for this issue to go away.

    Thanks again.

  6. #6
    Join Date
    Apr 2008
    Posts
    29

    Default

    krpnc;
    use "New Thread" green button right above the threads list for the forum.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.