Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Counts in Preview and Table Output seem wrong

  1. #1
    Join Date
    Mar 2016
    Posts
    1

    Default Counts in Preview and Table Output seem wrong

    I have two questions that are possibly related to each other:

    I've created a Transformation in PDI to copy data from one database to another, with some steps in between to fit the target schema.

    1) When I "Preview" the various steps I get incorrect and inconsistent counts. For example, I have a table with 123 rows. None of my steps have filters/where clauses (at least none that I intend).
    When I click Preview, I get row counts all over the place - 3 records, 74 records, 123 records etc. I can Preview the same step multiple times and get different counts.
    - Is this known behavior?
    - Is there a configuration setting that causes this to happen, which I could change it to force the correct number? The Number of Rows to Preview in the Launcher is already set to 1000, which is more than enough for the examples in question.

    2) The final step of my Transformation is Table Output. Again I expect 123 records to be output. For some reason it only writes 122. When I look at the Step Metrics tab in the Execution Results it clearly shows Input 123, Output 122. That matches what I get when I query the DB using another program - 122 rows inserted, but it should be 123.
    Any guesses or suggestions as to why this happens or how to solve it?

    Thanks.

    UPDATE: About question 2), I figured out that the table I am inserting into has a unique constraint that's not present on the source table, and there was one duplicate value, which explains why I ended up with 122 rows instead of 123. Is there any way to make issues like that more visible in PDI? For example can I make it raise an error rather than inserting with no complaints? By comparison when I write a manual insert Postgres will complain:

    PostgreSQL said: duplicate key value violates unique constraint "ix_mycol_name"
    Detail: Key (mycol)=('duplicate value') already exists.
    Last edited by skarger; 03-23-2016 at 04:48 PM.

  2. #2

    Default

    You could capture errors from the step and either output them to a file or to a table.

    Hugo

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.