Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Only pass unique rows?(Verifies keys only) instead of Unique rows step

  1. #1
    Join Date
    Aug 2015
    Posts
    313

    Default Only pass unique rows?(Verifies keys only) instead of Unique rows step

    Hi Sir,Madam,

    I am confusing about sort rows step and unique rows step. Please help on my scenario. I need to load millions of records and i am using ( Table input -> sort rows -> unique rows -> update) steps.

    I haven't use order by in table input sql query thats why i used sort rows step and applying unique rows then. We have option in sort rows step (Only pass unique rows?(Verifies keys only).

    I verified the this thread and not yet getting conclusion(http://forums.pentaho.com/showthread...-Sort-w-unique).

    Final suggestion need from your end, i can ignore unique rows step if i choose (Only pass unique rows?(Verifies keys only) in sort rows step ? or are there any issues if i follow this way ?

    Thank you,
    Santhi

  2. #2
    Join Date
    Jan 2015
    Posts
    107

    Default

    Are the fields you are sorting by enough to determine uniqueness or do you want to check ALL fields?

    Using Sort Rows by itself seems the most efficient option, but I don't know what the step will do if you define (example) 60 fields to sort by.

  3. #3
    Join Date
    Aug 2015
    Posts
    313

    Default

    Quote Originally Posted by Isha Lamboo View Post
    Are the fields you are sorting by enough to determine uniqueness or do you want to check ALL fields?
    yes , i want to determine uniqueness on key column and not for all columns.
    Quote Originally Posted by Isha Lamboo View Post
    Using Sort Rows by itself seems the most efficient option, but I don't know what the step will do if you define (example) 60 fields to sort by.
    I dont want to sort all 60 fields only one column

  4. #4
    Join Date
    Jan 2015
    Posts
    107

    Default

    Then just use Sort Rows with pass unique rows only.

    Do note that for each unique key, you will get a random record, whichever one the database coughs up first for each key.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.