Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Filter with Sort step needed ?

  1. #1
    Join Date
    Nov 2016
    Posts
    25

    Exclamation Filter with Sort step needed ?

    Hi All,

    I have 2 doubts.

    1. We have 10-12 transformations developed by ex-developers. Now when we look at those transformations there's always a sort step used before filter. We are pretty sure that it is not needed and can be removed as filter can work for any data and it does not require any pre-sorting. But as it is repeated everywhere, just wanted to know whether it is really required and results in better performance and/or data OR its just a bad practice that was followed and sort step can be removed completely ?

    2. Also, it is almost everywhere -
    Table input 1 --> Sort Step 1 --
    --> Merge Join
    Table input 2 --> Sort Step 2 --

    Just wanted to confirm that we can use ORDER BY clause inside Table Input step instead of using sort step explicitly ? Will this improve performance and still give proper data ?

    Thanks in advance,
    Sanket Kelkar.

    PDI CE 6.1
    Windows server standard
    Oracle 11g
    TIA,
    Sanket Kelkar.

    PDI CE 6.1
    Oracle 11g
    Windows Server 2012

  2. #2
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    (1) You are right, Filter-Rows doesn't require rows to be sorted, but other steps do. Look out for Group-By, Unique-Rows, ..., downstream.

    (2) You can use ORDER-BY to establish a sort order, but if a Merge-Join step is used, make sure the database and Java use the same collation for sorting. Usually, it's safer to use Sort-Rows in those scenarios.
    So long, and thanks for all the fish.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.