Hitachi Vantara Pentaho Community Forums
Results 1 to 2 of 2

Thread: Transformations steps in Pentaho DI vs SQL Oracle

  1. #1

    Default Transformations steps in Pentaho DI vs SQL Oracle

    I have a question about ETL with Pentaho DI.

    When designing an ETL we can do data transformation steps in database (procedures or SQL) and we can do this steps in a ETL tool like Pentaho DI, as far as I know there are steps that have better performance in ETL tool and others have better performance in database. For example there are ETL tools that make the sorts or aggregation faster than database procedures or SQL.I would like to know if there are more steps that go different in a ETL tool than in a database, in order to have some criteria to make a design decision. I am working with Pentaho DI and Oracle.

    Also I would like to know the differences in performance of executing row by row and pipelines the rows through next setps and working in whole datasets processing steps like SQL.

    Thanks in advance,

  2. #2
    Join Date
    Jul 2009
    Posts
    476

    Default

    The short answer is, it depends on your situation.

    Generally speaking, operations will be faster inside the database. If your data transformations are just selects, inserts, updates and deletes, then the database is probably the best place to do that. However, if your transformations have a more complex workflow, including things like these:

    * Multiple data sources, such as text files, web services, other databases
    * Conditional logic
    * Dimension table maintenance
    * Various types of output
    * Shell scripts
    * Lots of transformations tied together
    * and more

    then a tool like PDI becomes more attractive, because it's easier to understand and maintain.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.