Hitachi Vantara Pentaho Community Forums
Results 1 to 7 of 7

Thread: Kettle 4.4 takes very long for loading and writign to repository

  1. #1

    Default Kettle 4.4 takes very long for loading and writign to repository

    Hi,

    I am using Pentaho in the following configuration:

    - PostgreSQL server on a unix VM
    - PDI installation on a windows VM (both VMs are hosted on the same hardware)
    - local PDI installation on my notebook

    When accessing a repository from the windows VM PDI installation, everything runs fine and smooth. When accessing the repository from the local PDI installation, it takes a long time to read and write the transformations and jobs from the repository (around factor 20).

    When accessing other data on the SQL server from my local PC, everything is fine. It only seems to be very 'laggy' with PDI and accessing the repository.

    Are there any specific settings that I have to adjust in order to improve the performance?
    Thanks for every hint!

    Max

  2. #2

    Default

    Nobody got an idea?

  3. #3

    Default

    Well, we're sitting in the same boat here, Bobse. When I use the repo on a locally running MySQL server it is fine but when using the remote MySQL-server through OpenVPN - good lord almighty - the speed down factor is around 30!

    So now I took the time to have a closer look at what is going on and I must disappoint you as I don't have any solution for this issue (yet). But what I figured out is that Kettle goes crazy on my poor DB.

    Name:  test-job.gif
Views: 39
Size:  8.4 KB

    The logged SQL queries and statements involved in setting up the job and transformation you see on the picture and then deactivating the connection in the transformation (and nothing else) encompasses according to the log (attached):

    2'552 commands of type "Query"
    1'658 commands of type "Execute"
    851 commands of type "Prepare"

    ... no kidding - this is inefficient by design I guess and usually compensated thanks to fast computing and networks. But if there is something lagging it hits you like Tom a piano falling down from 5th floor.

    It is also kind of funny that the exported repo (XML document) storing above job is sized mere 18 KB - while the log for its creation keeps 427 KB data. This seems a bit like trying to conduct brain surgery with an axe ...

    Currently the best work-around solution seems to be to do the development local and then deploy the repository to the production server.

    In case I find a better solution I am going to post it on my web-site.

    Regards

    Raffael
    Attached Files Attached Files
    Last edited by joyofdata; 11-08-2013 at 10:54 AM.

  4. #4

    Default

    Hi Raffael,

    Thanks for the info, guess I will go with the local development and then uploading it to the database :/
    BTW, nice blog!

    Bobse

  5. #5

    Default

    I DO (mostly) "enjoy" working with Kettle and I am looking forward to version 5 (BTW, why is the CE not ready yet while PDI 5 is available already?) but on those issues I would very much appreciate some serious and insightful comment by Matt Casters or maybe even Lord marabu 2nd Earl of Cunningham ... yes, I think that there is some tendency on this board to ignore posts that address weaknesses of the product - and that is very sad.
    Last edited by joyofdata; 11-08-2013 at 11:04 AM.

  6. #6

    Default Pentaho Kettle 5 comes with improved database repository performance

    Pentaho Kettle 5 comes with improved database repository performance - See more at: http://www.joyofdata.de/blog/kettle-...nce/#more-1781

  7. #7

    Default

    Thanks for the info, will take a look at it the next days

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.