Hitachi Vantara Pentaho Community Forums
Results 1 to 10 of 10

Thread: Table Output Question

  1. #1
    Join Date
    Oct 2006
    Posts
    24

    Default Table Output Question

    I've been having difficulty optimizing my loads. Out of the box i was getting about 200 r/s, which just isn't good enough.

    I've gotten all of my widgets up to the table output widget passing data at a rate
    of 3000 r/s, but it just stalls at the table output.

    I've changed the commit size and have checked batch update for inserts and the data screams through, but even after all the data is loaded, the table output doesn't get to a 'finished' state until minutes after all the data has moved.

    What am i doing wrong? What other information do i need to post for proper help?

    mike

  2. #2
    Join Date
    May 2006
    Posts
    4,882

    Default

    I think you've hit a known problem in 2.4.1-M1 (the bare download). Get yourself a new kettle.jar and try again please.

    For other speed increases... make sure that you co-locate the execution of your transformation with your databases. And possibly start up more copies of your table output: right click on the table output step and use "change number of copies to start"

    Regards,
    Sven

  3. #3
    Join Date
    Oct 2006
    Posts
    24

    Default still issues

    I was actually using 2.3.1

    So I downloaded 2.4.0 build 742. This time the batch updates for insert doesn't appear to be working at all. The best times I get are 300 r/s. It's as if that option is being completely ignored. I know I have to be doing something wrong, but I see no other options to click here....help.

    As a side note the change number of copies to start gives me an error, i have an exclusive lock on the table with the first instance, and any other instance just errors out.

  4. #4
    Join Date
    May 2006
    Posts
    4,882

    Default

    I didn't mean 2.4.0, but the 2.4.1-M1 with the latest kettle.jar, that should have everything back working again.

    What kind of database are you writing to and which error do you get?

    Regards,
    Sven

  5. #5
    Join Date
    Oct 2006
    Posts
    24

    Default error

    Alright...I'm downloading that new version now...i'll post results in a few minutes

    Sybase

    2007/03/12 12:52:50 - SRC_ANI_TMP.1 - ERROR (version 2.4.0, build 742 from 2007/01/27 12:29:00) : [Sybase][ODBC Driver][Adaptive Server Anywhere]Cannot open the requested object for write in the current transaction (78818431). Another user has write access in transaction 78818429.

  6. #6
    Join Date
    Oct 2006
    Posts
    24

    Default 2.4.1 M1

    Alright, I donwloaded this version and am still seeing the same type of results.

    If I set my commit size to 10000, and click on batch update for insert...the output table reads data until the 10000 row mark then appears to wait for roughly 45 seconds before it outputs (writes) the data to the table. I don't understand the wait time....are there any other options i should check?

  7. #7
    Join Date
    May 2006
    Posts
    4,882

    Default

    Now after v2.4.1 as a last step... download kettle.jar from http://www.javaforge.com/displayDocu...ar?doc_id=3719 and put that one in the lib directory overwriting your current kettle.jar

    For starting duplicate steps... it doesn't work Sybase, that's one of the odd databases.

    Regards,
    Sven

  8. #8

    Default

    Quote Originally Posted by mkirkeide View Post
    I don't understand the wait time....are there any other options i should check?
    The 'wait' time is most certainly the time it is taking for the transaction to complete.

    I cannot comment if 45 seconds is 'slow' or not since this is highly dependent on many factors such as dB settings, network load, server load ... etc.

    Regards,
    JM

  9. #9
    Join Date
    Oct 2006
    Posts
    24

    Default New .Jar file

    I used the new .jar file, and I have the same results...I don't doubt it's a setting in Sybase that I need to configure...however, I'm working with a very junior DBA, so it's a guessing game at this point....has anybody had any experience with Sybase and can point me in a direction???

    I'm pretty sure it has to do with commiting. I have a lookup table pulling from sybase and it's getting records at over 1000 r/s...it's just the table output widget.

    When I uncheck the batch updates option, it slows down to 100 r/s.

    I appreciate the help so far.....


    Mike

  10. #10
    Join Date
    Oct 2006
    Posts
    24

    Default Sybase

    This appears to be some sort of sybase issue...i'm going to repost another topic on this.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.