Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: repository issues running in clustered mode v4.0

  1. #1
    Join Date
    Oct 2010
    Posts
    6

    Default repository issues running in clustered mode v4.0

    I have a job which runs several transformations in sequence.
    The first transformation reads from a database table which tells it what date for the source records we are interested in, and what record we need to start reading from. It then writes this information to a file.
    The second transformation reads the file from the first transformation, performs the query and starts processing the records to fill up the fact table.
    The last transformation updates the values in the database to increment the next record set to read.
    Then we start the whole thing again until there are no more records left.

    The second transformation is the one which is clustered.
    If I run the job in non clustered mode everything works fine and it can process 50,000 rows in 12 minutes. In clustered mode it can process 50,000 rows in about 1-2 minutes, however when processing the second batch of records, it will have the following error on the master server:
    ERROR 20-01 15:41:59,891 - 1 - Error disconnecting from database :
    Unable to commit repository connection

    Error comitting connection
    at java.lang.Thread.run (Thread.java:619)
    at org.pentaho.di.trans.step.RunThread.run (RunThread.java:88)
    at org.pentaho.di.trans.step.BaseStep.markStop (BaseStep.java:2226)
    at org.pentaho.di.trans.Trans$1.stepFinished (Trans.java:779)
    at org.pentaho.di.trans.Trans.fireTransFinishedListeners (Trans.java:937)
    at org.pentaho.di.www.AddTransServlet$1.transFinished (AddTransServlet.java:128)
    at org.pentaho.di.repository.kdr.KettleDatabaseRepository.disconnect (KettleDatabaseRepository.java:1437)
    at org.pentaho.di.repository.kdr.delegates.KettleDatabaseRepositoryConnectionDelegate.disconnect (KettleDatabaseRepositoryConnectionDelegate.java:243)
    at org.pentaho.di.repository.kdr.delegates.KettleDatabaseRepositoryConnectionDelegate.commit (KettleDatabaseRepositoryConnectionDelegate.java:268)
    at org.pentaho.di.core.database.Database.commit (Database.java:656)
    at org.pentaho.di.core.database.Database.commit (Database.java:676)


    INFO 20-01 15:42:58,784 - RepositoriesMeta - Reading repositories XML file: /export/home/jstor/data-integration/./.kettle/repositories.xml

    At this point the carte servers do not close their internal ports and everything has to be restarted as everyone is in a wait state.

    The big problem is of course the exception on the master server.
    I do have the book Pentaho Kettle Solutions (great resource BTW, I recommend it), but can't seem to figure out what's going wrong here.

    Thanks

  2. #2
    Join Date
    Oct 2010
    Posts
    6

    Default

    And I forgot to mention the database is mySQL 5.0.24

  3. #3
    Join Date
    Nov 2008
    Posts
    143

    Default

    I'd say file a JIRA issue, for it seems that the thread is not validating if it is running in a clustered mode before closing connections.

  4. #4
    Join Date
    Oct 2010
    Posts
    6

    Default

    Under 4.1.1 I still get the error, however it does not stop the transformation from running. Eventually of course it maxes out the number of allowed connections to the DB.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.