Hitachi Vantara Pentaho Community Forums
Results 1 to 4 of 4

Thread: Performance problems saving to Mysql DB

  1. #1
    Join Date
    Jan 2006
    Posts
    313

    Default Performance problems saving to Mysql DB

    I use Metadataeditor 1.6.0.GA.162

    I clean delete all tables form the metadata database, in order to get new tables when metadataeditor starts up.
    I save and give it a name.
    I create a connection and load 100 tables. It takes 50 seconds to load it from an AS/400 DB2/400 database. Not fast but ...
    When i save again it takes over 5 minutes to save the whole to the database.
    Does anyone have better performance using mysql5?


    repository.properties is
    Code:
    # Class name of MDR storage plugin factory
    org.netbeans.mdr.storagemodel.StorageFactoryClassName=org.netbeans.mdr.persistence.jdbcimpl.JdbcStorageFactory
    # JDBC driver for MySQL
    MDRStorageProperty.org.netbeans.mdr.persistence.jdbcimpl.driverClassName=org.gjt.mm.mysql.Driver
    # URL for database storage
    MDRStorageProperty.org.netbeans.mdr.persistence.jdbcimpl.url=jdbc:mysql://bewymysql:3306/cwm_repo_test?defaultFetchSize=500&useCursorFetch=true
    # user name
    MDRStorageProperty.org.netbeans.mdr.persistence.jdbcimpl.userName=cwm_repo_test
    # password
    MDRStorageProperty.org.netbeans.mdr.persistence.jdbcimpl.password=xxxxxxxxx
    # schema name (used as a table prefix)
    MDRStorageProperty.org.netbeans.mdr.persistence.jdbcimpl.schemaName=cwm
    # turn on debugging!
    MDRStorageProperty.org.netbeans.mdr.persistence.jdbcimpl.debugPrint=false
    # data type mappings...
    MDRStorageProperty.org.netbeans.mdr.persistence.jdbcimpl.datatype.mofid=BIGINT
    MDRStorageProperty.org.netbeans.mdr.persistence.jdbcimpl.datatype.streamable=BLOB
    MDRStorageProperty.org.netbeans.mdr.persistence.jdbcimpl.datatype.string=VARCHAR(750)
    MDRStorageProperty.org.netbeans.mdr.persistence.jdbcimpl.datatype.int=INT
    # Query for duplicates to prevent transaction aborts
    # This can be disabled for some future PostgreSQL version with real subtransaction support
    MDRStorageProperty.org.netbeans.mdr.persistence.jdbcimpl.queryDuplicates=true

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    The problem is in the Sun MDR logic. It uses tables to run the Metadata Object Facility (MOF).
    Tables are dynamically created and destroyed to do so. That's causing the slowdown.
    Personally, I would run on the local machine (from file) until I really had to push it into a MySQL repository.
    Even safer is to export to XMI after everything is OK and then import it into the MySQL repository.

    That way you can edit you models at ... euh... more reasonable speed.
    BTW, this performance drop is the price we pay for going with the CWM model.
    We are investigating alternatives, but it's a "nobody ever got fired for chosing CWM" situation ;-)

    Take care,

    Matt

  3. #3
    Join Date
    Oct 2007
    Posts
    1

    Default

    Hi,
    Could you elaborate a bit on "going with the CWM model" part. To my best understanding CWM is just a framework
    (or a metamodel, if you wish) so I'd like to understand what do you mean by 'CWM model' here. Do you have any
    technical details of your metadata implementation available? A whitepaper or an ER diagram perhaps?

    Best regards,
    marx

  4. #4
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    CWM is indeed a metamodel. However, to persist/store a metamodel, you need to store it in a (Level 3) Metadata Object Facility, MOF.

    These are the levels you can discern:

    Level 0 : Data
    Level 1 : Metadata (describing data)
    Level 2 : Metamodel (describing metadata)
    Level 3 : MOF (describing metamodels)

    Typical examples of metamodels that can be stored in a MOF are the CWM and UML.

    Note: There is no level 4 since a MOF can be stored in another MOF.

    The MOF used here is the MetaData Repository (MDR) from Sun Microsystems Netbeans project.
    --> http://mdr.netbeans.org/

    HTH,

    Matt

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.