Hitachi Vantara Pentaho Community Forums
Results 1 to 6 of 6

Thread: Automatically import repository from XML file

  1. #1
    Join Date
    Mar 2007
    Posts
    14

    Default Automatically import repository from XML file

    Hi,

    We are trying to build an automatic test framework against the the Kettle ETL development effort. One of the critical thing is to be able to import the latest kettle repository from the XML file.

    We know we can export a repository to a XML file via Pan however the only way to import XML file into the repository is through the repository explorer UI in spoon. It seems the export/import cycle is incomplete for automation purpose.

    I know I can submit a feature request for this and you can consider to include it in the product depends on the priority. However we are kind of in an urgent need for this.

    So my question is, if we want to implement our customize code to achieve this goal based on the open source, how hard is it and where to start? Appreciated if anyone can give us a pointer on where to start with.

    Thanks a lot!

    Regards,

    Tao

  2. #2
    Join Date
    Jul 2007
    Posts
    247

    Default

    Are you sure you need XML files to do this?

    I'm not sure if I got it right but I think you're trying to automatically set up a second system/installation on basis of your current kettle repository?!

    You mentioned exporting... is your current repository stored in a database? If so, why don't you create a script that dumps all the repository tables and imports them in the target database again. Maybe there's also a database link between the two databases so no dumping is necessary and you could work with direct inserts...

    To me this sounds a lot easier than to expand pan's functionality.

    Just an idea though...

    Regards,
    Ben

  3. #3
    Join Date
    May 2006
    Posts
    4,882

    Default

    Personally I would copy pan, rename it. Remove all code not related to exporting to xml, then take spoon and follow the code to import xml files.

    Regards,
    Sven

  4. #4
    Join Date
    Mar 2007
    Posts
    14

    Default

    Thanks for your response and advise.

    Let me make myself more clear:

    1. Our Kettle repository is in the database, however, we will export the repository to a XML file for easy store, source control and distribution.

    2. In production environment, we will import the XML into the production database via spoon UI before we can use it.

    3. For automatic test purpose, we would like to get the latest XML file and import it into a new database to test the development. That's where we need an automatic way to import repository from XML to database which is missing in current Kettle.

    4. We could also take the db dump from the development and use it in the auto test to by pass the XML export/import process. However, first it is hard to have source control on db dump and second, we support multiple databases (SQL Server, Oracle and DB2). If we take db dump approach we have to take db dump for each database while using XML can avoid this.

    Thanks,

    Tao

    Quote Originally Posted by BeLienig View Post
    Are you sure you need XML files to do this?

    I'm not sure if I got it right but I think you're trying to automatically set up a second system/installation on basis of your current kettle repository?!

    You mentioned exporting... is your current repository stored in a database? If so, why don't you create a script that dumps all the repository tables and imports them in the target database again. Maybe there's also a database link between the two databases so no dumping is necessary and you could work with direct inserts...

    To me this sounds a lot easier than to expand pan's functionality.

    Just an idea though...

    Regards,
    Ben

  5. #5
    Join Date
    Mar 2007
    Posts
    14

    Default

    The repository import related code is in be.ibridge.kettle.repository.dialog.RepositoryImportProgressDialog.java. Is this the right code to look at? Is there other exposed interface or utility in the source that we could use for this import functionality or we just have to convert the import method in this class and rip off any UI related code as well as resolving context such as current repository and repository directory somehow?

    Thanks,

    Tao

    Quote Originally Posted by sboden View Post
    Personally I would copy pan, rename it. Remove all code not related to exporting to xml, then take spoon and follow the code to import xml files.

    Regards,
    Sven

  6. #6
    Join Date
    May 2006
    Posts
    4,882

    Default

    method importAll() in there looks like a promising start.

    Regards,
    Sven

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.