Hitachi Vantara Pentaho Community Forums
Results 1 to 5 of 5

Thread: how to import data from tables on several servers?

  1. #1

    Default how to import data from tables on several servers?

    Hi, All

    I'm really quite fresh to Kettle. Now I'm blocked again. my problem is:

    I have around 100 servers, and each server locates a database, whose names and table structures are exactly the same. I need to import data from those databases on each server which has their own ips.

    what I'm doing is to create 100 connections to each database on each server (only IP are different), and apply each connection to a table. The huge task is still going, but I'm thinking it's heavily stupid.

    Can I use some simple method to import data from these tables (if the ips can be looped, then it should be great)? Any helps on this should be greatly appreciated.

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Hello Shangai!!

    That is indeed possible.
    You can use variables in your database connection.
    Create one connection with ${DB_HOST} as the hostname in it.
    In a job, create a transformation that sets 100 rows with 100 ip-addresses in it.
    The next entry in the job is a "Job" sub-job entry that loops over these 100 ip-addresses.
    In that sub-job entry, set an environment variable DB_HOST and then execute the desired transformation(s).

    That does the trick. It's actually easier to do than to describe it. :-)

    Good luck!

    Matt

  3. #3

    Default

    Thanks for your quick help very much.

    but please forgive my slow understanding. I'm still blocked on the sub-job entry. though I know how to set environment variables, I really don't know how to start the loop. where's the loop in the new job?

    expecting more helps, thanks.

  4. #4
    Join Date
    May 2006
    Posts
    4,882

    Default

    Have a look in your installation at samples\jobs\process all tables ... it does something similar with tables.

    Regards,
    Sven

  5. #5

    Default

    Thanks Sven and Matt, I finally got it solved.

    This loop is really useful.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.