PDA

View Full Version : Easy switching between Production & Test



jbleuel
02-07-2006, 12:21 PM
I added a change request (1517): http://www.javaforge.com/proj/tracker/itemDetails.do?task_id=1517&navigation=true


Often you have a PRODUCTION and TEST system (for us ETLers mostly a different database connection).



It would be good to test a transformation and use the same transformation for production (after switching to production).



Proposal: Use the same connection for test and production - so this is transparent to the transformation steps and hops. Then we would define two properties for each for "production (default)" and "test (optional)" for all needed. We could use the same Database-property and different user id, password etc. (in SAP also the client and system).



When it is a TEST it should be marked to the user may be with a watermark in the background (as before Kettle was open source there was a "not licensed" mark ;-)



Where could we switch between TEST and PRODUCTION? What makes sense?



Regards,



Jens



P.S.: Nice to have for this, too: http://www.javaforge.com/proj/tracker/itemDetails.do?task_id=1420&navigation=true

jescalante
04-08-2010, 03:16 PM
Was anything every done with this post. I am looking for a similar solution. Thanks. Jodi

MattCasters
04-08-2010, 04:53 PM
It was done years ago in the form of variables that can be set differently for the various environments.

Peter Schmidt
04-09-2010, 10:40 AM
The way that we solved this was to set up a JNDI connection, lets say:

dataWarehouse
sourceDB

Then it is just a matter of rotating out and/or configuring the appropriate jdbc.properties.

So, for example in your test environment, the JNDI connections are configured to hit test db's

MattCasters
04-09-2010, 01:22 PM
Using variables works the same. There really is no need messing with the weird JNDI files.

jbeal
04-09-2010, 04:49 PM
Thankfully most all of the dialog fields that one might want to supply dynamically let you interpolate parameters.

I've seen some people trying to use KETTLE_ variables with mixed success.

A general solution that I have used for supplying values per environment from a local file leverages existing config information shared among multiple technologies. Basically just a simple .ini style file - easily parsed in the scripting language of your choice.

I was getting familiar with PDI when the need to pass environment-specific parameters arose, else I would have probably just parsed the .ini file within a Pentaho transform and set variables for the job.

As it is I use a simple perl script (yes, perl) that parses the ini file, and invokes kitchen.sh, passing named parameters into the job with a dynamically generated command line. (Also an input file name not known until the processing needs to run). Digests piped output to log file.

Then you can just tailor an ini file for your various environments.

Not fancy but it works...

MattCasters
04-09-2010, 05:11 PM
You can do the same with Kettle since we have a step that reads properties files.
Setting the variables using JavaScript is easy enough.
If you do that in a transformation in a job as the first thing before anything else, you're set.

jbeal
04-09-2010, 05:16 PM
Thanks Matt, I'll probably switch to reading them that way when I can find time. :)

Regards,
Jeremy

dhartford
04-12-2010, 01:05 PM
The way that we solved this was to set up a JNDI connection, lets say:

dataWarehouse
sourceDB

Then it is just a matter of rotating out and/or configuring the appropriate jdbc.properties.

So, for example in your test environment, the JNDI connections are configured to hit test db's

This is also how I solved this (and in my scenario, DEV->STAGING/TEST->PRODUCTION, so 3 environments).

Either JNDI or variables will work, as one of my constraints is that the file *DOES NOT CHANGE* as you move it between environments.

And, I think with db-repos, shared connections also work the same (i.e. if you have seperate repos for TEST and PRODUCTION, but the same named connection, I think that'll work too).

jbeal
04-12-2010, 05:02 PM
Ahh the unmuted file makes it tricker...

dhartford
05-21-2010, 03:20 PM
Ahh the unmuted file makes it tricker...

It's just the transformation or job file that can not change - variables passed to it, kettle.property env variables, same JNDI name but with different configurations are all valid approaches to work around the issue.