Generic transformation

[ranadeepb]

Hi,

I want to create a generic transformation that I can run for several data sources.

For example, I have some files to load in a database in the same way.

Is it possible to create one transformation that can take different file formats and run the load? I am able to pass database name, table name and other connection information as variables but cannot figure out how I can pass the file layout.

Any help will be appreciated
Ranadeep

[sboden]

The simple answer is no.

There are ways around it but they're more trouble than they're worth.

Regards,
Sven

[fabianS]

Only way I can think of would be to generate the ktr file. If you limit yourself on the amount of flexibility it might actually be possible...

[ranadeepb]

Appreciate the replies. I have two followup questions -

Has anyone tried this?

Is there any plan for PDI to have a functionality in near future where it is possible to use an external file with the layout so that in the "fields" tab you can either mention each field or use a file that contains the layout?

Thanks
Ranadeep

[fabianS]

We have considered doing something like that (our database structure is volatile on the number of columns and we do sport something around 400 columns in there).

We thought of simply changing the XML file that pops out of Spoon and injecting the list of columns.

Until now we did not have enough pressure to do it and it IS a mess. ;-)

You may try to create your transformation on-the-fly - there are ways to use the kettle-core.jar and create a transformation via programming...

[sboden]

@ranadeepb ... No short term plans probably ... doing what you want to have breaks some stuff in other places. Unless you're ready to jump through a LOT of hoops, PDI likes its metadata to be "fixed".

A lot of steps e.g import meta-data into the step itself when the developer does a "get fields", if you use such a step e.g. the meta-data won't be able to change on the fly afterwards (or these steps would abort on "wrong meta data").

Regards,
Sven