Job 'Batch ID' with multiple nested Jobs...

[JMOSSER_NISC]

I think I may have found a bug regarding how 'Batch ID' is passed from parent to grandchild Jobs. Or perhaps this is a purpose built feature and I am wasting my typing beyond this point...

In any case, I am using KETTLE 2.4.0, targeting Oracle and sourcing from Progress (if the dB stuff matters?).

The project I am working on has the need for 3 to 4 'layers' of Jobs. This allows us to run parts of the overall process separately/manually/repeatedly when needed during the testing phase.

At the topmost level is a main 'controller' (parent) job that is responsible for reporting process success or failure via an email. This parent job is also configured for logging of the child/grandchild/N-th_Child jobs and transformations.

At some level (N-deep) in the process tree, I have transformations that may need to log 'business based' exceptions. With these exceptions I was hoping to output the 'Batch ID' of the 'controller' job so that relation to the Job log table would be cake.

It appears the 'Batch ID' from a given Job is only propagated to the immediate children and grandchildren Jobs and Transformations. One other way to think of this (perhaps the correct way) is siblings and children of the Parent Job are the only things that get the 'Batch ID' of the Parent Job.

All of this rambling boils down to the question: Is this the intended behavior?

I have attached example Jobs and Transformations (2.4.0 friendly) that will display information via JavaScript Alert(...) calls.

To run the example, extract all of the files to the same folder and open/execute the 'TEST_BATCH_ID_PASSING' Job.

You may choose to enable the logging as well in the Job Properties dialog (CTRL-J). I have found that writing log info to a dB does not make a difference for this test. What you will see in the first set of JavaScript popups if logging is not enabled is a value of -1 for the 'Batch ID'... if logging is enabled, you will get a larger value.

The first three popups are from transformations invoked at a child/grandchild level in the tree (or sibling/child depending on your perspective). The 4th popup is where the 'Batch ID' is no longer propagated, you will notice that it goes to zero. The 5th and 6th popups are siblings of the 2nd and 3rd and should display the same values.

While typing this mess I have thought of one way to make this work with existing KETTLE features. In the top-most parent job, invoke a transformation that sets a 'global' variable to the value of the 'Batch ID'. That would work reasonably well.

I'm still left wondering if a parent 'Batch ID' should propagate 'down' infinitely unless 'overridden' by some descendant's logging configuration?

KETTLE devs, what are your thoughts on this?

Thanks for having this forum allowing me to rant to myself...

Regards,
JM

[MattCasters]

Isn't there a checkbox in the Job settings, something like "pass batch id to sub-jobs"?
Mmm, there must be something like that somewhere. :-)

[JMOSSER_NISC]

Isn't there a checkbox in the Job settings, something like "pass batch id to sub-jobs"?
Mmm, there must be something like that somewhere. :-)

Yes, that feature exists. However, even if all of the 'pass batch id' settings are enabled for all Jobs in the tree the 'Batch ID' is only propagated to child and grand-child Jobs and Transformations of the parent that 'owns' the 'Batch ID'. The child and grand-child items do not appear to propagate the 'Batch ID' to their children and grand-children.

I think my observation could be summed up by saying: The 'Batch ID' is not propagated 'infinitely down' even if all 'pass batch id' options are set.

Again, I'm not sure if this was ever the intended behavior of this feature and that is what I hope to determine by asking about it.

Regards,
JM

[BeLienig]

I filed in a bug report, see http://jira.pentaho.org/browse/PDI-360 for details.

BTW: any chance that this bug gets fixed until 3.0 will be out?


Regards,
Ben

[MattCasters]

I have to check the sample, but I think I already fixed it, yes.

[BeLienig]

Looks like it didn't make it to 3.0.1, I get the same result when using 3.0.1 build 534.

The JIRA tracker is also still open... any chance this get's fixed soon?

Though I would prefer PDI-361 to get implemented ;-)


Thx in advance,
Ben