Java Plugin: different output between Preview and Run of a transformation

[danielh]

Hey, I developing a java plugin which should create phonetic codes from strings. The plugin read the inputstream, for example two columns with surname and name, these string were transformed with different phonetic algorithms and for each algorithm and column there are will generated a new column with the codes.
Now, when I do a preview of the transformation in pentaho kettle everything works great and the source columns and the new generated columns were written in the output textfile. But when I run the transformation normally I got just the two source columns in the output textfile but not the new generated columns.
I haven't any clue whats going wrong.
Are there two different routines/classes used? I really hope somebody can help me.

Here is my Step Java Class.

7409Kind regards
Daniel

[Mark.James.Cloud2Land]

Hi Daniel,

Just so I'm clear your plugin takes an arbitrary number of fields and creates a new phonetic field for every one? and it can apply multiple algorithms? so if you select 3 algorithms you'd have 4 x <original number of fields> in the output

From a quick scan it doesn't look like you're adding the new fields (columns) meta values to data.outputRowMeta (used in putRow to pass your data to the next step)

in your if(first) block try something like:



...
if (temp_check_alg[1]==true) {
ArrayList<String> col_sound = createColumnsNames("Koelner Phonetik");
for (String newFieldName : col_sound)
data.outputRowMeta.addValueMeta(new ValueMeta(newFieldName, ValueMeta.TYPE_STRING))
...

[danielh]

Hey Mark,
thx for your solution. Yeah you're right, you can choose multiple algorithms and for each you get a new column. I tried your solution, but it is still the same problem. I have no idea, why it runs well in debug-mode but not the normal way. I can assume the columns are created correctly. What else can lead to this problem?

Regards
Daniel

[Mark.James.Cloud2Land]

Hi,

I'm guessing you have your plugin connected direct to a text output step? that step will need to know which fields to expect if you right click it and "show input fields" is it correct? similarly for your step -> show output fields. FYI, the PDI preview does actually run the transformation so you can't be far off.

[danielh]

Your guess is right. The output fields of my plugin show the two source fields and a field called phonetic algorithm (the actual name of step). So the new columns are created. Here is a screenshot:
7413

Why is a field with name of the stepname created but not the neccessary output fields?
Regards,
Daniel

[Mark.James.Cloud2Land]

I've had a read of your step code and I can't see where you're adding the new ValueMetas to data.outputRowMeta?? if you add the following before putRow(...) what do you get?



log.logBasic("output meta len = "+data.outputRowMeta.size());
log.logBasic("output data len = "+outputRow.length);

[danielh]

When I run the transformation I got this output:
2011/07/08 14:25:19 - phonetic transformation.0 - output meta len = 2
2011/07/08 14:25:19 - phonetic transformation.0 - output data len = 2

BUT when I run the preview I got this:
2011/07/08 14:27:21 - phonetic transformation.0 - output meta len = 6 (2 Inputfields with 2 algorithm = 6 outputfields)
2011/07/08 14:27:21 - phonetic transformation.0 - output data len = 6

this is the actual Stepcode

I try your solution at line 55
7423I have no clue why there is still a difference?

Regards,
Daniel

[Mark.James.Cloud2Land]

Have you implemented preview in your dialog? if so can you attach that too. also can you export your trans as a ktr and attach that, possibly something's not saving correctly in the meta. it is a strange problem though.

[danielh]

I didn't implement a preview in my dialog. You can just give a stepname and choose from different algorithms.

7424
7425
7426Iam really glad, that you try to figure out my problem.

Regards,
Daniel

[Mark.James.Cloud2Land]

Hi,

Well I'm not sure why preview works but I see why execute doesn't, it's reading from the saved version of your meta data (eg. a repository entry) and your meta class isn't saving all the data you're configuring with your dialog. have a look at the code for a org.pentaho.di.trans.step.<anything>Meta class, you need to implement getXML, readData, readRep and saveRep so that the information you're setting in the dialog is actually saved in the repository/ktr file.

[danielh]

Hey Mark,

finally I implemented the getXMl and loadXML methods. But now I have another Error when I try to run my transformation.
This is my actual classes

7468
7469
7470
7471The Meta Data is correctly written in the xml/ktr file and I see the new outputfields within the next input fields but when I run/preview my transformation I got this error from the TextFileOutput:

2011/07/24 15:48:27 - Text file output.0 - ERROR[...] org.pentaho.di.core.exception.KettleStepException:
2011/07/24 15:48:27 - Text file output.0 - ERROR[...] Field [Koelner_Phonetik_name_01] couldn't be found in the input stream!
2011/07/24 15:48:27 - Text file output.0 - ERROR[...] org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.processRow(TextFileOutput.java:119)
It doesnt matter whether I run "getFields" in TextfileOutput or not!

What could lead to this error. I actually google it but didn't find a solution or hint.

Regards Daniel

[Mark.James.Cloud2Land]

Hi,

I'm guessing you're adding the new fields to the step's output data but not to the output row's meta, you have to add the new field definitions to that. (name, type, format...)

[danielh]

Isn't that what I am doing in the getFields() methode in StepMeta? If yes, I already added the new fields to the meta data and they appear in the ktr/xml file. Thats why I am confused about the error message.

Regards
Daniel

[Mark.James.Cloud2Land]

Hi,

That's saving them ok, but you're not "using" them in the main step class. In PhoneticStep's if (first) block you need to add the new fields meta to data.outputRowMeta. e.g data.outputRowMeta.addValueMeta(new ValueMeta(newFieldName, ValueMeta.TYPE_STRING));

[danielh]

Hey Mark,
okay that make sense and I implemented the addValueMeta method in the first block, but now I got another error message.

[...]Text file output.0 - ERROR org.pentaho.di.core.exception.KettleStepException:
Text file output.0 - ERROR: Error writing line
Text file output.0 - ERROR: 0
Text file output.0 - ERROR: org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.writeRowToFile(TextFileOutput.java:253)
Text file output.0 - ERROR: org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.processRow(TextFileOutput.java:168)When I run the transformation in Safe Mode I got this error message:

Text file output.0 - ERROR: Unexpected error :
Text file output.0 - ERROR: org.pentaho.di.core.exception.KettleException:
Text file output.0 - ERROR: Safe mode check noticed that the length of the row data is smaller (0) than the row metadata size (6)
Text file output.0 - ERROR: org.pentaho.di.trans.step.BaseStep.getRow(BaseStep.java:1568)
Text file output.0 - ERROR: org.pentaho.di.trans.steps.textfileoutput.TextFileOutput.processRow(TextFileOutput.java:75)All I figured out is, that there is a probably a problem with mixing rows, but how can I solve this issue?

Regards,
Daniel