Hitachi Vantara Pentaho Community Forums
Results 1 to 7 of 7

Thread: Strange plugin behaviour when generating multiple output rows from a single input row

  1. #1
    Join Date
    Feb 2011
    Posts
    6

    Default Strange plugin behaviour when generating multiple output rows from a single input row

    I'm writing a Kettle Lookup plugin for our content management software, and I've come across some odd behaviour when my lookup plugin generates multiple output rows from a single input row.

    Basically what happens is, if I add a step (e.g. Select Values or Text File Output) to receive the output from my Lookup plugin, the receiving step, instead of receiving all the multiple output rows, seems to receive the last record multiplied several times. For example, if my Lookup plugin has an input of 1 row and outputs 10 rows, the receiving step seems to get the last row 10 times!

    But what's even odder is that if I link the output from my Lookup to a second step, so that the data is copied to two output steps, the second step receives the correct data (i.e. all 10 output rows), whereas the first step still gets the last output row copied 10 times.

    I've only tried this out with Select Values and Text File Output plugins, so it may be a problem limited to them.

    Does anyone know what might be causing this behaviour?

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    You forgot to clone the input row? (Object[] gets re-used and so you end up changing the same row 10 times)

  3. #3
    Join Date
    Feb 2011
    Posts
    6

    Default

    Quote Originally Posted by MattCasters View Post
    You forgot to clone the input row? (Object[] gets re-used and so you end up changing the same row 10 times)
    I thought that's what I was doing. Here's my code:

    Code:
    for(StructureContext ctx : results) {
       String namePath = ctx.getNamePath().toString();
       String idPath = new IdNamePath(ctx.getIdPath().getElementsAsIntArray()).toString();
       String name = ctx.getName();
       int id = ctx.getId();
    				        	
       Object[] outputRow = RowDataUtil.addRowData(r, data.previousMeta.size(), new Object[] {namePath, idPath, name, Integer.toString(id), Integer.toString(numMatches), message});   	
       putRow(data.outputMeta, outputRow);  
    }
    Isn't outputRow going to be a different array with different data each time I call addRowData?

    (By the way, I'm really impressed with the PDI tool. It's nice to come across something so well designed. I spent several weeks evaluating most of the other free ETL tools before settling on yours, but I must admit I'd already made my decision within about 10 minutes of sitting down and using PDI ).
    Last edited by ncurrie; 02-09-2011 at 10:44 AM.

  4. #4
    Join Date
    Feb 2011
    Posts
    6

    Default

    Quote Originally Posted by MattCasters View Post
    You forgot to clone the input row? (Object[] gets re-used and so you end up changing the same row 10 times)
    Ah, I see. I cloned the inputRow and it all worked. Thanks :-).

    The working code looks like this:

    Code:
    Object[] inputRow = r.clone();
    Object[] outputRow = RowDataUtil.addRowData(inputRow, data.previousMeta.size(), new Object[] {namePath, idPath, name, Integer.toString(id), Integer.toString(numMatches), message});

  5. #5
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Or slightly safer (binary objects and such):

    Code:
    Object[] inputRow = getInputRowMeta().cloneRow(r);
    Alternatively, you can use

    Code:
    Object[] inputRow = RowDataUtil.createResizedCopy(r, nrOutputColumns);
    HTH,

    Matt

  6. #6
    Join Date
    Feb 2011
    Posts
    6

    Default

    Quote Originally Posted by MattCasters View Post
    Or slightly safer (binary objects and such)
    Excellent. Thanks for the tip.

  7. #7
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Another tip. In general, always work with the RowMetaInterface methods to get data from a row. That way you'll always be sure to get the result you want.
    For example, don't assume that a field is a string, call getInputRowMet().getString() to get the value. That way, the data is converted if needed.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.