Hitachi Vantara Pentaho Community Forums
Results 1 to 9 of 9

Thread: Synchronize After Merge - Anyone used this?

  1. #1
    Join Date
    Jul 2009
    Posts
    17

    Default Synchronize After Merge - Anyone used this?

    Hi
    Just wondering if anyone has used the new Synchronize After Merge Step in PDI 3.2. I've been trying to use this but it seems to give me errors. There's very little in these forums about this step and no documentation in the wiki.

    Just wondering if anyone has used this step and is familiar with its error messages. I get these errors:

    2009/08/04 13:30:17 - Synchronize after merge.0 - ERROR (version 3.2.0-GA, build 10572 from 2009-05-12 08.45.26 by buildguy) : It was not possible to find operation field [null] in the input stream!
    2009/08/04 13:30:17 - Synchronize after merge.0 - ERROR (version 3.2.0-GA, build 10572 from 2009-05-12 08.45.26 by buildguy) : Unexpected error :
    2009/08/04 13:30:17 - Synchronize after merge.0 - ERROR (version 3.2.0-GA, build 10572 from 2009-05-12 08.45.26 by buildguy) : org.pentaho.di.core.exception.KettleStepException:
    2009/08/04 13:30:17 - Synchronize after merge.0 - ERROR (version 3.2.0-GA, build 10572 from 2009-05-12 08.45.26 by buildguy) : It was not possible to find operation field [null] in the input stream!
    2009/08/04 13:30:17 - Synchronize after merge.0 - ERROR (version 3.2.0-GA, build 10572 from 2009-05-12 08.45.26 by buildguy) :
    2009/08/04 13:30:17 - Synchronize after merge.0 - ERROR (version 3.2.0-GA, build 10572 from 2009-05-12 08.45.26 by buildguy) : at org.pentaho.di.trans.steps.synchronizeaftermerge.SynchronizeAfterMerge.processRow(SynchronizeAfterMerge.java:725)
    2009/08/04 13:30:17 - Synchronize after merge.0 - ERROR (version 3.2.0-GA, build 10572 from 2009-05-12 08.45.26 by buildguy) : at org.pentaho.di.trans.step.BaseStep.runStepThread(BaseStep.java:2889)
    2009/08/04 13:30:17 - Synchronize after merge.0 - ERROR (version 3.2.0-GA, build 10572 from 2009-05-12 08.45.26 by buildguy) : at org.pentaho.di.trans.steps.synchronizeaftermerge.SynchronizeAfterMerge.run(SynchronizeAfterMerge.java:1014)

    The error suggests that there is a null in the input stream but my previous step is providing no nulls.

    If anyone is familiar with this step and could help, I'd be grateful.

    Thanks in advance

    Tony

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    It's obvious you didn't specify the field that contains the operation.

  3. #3
    Join Date
    Jul 2009
    Posts
    17

    Default

    Matt
    Thanks for the tip!

    I saw the Operation section on the Advanced Tab, but I incorrectly assumed that the separate Advanced tab meant that the step worked without using this tab.

    My previous understanding was that this step was able to determine by the lookup and the update fields which fields to insert/update/delete - i.e. kind of like the insert/update step but with the ability to delete rows that were not present in the previous step.

    Now I understand - this step requires the Operation field - which tells the step whether to insert/update/delete. In my circumstance I don't have an Operation field, so I think I will have to create a separate transformation to remove the rows that are no longer needed.

    Thanks again Matt

    PS Very impressed with Kettle.

  4. #4
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    FYI, it's called " Synchronize After Merge" because the "Merge" step generates the operation field for you. It detects new, identical, updated and deleted rows after comparing two sorted data streams.
    As such, with these 2 steps and a little bit of extraction logic you can synchronize 2 database tables.

  5. #5
    Join Date
    Jul 2009
    Posts
    17

    Default

    Matt

    Thanks for the explanation - this step does actually then do inserts/updates/deletes - if preceeded by a "Merge Rows (diff)" step. I now understand how it works and I have used this successfully to synchronize tables with inserts, updates and deletes.

    Synchronize After Merge is indeed a very useful transformation step. I have taken the liberty of documenting this step in the Wiki to assist others.

    http://wiki.pentaho.com/display/EAI/Synchronize+after+merge


    Tony
    Last edited by mfantcook; 08-09-2009 at 07:53 PM.

  6. #6
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    Thank you!!

  7. #7

    Smile Thanks

    Thanks Tony for updating the documentation


    Take care

    Samatar
    Samatar

  8. #8
    Join Date
    Dec 2013
    Posts
    2

    Default

    Hi,
    I got the same problem, solved it by this way : put a switch/case + specify operation values in the Advanced Tab

    Operation fieldname = flagfield
    Insert when value equal = new
    Update when value equal = changed
    Delete when value equal = deleted

    Name:  Capture.jpg
Views: 1318
Size:  12.7 KB

  9. #9
    Join Date
    Dec 2013
    Posts
    2

    Default

    Hi Tony,

    I got the same problem, solved it by this way : put a switch/case + specify operation values in the Advanced Tab

    Operation fieldname = flagfield
    Insert when value equal = new
    Update when value equal = changed
    Delete when value equal = deleted
    Name:  Capture.jpg
Views: 1241
Size:  12.7 KB

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.