Hitachi Vantara Pentaho Community Forums
Results 1 to 3 of 3

Thread: Aggregation of input rows into multiple chunks

  1. #1
    Join Date
    Oct 2009
    Posts
    8

    Default Aggregation of input rows into multiple chunks

    Hi Folks,

    i'm struggling with a little problem. Consider the following input rows, each containing a single numerical value

    Code:
    0
    1
    ...
    999
    1000
    I perform aggregation overs theses rows in order to query a webservice operation in a bulk fashion:

    Code:
    0,1,...,999,1000
    However, the webservice responses become too big, if i perform a bulk query for all values at once. So what i would like to do is to split the input into rows into multiple chunks, each representing an aggregation over a part of the input rows:

    Code:
    output row 0: 0,1,..., 98, 99
    output row 1: 100,101,..., 198, 199
    ...
    Of course, the chunk size should be configurable.

    I have no idea how to do this in PDI - I appreciate your help

  2. #2
    Join Date
    Nov 1999
    Posts
    9,729

    Default

    So first of all, you need to create a group. The easist would probably to do an integer division of this number.
    So add a constant value groupSize, set it to 50 for example.
    Add a "calculator" step, A/B, A=value, B=groupSize, Integer result
    Then add a "Group by" step, concatenate separated by ,

    Create and use a group.ktr

  3. #3
    Join Date
    Oct 2009
    Posts
    8

    Default

    Ah, that sounds like a good solution. Thanks!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.