Hitachi Vantara Pentaho Community Forums
Results 1 to 7 of 7

Thread: Running sum/count without aggregation

  1. #1
    Join Date
    Apr 2007
    Posts
    2,010

    Default Running sum/count without aggregation

    Hi,

    Came across something today which sounds simple, but turns out I can't see a native way to do it!

    Essentially i want a running total/count, possibly I want it conditional in the future.

    So group rows step cannot be used because I don't want to do any grouping. At most all i want to do is increment a counter when a field is a certain value.

    It's easily done in the UDJC step, or even in javascript if you can take the performance hit but this just feels wrong. Any other ways?

    Dan

  2. #2
    Join Date
    Apr 2008
    Posts
    1,771

    Default

    Maybe you can try "Output steps metrics" step.
    -- Mick --

  3. #3
    Join Date
    Apr 2007
    Posts
    2,010

    Default

    Ooh; an interesting one. A quick glance at it however and it only passes rows once the "step" you're interested in has finished. I need this to stream through. i.e. i just want a count of 2 on the 2nd row, i dont want to wait and see what may come next.

    Can see that being pretty useful though, especially in process reporting etc

  4. #4
    Join Date
    Apr 2008
    Posts
    1,771

    Default

    In most of the Input steps, there's an option to add the ROWCOUNT as a field.
    -- Mick --

  5. #5
    Join Date
    Jun 2012
    Posts
    5,534

    Default

    Quote Originally Posted by codek View Post
    Essentially i want a running total/count, possibly I want it conditional in the future.
    I once described a suitable concept here, I think.
    So long, and thanks for all the fish.

  6. #6
    Join Date
    Apr 2008
    Posts
    4,696

    Default

    You could filter rows, and then use a Group By step with the Cumulative Sum option (along with return all rows), and merge it back together again....
    **THIS IS A SIGNATURE - IT GETS POSTED ON (ALMOST) EVERY POST**
    I'm no expert.
    Take my comments at your own risk.

    PDI user since PDI 3.1
    PDI on Windows 7 & Linux

    Please keep in mind (and this may not apply to this thread):
    No forum member is going to do your work for you. We will help you sort out how to do a specific part of the work, as best we can, in the timelines that our work will allow us.
    Signature Updated: 2014-06-30

  7. #7
    Join Date
    Apr 2007
    Posts
    2,010

    Default

    thanks, the group rows step will do it for me with the all rows option after all - i had somehow misunderstood that step.
    I wanted to avoid splitting and merging because of the risk of a hang in that scenario.
    I did notice the group rows step can be quite slow, but that largely seems to be due to memory usage. If you reduce the rowset size it can improve the performance quite a bit.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
Privacy Policy | Legal Notices | Safe Harbor Privacy Policy

Copyright © 2005 - 2019 Hitachi Vantara Corporation. All Rights Reserved.