PDA

View Full Version : [Mondrian] Re: Eigenbase perforce change 8645 for review



John V. Sichi
02-05-2007, 12:36 AM
Richard, looks like you accidentally clobbered Zelaine's addition of the
IterationLimit parameter description when you merged?

JVS

Richard Emberson wrote:
> http://p4web.eigenbase.org/@md=d&c=6PU@//8645?ac=10
>
> Change 8645 by emberson (AT) bortei (DOT) head on 2007/02/03 15:59:41
>
> MONDRIAN
> Added ObjectFactory and Java5 memory monitoring support.
> Now when using Java5, if a query is using "too much" memory
> a MemoryLimitExceededException is thrown rather than
> an OutOfMemoryError.
> 281,288d284
> < <td><a href="api/mondrian/olap/MondrianProperties.html#IterationLimit">
> < <code>mondrian.rolap.IterationLimit</code></a></td>
> < <td>int</td>
> < <td>0</td>
> < <td>If > 0, the maximum number of iterations allowed when evaluating an
> < aggregate. If 0, there is no limit.</td>
> < </tr>
> < <tr>
_______________________________________________
Mondrian mailing list
Mondrian (AT) pentaho (DOT) org
http://lists.pentaho.org/mailman/listinfo/mondrian

John V. Sichi
02-05-2007, 06:40 AM
Another problem with this checkin. When I run "ant test" with Derby on
Linux and Sun JDK 1.5 (which defaults to 64M max heap), lots of tests
error out with MemoryLimitExceededException. Are you sure it isn't
somehow checking *before* garbage collection?

When I set mondrian.util.memoryMonitor.enabled=false, all is well again.

JVS

John V. Sichi wrote:
> Richard, looks like you accidentally clobbered Zelaine's addition of the
> IterationLimit parameter description when you merged?
>
> JVS
>
> Richard Emberson wrote:
>> http://p4web.eigenbase.org/@md=d&c=6PU@//8645?ac=10
>>
>> Change 8645 by emberson (AT) bortei (DOT) head on 2007/02/03 15:59:41
>>
>> MONDRIAN
>> Added ObjectFactory and Java5 memory monitoring support.
>> Now when using Java5, if a query is using "too much" memory
>> a MemoryLimitExceededException is thrown rather than
>> an OutOfMemoryError.
>> 281,288d284
>> < <td><a
>> href="api/mondrian/olap/MondrianProperties.html#IterationLimit">
>> < <code>mondrian.rolap.IterationLimit</code></a></td>
>> < <td>int</td>
>> < <td>0</td>
>> < <td>If > 0, the maximum number of iterations allowed when
>> evaluating an
>> < aggregate. If 0, there is no limit.</td>
>> < </tr>
>> < <tr>
>

_______________________________________________
Mondrian mailing list
Mondrian (AT) pentaho (DOT) org
http://lists.pentaho.org/mailman/listinfo/mondrian

Richard Emberson
02-05-2007, 10:50 AM
I suspect that because Derby uses more memory, the queries
actually come near to out-of-memory.
The memory monitor is set to trigger when memory used
is 90% of the maximum memory
(mondrian.util.memoryMonitor.percentage.threshold=90 by
default).
It triggers after garbage collection, from the
java.lang.management.MemoryPoolMXBean documentation for the
setUsageThreshold method:

* Sets the threshold of this memory pool to the given <tt>threshold</tt>
* value if this memory pool supports the usage threshold.
* The usage threshold crossing checking is enabled in this memory pool
* if the threshold is set to a positive value.
* The usage threshold crossing checking is disabled
* if it is set to zero.

Note, I use percentage rather than absolute memory size while the
above Java5 API uses absolute memory size.
The memory "usage" is how much is actually being used.

Setting the property value to 0% is the same as setting the
usage threshold to zero.

I could be that 64M is too small when using Derby if one wishes
to use 90% as the notification limit.

As a general note, whether to set it at 90% or 95%, etc,
depends upon how much memory can possibly be consumed between
calls to Query.checkCancelOrTimeout. Writing code that is
a good Java5-memory-monitoring citizen will take some time
to figure out the do's and don'ts.
For example, both StringBuffer and StringBuilder are NOT
good citizens, they both use the AbstractStringBuilder.expandCapacity
method which takes the current length of the internal
array of chars and doubles it - this is not an incremental
grow strategy, rather its a undisciplined memory
allocation. For large StringBu*ers there is no chance of the
memory monitor to notify the client if they have to grow.
A good citizen String-maker would use an array of arrays of
chars and slowly grow the number of small (say, 50k) arrays
it uses when pushed to hold very large strings.

Richard


John V. Sichi wrote:
> Another problem with this checkin. When I run "ant test" with Derby on
> Linux and Sun JDK 1.5 (which defaults to 64M max heap), lots of tests
> error out with MemoryLimitExceededException. Are you sure it isn't
> somehow checking *before* garbage collection?
>
> When I set mondrian.util.memoryMonitor.enabled=false, all is well again.
>
> JVS
>
> John V. Sichi wrote:
>> Richard, looks like you accidentally clobbered Zelaine's addition of
>> the IterationLimit parameter description when you merged?
>>
>> JVS
>>
>> Richard Emberson wrote:
>>> http://p4web.eigenbase.org/@md=d&c=6PU@//8645?ac=10
>>>
>>> Change 8645 by emberson (AT) bortei (DOT) head on 2007/02/03 15:59:41
>>>
>>> MONDRIAN
>>> Added ObjectFactory and Java5 memory monitoring support.
>>> Now when using Java5, if a query is using "too much" memory
>>> a MemoryLimitExceededException is thrown rather than
>>> an OutOfMemoryError.
>>> 281,288d284
>>> < <td><a
>>> href="api/mondrian/olap/MondrianProperties.html#IterationLimit">
>>> < <code>mondrian.rolap.IterationLimit</code></a></td>
>>> < <td>int</td>
>>> < <td>0</td>
>>> < <td>If > 0, the maximum number of iterations allowed when
>>> evaluating an
>>> < aggregate. If 0, there is no limit.</td>
>>> < </tr>
>>> < <tr>
>>
>
>


--
Quis custodiet ipsos custodes:
This email message is for the sole use of the intended recipient(s) and
may contain confidential information. Any unauthorized review, use,
disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply email and destroy all
copies of the original message.
_______________________________________________
Mondrian mailing list
Mondrian (AT) pentaho (DOT) org
http://lists.pentaho.org/mailman/listinfo/mondrian

John V. Sichi
02-05-2007, 02:21 PM
Eigenchange 8663 adds a bit of framework for raising alerts when
functions cannot be implemented as native, even though doing so would
have been beneficial. (There are cases where we know native evaluation
wouldn't be a good idea; no alert for those.)

For now I've only done NonEmptyCrossJoin, since that's probably the most
important to be done "wicked fast". Documentation is in the usual
places. Default behavior is OFF (no alerting); you can choose WARN (log
a warning using RolapUtil class logger) or ERROR (fail the query).

Using WARN gives you a way to find queries which need optimization by
scanning the log; using ERROR provides a governance mechanism in cases
where you never want non-native evaluation (also handy for testing).

In the ERROR case, it throws a new NativeEvaluationUnsupportedException.
I made this inherit from ResultLimitExceededException (kind of like
QueryCanceledException, where the limit in question is the user's
patience) so that it is guaranteed to propagate through to a UI which
wants to catch it and display a non-internal error.

JVS
_______________________________________________
Mondrian mailing list
Mondrian (AT) pentaho (DOT) org
http://lists.pentaho.org/mailman/listinfo/mondrian