PDA

View Full Version : RE: [Mondrian]Re:VirtualCubeTest.testCalculatedMemberAcrossCubesfailingonSMP



Pappyn Bart
01-24-2007, 04:22 AM
Hierarchy cache is already controllable with the data source listener plug-in.

Indeed, there are a lot of entry points, especially when it comes down to non mdx query executing
related stuff. xmla is the easiest one, because behavior is controlled by mondrian itself, but jpivot
native is calling things directly.

RolapConnection is already storing a jdbc connection, so it is better to use this one instead of inventing a new
principle. And since a transaction should live as long as a RolapConnection, that should be the right path
to take. For now, agg loaders are not able to use this one, but patching is easy. The query object that
contains the connection is already available in FastBatchingCellReader.loadAggregates(), so it is only a
matter of taking it further down.

For the moment RolapUtil.executeQuery only allows one query at a time to be executed. At the moment this
is a bottleneck in multi-user applications. But it was required since all threads where sharing the same jdbc connections.

For non-mdx query related functions, regarding xmla discover, mondrian native calls to hierarchy cache, there is a bigger
problem, since there is no single entry point and I cannot see everything clear for the moment. It is not clear how long
transactions should live and how to make sure hierarchy cache is following the data integrity of the mdx query and is in sync.

There is also a problem about aggregate statistics about size and volume. After my upcoming changes, mondrian will be
able to flush the cache, but the statistics will not be loaded again, so mondrian might take the wrong decisions.

Another thought : For the moment, all threads executing the queries are filling the cache. Query logic and data loading
is now very much woven in each other and is tightly coupled. This makes things difficult and results in hierarchy cache and aggregate cache
live their own lives and not being in sync. I think in the future it might be better to have threads executing queries and asking a central cache manager
for required data. The query threads would just wait until the cache comes available, the central cache manager is in control of loading
all the data (both aggregates as hierarchy data) and checking the data source for modifications. The central cache manager could have
multiple worker threads to load aggregates and hierarchy data, to be able to scale better on SMP hardware and multi-user applications.
This way, most problems would be vanished and there would be a much better control of data integrity. It would be much easier to control
database transactions and jdbc connections. Central cache management would also open doors for cold start support and so on...

For Julian : Could this become a roadmap topic ?

Bart


________________________________

From: mondrian-bounces (AT) pentaho (DOT) org [mailto:mondrian-bounces (AT) pentaho (DOT) org] On Behalf Of michael bienstein
Sent: dinsdag 23 januari 2007 17:16
To: Mondrian developer mailing list
Subject: Re : Re : Re :[Mondrian]Re:VirtualCubeTest.testCalculatedMemberAcrossCubesfailingon SMP


My comments in your text.


----- Message d'origine ----
De : Pappyn Bart <Bart.Pappyn (AT) vandewiele (DOT) com>