PDA

View Full Version : Re : [Mondrian] Multithreading, Parallel Batch / Query Execution



Tushar Apshankar
04-27-2007, 10:30 AM
Hi,
To achieve Parallel Batch / Query Execution, we need to give execution
tasks to separate threads.
We could think of 2 options to achieve it. Both of them have their pros
and cons in terms of performance and ThreadLocal variable management.

1) New Thread Creation:
Pros: We change the thread local variable to inheritable thread
local and the child thread automatically get the thread local values of
the parent thread.
Cons: Creating new threads is not a good idea. Too many threads
running in a system should not be a problem though, as we can limit the
number of Threads created using java.util.concurrent.Semaphore/BigInteger
etc.
2) Thread Pool:
Pros: It would be faster (It looks like a valid assumption.)
Cons: We will have to do some manipulation to pass the thread
local variables to worker threads of Thread Pool. All ThreadLocal
variables of a Thread are stored in a ThreadLocalMap. Thread class does
not provide getter/setter for ThreadLocalMap. We could still get/set the
ThreadLocalMap using reflection. This solution is working fine.

Is there any other option? If not, which one of the above looks better
approach to you?

Please let us know your thoughts.
Thanks and regards,
Tushar






Tushar Apshankar <tapshank (AT) thoughtworks (DOT) com>
Sent by: mondrian-bounces (AT) pentaho (DOT) org
04/25/2007 06:27 PM
Please respond to
Mondrian developer mailing list <mondrian (AT) pentaho (DOT) org>


To
Mondrian developer mailing list <mondrian (AT) pentaho (DOT) org>
cc

Subject
Re : [Mondrian] Multithreading, Parallel Batch / Query Execution







Hi,

To improve performance, we started looking at ReentrantReadWriteLock and
ConcurrentHashMap (from java.util.concurrent) package.
We are also looking at ThreadPoolExecutor and related classes to implment
parallel query / batch execution (most of these classes are also available
in java.util.concurrent as well as backport jar).
Since compatibility with Java 1.4 is essential, we started looking at the
build script retroweave target that uses Retroweaver 1.2.4.

Retroweaver without our changes works fine. We ran into problems while
using the current retroweaver version and build target after our changes.

We have a simple class LockTest to reproduce the problem:
=============================================================================================

package mondrian;

import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.locks.ReentrantReadWriteLock;
import java.util.concurrent.locks.Lock;

public class LockTest {
ConcurrentHashMap<String, String> cache = new
ConcurrentHashMap<String, String>();
ReentrantReadWriteLock lock;
Lock r ;
Lock w ;
public LockTest()
{
cache = new ConcurrentHashMap();
lock = new ReentrantReadWriteLock();
System.out.println("Before lock = " + lock);
w = lock.writeLock();
r = lock.readLock();
System.out.println("After lock = " + lock);
}

public static void main(String[] args) {
LockTest lt = new LockTest();
System.out.println("i am inside main method");
}
}
=============================================================================================

When we retroweave it with retroweaver 1.2.4 (using ANT as well as
retroweaver UI) and run the weaved class using Java 1.4.2, we get
java.lang.NoSuchMethodError. Stack is:

java.lang.NoSuchMethodError:
edu.emory.mathcs.backport.java.util.concurrent.locks.ReentrantReadWriteLock.writeLock()

Ledu/emory/mathcs/backport/java/util/concurrent/locks/ReentrantReadWriteLock$WriteLock;

[Loaded java.lang.StackTraceElement from C:\j2sdk1.4.2_14\jre\lib\rt.jar]
at mondrian.LockTest.<init>(LockTest.java:26)
at mondrian.LockTest.main(LockTest.java:34):

We have backport jar in our class path (please note that we are NOT
getting ClassNotFoundExcecption).
Did someone face this issue earlier? Please let us know if we are missing
something here.

We also tried out retroweaver 2.0 without success.

As an alternative, we tried Retrotranslator(
http://retrotranslator.sourceforge.net) and everything worked in first
attempt. We propose to create a new target for Retrotranslator and add the
related libraries. Please let us know your thoughts on this

Thanks and regards,
Tushar, Ajit
_______________________________________________
Mondrian mailing list
Mondrian (AT) pentaho (DOT) org
http://lists.pentaho.org/mailman/listinfo/mondrian


_______________________________________________
Mondrian mailing list
Mondrian (AT) pentaho (DOT) org
http://lists.pentaho.org/mailman/listinfo/mondrian