Hi to all,
i'm trying to set up some demos with PDI+AEL->Hadoop+Spark

I have done a simple job with one transf that has 1 generate rows, 1 Memory Group By and 1 Write to log.

I start the Job on my client PC and set the Transformation to run on through AEL on a Spark Server

So my set up is
- my pc that run the job
- a server with hadoop+spark+pentaho ael daemon

conf:
Pentaho 8.1 CE
hadoop-3.1.2
spark-2.4.3

As far as i know AEL re-map on spark some steps, generate rows and write to log are not those kind of re-mapped steps, Memory Group By it's one of these.
Everything fine as far as I don't use re-mapped steps, daemon get called and the transf run on a single node of spark and that's ok.

If i use Memory Group By i get an error on my pc that run the job and then the transf fails, the error is copied at the end of the post.
The daemon on remote machine simpy says:
2019-06-14 10:47:54.349 INFO 6864 --- [io-53000-exec-9] o.p.a.d.spark.RequestServerEndpoint : Received Stop Message from driver
2019-06-14 10:47:54.362 INFO 6864 --- [io-53000-exec-2] o.p.a.d.spark.RequestServerEndpoint : Session 12 closed because of CloseReason: code [1000], reason [Transformation Complete Successfully]
2019-06-14 10:47:54.395 INFO 6864 --- [io-53000-exec-1] o.p.a.d.spark.RequestServerEndpoint : Received Driver Session Closed Message: Driver Session not Reused
2019-06-14 10:47:54.429 INFO 6864 --- [io-53000-exec-1] o.p.a.d.spark.RequestServerEndpoint : Session 13 closed because of CloseReason: code [1000], reason [Driver Finalized]



Any hint?
Thank you

2019/06/14 12:40:30 - Memory Group by.0 - ERROR (version 8.1.0.0-365, build 8.1.0.0-365 from 2018-04-30 09.42.24 by buildguy) : None.get
2019/06/14 12:40:30 - Memory Group by.0 - ERROR (version 8.1.0.0-365, build 8.1.0.0-365 from 2018-04-30 09.42.24 by buildguy) : org.pentaho.di.engine.api.remote.ExecutionException: None.get
2019/06/14 12:40:30 - Memory Group by.0 - at scala.None$.get(Option.scala:347)
2019/06/14 12:40:30 - Memory Group by.0 - at scala.None$.get(Option.scala:345)
2019/06/14 12:40:30 - Memory Group by.0 - at org.pentaho.di.engine.spark.impl.ops.groupby.mapper.MemoryGroupByMetaDataMapper.mapPentahoSparkField(MemoryGroupByMetaDataMapper.java:175)
2019/06/14 12:40:30 - Memory Group by.0 - at org.pentaho.di.engine.spark.impl.ops.groupby.mapper.MemoryGroupByMetaDataMapper.lambda$mapAggregateFields$0(MemoryGroupByMetaDataMapper.java:96)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.stream.IntPipeline$4$1.accept(IntPipeline.java:250)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
2019/06/14 12:40:30 - Memory Group by.0 - at org.pentaho.di.engine.spark.impl.ops.groupby.mapper.MemoryGroupByMetaDataMapper.mapAggregateFields(MemoryGroupByMetaDataMapper.java:104)
2019/06/14 12:40:30 - Memory Group by.0 - at org.pentaho.di.engine.spark.impl.ops.groupby.mapper.MemoryGroupByMetaDataMapper.fromPdiToAelGroupByMeta(MemoryGroupByMetaDataMapper.java:70)
2019/06/14 12:40:30 - Memory Group by.0 - at org.pentaho.di.engine.spark.impl.ops.groupby.MemoryGroupBySparkOperation.mapToGroupByAelMeta(MemoryGroupBySparkOperation.java:82)
2019/06/14 12:40:30 - Memory Group by.0 - at org.pentaho.di.engine.spark.impl.ops.groupby.BaseGroupBySparkOperation.apply(BaseGroupBySparkOperation.java:110)
2019/06/14 12:40:30 - Memory Group by.0 - at org.pentaho.di.engine.spark.impl.execution.TaskObservable.applyOperation(TaskObservable.java:264)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.Optional.ifPresent(Optional.java:159)
2019/06/14 12:40:30 - Memory Group by.0 - at org.pentaho.di.engine.spark.impl.execution.TaskObservable$OperationSubscriber.lambda$addOutput$1(TaskObservable.java:355)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140)
2019/06/14 12:40:30 - Memory Group by.0 - at org.pentaho.di.engine.spark.impl.execution.TaskObservable$OperationSubscriber.addOutput(TaskObservable.java:354)
2019/06/14 12:40:30 - Memory Group by.0 - at org.pentaho.di.engine.spark.impl.execution.TaskObservable$OperationSubscriber.lambda$setOutput$3(TaskObservable.java:399)
2019/06/14 12:40:30 - Memory Group by.0 - at java.lang.Iterable.forEach(Iterable.java:75)
2019/06/14 12:40:30 - Memory Group by.0 - at org.pentaho.di.engine.spark.impl.execution.TaskObservable$OperationSubscriber.setOutput(TaskObservable.java:399)
2019/06/14 12:40:30 - Memory Group by.0 - at org.pentaho.di.engine.spark.impl.ops.GenericSparkOperation.apply(GenericSparkOperation.java:114)
2019/06/14 12:40:30 - Memory Group by.0 - at org.pentaho.di.engine.spark.impl.execution.TaskObservable.applyOperation(TaskObservable.java:264)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.Optional.ifPresent(Optional.java:159)
2019/06/14 12:40:30 - Memory Group by.0 - at org.pentaho.di.engine.spark.impl.execution.TaskObservable$OperationSubscriber.lambda$addOutput$1(TaskObservable.java:355)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2019/06/14 12:40:30 - Memory Group by.0 - at org.pentaho.di.engine.spark.impl.execution.JobGroupDriverTask.lambda$new$0(JobGroupDriverTask.java:51)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019/06/14 12:40:30 - Memory Group by.0 - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019/06/14 12:40:30 - Memory Group by.0 - at java.lang.Thread.run(Thread.java:748)