PDA

View Full Version : Mongo input help - join mongo collections



tnewman
02-26-2013, 02:16 PM
Hi,

I am pretty new to the mongo input step in PDI. I have the following mongo query :-

db.locations.find().forEach(function(l) {
for (var i=0; i<l.sourceUrls.length; i++) {
if (!l.sourceUrls[i].isCompetitor) {
var c = db.collectors.findOne({url: l.sourceUrls[i].url});
if (c) {
if (c.numFailures > 1) {
var csv = l.code + ',' + l.name (http://l.name/) + ',' + l.sourceUrls[i].url + ',' + c.numFailures;
printjson(csv);
}
}
}
}
});

but I don't see how I can get this to work in the mongodb input step. It have multiple joins to collections.

I have tried various versions of quoting but I find the documentation on what to quote lacking.


The output fields (l.code + ',' + l.name (http://l.name/) + ',' + l.sourceUrls[i].url + ',' + c.numFailures) should go into the 'field expression' - is this correct?

It looks like the current Mongo input component cannot handle arrays - is this correct?

Any guidance would really help me out.

Thanks,

Mark
03-03-2013, 11:06 PM
Hi,

The MongoDB input step does not handle joins between collections. Furthermore, it looks like the Java driver for MongoDB does not support the forEach() cursor method:

http://api.mongodb.org/java/2.10.1/com/mongodb/DBCursor.html

Nor is forEach possible via an operator to find().

The version of MongoDB input that will be available in PDI 5.0 has support for extracting from arrays and unwinding arrays. It allows you to specify field paths in much the same way as you can in the JSON step. It also has support for executing aggregation pipeline queries. Neither of these are going to help you with joins though. However, the new array unwinding plus PDI's join steps might accomplish what you want.

Cheers,
Mark.