PDA

View Full Version : Metadata



tfindlay
07-14-2005, 02:12 AM
Hi,

I've had a skim over the roadmap and everything, and am currently using some of the tools that are targeted for this project.

My only concern is, I havent seen any common metadata layer ?

It's pertenant in my mind, as I'm currently maintaning a SAS Metadata server, Webfocus metadata and Cognos metadata and it made me realise that there arent any good open source metadata servers out there.

Certainly nothing that covers business and technical metadata, or anything that has the scope for an whole-of-enterprise setup ... well, I should say, not that I know of.

With all the tools we're looking at in Pentaho, a metadata layer would be essential in tying it all together.

Food for thought.

Tim.

mnutty
07-14-2005, 10:39 AM
This is interesting and I have been thinking of it too. Maybe in a slightly different way.

I think the reason that there is no OSS Metadata layer is because there was no need. There was nothing like Pentaho. Parts and pieces but no combined tool.

In the reporting world (BIRT, JasperReports and such) mapping could be done via a domain layer (JDO, Hibernate, EJBs, etc). This is also done with applications (ie OLTP) It would be nice if this could be reused in the other areas such as ETL and Analysis. I had suggested this in the Mondrian forums but only got negative responses.

I've seen a little of what COGNOS has but I think it is very data centric and probably isn't for everything in the enterprise (I could be wrong - these are just my thoughts).

I would like to see this worked out so there is little duplication of effort but works well for the different "stacks" (ie ETL, Reporting, OLTP, ...).

I definitely think this is something that would be useful in Pentaho. I would love to hear others' thoughts .

ngoodman
08-02-2005, 03:01 PM
There is already a good "base" for this in the Common Warehouse Metamodel. I believe Mondrian uses parts of this as well, however I've never looked that closely.

I think that CWM is "broad" enough that there is much ambiguity in using it between products. The OSS community could really bring the standard into usefulness by using it as the basis for it's BI/DW/ETL metadata repository...

Anyhow... something to mull... I believe the OMG also has Java classes that implement the model, but I'm not sure about that either...

Nick

mnutty
08-04-2005, 04:34 AM
Sounds like a good idea.

I had not seen that in Mondrian. I'll have to dig for it.

Did a quick google and it looks like JOLAP "uses the CWM OLAP metamodel". On the down side, JOLAP (sadly) seems to be going no where. I've not heard anything new about it lately. Anything I find is old news. Mondrian does support JOLAP, somewhat, and that might be the connection to CWM.

swood
08-04-2005, 04:49 AM
Mondrian does use CWM as part of its JOLAP implementation - I think that is part of the JOLAP spec. Given that JOLAP does appear to be going nowhere, the JOLAP API in Mondrian has not been used by the Mondrian community.


Sherman

mnutty
08-04-2005, 08:16 AM
Maybe this project will give more exposure to Mondrian/JPivot and thus get the big vendors to start moving on JOLAP again.

That or look for IBM or Oracle to buy the main company behind Mondrian and/or JPivot. :(

tfindlay
09-05-2005, 08:24 PM
Keep in mind, that metadata should go ALOT further than OLAP, Id really love to see Octopus extended to support a metadata layer in ETL, and Jasper utilizing this also.

Maybe I'm a dreamer, but when I think of a integrated platform, I really think of common threads between naming conventions, or business terms for things. In our business we have different business units that would give you a different definition of a "Standard Claim", this is where the power of a COMMON metadata layer is.

Even if JOLAP were ramped up, I dont think this will support a common metadata layer/repository for all the reporting components in the pentaho platform.

(I should note, I have not used JOLAP, so am speculating about its use).

ngoodman
09-05-2005, 08:40 PM
tfindlay wrote:

Keep in mind, that metadata should go ALOT further than OLAP, Id really love to see Octopus extended to support a metadata layer in ETL, and Jasper utilizing this also.


CWM has provisions for ETL mappings (flows), transformations, etc etc. However, as previously mentioned it's not gained much traction. There's a book on it (not very useful, just a guided view of the standard) as well which has been collecting dust on my shelf... :)

Nick

jdixon
09-06-2005, 07:37 AM
We agree that metadata should go much further than OLAP. Our intention is to integrate metadata into


Report queries
Report drilling
Dashboard queries
Dashboard drilling
Pivot table queries
Pivot table drilling
ETL


It is true that meta-data initiatives have not gained much traction in products designed for SMEs (small to medium sized enterprises) and Open Source projects. I think largely because its hard.

There are a collection of standards that are all ultimately based on OMG's Meta-Object Facility (MOF).

Common Warehouse Metamodel (CWM) is based on MOF.
Sun's Java Metadata Interface (JMI - JSR40) is based on MOF and can map CWM to Java interfaces.
Java OLAP Interface (JOLAP - JSR69) uses the CWM OLAP metamodel and leverages JMI
Java Data Mining API (JDMAPI - JSR73) is based on the CWM Data Mining metamodel


Between them these standards are backed by HP, IBM, Oracle, Unisys, Hyperion, SAS, Sun, SPSS, SAP. Getting these groups to cooperate and agree on anything is a major achievement.

It is certainly not trying re-create or compete with these standards.
Our initial research will be to attempt a integration of reporting, dashboards, pivot views and ETL based on these standards.

tfindlay
09-06-2005, 08:18 AM
Sounds great!

I think you've hit the nail on the head (so to speak). The biggest problem is understanding the standards that are out there. I'll be the first to admit I dont fully understand them. I think with better knowledge about how they work and how to use them, we'll see bigger compliance. I think the corporates that work in this space will be far more likely to comply with the standards if Open Source led the way.

IMO, as long as noone complies with the standard, big companies will happily detour from it as a form of pseudo lock-in, it's only when someone follows the standard that the corporate players are held to it. Not to mention, it makes it easier for them to implement it when someone shows them how. :)

I guess that leads to the obvious question - How can Pentaho encourage these standards in projects like Octopus and some of the others that are lagging in this area ?

... I guess the obvious answer is, how prepared are developers to get on board and make it happen! As per your point however, I'd get on board myself and start implementing some support for CWM or OMG's MOF, if I understood it properly!

I wonder if I can convince my boss to send on a fancy training course ? haha! (not likely).

Prehaps if OMG & CWM were to write up some simpler, clearer docs about their standards and how to implement them, with better education it'd be easier to promote them in Open Source projects, and get people interested. I know I would be!

MattCasters
11-21-2006, 04:33 PM
Well Tim, you have been heard.
Check out "Pentaho metadata" & if you have questions join us on the forum with the same name.

Matt