newbie Weka question on attribute selection

[tomas1453]

Hello everybody

I would like to solve a machine learning problem and I am looking for some advice on what method to use

I have the following problem, I have to arrays :
1) X(i,j,k) with values between 0 and 1, including fractional values
2) Y(i,k,k) with discreet values 0 and 1

i ~= 1..100,j ~= 1..30, k ~= 1..20
(these arays represent the linear relaxation (X array) and near optimal binary solution (Y) of a difficult real world problem)

I am convinced that there is some structure in this problem that could be extracted as rules such as :
IF X(i-1,j,k)==1 AND X(i+1,j,k)==1 AND X(i,j,k-1)==1
THEN
Y(i,j,k)==1

It seems to me that these kind of rules could be extracted through building a decision tree (I have done a quick test with Weka and it seems to work). However what bothers me is that I need to select attributes before running the decision tree learning algorithm. I would like to explore a wide range of possible rules and I don't really know how to select attributes. My ideal ML algorithm would first select a suitable set of attributes and then give me the rule that I want (or do it at the same time).

What should I look at in Weka to be able to solve such a problem? Is there anything I should look at outside of Weka?

Thx a lot

[Mark]

Hi,

When you say you need to select attributes I'm guessing that you really mean that you need to construct a feature space suitable for propositional learning algorithms to operate on. Once you have a space of features, then methods like decision trees automatically select the most predictive attributes from this space while constructing the tree. Constructing the representation is a manual task for the domain expert. There is a subfield of machine learning, called constructive induction, that attempts to construct new features for use by learning algorithms. They still need a base set of features to operate on however.

Cheers,
Mark.

[tomas1453]

Hello Mark

Thanks a lot for your answer, it was very helpful.

Having done some research on the internet on constructive induction, it seems to me that I could use the Multifactor dimensionality reduction algorithm that is part of the Weka GC package. The set of possible features that I could define is fairly large but it seems like is worth a try. (If I understood how the algorithm works there might be one additional benefit - it seems like the MDR algorithm might be able to detect extraneous variables in the mathematical model my solutions came from...)

Do you have any experience with this? Do you think it would work?

(To give a little background, the two arrays are solutions -linear relaxation, discrete - of a complicated real world model that takes quite some time to solve using exact mathematical optimization methods. On the other hand a human expert is capable of finding a solution through his understanding of the underlying relations of the real world elements that are not necessarily well described in the model. Unfortunately I don't really have sufficient access to pick the brain of the expert so I would like to discover some rules that could be implemented in a heuristic solution algorithm that works at least part of the time... I realize that it is a long shot, but I am getting a little desperate with this problem so I am willing to test ideas that seem only half intelligent...)

Thanks a LOT!

Tomas

[Mark]

Hi Tomas,

I'm afraid that I've never tried MDR (or the Weka GC package for that matter). Depending on whether human-interpretable rules are an important outcome of learning for you, you could also consider applying support vector machines to your prediction problem. SVMs have the ability to deal with very large feature spaces and are quite robust with respect to the curse of dimensionality. Furthermore, kernel functions can be used to construct new feature spaces efficiently in order to handle non-linear problems.

Cheers,
Mark.