Naive Bayes training set optimization - machine-learning

Question

I am working on a naive bayes classifier that takes a bunch of user profile data such as:
Name
City
State
School
Email Address
URLS { ... }
The last bit is a bunch of urls that are search results for the user gathered by a google search for the user by name. The objective is to decide if the search result is accurate(ie. it is about the person) or inaccurate. In order to do this, each piece of the profile data is searched within each link in the url array and a binary value is assigned per attribute if that profile data (ex. City) is matched on a page. The results are then represented as a vector of binaries (ie. 1 0 0 0 1 means Name and Email address was matched on the url).
My questions revolves around creating the optimal training set. If a person's profile has incomplete information (such as missing email adddress), should that be a good profile to use in my training set? Should I be only training on profiles with full training information? Would it make sense to make different training sets (one for each combination of complete profile attributes) and then when i am given a user's url to test with, i determine which training set to use based on how much user profile is on record for the test person? How can i go about this?

Answer 1

In general, there is no "should". Whichever way you create a model, the only thing which matters is its performance, no matter how you created it.
However, it is highly unlikely you'd be able to create a proper model with hand-picked training set. The simple idea is that you should train your model on data which looks exactly like live data. Will live data have missing values, incomplete profiles etc? So, you need your model to know what to do in such situations and, therefore, such profiles should be in the training set.
Yes, certainly, you can make a model composed of several sub-models, however you might run into problems with having not enough training data and overfitting. You'll have to create multiple good models for it to work, which is harder. I suppose it would be better to leave this reasoning to the model itself rather than trying to hand-hold it into the right direction, this is what machine learning is for - save you the trouble... But there is really no way to say before trying it on your data set. Again, whatever works in your particular case is right.

Answer 2

Because you're using Naive Bayes as your model (and only because of that) you can benefit from the independence assumption to use every piece of data you have available and only consider those present in the new sample.
You have features f1...fn, some of which may or may not be present in any given entry. The posterior probability p( relevant | f_1 ... f_n ) decomposes as:
p( relevant | f_1 ... f_n ) \propto p( relevant ) * p( f_1 | relevant ) * p( f_2 | relevant ) ... p(f_n | relevant )
p( irrelevant | f_1...f_n ) is similar. If some particular f_i isn't present, just drop the terms from the two posteriors---given that they're defined over the same feature space probabilities are comparable, and can be normalised in the standard way. All you then need is to estimate the terms p( f_i | relevant ): this is simply the fraction of the relevant links where the i_th feature is 1 (possibly smoothed). To estimate this parameter simply use the set of relevant links where the i-th feature is defined.
This is only going to work if you implement yourself, as I don't think you can do this with a standard package, but given how easy it is to implement I wouldn't be concerned.
Edit: an example
Imagine you have the following features and data (they're binary, since you say that's what you have, but the extension to categorical or continuous is not difficult, I hope):
D = [ {email: 1, city: 1, name: 1, RELEVANT: 1},
{city: 1, name: 1, RELEVANT: 0},
{city: 0, email: 0, RELEVANT: 0}
{name: 1, city: 0, email: 1, RELEVANT: 1} ]
where each element of the list is an instance, and the target variable for classification is the special RELEVANT field (note that some of these instances have some variables missing).
You then want to classify the following instance, missing the RELEVANT field since that's what you're hoping to predict:
t = {email: 0, name: 1}
The posterior probability
p(RELEVANT=1 | t) = [p(RELEVANT=1) * p(email=0|RELEVANT=1) * p(name=1|RELEVANT)] / evidence(t)
while
p(RELEVANT=0 | t) = [p(RELEVANT=0) * p(email=0|RELEVANT=0) * p(name=1|RELEVANT=0)] / evidence(t)
where evidence(t) is just a normaliser obtained by summing the two numerators above.
To get each of the parameters of the form p(email=0|RELEVANT=1), look at the fraction of training instances where RELEVANT=1 which have email=0:
p(email=0|RELEVANT=1) = count(email=0,RELEVANT=1) / [count(email=0,RELEVANT=1) + count(email=1,RELEVANT=1)].
Notice that this term simply ignores instances for which email is not defined.
In this instance, the posterior probability of relevance goes to zero because the count(email=0,RELEVANT=1) is zero. So I would suggest using a smoothed estimator where you add one to every count, so that:
p(email=0|RELEVANT=1) = [count(email=0,RELEVANT=1)+1] / [count(email=0,RELEVANT=1) + count(email=1,RELEVANT=1) + 2].