This is based on my earlier question. Can I use machine learning algorithms to help me with understanding sentences?
(I will use a closely related example as I used in my previous question). For example, I want my algorithm/code to start a program based on what the user says. For instance, if he says "turn on the program," then the algorithm should do that. However, if the user says "turn on the car," the computer shouldn't turn on the program, obviously. (BUT how would a computer know?) I am sure there are hundreds of different ways to say "start" or "turn on the program." What I am asking is how can a computer differentiate between "program" and "car"? How can the algorithm know that in the first sentence, it has to start the program, but not in the second one? Is there a way for the algorithm to know what the sentence is talking about?
Could I use an unsupervised learning algorithm for this, that is, one that can learn what the sentence is about?
Thanks
Natural language understanding is a very hard problem and many researchers are working on it. To begin with, basic Natural language understanding systems start off as rule based. You manually write down rules that will be matched against an input, and if a match is found, you fire a corresponding action. So, you restrict the format of your input and come up with rules while keeping them as general as possible. For example, instead of matching the exact statement "turn on the program", you can have a rule such as: unless the word "program" occurs in the command, don't start the program, OR ignore every sentence unless it contains "program". And then, combine your rules to develop more complex "understanding". How to write/represent rules is another tough problem. You can start off with Regular Expressions.
Regarding various ways of expressing the action of "Start"ing something, you are going to look at Synonyms for "start", e.g. "begin". This can be obtained from a thesaurus and a commonly used resource for such tasks is WordNet
You need to figure out what information do you exactly want to extract from the sentence. Most natural language techniques are task specific, there isn't be a general one-size-fits-all natural language understanding tool.
no machine learning algorithms can learn without enough information input. If there are enough information about a car versus a program, then the learning algorithms may differentiate them. Machine learning group things that have similar properties and separate them into different group if thing has different properties.
Related
I'm curious about applying NLP to predict/evaluate someone's level of education (or adherence to correct grammar, spelling, etc.) by analyzing text written by them.
It would be something like: f(t) = s where t is a text and s is some score which rates the grammatical correctness of that text.
Does that exist? I don't know how to search for it. If it does, I'd like some references to relevant papers or algorithms.
It does not exist. "Grammatical correctness" is a vague concept anyway, as there is no complete grammatical description of any given language. Also, we all speak and write different variations of our language, which cannot be captured by a single grammar. A language is basically the union of all the individual variants that its speakers produce.
Leaving aside these linguistic philosophy issues, there is also no formal grammar of even a single variant of a language that you could use as a benchmark. I guess the nearest thing you could do is come up with a couple of heuristics and simple rules (which I assume commercial grammar checkers use), checking for example that reads always occurs after a third person singular noun. If you have a sufficient number of such heuristics, you can get an idea if a given text is grammatical according to the definition that grammaticality is equivalent to not breaking the rules you encoded.
However, language is very flexible, and hard to capture in rules. Sometimes a sentence might sound like an error, but then in a given context it is fine. If it was easy, someone would already have done it, and primary school teachers could focus their efforts on tasks other than teaching basic grammar...
You can probably capture a number of 'mistakes' easily, but I wouldn't want to guess what coverage you would get; there will be a lot of issues you cannot capture easily.
[I'm approaching this as an outsider to machine learning. It just seems like a classification problem which I should be able to solve with fairly good accuracy with Machine Larning.]
Training Dataset:
I have millions of URLs, each tagged with a particular category. There are limited number of categories (50-100).
Now given a fresh URL, I want to categorize it into one of those categories. The category can be determined from the URL using conventional methods, but would require a huge unmanageable mess of pattern matching.
So I want to build a box where INPUT is URL, OUTPUT is Category. How do I build this box driven by ML?
As much as I would love to understand the basic fundamentals of how this would work out mathematically, right now much much more focussed on getting it done, so a conceptual understanding of the systems and processes involved is what I'm looking to get. I suppose machine learning is at a point where you can approach reasonably straight forward problems in that manner.
If you feel I'm wrong and I need to understand the foundations deeply in order to get value out of ML, do let me know.
I'm building this inside an AWS ecosystem so I'm open to using Amazon ML if it makes things quicker and simpler.
I suppose machine learning is at a point where you can approach reasonably straight forward problems in that manner.
It is not. Building an effective ML solution requires both an understanding of problem scope/constraints (in your case, new categories over time? Runtime requirements? Execution frequency? Latency requirements? Cost of errors? and more!). These constraints will then impact what types of feature engineering / processing you may look at, and what types of models you will look at. Your particular problem may also have issues with non I.I.D. data, which is an assumption of most ML methods. This would impact how you evaluate the accuracy of your model.
If you want to learn enough ML to do this problem, you might want to start looking at work done in Malicious URL classification. An example of which can be found here. While you could "hack" your way to something without learning more about ML, I would not personally trust any solution built in that manner.
If you feel I'm wrong and I need to understand the foundations deeply in order to get value out of ML, do let me know.
Okay, I'll bite.
There are really two schools of thought currently related to prediction: "machine learners" versus statisticians. The former group focuses almost entirely on practical and applied prediction, using techniques like k-fold cross-validation, bagging, etc., while the latter group is focused more on statistical theory and research methods. You seem to fall into the machine-learning camp, which is fine, but then you say this:
As much as I would love to understand the basic fundamentals of how this would work out mathematically, right now much much more focussed on getting it done, so a conceptual understanding of the systems and processes involved is what I'm looking to get.
While a "conceptual understanding of the systems and processes involved" is a prerequisite for doing advanced analytics, it isn't sufficient if you're the one conducting the analysis (it would be sufficient for a manager, who's not as close to the modeling).
With just a general idea of what's going on, say, in a logistic regression model, you would likely throw all statistical assumptions (which are important) to the wind. Do you know whether certain features or groups shouldn't be included because there aren't enough observations in that group for the test statistic to be valid? What can happen to your predictions and hypotheses when you have high variance-inflation factors?
These are important considerations when doing statistics, and oftentimes people see how easy it is to do from sklearn.svm import SVC or somthing like that and run wild. That's how you get caught with your pants around your ankles.
How do I build this box driven by ML?
You don't seem to have even a rudimentary understanding of how to approach machine/statistical learning problems. I would highly recommend that you take an "Introduction to Statistical Learning"- or "Intro to Regression Modeling"-type course in order to think about how you translate the URLs you have into meaningful features that have significant power predicting URL class. Think about how you can decompose a URL into individual pieces that might give some information as to which class a certain URL pertains. If you're classifying espn.com domains by sport, it'd be pretty important to parse nba out of http://www.espn.com/nba/team/roster/_/name/cle, don't you think?
Good luck with your project.
Edit:
To nudge you along, though: every ML problem boils down to some function mapping input to output. Your outputs are URL classes. Your inputs are URLs. However, machines only understand numbers, right? URLs aren't numbers (AFAIK). So you'll need to find a way to translate information contained in the URLs to what we call "features" or "variables." One place to start, there, would be one-hot encoding different parts of each URL. Think of why I mentioned the ESPN example above, and why I extracted info like nba from the URL. I did that because, if I'm trying to predict to which sport a given URL pertains, nba is a dead giveaway (i.e. it would very likely be highly predictive of sport).
My question is related to the project I've just started working on, and it's a ChatBot.
The bot I want to build has a pretty simple task. It has to automatize the process of purchasing movie tickets. This is pretty close domain and the bot has all the required access to the cinema database. Of course it is okay for the bot to answer like “I don’t know” if user message is not related to the process of ordering movie tickets.
I already created a simple demo just to show it to a few people and see if they are interested in such a product. The demo uses simple DFA approach and some easy text matching with stemming. I hacked it in a day and it turned out that users were impressed that they are able to successfully order tickets they want. (The demo uses a connection to the cinema database to provide users all needed information to order tickets they desire).
My current goal is to create the next version, a more advanced one, especially in terms of Natural Language Understanding. For example, the demo version asks users to provide only one information in a single message, and doesn’t recognize if they provided more relevant information (movie title and time for example). I read that an useful technique here is called "Frame and slot semantics", and it seems to be promising, but I haven’t found any details about how to use this approach.
Moreover, I don’t know which approach is the best for improving Natural Language Understanding. For the most part, I consider:
Using “standard” NLP techniques in order to understand user messages better. For example, synonym databases, spelling correction, part of speech tags, train some statistical based classifiers to capture similarities and other relations between words (or between the whole sentences if it’s possible?) etc.
Use AIML to model the conversation flow. I’m not sure if it’s a good idea to use AIML in such a closed domain. I’ve never used it, so that’s the reason I’m asking.
Use a more “modern” approach and use neural networks to train a classifier for user messages classification. It might, however, require a lot of labeled data
Any other method I don’t know about?
Which approach is the most suitable for my goal?
Do you know where I can find more resources about how does “Frame and slot semantics” work in details? I'm referring to this PDF from Stanford when talking about frame and slot approach.
The question is pretty broad, but here are some thoughts and practical advice, based on experience with NLP and text-based machine learning in similar problem domains.
I'm assuming that although this is a "more advanced" version of your chatbot, the scope of work which can feasibly go into it is quite limited. In my opinion this is a very important factor as different methods widely differ in the amount and type of manual effort needed to make them work, and state-of-the-art techniques might be largely out of reach here.
Generally the two main approaches to consider would be rule-based and statistical. The first is traditionally more focused around pattern matching, and in the setting you describe (limited effort can be invested), would involve manually dealing with rules and/or patterns. An example for this approach would be using a closed- (but large) set of templates to match against user input (e.g. using regular expressions). This approach often has a "glass ceiling" in terms of performance, but can lead to pretty good results relatively quickly.
The statistical approach is more about giving some ML algorithm a bunch of data and letting it extract regularities from it, focusing the manual effort in collecting and labeling a good training set. In my opinion, in order to get "good enough" results the amount of data you'll need might be prohibitively large, unless you can come up with a way to easily collect large amounts of at least partially labeled data.
Practically I would suggest considering a hybrid approach here. Use some ML-based statistical general tools to extract information from user input, then apply manually built rules/ templates. For instance, you could use Google's Parsey McParseface to do syntactic parsing, then apply some rule engine on the results, e.g. match the verb against a list of possible actions like "buy", use the extracted grammatical relationships to find candidates for movie names, etc. This should get you to pretty good results quickly, as the strength of the syntactic parser would allow "understanding" even elaborate and potentially confusing sentences.
I would also suggest postponing some of the elements you think about doing, like spell-correction, and even stemming and synonyms DB - since the problem is relatively closed, you'll probably have better ROI from investing in a rule/template-framework and manual rule creation. This advice also applies to explicit modeling of conversation flow.
Can Machine Learning be used to validate statements or catch errors in text documents?
For example, if you teach a classifier that "You should eat apples twice per day", but in a document that you're testing on, the statement is "You should eat apples three times per day", can the statement be flagged?
Obviously you can build some rules-based software that catches these, but my question centers around training an ML model to catch these, as rules change.
I have looked at word2vec and NLTK and performed some tests with them, but can't connect the dots for teaching the classifier.
If it's possible, how would one go about it or provide some direction?
Thanks, Doug
(Got too long for a comment. )
Yes it can. However, it is freakingly complicated. This kind of reasoning and analysis is done by Watson for example. IBM is calling these cognitive computing. As you wrote rule based (or logical reasoning) systems can solve such tasks. So the question you should ask yourself is how you can extract the required facts from text. => NLP , Part Of Speech, Named Entity,... However the task is extremely hard because " not more then 100times" a day is not contradicting the sentence. So reasoning would require rich background knowledge.
As said it is an extremely broad topic. You would have to sketch the solution and then pick a tiny piece, which would be called a PhD thesis ;).
Which is illustrated in this nice image http://matt.might.net/articles/phd-school-in-pictures/
So looking with the right keywords for PhD thesis's turned up http://nakashole.com/papers/2012-phd-thesis.pdf . This one might provide you a few nights of reading.
If you want to try something hands on with NLTK I would generate parse trees for the sentences you want to analyse. Afterwards you could try to align these and check for overlaps and deviations. However I'm not sure how to draw conclusions. A slightly simpler version would be to match word by word. Something along Levenstein Distance calculations.
Let's say I want to build a search engine that goes through a text and finds sentences or paragraphs that could be turned into an image, video or 3d-animation. So sentences that contain information that could be expressed visually.
Ideally, this search engine would get better over time.
Is there already search engine that could to that?
If not, which type of things would I need to look at/consider? My point here being that I don't really know much about machine learning and search engines. I am trying to get a feeling of which areas of machine learning, information retrieval and so forth I would need to look at.
I don't expect long answers here, just things like "well, take a look at this type of machine learning" or "this part of information retrieval theory may be relevant".
Just to get a broad overview of what I would need to look at.
Natural Language Understanding
I don't know about any existing search engine doing that. But this can be done with the help of Natural Language Understanding and Semantic Parsing.
Have a look at Stanford's Natural Language Understanding course (discussion of the text-to-scene problem can be found here) for further details.
How semantic search works is, it analysis data and put them into a 3-D vector space. One it's done with the help of bid data and knowledge graph the algorithm will try to find data points that connect to the article, the authority of the author, website relevance, and a couple of other factors. Once these factors are factored in, it then tries to create co-relate data to create a layer of information interconnected to each other. Once these information's are gathered then it is used to arrive at a conclusion to decide how relevant the data is.