How to create a Recipe / Ingredient parser - parsing

Question

Given a line such as
1 pound of Beef
I want to extract the ingredient. Initially im only interested in the ingredient name.
Ive looked at rubys famous time parser Chronic and like its use of regexs.
def self.scan_for_month_names(token)
scanner = {/^jan\.?(uary)?$/ => :january,
/^feb\.?(ruary)?$/ => :february,
/^mar\.?(ch)?$/ => :march,
/^apr\.?(il)?$/ => :april,
/^may$/ => :may,
/^jun\.?e?$/ => :june,
/^jul\.?y?$/ => :july,
/^aug\.?(ust)?$/ => :august,
/^sep\.?(tember)?$/ => :september,
/^oct\.?(ober)?$/ => :october,
/^nov\.?(ember)?$/ => :november,
/^dec\.?(ember)?$/ => :december}
scanner.keys.each do |scanner_item|
return Chronic::RepeaterMonthName.new(scanner[scanner_item]) if scanner_item =~ token.word
end
return nil
end
However in my case Id probably have to create over 300 regexs for each individual ingredient.
I'd also have to take into account of synonyms such as Cilantro & Corriander Leaf
Ive never done parsing before but is the use of regexs here still the best way to go. I cant think of any other reasonable alternative.

Answer 1

Firstly, I'm assuming that the ingredients don't always take the form of QUANTITY UNIT of INGREDIENT - otherwise, this would be a very trivial task (just copy the substring after of
This is a difficult problem - the solution will not be simple.
I think using regex may not be the best approach here:
As you mention, you'll have to write a lot of expressions for each
ingredient
Your list of possible ingredients will always be limited
by the regex list, and you can't detect new ingredients without
compiling more.
it will be very difficult to parse some ingredients(cheese, 1 pound (parmesan))
I think that natural language processing is the way to go here. You have unstructured input, but in a very restricted context.
Perhaps counter-intuitively, I think the best way to find the ingredient may very well be to not look for it - look for everything else instead. If you assume that a line will always have
a numeral (quantity)
a unit (pounds, teaspoons, etc)
a ingredient
and that it's pretty easy to detect numerals and units, it should be straightforward to recognize those first and then extract the ingredient.
If you use a part-of-speech tagger, it's easy to identify relevant words:
[('1', 'LS'), ('pound', 'NN'), ('of', 'IN'), ('Beef', 'NNP')]
From there, you may want to use a classifier. For that, you'll need to label the ingredients manually on a good quantity of lines (say, hundreds). Some possibly good features to use:
position of the word in the line
presence in a precomputed ingredient dictionary (possibly using some partial string matching metric like Levenshtein's
output of part-of-speech tagger
words immediately before and after (if you have an 'of' before the word, there's a high probability it's a ingredient
I'm sure you'll be able to find countless others after working on a few lines.
Finally, I expect that some lines will be very difficult to work on. 1 pound of parmesan cheese, 1 pound of emmentaler: you'd have to infer that the second ingredient is a cheese, too.
As to software, if you can choose the language to use, python has the fantastic Natural Language Toolkit. I can't vouch for toolkits in other languages, but maybe someone else will.

Answer 2

I think I would start by running a series of regex checks against each line, and adjust the parsed text as you go. For example (pseudocode):
First, check for instruction:
/^(add|fold in|stir in|etc...)/
If you found an instruction, remove it from the line, set a flag, and continue:
instruction = $1
this_line = this_line.substring(instruction.length())
If an instruction was found, check to see if there was a subsequent instruction (like "and cover" or "and set aside")
/\b(and\s)(.*)$/
If found, strip that and insert it before the next line of the recipe
instruction = instruction.substring(0, instuction.length - $1.length - $2.length)
splice $2 into the array of lines immediately following this one
Next, maybe you'll check for a preposition:
/((?in)to\s(.+)/
If found, you might use that to check for equipment names, bowls, measuring cups, etc.
Even if you don't use it, you can probably remove it from the string you're parsing, to improve your matching.
Finally, the real work is done with the text that's left:
Check against /^(\d+\s+(?a\s)?\w+)\s*(?of\s*)?(.+)$/
Which should give you $1 containing the unit of measure and $2 containing the ingredient.
Lather. Rinse. Repeat.
After that, do whatever magic your app does with this information.

Answer 3

First of all, I suggest doing some searching to see if someone else has already created a solution to this problem which is good enough for you to use, rather than reinventing the wheel.
For instance, you may find this project to be interesting. It uses machine learning to attempt to parse ingredient phrases, including type of ingredients and amounts.
Other interesting projects also come up when googling for "ingredient parser".
If you are really determined to write this yourself, then I suggest that you do some research into the category of software tools known as a "parser generator", which is a tool which will allow you to write the language you want to recognize in an abstract form (a "grammar"), and then will generate code in your language of choice which will parse text according to that grammar and will identify specific subconstructs within it it very efficiently (much more efficiently than could be done by hundreds of regular expression matches).
For instance, a grammar used as input to a parser generator might look something like this:
// I am making up the following syntax for demonstration purposes, but it illustrates the
// sort of things that one could specify in a grammar, and is not terribly different from
// the grammar languages that real parser generators use.
//
// Note that everything in the curly braces is code to be inserted into the generated parser.
// Each such code block will be invoked when the preceding parsing rule is matched.
%declare { bool organic=false; bool dried=false; bool smoked=false; }
INGREDIENT ::= "organic" INGREDIENT { organic=true; }
| INGREDIENT "(" "organic" ")" { organic=true; }
| "dried" INGREDIENT { dried=true; }
| "smoked" INGREDIENT { smoked=true; }
| AMOUNT "of" INGREDIENT
| INGREDIENT "(" AMOUNT ")"
| BASE_INGREDIENT
BASE_INGREDIENT ::= ( WORD )* {
doSomethingWithBaseIngredient(organic, dried, smoked, $BASE_INGREDIENT);
}
AMOUNT ::= NUMBER ( VOLUME_UNIT | WEIGHT_UNIT )
VOLUME_UNIT ::= "cup" | "liter"
WEIGHT_UNIT ::= "mg" | "kg" | "pound"
NUMBER ::= [0-9]+
WORD ::= [a-zA-Z]+
... and so forth.
The parser generator, when run, would take this grammar as input, and would generate code in your desired programming language as output. This code would parse input text according to the grammar and would also set variables and/or call functions of yours as desired when certain parsing rules are matched. The parsers generated by such tools often use special parsing techniques (often involving large tables, state machines, and so forth) to parse very efficiently in a single pass without having to do any more work than necessary, and avoiding backtracking when possible.
Some common examples of parser generators are lexx/yacc, bison, and Antlr. Many others exist. (Personally, I have gotten good results with Antlr in the past, and am particularly fond of the fact that it can generate parsers in many different programming languages.) Many of these parser generators are mostly intended for use by compiler writers, but that does not mean that they can't be used for other purposes, such as recognizing the various forms that ingredients in recipes take.
This article provides an overview of parser generators, and this article contains a table of various parser generators and their attributes (output languages, etc.) as well as links on where to find more.