How do you add new documents to an inverted index - search-engine

Question

Consider an inverted index with positional records stored in a MySQL Database as:
Word (VARCHAR) | Documents (LONGTEXT)
-------------------------------------------------------------
Hello | {id: 11, freq: 4, pos: [18, 37, 43, 119]},
| {id: 19, freq: 2, pos: [17, 32]}
-------------------------------------------------------------
Now, a new document comes and most of its words are already indexed. What should be the index operation now? Basic approach seems that if the word is already present in the database, then fetch its documents and add the current document to it and update the record.
Is this sustainable as the number of documents increases reaching, say, millions? How do real world search engines like Solr, Xapain, Google, Bing etc. handle this?

Answer 1

When a new document is added to your collection, the operation would be to:
Assign the document an id, say 20, which uniquely identifies the document. This id is typically incremented by 1 for each new document added to the collection.
Make a list over all the words in the new document, and at what position they occur.
For the document Hi Hello Hello Bye, this would be:
Bye: {id: 20, freq: 1, pos: [15]}
Hello: {id: 20, freq: 2, pos: [3, 9]}
Hi: {id: 20, freq: 1, pos: [0]}
For any new word (Bye, Hi), add an entry to the database for that word. For any existing word in the database (Hello), add the new data to that value.
Below is how your database would look after adding the document.
Word (VARCHAR) | Documents (LONGTEXT)
-------------------------------------------------------------
Bye | {id: 20, freq: 1, pos: [15]}
Hello | {id: 11, freq: 4, pos: [18, 37, 43, 119]},
| {id: 19, freq: 2, pos: [17, 32]}
| {id: 20, freq: 2, pos: [3, 9]}
Hi | {id: 20, freq: 1, pos: [0]}
-------------------------------------------------------------
The quick answer to your other question is: Yes, this is sustainable for large indexes. Inverted indexes is typically optimized for lookup, using hash tables or binary trees, making retrieval practically independant on the size of the document collection.
For how large search engines handle this: I don't know about the details (even though I'd like to). They obviously use data cluster to spread the load over multiple servers (yes, I said spread the load. It wasn't intentional). I bet they have preprocessed a bunch of stuff, and cached common queries like "Stack Overflow" so there's already a solution page for that.