2.Breaking Words Apart to Find Them Again

Sarah wants to understand the actual mechanism, not just trust that it works. If "jaket" finding "jacket" isn't magic, what is it? She starts with the most basic question: how does a search engine even look at one product's text in the first place?

It doesn't read a product's description like a sentence. It takes it apart.

Tokenization is that first, literal step: splitting a piece of text into individual tokens — roughly, words. "A lightweight, waterproof jacket built for cold, wet trails" becomes a list: lightweight, waterproof, jacket, built, for, cold, wet, trails. This is done by an analyzer — the pipeline that turns raw, messy text into a clean list of tokens, usually including text normalization (lowercasing everything, so "Waterproof" and "waterproof" are treated as the exact same token — this course's own Search Playground genuinely does this, confirmed directly: searching "WATERPROOF" finds the same result as "waterproof").

Once every document's text is tokenized, the search engine builds an inverted index — and the name describes exactly what it is. A normal index goes document → its words. An inverted index flips that: it goes word → every document containing it. Token waterproof points to document 1. Token coat points to document 2. Search for waterproof, and the engine doesn't scan every document's text — it looks up that one token directly and gets back exactly the documents that contain it, instantly.

Two more real IR concepts worth naming honestly, not glossed over: stemming is reducing a word to its root form so "running," "runs," and "ran" all match as the same underlying concept — a real, genuine technique in production search engines. Stop words are common, low-information words ("a," "the," "for") that a search engine often strips out before indexing, since they add noise without adding meaning.

This course's own Search Playground, worth being precise about, does neither of those two specifically. It's real text normalization (case-insensitivity) plus real fuzzy/typo tolerance — which is a different mechanism from true stemming, confirmed directly: searching "insulating" or "insulation" finds nothing for the "Insulated Winter Coat," because the edit distance between those words is too large for fuzzy matching to bridge, and there's no real stemmer turning them into a shared root the way a production search engine like Elasticsearch would. "Jackets" matching "jacket" earlier wasn't stemming recognizing a plural — it was fuzzy tolerance catching a one-character difference. The distinction matters: this Playground's typo tolerance is real and genuinely useful, but it isn't the same mechanism as true stemming, and conflating the two would be teaching something false.