A Spotify-style search over six verticals and four languages with Elasticsearch

At work I built the search box of a travel and local-commerce platform in Andorra. You type “spa” and you get, in one response, a row of experiences, a row of hotels and a row of shops, the way Spotify answers “queen” with the band, its songs and the playlists that mention it. There are six verticals behind that box (experiences, hotels, restaurants, shops, editorial articles and CMS pages), four languages, and a query takes about 3 ms inside Elasticsearch.

This post is the tour: what Elasticsearch does with your text, how the indexes are written, how one request reaches six indexes, and the relevance bugs I found by measuring instead of guessing. It also covers the decision I would be asked about first in an interview: the indexes are never patched, they are rebuilt from scratch on every sync. At our scale that is the right call, and I explain why. Then I explain what I would build instead at a million documents per vertical, and point to the Vinted engineering posts that describe that kind of pipeline in production.

The problem#

The requirements fit in a paragraph. One GraphQL query takes the text, the verticals wanted and the browsing language, and returns hits grouped by vertical, each group with its total and its top cards. The catalogue is translated into Catalan, Spanish, English and French, but not evenly: there are no Catalan translation rows at all (the base columns are already Catalan), and some Spanish, English or French ones are missing. Someone browsing in Spanish will type a Catalan word, and the search has to understand them.

The scale is small, and that shapes everything below. Andorra is a country of under a hundred thousand people, and its catalogue is what you would expect: about 250 published experiences, 181 hotels, around 120 restaurants and 115 shops, plus the articles and the CMS pages. The experiences index takes 1.6 MB.

What made it hard was never the volume. It was the relevance: “restaurant encamp” returning 162 hotels because their amenities say “restaurant”, or “massatge” on the Spanish site finding one massage out of fourteen.

Elasticsearch in five minutes#

Elasticsearch is a distributed layer on top of Lucene. An index is a set of documents with a mapping, the schema that says how each field is stored and analyzed. An index is split into shards, each one a Lucene index, and each shard can have replicas. Ours are tiny, so every index has one primary shard.

The part that matters for search is what happens to text before it is stored. A text field goes through an analyzer: character filters that rewrite the raw string, a tokenizer that splits it into tokens, and token filters that transform them. The resulting terms go into an inverted index, a map from each term to the documents that contain it. A query goes through the same analyzer, so matching is a lookup of terms, not a scan of text.

That diagram is the real output of _analyze with explain: true against our mapping, and almost every step is there because of a bug:

• apostrophes turns ' into a space, so Oh'Kebab is searchable as kebab.
• place_phrases glues “Pas de la Casa” into one token. Otherwise it is “pas” plus three stopwords.
• asciifolding makes esqui match esquí. Nobody types accents in a search box.
• place_names is a keyword_marker with the towns and parishes, which the stemmer then leaves alone. Before it existed, the Catalan stemmer reduced Canillo to can.
• The stemmer is what makes esquiar and esquí the same term.

Each language gets its own analyzer and each multilingual field is stored four times, name.catalan, name.spanish and so on. The mapping is dynamic: strict and embedded in the Go binary with go:embed, so a field the mapping does not know makes Elasticsearch reject the document at sync time instead of indexing it silently.

The last concept is the alias, a name that points to one or more indexes. Our queries never name a physical index. They name search_experiences, which points to search_experiences_v20260825130120. That indirection is what makes the write path below possible.

The write path: rebuild, never patch#

Every sync builds each vertical’s index from scratch, next to the live one, and then moves the alias.

The order of the steps is the design:

1. Read everything first. Experiences come from MySQL, hotels and restaurants and shops from two gRPC microservices, articles and pages from the CMS. All of it is fetched in batches of 100 before anything is created in Elasticsearch, so a database blip, which is the likely failure, aborts the run without leaving an index behind.
2. Guard the count. If the number of documents read differs from the count taken at the start, the run stops. Swapping onto an emptier index than expected would take search down without a single error, which is worse than serving yesterday’s data.
3. Fill an index nobody can see. It is created without an alias, filled with _bulk, using the source id as _id so a retry is an idempotent overwrite, and then refreshed. The refresh is mandatory: bulk writes sit in a buffer and are not searchable until a refresh.
4. Swap atomically. The remove of the old index and the add of the new one go in the same _aliases call, which Elasticsearch applies as one cluster state change. No search ever sees the alias pointing to nothing.

POST /_aliases
{
  "actions": [
    { "remove": { "index": "search_experiences_v20260825125953", "alias": "search_experiences" } },
    { "add":    { "index": "search_experiences_v20260825130120", "alias": "search_experiences" } }
  ]
}

A rebuild that dies between creating the index and the swap leaves an index with no alias, invisible but taking disk. The next run starts by sweeping those, and the timestamp in the name is not decoration: the sweep skips anything younger than an hour, because that may be a rebuild still in flight.

The verticals run concurrently, each in its own goroutine and on its own index. A failure travels inside that vertical’s report instead of as an error, so one broken vertical never hides the others, and the endpoint only returns a 5xx when every vertical that ran failed. Cloud Tasks retries the whole call on a 5xx, and rebuilding the verticals that already worked would be wasted work.

Why rebuild instead of syncing changes#

This is the decision I would defend in a design review, so here are the reasons.

The documents are denormalized from eleven tables. An experience document carries its business name, category names, prices, images, slug, rating and option labels in four languages. Watching products.updated_at would miss most changes: renaming a category or hiding an experience from the website does not touch the product row. An incremental sync would need to know, for every table, which documents it feeds. A rebuild makes that whole class of bug impossible.

Deletions are free. An unpublished experience is simply missing from the new index. There is no tombstone logic to get wrong.

Every sync is also a migration. A change of mapping, analyzer or document shape ships with the next sync. No reindex scripts, no versioned migrations. That mattered a lot while I was tuning relevance, because most fixes changed the mapping.

It is cheap. A full rebuild of a vertical takes 4 to 9 seconds, nearly all of it reading MySQL. The catalogue changes a few times a day and nobody needs a new hotel to be searchable within the second, so a scheduled rebuild is fresh enough.

The price is known and accepted. Freshness is bounded by the schedule. The count and the batches run in separate transactions, so a write in the middle of a sync can trip the guard; with this write rate that is rare, and one REPEATABLE READ transaction around the read would remove it. And one flag is too volatile to bake into the index: active, whether an experience can be booked right now, is stored in every document but filtered in the query, so changing the criterion takes effect on the next search with no rebuild.

If there were a million documents per vertical#

Now change one number. Say each vertical had a million documents instead of a few hundred. The rebuild stops being the right answer, and the reasons are concrete:

• Memory. The index takes about 6.5 KB per document (1.6 MB for 252), so a million documents is around 6 GB of index. The sync reads the whole vertical into memory before it creates the index, on purpose, so the process that runs it would need memory of that order.
• Load on the source. Every run reads every row of eleven tables and walks two microservices end to end, whether one thing changed or nothing did.
• Freshness. A rebuild that takes an hour cannot run every five minutes. Changes wait for the next run.

The index size is not the problem. Elastic’s sizing guidance aims for shards of tens of gigabytes, so 6 GB is still one primary shard, and read throughput scales with replicas. What breaks is the write path. Here is what I would build instead.

Capture changes at the source, not by polling. Debezium reads the MySQL binlog, so every committed change becomes an event without touching application code. The hotels and listings microservices own their data, so they publish their own events with a transactional outbox, which writes the event in the same transaction as the change. The CMS sends webhooks.

Resolve the fan-out before building anything. This is where my case differs from a table that maps one-to-one to an index. A row change has to be turned into the ids of the documents it affects: a product update touches one experience, a category rename touches every experience in that category. The resolver publishes those ids to a search.changes topic, and the message carries no data, only “rebuild experience 812”.

Build full documents, never patches. A document builder consumes the ids, reads each document from the source with the same mappers the rebuild uses today, and publishes the full document to a compacted topic keyed by id. A document that is no longer published becomes a tombstone. Because the change events carry no data, their order does not matter: whichever event arrives last, the builder reads the current state. A burst of changes to the same document collapses into one read.

The compacted topic is the snapshot. With compaction and infinite retention, the topic keeps the latest version of every document, as Vinted does. That makes it the source for two sinks:

• The live sink reads the tail and writes with _bulk, using the record’s offset as an external _version, so a retried, older write is rejected instead of overwriting a newer one. Vinted’s partition incident shows the condition: the offset is only a valid version while the partition count of that topic never changes.
• A rebuild sink, for a new mapping, starts a new consumer group from offset 0 into a new index, with refresh_interval: -1 and no replicas while it loads, and restores both at the end. It never reads the database. Both sinks consume new events while the rebuild runs, so no change made during it is lost, which is the classic race of reindex-then-swap. When its lag is close to zero, the alias moves and the rebuild sink becomes the live one.

Reconcile, because incremental pipelines drift. A nightly job compares counts and checksums per vertical between the source and the index, and re-enqueues the ids that differ. It is the count guard of today, grown up.

What does not change is everything a query touches. The aliases, the versioned indexes, the strict mappings and the query builders stay as they are. The module is hexagonal and the use cases only see a port, so this is a new adapter on the write side, not a rewrite.

For the catalogue we have, all of this would be a Kafka cluster, Debezium, an outbox in two services and two consumers to keep alive, so that a few hundred documents per vertical can show up faster than nobody needs. That is why the code rebuilds, and why I know when it should stop.

The read path: six verticals in one round trip#

Each vertical has a query builder: its own fields, boosts and popularity signal. The searcher asks each builder for its query and writes all of them into one _msearch body, the newline-delimited format where a header line names the index and the next line is the query:

var body bytes.Buffer
encoder := json.NewEncoder(&body)
for _, vertical := range selected {
  header := map[string]string{"index": searcher.client.Alias(vertical.Index)}
  if err := encoder.Encode(header); err != nil {
    return nil, err
  }
  if err := encoder.Encode(vertical.Builder.Build(params)); err != nil {
    return nil, err
  }
}
responses, err := searcher.client.Msearch(ctx, &body)

Three details make it work:

• Responses come back in the order the queries were written, so response i belongs to vertical i, with no ids to match.
• Errors are per sub-search. A malformed restaurants query fails that group, which is logged and skipped, and the other five still render. The trade-off is that a broken query looks like an empty group, so an unexpected empty result means checking the logs first.
• Scores are not comparable across indexes. BM25 uses statistics of the index it runs in, so a 12 in hotels and a 12 in experiences mean different things. The order of the groups is fixed by the product, never derived from scores.

Every hit also reports which named query group it matched, strong or weak, and that drives the last step before the response, which the next section explains.

Relevance: what I measured#

Relevance was most of the work. Our product manager wrote a battery of test searches in seven areas: relevance, noise between verticals, normalization, languages, publication, edge cases and UX. I turned the first six into a Go test, behind a relevance build tag, that runs the real query builders against a local Elasticsearch: 144 cases, all passing. A second list of 218 searches is for checking by hand on the website. Every fix below was measured with that harness before and after.

Noise between verticals: strong and weak matches#

“Karting Mini Gran Premi” returned 79 hotels, because “gran” appears in their descriptions. Each vertical’s matchers are now split into two named groups. Strong covers fields someone curated to say what the thing is: name, tags, keywords, category, amenities, place. Weak is long text where a word can appear in passing: descriptions, menus, catalogues. Every hit reports which group matched (matched_queries). If any vertical made a strong match, the verticals that only made weak ones are emptied. That query now returns 0 hotels, 0 restaurants and 0 shops instead of 79, 45 and 18. With no strong match anywhere everything stays, because “tortilla” only exists in menus.

Two rules sit next to it. A query that names another vertical, such as “restaurant encamp”, only admits documents whose name holds every term; before, it returned 162 hotels, mostly because 154 of them list a restaurant among their amenities. And “andorra” is removed from the matchers unless it is part of “Andorra la Vella”, because it is in the names of 33 experiences and 77 shops.

Languages: try every language, weigh the one being browsed#

On the Spanish site, “massatge” found 1 massage out of 14: the only one with no Spanish translation, which fell back to the Catalan name. Now every multilingual field is also queried in the three languages not being browsed, weighted at 0.3 so that a match in the browsing language still wins. Those extra clauses require every term (minimum_should_match: 100%), because they multiply the fields by four and, with the default OR, “escape room” would match every document that mentions a room in any language. “massatge” now finds all 14.

Stopwords: one list for four languages#

Each analyzer used to remove its own language’s stopwords. The content mixes languages, so that leaked: a is not in the French list, and through french_text it pulled 41 of 252 experiences and 178 of 181 hotels. Every analyzer now uses the union of the Catalan, Spanish, English and French lists, plus a filter that drops one-letter tokens. It removes more words than any single list would, and the battery confirmed that no legitimate query lost results.

As-you-type needs asymmetric analysis#

The name has a search_as_you_type subfield, which indexes prefixes so matching while typing is cheap. Here stopwords must be removed at index time but not at search time. With them in the index, “moto de neu” (a snowmobile, which the catalogue does not have) returned 12 documents topped by Baño de bosque, through de. Removing them from the query as well breaks typing: ma and es are French function words, and stripping them took ma from 13 hits to 0 while the user was still typing “massatge”. With the index analyzer removing stopwords and the search analyzer keeping them, ma still finds 13 and “moto de neu” finds 0, which is the right answer.

Fuzziness: AUTO:5,8 and prefix_length 1#

The default fuzziness: AUTO is AUTO:3,6: one edit from three letters and two from six. One edit on a three-letter word matches almost anything: “via” reached “vi” (wine, in Catalan), and a painting-and-wine workshop showed up for “via ferrata”. Raising it to AUTO:5,8 keeps the real typos, since “masage” and “masatge” both still find all 14 massages. Fuzziness only applies to the short name fields; on long text, two edits match nearly everything. prefix_length is 1 and not 2, because at 2 “escursion” stops matching “Excursió”: the typo is in the second letter.

Popularity has to multiply#

Sales are the only popularity signal, and I added them with field_value_factor and boost_mode: sum. They did nothing, and the reason is a trap in the API: the modifier is applied to factor × sales, so with log1p it computes log(1 + factor·sales), not factor · log(1 + sales). The logarithm saturates, and no factor brings popularity close to text scores of 30 to 60. What was really ranking the results was name length, because BM25 favours short fields:

Multiplying is safe with log2p, which is log10(2 + sales): zero sales gives 0.30 instead of 0, so a new experience is damped, never erased, and the spread up to the best seller is about 5×, less than the spread of text scores. Relevance still wins: “museo” keeps the exact name match with 33 sales above a more popular experience that only mentions the museum in its description.

The boosters, a name holding every term and a flat 30 points per term covered by some short field, are added after the multiplication. Inside it, 41 sales tripled the score of a tour that merely names Escaldes, and it outranked the massage in Escaldes that “massatge escaldes” asked for. The 30 is measured too: 25 was not enough for that query, and 30 moved nothing else in the battery.

What is left#

The open question is no longer the term logic, it is quality. A fourteen-word sentence still reports 23 documents, because its words really are spread across the catalogue, and telling 23 weak matches from 23 good ones needs click data. Hotels have no popularity signal yet, so amenity searches like “piscina” rank 32 equally good matches arbitrarily. Rating or stars would fix that with the same multiply mechanics.

What I would keep from this project: measure relevance, never eyeball it. Every rule above started as a number that looked wrong in the harness, and every one of them broke something else the first time. And choose the write path for the scale you have, while knowing exactly which number would make you change it.