40_One_language_per_doc.asciidoc

One language per document

A single predominant language per document requires a relatively simple setup. Documents from different languages can be stored in separate indices — blogs-en, blogs-fr, etc. — which use the same type and the same fields for each index, just with different analyzers:

PUT /blogs-en
{
  "mappings": {
    "post": {
      "properties": {
        "title": {
          "type": "string", (1)
          "fields": {
            "stemmed": {
              "type":     "string",
              "analyzer": "english" (2)
            }
}}}}}}

PUT /blogs-fr
{
  "mappings": {
    "post": {
      "properties": {
        "title": {
          "type": "string", (1)
          "fields": {
            "stemmed": {
              "type":     "string",
              "analyzer": "french" (2)
            }
}}}}}}

1. Both blogs-en and blogs-fr have a type called post which contains the field title.

2. The title.stemmed sub-field uses a language-specific analyzer.

This approach is clean and flexible. New languages are easy to add — just create a new index — and because each language is completely separate, we don’t suffer from the term frequency and stemming problems described in [language-pitfalls].

The documents of a single language can be queried independently, or queries can target multiple languages by querying multiple indices. We can even specify a preference for particular languages with the indices_boost parameter:

GET /blogs-*/post/_search (1)
{
    "query": {
        "multi_match": {
            "query":   "deja vu",
            "fields":  [ "title", "title.stemmed" ] (2)
            "type":    "most_fields"
        }
    },
    "indices_boost": { (3)
        "blogs-en": 3,
        "blogs-fr": 2
    }
}

1. This search is performed on any index beginning with blogs-

2. The title.stemmed fields are queried using the analyzer specified in each index.

3. Perhaps the user’s accept-language headers showed a preference for English, then French, so we boost results from each index accordingly. Any other languages will have a neutral boost of 1.

Foreign words

Of course, these documents may contain words or sentences in other languages, and these words are unlikely to be stemmed correctly. With predominant-language documents this is not usually a major problem. The user will often search for the exact words — for instance, of a quotation from another language — rather than for inflections of a word. Recall can be improved by using techniques explained in [token-normalization].

Perhaps some words like place names should be queryable in the predominant language and in the original language, such as Munich and München. These words are effectively synonyms, which we will discuss in [synonyms].

Don’t use types for languages

You may be tempted to use a separate type for each language, instead of a separate index. For best results, you should avoid using types for this purpose. As explained in [mapping], fields from different types but with the same field name are indexed into the same inverted index. This means that the term frequencies from each type (and thus each language) are mixed together.

To ensure that the term frequencies of one language don’t pollute those of another, either use a separate index for each language, or a separate field, as explained in the next section.