70_Symbol_synonyms.asciidoc

Symbol synonyms

The final part of this chapter is devoted to symbol synonyms, which are unlike the synonyms we have discussed up until now. Symbol synonyms are string aliases used to represent symbols that would otherwise be removed during tokenization.

While most punctuation is seldom important for full text search, character combinations like emoticons may be very signficant, even changing the meaning of the the text. Compare:

• I am thrilled to be at work on Sunday.

• I am thrilled to be at work on Sunday :(

The standard tokenizer would simply strip out the emoticon in the second sentence, conflating two sentences that have quite different intent.

We can use the {ref}analysis-mapping-charfilter.html#analysis-mapping-charfilter[mapping character filter] to replace emoticons with symbol synonyms like emoticon_happy and emoticon_sad before the text is passed to the tokenizer:

PUT /my_index
{
  "settings": {
    "analysis": {
      "char_filter": {
        "emoticons": {
          "type": "mapping",
          "mappings": [ (1)
            ":)=>emoticon_happy",
            ":(=>emoticon_sad"
          ]
        }
      },
      "analyzer": {
        "my_emoticons": {
          "char_filter": "emoticons",
          "tokenizer":   "standard",
          "filter":    [ "lowercase" ]
          ]
        }
      }
    }
  }
}

GET /my_index/_analyze?analyzer=my_emoticons
I am :) not :( (2)

1. The mappings filter replaces the characters to the left of ⇒ with those to the right.

2. Emits tokens: i, am, emoticon_happy, not, emoticon_sad.

It is unlikely that anybody would ever search for emoticon_happy, but ensuring that important symbols like emoticons are included in the index can be very helpful when doing sentiment analysis. Of course, we could equally have used real words, like happy and sad.

Tip	The mapping character filter is useful for simple replacements of exact character sequences. For more flexible pattern matching, you can use regular expressions with the {ref}analysis-pattern-replace-charfilter.html[pattern_replace character filter].