Routing Lost on Single Node Cluster Recovery [community plugins involved]

[daviddyball]

Elasticsearch version: 2.3.3

Plugins installed: [ kiwionly/elasticsearch-image, mobz/head, cloud-aws]

JVM version (java -version):

openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-1~bpo8+1-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

OS version (uname -a if on a Unix-like system):
Docker image version elasticsearch:2.3.3 from Docker Hub

Description of the problem including expected versus actual behavior:
We had a single-node cluster get overloaded and the search queue backed up. We did a kill -TERM to restart it gracefully, but after it came up most of the documents in our index were unavailable by the GET API. After reading in lots of places that documents being unavailable in GET is usually related to routing issues, I discovered doing curl localhost:9201/my_index/document/{id}?routing=4 was able to return my documents. We've never used custom routing on our clusters (and this is a single-node anyway, so irrelevant). So I'm wondering what is causing this.

One thing that comes to mind is a bug in the elasticsearch-image plugin whereby an index has to be created with index.version.created always set to 1070499 regardless of the actual version of ES you're using to create the index (bug raised here: kiwionly/elasticsearch-image#17). I noticed that, after a restart of the node there are additional legacy.routing options set in the index _settings. Could this be causing the routing confusion?

While I appreciate that there is an obscure community plugin in the mix, I'd be really keen to know if there is a way to get back to a state where I no longer have to specify routing=4 on every GET request in order to see documents?

Steps to reproduce:

1. Install elasticsearch-image
2. Index settings

   {
     "my_index": {
       "settings": {
         "index": {
           "version": {
             "created": "1070499"
           },
           "uuid": "3UTJJ0CoR2WxjyfajGr36Q",
           "number_of_replicas": "0",
           "creation_date": "1498056689819",
           "number_of_shards": "5"
         }
       }
     }
   }
   

3. Mappings

   {
     "my_index": {
   	"mappings": {
   	  "document": {
   		"properties": {
   		  "image": {
   			"metadata": {},
   			"feature": {
   			  "JCD": {
   				"hash": [
   				  "BIT_SAMPLING"
   				]
   			  }
   			},
   			"type": "image"
   		  }
   		},
   		"_timestamp": {}
   	  }
   	}
     }
   }
   

4. Shutdown the node and restart it.
5. The index settings has the following added to it, which I believe is caused by the index originally being created with settings.index.version.created=1070499:

   "legacy": {
     "routing": {
       "use_type": "false",
       "hash": {
         "type": "org.elasticsearch.cluster.routing.DjbHashFunction"
       }
     }
   }
   

Provide logs (if relevant):

[tlrx]

Elasticsearch uses a hash function to spread the documents among the shards of the index. In your case, it uses the hash function of 2.3.3. But since you created the index with a old version (index.version.created: 1070499 corresponds to version 1.7.4), when the node is restarted elasticsearch thinks it's a old index and updates the index settings to use the hash function corresponding to 1.7.4 (DjbHashFunction). That just messed up all the Get requests.

You should not force a old version when creating an index (I think it's not possible anymore in newer versions). So your options are to either use an old elasticsearch version compatible with the community plugin or use a newer version of the plugin if it exists.

[daviddyball]

Makes complete sense. Thanks for the clarification. I've spoken with the plugin maintainer and it's not on the roadmap, so I'll work something out.