Index already exists exception

[abreitenfeld]

Get this exception when I try to start the latest (0.14.2) Annotator Store on a clean and empty installation of ElasticSearch 2.1.1

2015-11-11 09:58:25 12411 annotator.elasticsearch [INFO] Creating index 'annotator'.
2015-11-11 09:58:25 12411 elasticsearch [WARNING] PUT /annotator [status:400 request:0.004s]
2015-11-11 09:58:25 12411 annotator.elasticsearch [CRITICAL] Failed to create an Elasticsearch index
Traceback (most recent call last):
  File "run.py", line 114, in <module>
    main(sys.argv)
  File "run.py", line 67, in main
    annotation.Annotation.create_all()
  File "/home/user/Projects/annotator-store/annotator/elasticsearch.py", line 93, in create_all
    conn.indices.create(cls.es.index)
  File "/home/user/virtualenv/pyenv/local/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/user/virtualenv/pyenv/local/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 103, in create
    params=params, body=body)
  File "/home/user/virtualenv/pyenv/local/lib/python2.7/site-packages/elasticsearch/transport.py", line 307, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/home/user/virtualenv/pyenv/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 93, in perform_request
    self._raise_error(response.status, raw_data)
  File "/home/user/virtualenv/pyenv/local/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 105, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, u'index_already_exists_exception')

[Treora]

I do not know what is wrong, but perhaps try Elasticsearch 1.x, since more things in the code do not work yet with Elasticsearch 2.x (e.g. #130). @nickstenning or @tilgovi (or whoever with access), could you update the readme to reflect this?
Please close this issue if that fixes it.

[abreitenfeld]

I tested it with ElasticSearch 1.7.4 on a Xubuntu installation and everything seems to be fine.
But as I used the same setup (downloaded the binary from ElasticSearch) on a Ubuntu installation I still get the exception. Very confusing

[abreitenfeld]

I dived into the source code and found the error source in elasticsearch.py at line 97

if not (e.error.startswith('IndexAlreadyExistsException')
   or e.error.startswith('InvalidIndexNameException')):
        log.fatal("Failed to create an Elasticsearch index")
        raise

Are you checking the type of error by comparing strings? Seriously? So in my case this leads to an exception. Please fix this

[nickstenning]

Are you checking the type of error by comparing strings? Seriously? So in my case this leads to an exception. Please fix this

At the time this code was written there was no exception hierarchy in the python ElasticSearch client. There was no way to determine the type of the error except by inspecting the string value of the error.

More importantly, you are currently commenting on an issue for a piece of open source software. It was developed by a variety of people, some of whom contributed to it in their free time. You have paid no money to use it. You are under no obligation to use it. Please adjust your tone.

[abreitenfeld]

Okay I understand. I checked the source code of pyelasticsearch and they do it the same way like you. Actually I would prefer an exception-preventing approach: check if the index already exist before creating it. You can see my suggestion here FUB-HCC@21bb8de

If you go with my solution I can create a Pull Request.

[tilgovi]

Checking that the index doesn't exist first will not be bulletproof because the index might be created between the check and the call to create.

I would prefer that we remove the exception handling entirely and leave it to the operator to invoke a script to create the index interactively.

Making the creation implicit increases the likelihood that the index already exists and that multiple instances race to create an index. It's also better to be paranoid, I think, that perhaps you're writing to the wrong index if you're trying to create it and it already exists.

[tilgovi]

The change then, that I propose, is to move the exception code out of create_all and into the run.py script.

It might be reasonable to then perform the check that you have for existence in the run script, since it serves as an example CLI, @abreitenfeld.

I generally I believe it's better to let the errors bubble and catch them at the interface, where we know more about the user intent. A CLI script is a better place to catch than in the guts of the model code.

[abreitenfeld]

That sounds good to me. One information aside, I tested it with ElasticSearch 2.x. and immediately got some exceptions. Maybe you restrict the supported versions to 1.x in the meantime

[tilgovi]

I'm re-opening this because we can solve it with a simple ignore=400 keyword parameter. I also opened #133.