CSM BDS: Change allocation index mapping of text field to date type

[thanh-lam]

Describe the bug
Current cast-allocation index contains two string fields that are used for time range query:

data.begin_time
data.history.end_time

With elasticsearch 7.5.1+, time range queries on text fields do not work. These fields need to be mapped to date type.

It's possible that the index cast-allocation didn't have a customized mapping of data fields initially. So, filebeat reads the allocation records in csm_transaction log and puts them into elasticsearch indexes as text or string type.

This defect documents the procedure of loading a customized index mapping template for cast-allocation using the filebeat and curl commands.

To Reproduce
Steps to reproduce the behavior:

1. Log in to a BDS node to access elasticsearch service (local or remote).
2. Run curl to display cast-allocation index mapping:

curl http://XX.X.X.XX:9200/cast-allocation/_mapping?pretty

3. In the output, look for begin_time and end_time
4. Note that they are defined as text, for example:

            "begin_time" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },

Expected behavior
Those date fields should be defined as date, for example:

            "begin_time" : {
              "type" : "date"
            },

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Machine [CSM BDS, IST BDS clusters]
• Version [CSM 1.8.2]

Additional context
With these fields changed to date type, the date format to be written into them needs to have correct format as well. See #992.

Issue Source:
Beside addressing the index mapping issue, the procedure of how to use the filebeat command to load customized mapping template should be in CSM BDS readthedoc for when there's need in the future.

[thanh-lam]

Following are the steps to load a customized index mapping for cast-allocation using the filebeat command.

1. /etc/filebeat/filebeat.ym contains following at the end (no changes).
   Note: The section under "Elasticsearch template setting" could be added automatically by one of the commands.

name: "master"
setup.kibana:
  host: "X.XXX.XX.XX:5601"

output.logstash:
  hosts: [ "cXXXfXn0X:10523" ]

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

#==================== Elasticsearch template setting ==========================

setup.template.name: "filebeat-*"
setup.template.fields: "filebeat_fields.yml"
setup.template.overwrite: true

2. Obtain current cast-allocation index mapping from elasticsearch and save the output to a file (cast-allocation-map.json:

curl http://XX.X.X.XX:9200/cast-allocation/_mapping?pretty > cast-allocation-map.json

3. Edit cast-allocation-map.json and change the two fields: data.begin_time and data.history.end_time to type date as shown in defect description.
4. Copy cast-allocation-map.json to cast-allocation-template.json. Edit the new file and make sure the "index_patterns" attribute is assigned the value "cast-allocation" before the "mappings" section.

{
  "index_patterns": [
    "cast-allocation"
  ],
  "mappings": {

5. Run filebeat to disable logstash output and enable elasticsearch output:

filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["XX.X.X.XX:9200"]'

6. Run curl to load the customized template:

curl -XPUT -H 'Content-Type: application/json' http://XX.X.X.XX:9200/_template/cast-allocation [email protected]

7. Verify that the template was applied:

curl http://XX.X.X.XX:9200/_template/cast-allocation?pretty

8. Login to kibana GUI to delete existing cast-allocation index in elasticsearch and kibana. Then, create some new csm allocations to see new cast-allocation index in Elasticsearch. Now, create the index pattern in kibana to access new data.

[thanh-lam]

Edited the steps to add the last step because it's necessary to delete the old cast-allocation index in elasticsearch and kibana. Then, the new data will be mapped to the new data type.