Field type of geoip.ip and geoip.location changed after ELK upgrade

[kaisecheng]

This is an ​issuer transfer from logstash-plugins/logstash-filter-geoip#184

The field type of geoip.ip changed from ip to text and geoip.location changed from geo_point to object. This behavior causes Field type conflict .... The problem is suspected to relate to ELK upgrade to 7.12.1

Logstash information:

Please include the following information:

1. Logstash version (e.g. bin/logstash --version) 7.12.1
2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker) - DEB package from elastic repo
3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes) - systemd
4. How was the Logstash Plugin installed - bundled in DEB package

JVM (e.g. java -version): java 11.0.11

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

1. JVM version (java -version)
2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
3. Value of the JAVA_HOME environment variable if set.

... I'm using Debian repo's openjdk-11

OS version (uname -a if on a Unix-like system): Linux elk-node-1 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
I have problem that cca 1/4 events from apache / nginx / haproxy have tag _geoip_lookup_failure (some of them are RFC 1918 IPs) but the rest of events not have that tag. Below is image of Index pattern data with data types. In past (I think ~3 months ago) there were different types of some fields (geoip.ip changed from ip to text and geoip.location changed from geo_point into object). This behavior causes Field type conflict ... Due to Field type conflict I can't load map layer and even on new index patterns (eg. I've recently added new haproxy into different space and pattern) I can't create map layers bcs of wrong field type for geoip mapping

plugin setup

elasticsearch {
    hosts => "https://127.0.0.1:9200"
    ssl => "true"
    cacert => "<CERT_PATH>"
    ssl_certificate_verification => "false"
    index => "logstash-%{[app][name]}-%{[app][env]}-%{+YYYY.MM.dd}"
    ilm_policy => "<JUST_HOT_TO_WARM_PHASE>"
    user => "XXXXXXXXXX"
    password => "XXXXXXXXXX"
}

The index template copied from Kibana

{
  "geoip": {
    "properties": {
      "city_name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "continent_code": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "country_code2": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "country_code3": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "country_name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "dma_code": {
        "type": "long"
      },
      "ip": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "latitude": {
        "type": "float"
      },
      "location": {
        "properties": {
          "lat": {
            "type": "float"
          },
          "lon": {
            "type": "float"
          }
        }
      },
      "longitude": {
        "type": "float"
      },
      "postal_code": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "region_code": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "region_name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "timezone": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  }
}

cc @dusatvoj

[TheVastyDeep]

This is an ECS compatibility issue I believe. ECS is v8 by default, and [geoip] is not the target when ECS is enabled, so there is no reason for the ECS compatible template to make it a geo_point. You can always add your own template, or disable ECS using the ecs_compatibility option on the output.