Review log entries related to monitoring via internal collection

[robbavey]

Currently, when using monitoring in Logstash, a number of warnings and deprecation logs are emitted:

In regular logstash logs:

[2021-10-06T14:48:49,171][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}

and

[2021-10-06T14:48:49,343][WARN ][logstash.javapipeline    ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary

These are not actionable by users, they are emitted regardless of settings in logstash.yml if internal collection is enabled.

Additionally, in the deprecation log:

[2021-10-06T14:40:03,729][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
Please configure Metricbeat to monitor Logstash. Documentation can be found at: 
https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html

We should ensure that this message is accurate, and reflects the reality wrt using internal collectors with Logstash with 8.0 of Elasticsearch

[yomduf]

Hi,
These deprecation logs are emitted even without monitoring enabled :
We just need to have Elasticsearch output.
Logstash 8.5->ES 8.5

input {
    generator {

        lines => [ '{"key":"value","secondkey":"secondvalue"}' ]
        count => 1
    }
}
filter {
    json {
        source => "message"
    }
}

output {
    stdout {
        codec => rubydebug {
        }
    }
        elasticsearch {
    }

}

Related warning:

{"level":"WARN","loggerName":"logstash.outputs.elasticsearch","timeMillis":1669730430179,"thread":"[inputfile.conf]-pipeline-manager","logEvent":{"message":"Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type","es_version":8}}

[robbavey]

Created logstash-plugins/logstash-output-elasticsearch#1198 to track Elasticsearch output unnecessary output