Logstash throws a "Failed to install template" on events sent from Elastic Agent

[jsvd]

Moved from elastic/logstash#14055 (created by @dedemorton)

Logstash information:

Please include the following information:

1. Logstash version: 8.2.0
2. Logstash installation source: expanded from tar
3. How is Logstash being run: via command line

Plugins installed: (bin/logstash-plugin list --verbose)

logstash-codec-avro (3.3.1)
logstash-codec-cef (6.2.4)
logstash-codec-collectd (3.1.0)
logstash-codec-dots (3.0.6)
logstash-codec-edn (3.1.0)
logstash-codec-edn_lines (3.1.0)
logstash-codec-es_bulk (3.1.0)
logstash-codec-fluent (3.4.1)
logstash-codec-graphite (3.0.6)
logstash-codec-json (3.1.0)
logstash-codec-json_lines (3.1.0)
logstash-codec-line (3.1.1)
logstash-codec-msgpack (3.1.0)
logstash-codec-multiline (3.1.1)
logstash-codec-netflow (4.2.2)
logstash-codec-plain (3.1.0)
logstash-codec-rubydebug (3.1.0)
logstash-filter-aggregate (2.10.0)
logstash-filter-anonymize (3.0.6)
logstash-filter-cidr (3.1.3)
logstash-filter-clone (4.2.0)
logstash-filter-csv (3.1.1)
logstash-filter-date (3.1.14)
logstash-filter-de_dot (1.0.4)
logstash-filter-dissect (1.2.5)
logstash-filter-dns (3.1.4)
logstash-filter-drop (3.0.5)
logstash-filter-elasticsearch (3.11.1)
logstash-filter-fingerprint (3.3.2)
logstash-filter-geoip (7.2.12)
logstash-filter-grok (4.4.1)
logstash-filter-http (1.4.0)
logstash-filter-json (3.2.0)
logstash-filter-kv (4.7.0)
logstash-filter-memcached (1.1.0)
logstash-filter-metrics (4.0.7)
logstash-filter-mutate (3.5.6)
logstash-filter-prune (3.0.4)
logstash-filter-ruby (3.1.8)
logstash-filter-sleep (3.0.7)
logstash-filter-split (3.1.8)
logstash-filter-syslog_pri (3.1.1)
logstash-filter-throttle (4.0.4)
logstash-filter-translate (3.3.0)
logstash-filter-truncate (1.0.5)
logstash-filter-urldecode (3.0.6)
logstash-filter-useragent (3.3.3)
logstash-filter-uuid (3.0.5)
logstash-filter-xml (4.1.3)
logstash-input-azure_event_hubs (1.4.3)
logstash-input-beats (6.3.0)
└── logstash-input-elastic_agent (alias)
logstash-input-couchdb_changes (3.1.6)
logstash-input-dead_letter_queue (1.1.11)
logstash-input-elasticsearch (4.12.3)
logstash-input-exec (3.4.0)
logstash-input-file (4.4.0)
logstash-input-ganglia (3.1.4)
logstash-input-gelf (3.3.1)
logstash-input-generator (3.1.0)
logstash-input-graphite (3.0.6)
logstash-input-heartbeat (3.1.1)
logstash-input-http (3.5.0)
logstash-input-http_poller (5.3.0)
logstash-input-imap (3.2.0)
logstash-input-jms (3.2.1)
logstash-input-pipe (3.1.0)
logstash-input-redis (3.7.0)
logstash-input-s3 (3.8.3)
logstash-input-snmp (1.3.1)
logstash-input-snmptrap (3.1.0)
logstash-input-sqs (3.3.0)
logstash-input-stdin (3.4.0)
logstash-input-syslog (3.6.0)
logstash-input-tcp (6.2.7)
logstash-input-twitter (4.1.0)
logstash-input-udp (3.5.0)
logstash-input-unix (3.1.1)
logstash-integration-elastic_enterprise_search (2.2.1)
 ├── logstash-output-elastic_app_search
 └──  logstash-output-elastic_workplace_search
logstash-integration-jdbc (5.2.4)
 ├── logstash-input-jdbc
 ├── logstash-filter-jdbc_streaming
 └── logstash-filter-jdbc_static
logstash-integration-kafka (10.10.0)
 ├── logstash-input-kafka
 └── logstash-output-kafka
logstash-integration-rabbitmq (7.3.0)
 ├── logstash-input-rabbitmq
 └── logstash-output-rabbitmq
logstash-output-cloudwatch (3.0.10)
logstash-output-csv (3.0.8)
logstash-output-elasticsearch (11.4.1)
logstash-output-email (4.1.1)
logstash-output-file (4.3.0)
logstash-output-graphite (3.1.6)
logstash-output-http (5.5.0)
logstash-output-lumberjack (3.1.9)
logstash-output-nagios (3.0.6)
logstash-output-null (3.0.5)
logstash-output-pipe (3.0.6)
logstash-output-redis (5.0.0)
logstash-output-s3 (4.3.5)
logstash-output-sns (4.0.8)
logstash-output-sqs (6.0.0)
logstash-output-stdout (3.1.4)
logstash-output-tcp (6.0.2)
logstash-output-udp (3.2.0)
logstash-output-webhdfs (3.0.6)
logstash-patterns-core (4.3.2)

JVM (e.g. java -version):

bundled version

OS version (uname -a if on a Unix-like system):

Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_X86_64 x86_64

Description of the problem including expected versus actual behavior:

Sending events from Elastic Agent to Logstash (with TLS enabled) results in Failed to install template error.

According to @jsvd, this is a bug. Template management should be disabled by default when writing to data streams.

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1. Follow the steps in this guide to send data from Fleet-managed Elastic Agents to Logstash: https://www.elastic.co/guide/en/fleet/8.2/secure-logstash-connections.html
2. Notice that the console reports the following error:

[2022-04-22T00:11:04,026][ERROR][logstash.outputs.elasticsearch][elastic-agent-pipeline] Failed 
to install template {:message=>"Got response code '403' contacting Elasticsearch at URL 
'https://58f88fcaeb294e459908dae6e61807a4.us-west2.gcp.elastic-
cloud.com:443/_index_template/ecs-logstash'", 
:exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, 
:backtrace=>["/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:84:in `perform_request'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:324:in `perform_request_to_url'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:311:in `block in perform_request'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:398:in `with_connection'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:310:in `perform_request'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `block in Pool'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client.rb:408:in `template_put'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client.rb:85:in `template_install'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/template_manager.rb:29:in `install'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:in `install_template'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch.rb:494:in `install_template'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch.rb:318:in `finish_register'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch.rb:283:in `block in register'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/plugin_mixins/elasticsearch/common.rb:149:in `block in 
after_successful_connection'"]}