Add docs about configuring SSL for the Logstash output

[dedemorton]

Closes #1691

Preview links:

Configure SSL/TLS for the Logstash output
Fleet settings

Reviewers: Please respond to the questions I've added that begin with //REVIEWERS in the source files.

I want users to be able to find these docs after following the link we've put in the UI. So for now, I'm putting all the content about generating the certs and configuring the settings in the UI + logstash pipeline in one guide to make it easier for users to see which certs/keys get specified and where. This sort of buries the lede, though. I think it's good enough for beta, but we might want to revisit this alter.

Also, I ran into a couple of errors. Let me know if they are expected:

• [ERROR][logstash.outputs.elasticsearch][elastic-agent-pipeline] Failed to install template. Issue open here: Logstash throws a "Failed to install template" on events sent from Elastic Agent logstash-plugins/logstash-output-elasticsearch#1071

  Does the API key that Fleet generates include the privileges required on Logstash to install templates? Not sure this causes any problems for our use case, but the error might bother users:

[2022-04-22T00:11:04,026][ERROR][logstash.outputs.elasticsearch][elastic-agent-pipeline] Failed 
to install template {:message=>"Got response code '403' contacting Elasticsearch at URL 
'https://58f88fcaeb294e459908dae6e61807a4.us-west2.gcp.elastic-
cloud.com:443/_index_template/ecs-logstash'", 
:exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, 
:backtrace=>["/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:84:in `perform_request'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:324:in `perform_request_to_url'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:311:in `block in perform_request'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:398:in `with_connection'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:310:in `perform_request'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in `block in Pool'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client.rb:408:in `template_put'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/http_client.rb:85:in `template_install'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/template_manager.rb:29:in `install'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:in `install_template'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch.rb:494:in `install_template'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch.rb:318:in `finish_register'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/outputs/elasticsearch.rb:283:in `block in register'", 
"/Users/dedemorton/BuildTesting/8.2.0_3b2b9b86/logstash-
8.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-
java/lib/logstash/plugin_mixins/elasticsearch/common.rb:149:in `block in 
after_successful_connection'"]}

• If I try to validate the Logstash server's cert with curl -v --cacert ca.crt https://localhost:5044/ I get the following output, which might be expected because ssl_verify_mode is set to force_peer. Maybe I need to do this differently? Anyhow, it looks like the exception isn't handled in Logstash. Is this to be expected, or have I done something wrong with my TLS setup?

LS dev confirmed that the TLS setup in the docs is OK. This is just a testing thing.

*   Trying ::1:5044...
* Connected to localhost (::1) port 5044 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: ca.crt
*  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS alert, bad certificate (554):
* error:1401E412:SSL routines:CONNECT_CR_FINISHED:sslv3 alert bad certificate
* Closing connection 0
curl: (35) error:1401E412:SSL routines:CONNECT_CR_FINISHED:sslv3 alert bad certificate

Message from Logstash:

[2022-04-22T01:42:48,523][INFO ][org.logstash.beats.BeatsHandler][elastic-agent-pipeline][aa0aff69ed4a1d8b3f0e1aa7164ce2c3da53304e3db1127ef78b7a590300d019] [local: 0.0.0.0:5044, remote: 0:0:0:0:0:0:0:1:55348] Handling exception: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain (caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain)
[2022-04-22T01:42:48,525][WARN ][io.netty.channel.DefaultChannelPipeline][elastic-agent-pipeline][aa0aff69ed4a1d8b3f0e1aa7164ce2c3da53304e3db1127ef78b7a590300d019] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
	at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
	at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:339) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:295) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:286) ~[?:?]
	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:390) ~[?:?]
	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:375) ~[?:?]
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) ~[?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) ~[?:?]
	at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1512) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1526) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1390) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1280) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	... 17 more

[apmmachine]

A documentation preview will be available soon:

• 📚 HTML diff
• 📘 Fleet and Elastic Agent Guide
• 📙 Observability Guide

[kpollich]

I took a look at the docs previews and things are looking great here! Thanks for your hard work on this ❤️

I answered what I could from the reviewer questions, but I'm similarly not 100% confident in my knowledge around SSL certificate specifics. Hopefully what I've provided here is helpful.

Thanks again!

[dedemorton]

Merging now, but we can iterate over this after beta if more changes are required.