diff --git a/docs/en/ingest-management/fleet/fleet-settings.asciidoc b/docs/en/ingest-management/fleet/fleet-settings.asciidoc index 24f0913fb6..cbc4291619 100644 --- a/docs/en/ingest-management/fleet/fleet-settings.asciidoc +++ b/docs/en/ingest-management/fleet/fleet-settings.asciidoc @@ -5,23 +5,17 @@ NOTE: The settings described here are configurable through the {fleet} UI. Refer {kibana-ref}/fleet-settings-kb.html[{fleet} settings in {kib}] for a list of settings that you can configure in the `kibana.yml` configuration file. -Configure {fleet} settings to apply global settings to all {agent}s enrolled in -{fleet}: +On the *Settings* tab in *Fleet*, you can configure global settings available +to all {agent}s enrolled in {fleet}. This includes {fleet-server} hosts and +output settings. -. In {kib}, open the main menu, then click *Management > {fleet} > Settings*. +[discrete] +[[fleet-server-hosts-setting]] +== {fleet-server} host settings -. Under *{fleet-server} hosts*, click *Edit hosts* and specify one or more -host URLs: -+ --- -[cols="2*> +* <> -// ============================================================================= +[discrete] +[[es-output-settings]] +=== {es} output settings + +Specify these settings to send data over a secure connection to {es}. + +[cols="2*>. + +[cols="2*> -|Configure global settings for all {agent}s managed by {fleet}, including -{fleet-server} hosts and {es} output settings. +|Configure global settings available to all {agent}s managed by {fleet}, +including {fleet-server} hosts and output settings. |<> |Enroll, unenroll, upgrade, and view {agent} status and logs. diff --git a/docs/en/ingest-management/images/add-logstash-output.png b/docs/en/ingest-management/images/add-logstash-output.png new file mode 100644 index 0000000000..39eaf6e533 Binary files /dev/null and b/docs/en/ingest-management/images/add-logstash-output.png differ diff --git a/docs/en/ingest-management/images/agent-output-settings.png b/docs/en/ingest-management/images/agent-output-settings.png new file mode 100644 index 0000000000..e9b2347106 Binary files /dev/null and b/docs/en/ingest-management/images/agent-output-settings.png differ diff --git a/docs/en/ingest-management/images/ca-certs.png b/docs/en/ingest-management/images/ca-certs.png new file mode 100644 index 0000000000..a3c63403a2 Binary files /dev/null and b/docs/en/ingest-management/images/ca-certs.png differ diff --git a/docs/en/ingest-management/images/client-certs.png b/docs/en/ingest-management/images/client-certs.png new file mode 100644 index 0000000000..c71b91d0bb Binary files /dev/null and b/docs/en/ingest-management/images/client-certs.png differ diff --git a/docs/en/ingest-management/images/logstash-certs.png b/docs/en/ingest-management/images/logstash-certs.png new file mode 100644 index 0000000000..e3568606f6 Binary files /dev/null and b/docs/en/ingest-management/images/logstash-certs.png differ diff --git a/docs/en/ingest-management/images/logstash-output-certs.png b/docs/en/ingest-management/images/logstash-output-certs.png new file mode 100644 index 0000000000..7bfc7f2bb2 Binary files /dev/null and b/docs/en/ingest-management/images/logstash-output-certs.png differ diff --git a/docs/en/ingest-management/security/logstash-certificates.asciidoc b/docs/en/ingest-management/security/logstash-certificates.asciidoc index 4065851a93..97ce4f813e 100644 --- a/docs/en/ingest-management/security/logstash-certificates.asciidoc +++ b/docs/en/ingest-management/security/logstash-certificates.asciidoc @@ -1,7 +1,280 @@ [[secure-logstash-connections]] -= Configure SSL/TLS for {ls} += Configure SSL/TLS for the {ls} output -coming[8.2.0] +beta[] To send data from {agent} to {ls} securely, you need to configure Transport -Layer Security (TLS). +Layer Security (TLS). Using TLS ensures that your {agent}s send encrypted data +to trusted {ls} servers, and that your {ls} servers receive data from trusted +{agent} clients. + +[discrete] +[[secure-logstash-prereqs]] +== Prerequisites + +* Make sure your https://www.elastic.co/subscriptions[subscription level] +supports output to {ls}. + +* On Windows, add port 8220 for {fleet-server} and 5044 for Logstash to the +inbound port rules in Windows Advanced Firewall. + +* If you are connecting to a self-managed {es} cluster, you need the CA +certificate that was used to sign the certificates for the HTTP layer of {es} +cluster. For more information, refer to the +{ref}/configuring-stack-security.html[{es} security docs]. + +[discrete] +[[generate-logstash-certs]] +== Generate custom certificates and private keys + +You can use whatever process you typically use to generate PEM-formatted +certificates. The examples shown here use the `certutil` tool provided by {es}. + +TIP: The `certutil` tool is not available on {ecloud}, but you can still use it +to generate certificates for {agent} to {ls} connections. Just +https://www.elastic.co/downloads/elasticsearch[download an {es} package], +extract it to a local directory, and run the `elasticsearch-certutil` command. +There's no need to start {es}! + +. Generate a certificate authority (CA). Skip this step if you want to use an +existing CA. ++ +-- +[source,shell] +---- +./bin/elasticsearch-certutil ca --pem +---- + +This command creates a zip file that contains the CA certificate and key you'll +use to sign the certificates. Extract the zip file: + +image::images/ca-certs.png[Screen capture of a folder called ca that contains two files: ca.crt and ca.key] +-- + +. Generate a client SSL certificate signed by your CA. For example: ++ +-- +[source,shell] +---- +./bin/elasticsearch-certutil cert \ + --name client \ + --ca-cert /path/to/ca/ca.crt \ + --ca-key /path/to/ca/ca.key \ + --dns your.host.name.here \ + --ip 192.0.2.1 \ + --pem +---- + +Where `dns` and `ip` specify the valid host names and IP addresses where the generated certificates are valid. + +Extract the zip file: + +image::images/client-certs.png[Screen capture of a folder called client that contains two files: client.crt and client.key] +-- + +. Generate a {ls} SSL certificate signed by your CA. For example: ++ +-- +[source,shell] +---- +./bin/elasticsearch-certutil cert \ + --name logstash \ + --ca-cert /path/to/ca/ca.crt \ + --ca-key /path/to/ca/ca.key \ + --dns your.host.name.here \ + --ip 192.0.2.1 \ + --pem +---- + + +Extract the zip file: + +image::images/logstash-certs.png[Screen capture of a folder called logstash that contains two files: logstash.crt and logstash.key] +-- + +. Convert the {ls} key to pkcs8. For example, on Linux run: ++ +[source,shell] +---- +openssl pkcs8 -inform PEM -in logstash.key -topk8 -nocrypt -outform PEM -out logstash.pkcs8.key +---- + +Store these files in a secure location. + +[discrete] +[[configure-ls-ssl]] +== Configure the {ls} pipeline + +TIP: If you've already created the {ls} `elastic-agent-pipeline.conf` pipeline +and added it to `pipelines.yml`, skip to the example configurations and modify +your pipeline configuration as needed. + +In your Logstash configuration directory, open the `pipelines.yml` file and +add the following configuration. Replace the path to your file. + +[source,yaml] +---- +- pipeline.id: elastic-agent-pipeline + path.config: "/etc/path/to/elastic-agent-pipeline.conf" +---- + +In the `elastic-agent-pipeline.conf` file, add the pipeline configuration. Note +that the configuration needed for {ess} on {ecloud} is different from +self-managed {es} clusters. If you copied the configuration shown in {fleet}, +adjust it as needed. + +{ess} example: + +[source,text] +---- +input { + elastic_agent { + port => 5044 + ssl => true + ssl_certificate_authorities => ["/path/to/ca.crt"] + ssl_certificate => "/path/to/logstash.crt" + ssl_key => "/path/to/logstash.pkcs8.key" + ssl_verify_mode => "force_peer" + } +} + +output { + elasticsearch { + cloud_id => "xxxx:xxxxxxxxxxxxxxxxxxxxxxxxxxxxx=" <1> + api_key => "xxxx:xxxx" <2> + data_stream => true + ssl => true <3> + } +} +---- +<1> Use the `cloud_id` shown on your deployment page in {ecloud}. +<2> In {fleet}, you can generate this API key when you add a {ls} output. +<3> {ess} uses standard publicly trusted certificates, so there's no need +specify other SSL settings here. + +Self-managed {es} cluster example: + +[source,text] +---- +input { + elastic_agent { + port => 5044 + ssl => true + ssl_certificate_authorities => ["/path/to/ca.crt"] + ssl_certificate => "/path/to/logstash.crt" + ssl_key => "/path/to/logstash.pkcs8.key" + ssl_verify_mode => "force_peer" + } +} + +output { + elasticsearch { + hosts => "https://xxxx:9200" + api_key => "xxxx:xxxx" + data_stream => true + ssl => true + cacert => "/path/to/http_ca.crt" <1> + } +} +---- +<1> Use the certificate that was generated for {es}. + +To learn more about the {ls} configuration, refer to: + +* {logstash-ref}/plugins-inputs-elastic_agent.html[{agent} input plugin] +* {logstash-ref}/plugins-outputs-elasticsearch.html[{es} output plugin] +* {logstash-ref}/ls-security.html[Secure your connection to {es}] + +When you're done configuring the pipeline, restart {ls}: + +[source,shell] +---- +bin/logstash +---- + +[discrete] +[[add-ls-output]] +== Add a {ls} output to {fleet} + +This section describes how to add a Logstash output and configure SSL settings +in {fleet}. If you're running {agent} standalone, refer to the +<> configuration docs. + +. In {kib}, go to *Fleet > Settings*. + +. Under *Outputs*, click *Add output*. If you've been following the {ls} steps +in {fleet}, you might already be on this page. + +. Specify a name for the output. + +. For *Type*, select *Logstash*. + +. Under *Logstash hosts*, specify the host and port your agents will use to +connect to {ls}. Use the format `host:port`. + +. In the *Server SSL certificate authorities* field, paste in the entire +contents of the `ca.crt` file you <>. + +. In the *Client SSL certificate* field, paste in the entire contents of the +`client.crt` file you generated earlier. + +. In the *Client SSL certificate key* field, paste in the entire contents of the +`client.key` file you generated earlier. + +[role="screenshot"] +image::images/add-logstash-output.png[Screen capture of a folder called logstash that contains two files: logstash.crt and logstash.key] + +When you're done, save and apply the settings. + +[discrete] +[[use-ls-output]] +== Select the {ls} output in an agent policy + +{ls} is now listening for events from {agent}, but events are not streaming into +{es} yet. You need to select the {ls} output in an agent policy. You can edit +an existing policy or create a new one: + +. In {kib}, go to *Fleet > Agent policies* and either create a new agent policy +or click an existing policy to edit it: ++ +* To change the output settings in a new policy, click *Create agent policy* +and expand *Advanced options*. +* To change the output settings in an existing policy, click the policy to edit +it, then click *Settings*. + +. Set *Output for integrations* and (optionally) *Output for agent monitoring* +to use the {ls} output you created earlier. You might need to scroll down to see +these options ++ +[role="screenshot"] +image::images/agent-output-settings.png[Screen capture showing the Logstash output policy selected in an agent policy] + +. Save your changes. + +Any {agent}s enrolled in the agent policy will begin sending data to {es} via +{ls}. If you don't have any installed {agent}s enrolled in the agent policy, do +that now. + +There might be a slight delay while the {agent}s update to the new policy +and connect to {ls} over a secure connection. + +[discrete] +[[test-ls-connection]] +== Test the connection + +To make sure {ls} is sending data, run the following command from the host where +{ls} is running: + +[source,shell] +---- +curl -XGET localhost:9600/_node/stats/events +---- + +The request should return stats on the number of events in and out. If these +values are 0, check the {agent} logs for problems. + +When data is streaming to {es}, go to *Observability* and click +*Metrics* to view metrics about your system. + +//TODO: WE should add some troubleshooting advice like what to do if the +//connection is refused. But maybe after beta drops.