[Request]: apm_writer user does not have enough permissions

[WilliamDEdwards]

What documentation page is affected

When setting up APM server, it needs to connect to Elasticsearch.

Following the principle of least privilege, https://www.elastic.co/guide/en/observability/8.14/apm-privileges-to-publish-events.html guides users in using a dedicated apm_writer user for this purpose, which is then specified in /etc/apm-server/apm-server.yml.

However, the specified permissions do not suffice.

On startup, APM server logs:

Jun 09 10:47:47 elasticsearch-test.cyberfusion.cloud apm-server[170123]: {"log.level":"error","@timestamp":"2024-06-09T10:47:47.193+0200","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.waitReady","file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: error querying cluster_uuid: status_code=403","service.name":"apm-server","ecs.version":"1.6.0"}

This can be fixed by adding the monitor cluster privilege. This is definitely a documentation issue.

However, even then, the following is logged also:

Jun 09 10:48:17 elasticsearch-test.cyberfusion.cloud apm-server[170123]: {"log.level":"error","@timestamp":"2024-06-09T10:48:17.201+0200","log.logger":"agentcfg","log.origin":{"function":"github.com/elastic/apm-server/internal/agentcfg.(*ElasticsearchFetcher).Run.func1","file.name":"agentcfg/elasticsearch.go","file.line":150},"message":"refresh cache error: refresh cache elasticsearch returned status 403","service.name":"apm-server","ecs.version":"1.6.0"}

This cannot be resolved by adding the all cluster and indice privilege. The error does not occur when using the built-in elastic user, though, so perhaps that user has a special state? I can't tell if this is a bug, or a documentation issue.

What change would you like to see?

See above.

Additional info

No response

[colleenmcginnis]

👋 @simitt could you connect me with someone on your team who could help me find the solution here? I've read through a handful of discuss threads and GitHub issues, but the specific privileges that are needed here are still not clear.

Also, in my search for answers I did come across elastic/apm-server#10057. Is this related? Maybe I can address that issue at the same time?

[endorama]

@WilliamDEdwards You should be able to get rid of that refresh cache error by adding permissions as described here: https://www.elastic.co/guide/en/observability/8.14/apm-privileges-agent-central-config.html
That request is part of central configuration management, which is not included in the permissions granted to the "apm_writer" role created following the documentation page you linked.

@colleenmcginnis I think the documentation require some clarification on the what permissions are needed on different use cases.
Under feature roles we mention: "Typically, you need to create the following separate roles:" and then there is a list of roles to be created. From that list is not clear if those roles are optional or required. My current understanding is that the Writer role is required and other are optional. But Central configuration management is enabled by default, so that role is "required" too, unless that functionality is disabled (and I think this needs to be clarified).

I'm reviewing that area of the documentation so I can provide more guidance.