Systemd unit options break GCE credentials for config options

[octete]

I would like to make sure at the beginning that this is not a contribution to the project, but merely a bug report.

Affected Puppet, Ruby, OS and module versions/distributions

• Puppet: 6.10.1
• Ruby: 2.0.0p648
• Distribution: Centos7
• Module version: 8.2.0

How to reproduce (e.g Puppet code you use)

PR #415 introduced some changes to the systemd unit that prevents prometheus from working correctly in GCE (Google Cloud Engine) and reading a service account for its service discovery.
As specified in https://prometheus.io/docs/prometheus/latest/configuration/configuration/#gce_sd_config, prometheus will try to read the service account from:

1. a JSON file specified by the GOOGLE_APPLICATION_CREDENTIALS environment variable
2. a JSON file in the well-known path $HOME/.config/gcloud/application_default_credentials.json
3. fetched from the GCE metadata server

The sytemd unit now has an option: ProtectHome= which, if set to true will

If true, the directories /home, /root, and /run/user are made inaccessible 
and empty for processes invoked by this unit

This causes prometheus to basically stop at section 2. in the list above and it never falls back to section 3 (fetch from the GCE metadata server). I presume it's because the OS returns a different kind of error back to prometheus than 'file not found', etc. You can see the code that's used here: https://github.com/golang/oauth2/blob/9f3314589c9a9136388751d9adae6b0ed400978a/google/default.go#L76

Furthermore, there's no option to allow for an environment variable to point to a different service account file. i.e.

Environment=GOOGLE_APPLICATION_CREDENTIALS=/etc/gce_credentials/gce.json

which would be allowed, as it is not in /home/

What are you seeing

Prometheus just doesn't pick up the config from the default location $HOME/.config/gcloud/application_default_credentials.json nor it hits the default metadata server (point 3). Thus, service discovery does not work in GCE.

What behaviour did you expect instead

That access to /home/prometheus/.config/gcloud/application_default_credentials.json is allowed, by turning the ProtectHome setting off, or changing its behaviour. I can't also see why I shouldn't be able to configure a prometheus server to write its data to /home/prometheus. Also, that if we don't have access to /home/ the process should be able to continue and fetch credentials from the metadata server.

Furthermore, it would be nice if we could have a templated option in the systemd file that would allow setting the variable GOOGLE_APPLICATION_CREDENTIALS so that it can point somewhere else than the default value in $HOME/.config/gcloud/application_default_credentials.json.

Output log

Jan 24 12:20:37 prometheus-backup prometheus[10553]: level=error ts=2020-01-24T12:20:37.477Z caller=manager.go:342 component="discovery manager scrape" msg="Cannot create service discovery" err="error setting up communication with GCE service: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: open /home/prometheus/.config/gcloud/application_default_credentials.json: permission denied" type=*gce.SDConfig
Jan 24 12:20:37 prometheus-backup prometheus[10553]: level=error ts=2020-01-24T12:20:37.477Z caller=manager.go:342 component="discovery manager scrape" msg="Cannot create service discovery" err="error setting up communication with GCE service: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: open /home/prometheus/.config/gcloud/application_default_credentials.json: permission denied" type=*gce.SDConfig
Jan 24 12:20:37 prometheus-backup prometheus[10553]: level=error ts=2020-01-24T12:20:37.477Z caller=manager.go:342 component="discovery manager scrape" msg="Cannot create service discovery" err="error setting up communication with GCE service: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: open /home/prometheus/.config/gcloud/application_default_credentials.json: permission denied" type=*gce.SDConfig

Any additional information you'd like to impart