cf-runagent fails with "EXEC denied due to ACL for file: /usr/bin/dash"

[craigcomstock]

I want my git server to have a post-receive hook on my masterfiles repo which causes the policy to be updated and run on my policy server.

I thought I could do this with cf-runagent something like this:

cf-runagent --hail 192.168.1.197 -f update.cf --inform  # update policy from git since I have VCS configured
cf-runagent --hail 192.168.1.197 --inform  # run the updated policy

I have an ACL in def.json which seems like it should allow connections between agents and the policy server.

"acl": ["192.168.0.0/16"],

I try cf-runagent like this from the git server (192.168.1.174) to the policy server (192.168.1.197):

cf-runagent --hail 192.168.1.197 -f update.cf --inform

On the git server I get an error:

    info: ........................................................................
    info: Hailing 192.168.1.197 : 5308
    info: ........................................................................
192.168.1.197> !!  Unspecified server refusal (see verbose server output)

and on the policy server I stopped cf-serverd with systemctl

systemctl stop cf-serverd.service

and then ran it to see the details:

cf-serverd --verbose --no-fork

And the error seems strangely to do with ACL on /usr/bin/dash???

 verbose: 192.168.1.174>   TLS version negotiated:  TLSv1.2; Cipher: AES256-GCM-SHA384,TLSv1.2   
 verbose: 192.168.1.174>   TLS session established, checking trust...                            
 verbose: 192.168.1.174>   Setting IDENTITY: USERNAME=root                                              
 verbose: 192.168.1.174>   Received public key compares equal to the one we have stored                                                                                                                             
 verbose: 192.168.1.174>   SHA=73daa7e3dccd8748a6cad2f571b5d2677eacf5e351aced8abf6f2ca2aa3527b8: Client is TRUSTED, public key MATCHES stored one.
 verbose: 192.168.1.174>        Received:    EXEC
    info: 192.168.1.174>   EXEC denied due to ACL for file: /usr/bin/dash
 verbose: 192.168.1.174>   REFUSAL to user='root' of request: EXEC
    info: 192.168.1.174>   Closing connection, terminating thread

Why is it dealing with /usr/bin/dash?

Am I running cf-runagent properly to cause a policy update?

[amousset]

Let's have a closer look here (disclaimer: I'm not very familiar with masterfiles). The command that runs when triggered by cf-runagent is cfruncommand defined in body server control:

(in controls/cf-serverd.cf)

        cfruncommand => "$(def.cf_runagent_shell) -c \'
                           $(sys.cf_agent) -I -D cf_runagent_initiated -f $(sys.update_policy_path)  ;
                           $(sys.cf_agent) -I -D cf_runagent_initiated";

so def.cf_runagent_shell is where /usr/bin/dash comes from.

We also see that the command handles both policy update and agent execution, so no need for two cf-runagent commands. Furthermore, the -f flag in cf-runagent does not select a remote file but the local policy files loaded by cf-runagent.

Let's find the source of our permission problem. In order to be allowed to run the cfruncommand, it must be allowed by an ACL specifying who is authorized to run it:

(still in controls/cf-serverd.cf)

      "$(def.cf_runagent_shell)"
      handle => "server_access_grant_access_shell_cmd",
      comment => "Grant access to shell for cfruncommand",
      admit => { @(def.policy_servers) };

We see that the property controlling who is able to trigger a run is policy_server and not acl (used for client policy downloads, which is different).

So the cf-runagent target system should have the source server in def.policy_server, and it should work!

[nickanderson]

Yep

[craigcomstock]

Thanks @amousset I agree with your analysis. I do already have the hostname in def.policy_server but will try adding its IP address as well and see if that makes a difference. I recently changed the hostname as well so maybe there is a problem with DNS in this case. :)

[nickanderson]

What if you add the IP of the git server to this def.cf_runagent_shell access promise? Or did you do that and I missed it?

[olehermanse]

Nick is correct here, I tried to explain in some of the other threads below.

[craigcomstock]

Adding the ip address didn't seem to help. I did notice that the message mentions the user

EXEC denied due to ACL for file: /usr/bin/dash

so I added

"control_server_allowusers_policy_server": ["root"],

as a var in def.json.

in --verbose log of cf-serverd though I see /usr/bin/dash should admit the proper ip and EXEC should allow the proper user.

 verbose: Users from whom we accept cf-runagent connections (allowusers):
 verbose:       USER: root
 verbose: Access control lists:
 verbose:       Path: /usr/bin/dash
 verbose:               admit_ips: 192.168.1.197

as well as the same entry for classic protocol

 verbose: Access control lists for the classic network protocol:
--- snip ---
verbose:       Path: /usr/bin/dash
 verbose:               admit: 192.168.1.197

--debug for cf-serverd gives a bit more info:

debug: 192.168.1.174>   acl_CheckPath: '/usr/bin/dash' found in ACL entry '/usr/bin/dash', admit=false

It seems that the function access_CheckResource() is returning false even though the ipaddr is in acl.admit.ips:

I looked at the C code a bit to understand better what these messages mean versus what the logs seem to say. Then ran with gdb.

# gdb cf-serverd
(gdb) b acl_CheckPath
(gdb) b access_CheckResource
(gdb) run --no-fork
--- snip ---
Thread 3 "cf-serverd" hit Breakpoint 1, access_CheckResource (acl=0x55555575f1b8, found=0x0, ipaddr=0x55555573d088 "192.168.1.174", hostname=0x55555573d0c8 "",
    key=0x7fffe8019b50 "SHA=73daa7e3dccd8748a6cad2f571b5d2677eacf5e351aced8abf6f2ca2aa3527b8", username=0x0) at server_access.c:75
(gdb) p StrList_At(acl->admit.ips, 0)                                
$5 = 0x5555555ed408 "192.168.1.197"   

So it would seem that the ip should be admitted. Not sure why just it isn't just yet but will look at this more another time.

[olehermanse]

Thread 3 "cf-serverd" hit Breakpoint 1, access_CheckResource (acl=0x55555575f1b8, found=0x0, ipaddr=0x55555573d088 "192.168.1.174", hostname=0x55555573d0c8 "",
    key=0x7fffe8019b50 "SHA=73daa7e3dccd8748a6cad2f571b5d2677eacf5e351aced8abf6f2ca2aa3527b8", username=0x0) at server_access.c:75
(gdb) p StrList_At(acl->admit.ips, 0)                                
$5 = 0x5555555ed408 "192.168.1.197"   

Note here that the IPs differ "192.168.1.197" != "192.168.1.174". The normal way of using cf-runagent is by allowing non-policy servers to accept EXEC requests from the policy server. You are doing the opposite.

[craigcomstock]

Right. I was reading that wrong. 197 is the hub and 174 is the other host.

[olehermanse]

@craigcomstock Although it might not answer your question directly, this blog post has some details about how cf-runagent and cf-serverd works: https://cfengine.com/company/blog-detail/unpriviledged_cf-runagent/

[craigcomstock]

Right. Sadly my error is about the ACL for a specific path, /usr/bin/dash, which so far I can't figure out why it is not allowed since there is an entry in the cf-serverd log about the right entry being present for the right path and IP. I'll have to step debug through the code I think to understand where either the code is bugged or I am doing something wrong with my configuration. Maybe in the process I will find the spot that needs more debug logging that will help the next person. :)

[olehermanse]

I try cf-runagent like this from the git server (192.168.1.174) to the policy server (192.168.1.197):

@craigcomstock I think this is the problem.

The normal setup would be to run cf-runagent on the policy server, to a remote CFEngine client. This works by default, if you enable it.

(As pointed out by @nickanderson) To enable the opposite, you will have to make some additional changes, most notably to admit the git server IP:

      "$(def.cf_runagent_shell)"
      handle => "server_access_grant_access_shell_cmd",
      comment => "Grant access to shell for cfruncommand",
      admit => { @(def.policy_servers) }; # ADD GIT SERVER IP HERE, IT IS NOT THE POLICY SERVER

Additionally, note that MPF has different variables for what users to admit depending on whether it's a policy server or not:

    # The MPF provides different variables for configuring which users are
    # allowed to initiate execution via cf-runagent.
    !policy_server::
      allowusers => { @(def.control_server_allowusers_non_policy_server) };

    policy_server::
        allowusers => { @(def.control_server_allowusers_policy_server) };

This is explained in my blog post linked above, under "Allow the user to make EXEC requests":

https://cfengine.com/company/blog-detail/unpriviledged_cf-runagent/

[craigcomstock]

I tried adding the ip addr to the admit attribute and that works.

I had some policy laying around that payed attention to policy_servers in def.json but I had ripped the policy but not the def.json bits. I was doing some fail-over testing with a backup policy_server.

So I added my "other" host to policy_servers in def.json thinking it would be included in that slist, but nothing in masterfiles payed attention to policy_servers in def.json. :(

So I added a patch and things work! :) So problem solved.

diff --git a/controls/def.cf b/controls/def.cf
index 3033ff73..286ba0a1 100644
--- a/controls/def.cf
+++ b/controls/def.cf
@@ -363,7 +363,8 @@ bundle common def
       "policy_servers" slist => { "$(sys.policy_hub)", "@(standby_servers)" };
 
     !enable_cfengine_enterprise_hub_ha::
-      "policy_servers" slist => {"$(sys.policy_hub)"};
+      "policy_servers" slist => {"$(sys.policy_hub)"},
+        unless => isvariable(policy_servers);
 
     enterprise_edition.policy_server::
       "control_hub_exclude_hosts"

I'll stick with this since I do want this fail-over scenario to work. :)

Thanks for all the help @olehermanse @nickanderson @amousset . Sorry I had some wonky details in the mix. :)