Reduce dependencies for individual usage

[patrickheeney]

I found debops via ansible galaxy and I first attempted to use a bunch of the roles in my playbooks. However I later find out they all have a unique set of dependencies that really require the whole debops package. This doesn't work in my use case as I need to stay with stock ansible as close as possible. Debops is one of the best collections of ansible roles that I have come across and I would like to help contribute to making these roles better versus forking them.

It is my hope that you would consider going "native" on some of the functionality. This may involve some alternative approaches to functionality or collectively working with ansible to try and get some missing functionality in core. The end goal is to be able to use these roles independently and in insolation of the playbooks repository. These are some of the problems I encountered when using devops.

Pre / post role hooks

I definitely understand the use case for these, but I don't know that debops should cater to it. This seems like it could easily be accomplished with stock ansible with something like:

roles:
  - { role: custom-pre-mysql }
  - { role: debops.mysql, tags: [mysql] }
  - { role: custom-post-mysql }

If you want something to happen before or after a role, you just include your own role, before or after. If you want to override functionality, you can do it in the post role.

Alternatively, maybe you can add some feature detection:

- name: DebOps pre_tasks hook
  include: "{{ lookup('task_src', 'mysql/pre_main.yml') }}"
  when: debops_env == 'native'

That way it should work without it, and when ran in debops environment you can pass in the extra vars, put it in group vars, hosts, etc.

Template src

If I understand this correctly the goal is to be able to override templates to provide your own? I have seen this in some roles, but not others. So it seems this may be legacy or new functionality? that is not consistent across all roles yet?

I am not familiar with ansible internals but this seems like something better suited at the ansible level to be able to override any templates. For example {{ lookup("template", "etc/fail2ban/jail.local.d/default.local.j2") }} should have a way in the config to search custom paths for that file. if this doesn't exist, and hasn't been proposed, then we should voice our support or submit a pull request to introduce this functionality to core.

In the interim, you could always convert it to a var like fail2ban_template_jail_local: 'etc/fail2ban/jail.local.d/default.local.j2'. This would allow overrides. On templates with loops, you could specify a folder instead template: src={{ fail2ban_template_jail_local + item.name }}.

Also, maybe you can consider the feature detection mentioned above? One that loads your template_src when in the right environment, otherwise use a normal template that doesn't allow it to be overwritten.

Other role dependencies

I was also wondering about mysql for example, requiring secret, ferm, and tcpwrappers. It seems these dependencies offer some security or other features, but are not actually required by the role. Is there another way to have these dependencies installed and configured when they are part of a playbook that has them, but not a hard dependency on the individual role.

I am also not sure how this works if we try to override all the ferm_input_list, but if there are 5 roles with ferm as a dependency that add more inputs, do the group vars override all of that or also add to it?

Maybe the dependencies can be removed and replace with documentation suggesting the installation of ferm and its inputs when using mysql? Or maybe there is a way to make the dependencies optional when you want to use mysql but not ferm or tcpwrappers?

[drybjed]

Thanks for bringing up this issue. However, I'm afraid that the problems described by you are not solvable with currently available Ansible v1. Ansible v2 brings additional features like try/execept blocks, which might help with solving them, however v2 is still unstable. When it's released, I'll definitely will look into what new features are available and how to use them within DebOps.

Actually, the way into the project you described is very common; many people find out specific Ansible roles from DebOps, either through Ansible Galaxy or through Google, and start using them. After a while they realize that there's much larger project behind them. At this point, I suspect, people have a choice - either start using main DebOps playbooks and add their own roles/playbooks on top of it, or try to combine several roles together into what they need.

Different Ansible roles within DebOps are designed to work together to form a complete solution that manages a set of hosts. This approach has pros and cons on top of language limitations imposed by Ansible. On one hand, if you use the whole package, setting up services that work together is relatively easy. On the other hand, using some roles separately from others becomes very hard or impossible without modifying them.

Let's take your MySQL example. To setup the database server, debops.mysql role by itself should be enough - everything is presumed to work on localhost, so there's no need to open firewall ports and allow access to the database from other hosts... But DebOps is designed to use multiple hosts, so sooner or later (probably sooner) there comes a time where you want to setup more webservers behind a load balancer, and then database server will probably be moved to a separate host to be accessed remotely from all other hosts and spread the load. And now you need to configure access through the firewall and TCP Wrappers.

You may ask why - the reason is, DebOps is designed to be used in production environment - live servers exposed to the Internet. Because of that, by default, firewall automatically is set up to deny everything and only allow specific connections. TCP Wrappers also are set up to deny connections to different services, depending on if a service is deemed to be publicly available or not - the MySQL server, being a critical service, is configured to disallow access from hosts other than specific ones, which means that remote access is disabled by default.

So we have two services - iptables firewall and TCP Wrappers - which are critical to the security of the host (remember, production environment). But as you correctly noticed, it's very easy to make a mistake and overwrite existing configuration of each one of them. That's where separate roles, debops.ferm and debops.tcpwrappers, come in. Each one is designed to manage its service and allow other roles, using role dependency variables, to add their own rules which then are combined together so that both firewall and TCP Wrappers are correctly configured and their settings are not clobbered. It all happens transparently, and from your point of view it's as easy as specifying that this and this CIDR subnet should have access to MySQL server. Everything else is done by the above roles.

Other roles also use debops.ferm and debops.tcpwrappers. Actually, as you can probably see on the DebOps role dependency graph, debops.ferm is one of the most used role dependencies, because firewall is so essential in the multi-host, decentralized environment. Thanks to being a separate role, debops.ferm can consistently maintain the firewall no matter what set of services is enabled on a particular host. In fact, almost all DebOps roles could be enabled on just 1 host (there are some conflicts, like LXC vs OpenVZ, but that's due to kernel limitations), and it all would most likely correctly be installed and work together.

Could debops.ferm be replaced by something else, for example a shorewall, firewalld or firehol role? Most definitely yes. Some roles add ferm configuration files directly, but that could be changed to switch completely to dependency variables solution with some effort. Most of the roles use role dependency variables, which are treated as a kind of "public role API" - if you would design an alternative role which used the same syntax as debops.ferm for its dependency variables, I don't see a problem with replacing it, if needed. My choice of ferm as a base of iptables management was dictated by easy way to include different configuration files in it, relatively simple syntax and almost no magic automation in the ferm itself, which allowed me to design the iptables rules as I saw fit. It works pretty well so far, you should try it.

You may ask about the debops.secret role - it has a very specific purpose, essentially allows DebOps to create random passwords, credentials, and other confidental data as needed on the fly - you don't need to painstakingly create each and every random password yourself. I wish Ansible had better support for centralized, automated management of random credentials - debops.secret is the next best thing I could come up with.

I hope that you now understand why above roles are set up as role dependencies of debops.mysql and other such roles. Unfortunately, Ansible does not offer a way to make role dependencies optional - all I could do is add a way to disable debops.ferm and debops.tcpwrappers roles from managing their services, using ferm and tcpwrappers inventory variables. If you set these to False, roles will still be evaluated by Ansible, but they will not manage their respective services. This allows you to globally disable firewall and TCP Wrappers management by DebOps roles.

How about removing them completely instead (debops.ferm and debops.tcpwrappers, debops.secret needs to stay, otherwise Ansible refuses to run complaining about missing variable)? I could of course do that, basically remove them from every role and leave their execution only in the common.yml playbook. At this point, of course, you would need to manage the firewall and TCP Wrappers yourself (through the role configuration variables), adding rules to allow specific hosts access to specific ports, etc. Or, just disable firewall and TCP Wrappers completely and allow unrestricted access from all over the Internet (production environment, remember?). Are you sure you want to do that? It is doable... but will probably bring you more work you need to handle, instead of letting Ansible do it for you, as it is currently.

Besides, firewall and TCP Wrappers roles are not the only ones designed to be used as dependencies. debops.nginx, being the main webserver used in DebOps. is used as a role dependency by other roles to set up itself as a frontend or reverse proxy. debops.mysql was recently redesigned as two roles, debops.mariadb (client) and debops.mariadb_server, which allow for more flexibility to set up decentralized database infrastructure. The list goes on and on.

It all comes with a price. DebOps is designed to be used as a general purpose set of playbooks, to manage many different environments. Unfortunately, Ansible allows you to configure only so much. In several roles it was deemed useful to have more flexibility that the upstream project provides - for example some people asked for a way to use their own custom template files instead of the ones provided by DebOps, to avoid issues with quickly changing defaults. Others needed additional configuration steps performed at various points in the playbook, but wanted to keep using the existing roles due to their functionality.

These things are not easy to to do with current Ansible features. You either need to fork the role you're interested in and maintain your own changes, merging updates as they come along, or DebOps would need to be expanded to cover extra features. For example your pre / post role hook suggestion would require adding pre- and post- roles to each role that uses them, they would be most likely empty, but you would be able to override them using existing functionality in Ansible, by adding your own roles in roles_path variable. That would need to be done if you want to use the default DebOps playbook, and is irrelevant if you use your own playbooks. The conditional execution of include task still requires that the lookup plugin is present, otherwise Ansible refuses to run - this will probably get better with Ansible v2, when it is released. Includes will then be evaluated dynamically during playbook run, which hopefully will allow for neat tricks like dynamically inserted tasks.

As for template_src and file_src lookup plugins, adding them to Ansible could probably be useful, but I'm not sure if Ansible maintainers are comfortable with opening up the access to Ansible internals the way these lookup plugins do it. Right now they are contained in their own, separate, DebOps environment, giving access to Ansible directly might bring unintended consequences. Using separate variable for each template or file in their place will most likely quickly get out of control, and some combinations still won't be possible.

At the moment different DebOps roles are somewhat self-contained, with communication between being done through a carefully designed set of channels (Ansible local variables and role dependency variables). All of this is done to keep the roles from interfering with each other, as well as keep the overall host state consistent and idempotent. I plan to introduce a separate role which could be used to manage system and cluster wide configuration, and allow another way to centralize management of specific functionality, like LDAP support. Perhaps that will open up more options related to customization of how the project can be used.

When you look at DebOps as a whole instead of a set of separate Ansible roles, you can probably see how it all comes together to form a consistent way to manage your Debian infrastructure. Currently many parts are missing, some are outdated, need documentation. Still, people use it in production, myself included. I'm slowly adding new functionality and heading in a particular direction. I'm invinting you to come along - right now ride might get bumpy, but we will get there. Eventually.

[patrickheeney]

I definitely understand the value and goal of DebOps as a whole. My question to you is, do you see the value of being able to use roles outside of DebOps? Secondly, is the goal to provide a monolithic debian solution, or a some what decoupled solution made up of individual roles that combine into a larger solution? From abstracting ferm and tcpwrappers, it sounds to me like the latter, but some of the dependencies say otherwise.

I don't have a problem with the role dependencies as much as I do the "hidden" dependencies of hooks, template_src, file_src, etc. These dependencies you don't know about until you load them into your playbook and you get an error. Then after some googling you discover the playbooks repository which has these lookup functions and then you copy those pieces over. You run it again to find out those pieces require debops package. This was my on boarding experience. All of this is to provide the ability to supply your own template or hooks, which I don't need. In the current form, it is not possible to use them outside of the whole DebOps package, because of the hidden dependencies. So if you see the value in being able to use these independently, then I'd really like to help contribute and find a workaround so I can contribute to the community.

I understand waiting for ansible 2 and hopefully the block plugin will help. However, some of this functionality doesn't sound like it has been proposed to ansible, so it is unlikely they would implement anything similar. I think what was built here is really great, and I would definitely support this on the mailing list, github issues, twitter, etc to try and see some of it in core.

If the lookup plugins are still evaluated and will error even with conditionals, what about something like the snippet instead?

- name: DebOps pre_tasks hook
  include: "{{ debops_hook_mysql_pre_main }}"
  when: debops_hook_mysql_pre_main is defined

I haven't tested this, but it seems to like it could work and doesn't use the lookup plugin? What can be accomplished in the hooks, that you can't accomplish by putting your own role before or after the debops one?

What about using variables for templates instead of the lookup? Why would this solution not work? I have only briefly thought of this, so there may be many other ways to accomplish this functionality. Perhaps this is a solution we can discuss and propose it to ansible core? An easy way to override templates?

I just want to re-iterate that my use case is to just plug in mysql for example. I am definitely ok with it coming with ferm and tcpwrappers if it has some options to disable them. I can't do that now because of the hooks and templates, which I don't have a need for. I understand others do, so I am hoping there is a workaround to cater to both use cases.

[drybjed]

(Patrick went on to #debops IRC channel and we explained how the lookups issue works in regards to DebOps and Ansible).

I'm definitely interested in closer integration of various Ansible roles within DebOps together as a whole, instead ot keeping them able to be used on their own. This makes more sense in many cases and allows different roles to be interoperable within the project, but not necessarily with other, external roles. I don't plan actively to hinder role use outside of DebOps, but if it would be benefical to do so in a particular role, I would definitely consider it as an option.

As for additional Ansible plugins required by DebOps, even if they could be included in Ansible core, it won't happen right away, so roles still would need to use them as is for some time. We will see what Ansible v2 will bring to the table.

[patrickheeney]

I just wanted to share where I ended up after a week of exploring these concepts in case someone finds this.

The debops role dependencies in meta/main.yml are critical because of how ansible treats the variables. Essentially you can't access the role variables in your group_vars or custom role because there is a chance when using tags or other ansible features that they won't be loaded. The role dependency variables are the only sane way to manage this.

I have put forth a new proposal to change the hooks in #182 which should remove the most frequently used dependencies if accepted. This would make debops have a larger appeal as you can utilize the roles on their own.

I am closing this as we have covered most of the pain points I experienced.