Allow options via environment variables #118

[stevladimir]

Sketched up the solution.

It implies changes to core parser type, but tried to keep changes to API as minimal as possible. Atm it lacks necessary changes to parser docs generation and maybe some other things.

There are also few open questions:

1. Environment vs cmd parsers precedence.
   Widely accepted convention is cmd options taking precedence over environment variables, however, in issue discussion this question was raised. So need to decide whether it should be customizable.
2. IMO it would be nice to replace metavar for $FOO when using environ "FOO". In that case we also minimize changes to help text as it becomes self-documenting.
3. Flags and environ.
   Smth like [ -n "${FOO:+x}" ] is often used and works as a boolean flag "is variable is set and not null". But this is not the only way, so we should make this customizable.
   One of the solutions is to change environ type to ReadM a -> String -> Mod f a. This will also make possible to use different readers for environment and cmd parser, but I personally don't like this.
   We can also have separate combinators envOpt :: String -> Mod OptionFields a and envFlag :: String -> ReadM a -> Mod FlagFields a. Or just add envReader :: ReadM a -> Mod FlagFields a and have some default reader for flags.

[HuwCampbell]

Thanks for contributing and interesting work.

I won't comment now about the PR code – I will have a look at the implementation in more depth in a coming weekend.

Even though this is not a lot of code, it's a big change which broadens the scope from command line parsing to general configuration.

There are things to think about, such as:

• How should one express the help text?
• Should EnvP be a top level parser? If so, why is the API a modifier?
• What should environment variables be able to override or be overriden by?
• What are all the ways this can interact with the various option types (command, arguments, flags)? and
• How big is the marginal gain over the method expressed in Allow options via environment variables #118?

Lots to consider.

Kind regards,
Huw

[stevladimir]

Thank you for your response and awesome library btw.

I agree that there are "Lots to consider". That's why I wanted to get your opinion on the approach before diving into details.

How should one express the help text?

There can be few options:

• Append environment variables info to option description. Something like --foo My fancy option. Environment variables: FOO.
• Separate ENVIRONMENT section.
• Using $FOO as metavar. Though this cannot be an ubiquitous solution as people might want to override metavar, it still looks as good default.
  But I don't think there is single proper way, so guess we need to be flexible.

Should EnvP be a top level parser? If so, why is the API a modifier?

I should note, that I'm still not so much familiar with internals of the library, so it is quite possible that there can be more elegant way.

Why modifier? In most cases, environment variable is "linked" to some cmd option, so it looks reasonable to make it option modifier rather then separate top-level parser.

Regarding EnvP...
As far as I got there are 2 separate steps: parser construction and running of the parser. We don't have access to environment at the first step. Well, we can as it is demonstrated here (and I have snippet very similar in spirit as a workaround for this problem). But it requires manual passing of environment around, which is the thing one could live with. More important problem though is docs generation.

Adding environ modifier implies less changes from end-user point of view. You simply tag desired options with environ and replace execParser with execParserEnv. But as the end-result of the parser construction is Parser, I see no other way then extending this type to be able to "consume" variables when running parser. The biggest drawback is that it is a breaking change for those who use Parser directly.

What should environment variables be able to override or be overriden by?

I believe CMD > ENV is widely accepted convention (at least I can't remember cases where it is not so; would be thankful if anyone gave an example). In either case we can make this customizable global or per option. Though the latter would be very confusing to me.

What are all the ways this can interact with the various option types (command, arguments, flags)?

I don't see how they can interact. I mentioned in PR comment we likely need similar modifier for flag. There are some open questions, but I don't think they are really blocking. I would avoid extending this feature to argument though as there will be very confusing interaction with arguments count and order.

How big is the marginal gain over the method expressed in Allow options via environment variables #118?

Good question. I personally felt lack of this feature many times. Many tools (systemd, cron, docker and modern deployment infrastructure at all etc) are built around environment as source of customization values. On the other hand for local development and use cmd options are often more convenient. So you end up with boilerplate wrapper scripts like:

[ -z "$FOO" ] && die "No $FOO"
[ -z "$BAR" ] && die "No $BAR"

go ${FOO:+ --foo "$FOO"} ${BAR:+ --bar "$BAR"}

Beside for being tedious and error-prone you cannot list possible environment variables in help text (and documentation you could generate from parser description) without manually adding them to help text.

Thus I would say there are 2 key benefits:

1. You can trivially add environment support with single line of code per option.
2. This is nicely reflected in help texts, error messages (say, for missing variable) and so on.

How big is this gain? For me (and other people voted for initial request) it is worth changes, but it is up to you to decide keeping in mind all those things important for such popular library like compatibility, maintainability, lightweightness and so on. And I know you are very reasonably keen on this.

Another question that has been raised and should be kept in mind: is this in scope of this library at all? I agree with this comment. As environment and cmd options often closely interact and complement each other it is reasonable to support both. Ofc if to view optparse-applicative as getopt in Haskell environment is definitely out of scope. But I would say it is classical XY-problem. The real problem is not option parsing as such, but supplying of user values to programs. Cmd options is one way, environment another. If we can easily accommodate both, why not?

[thejohncrafter]

Hello,

I'd be interested in completing the implementation, how can I help?

[stevladimir]

@thejohncrafter thank you for raising this!

I think there are still some open questions and the first one is: do we want these changes at all?
These changes can be breaking for someone (though I assume for very low percentage of users) touching one of the core optparse-applicative types, so there are should be clear benefits to take such a risk.

The question is quite debatable, so feel free to share your thoughts on this.

I would like to come to the agreement on this point before addressing other questions mentioned in the discussion.

@HuwCampbell what do you think?

[thejohncrafter]

@stevladimir thank you for starting this PR!

I think the ability to read environment variables is relevant for this library. To be honest, I need this for a project of mine, and it looks like I can't do it properly without modifying optparse-applicative. And I thought that since that means that I should fork the project, I might as well PR...

The whole other question is that of the wording. I haven't been in the details of the library for now, so I can't tell what is the best way to implement this.
I'll take a proper look and get back here with more precise proposals :)

[stevladimir]

I think the ability to read environment variables is relevant for this library.

I think so too, but it is important that maintainers of this library also support this point of view.

To be honest, I need this for a project of mine, and it looks like I can't do it properly without modifying optparse-applicative. And I thought that since that means that I should fork the project, I might as well PR...

Basically, you can use this PR if you only need an ability to pass options via environment. It is missing help texts and so on, cause there is still some uncertainty about how to handle this better.

[HuwCampbell]

I'm on holiday at the moment. Back around the 15th.

In general I'm weighing compatibility, utility over known work arounds, and the api. I'm kind of hopeful there's a neat solution.

We should look hard at decline, clap, and nest before getting too far into implementation.

[stevladimir]

We should look hard at decline, clap, and nest before getting too far into implementation.

Let's figure out a list of things to reason about and go over it. I will start:

1. How should one express the help text?
   I'm in favor of being flexible here and providing few options (probably few at the same time):

   a) Append environment variable info to option description

   --foo METAVAR    Some description. Overrides FOO environment variable.
   

   b) Override metavar name.

   --foo $FOO
   

   c) Totally separate section for environment variables.

   ENVIRONMENT
   
   FOO Same as --foo
   

2. Environment variables for flags.
   Again we should be flexible and give an end-user a way to decide, probably smth like String -> Either String Bool reader: some users would like just to simply check whether variable is set or not, in other cases people would like to allow only yes/no|true/false and error otherwise.

3. Cmd option vs environment option preference.
   I've never observed environment options having higher precedence, but this argument was raised in issue discussion, so we have to keep it in mind.