Look into alternative approach for configuration

[lorenyu]

quick discussion with @jamesbursa on whether we should consider moving non-secret configs to config files

also look at https://flask.palletsprojects.com/en/2.2.x/config/

[jamesbursa]

Some rough thoughts on this. Using environment variables for configuration, as currently, has some disadvantages:

• Changing the app configuration requires infra changes (e.g. in container definitions in terraform), which can mean cross-team coordination, sign-off etc.
• Embedding os.getenv calls or equivalent in the code is fragile - if the env var is missing it can have unexpected behaviour, only strings are supported etc.
• Unit tests become more fragile - they can change behaviour if env vars are present in the developers environment, monkeypatching is needed if target code uses getenv
• There are multiple ways & places to set them (sometimes hidden or not obvious) - in the shell, docker-compose.yml, terraform / AWS, developer's IDE config, monkeypatching in tests

[jamesbursa]

A possible vision of how a better system could look.

Configuration files

Somewhere is the codebase is a directory of configuration files, one per deployed environment (prod, stage, test, etc.), plus one for local development and one for the test suite (last two might be the same).

Each file is Python (or yaml or json), containing configuration in a simple format:

db_host = "localhost"
db_schema = "api"
bucket_name = "api-bucket-test"
logging_level_override = {"api": "INFO", "sqlalchemy": "WARNING"}
...

These files get included in the container with the code. When the system starts, it loads the correct config depending on some detection of which environment it's running in. That could be an env var, or a required command line flag, with the name or path of the config file.

Questions:

• Should it be more structured, typed, and hierarchical?
• Some way to have a default and only specify the differences?
• Some way to deploy new config independent of code? Or is it really an advantage to keep them in sync?

Usage in the code

Two possible approaches:

• A central config module that other modules can retrieve the setting from.
• Something more module-based. Each module defines its own config setting class. It's read from the file when the program starts. Or in tests, defined with fixed values in the code.