Standardize how to do environment variable parsing & substitution

[karrtikr]

There're two parts to this issue:

Parsing

We should probably always follow dotenv syntax:

• Circuit Python: https://docs.circuitpython.org/en/latest/shared-bindings/dotenv/index.html#
  It's almost very similar to what we do; except for the multiline support and comments on the same line, so good news is it won't break us if we decide to align.

• Python dot-env: https://pypi.org/project/python-dotenv/
  Match with how it does variable substitution: Standardize how to do environment variable parsing & substitution #18307

Substitution

There are currently two ways of substituting variables in a .env file:

VAR2="value;${env:VAR1}"

and

VAR2="value;${VAR1}"

and a combination of both has to used to resolve all variables in different situations. However, this is not standard syntax. Again we should probably rely on the dotenv syntax always.

• ${env:...} only works for process env variables (for eg. PATH) when debugging or when resolving env variables in settings.
  • Exception: There's a bug due to which debugger doesn't resolve ${env:...} in .env files when debugging a Python file: Environment variable definitions file substitution does not work when debugging a Python file vscode-python-debugger#159. Make sure Environment variable definitions file substitution does not work when debugging a Python file vscode-python-debugger#159 is taken care of with this issue.
• "Recursive" variable substitution does not work with ${env:...}. It works only with ${...}. Need to correct the docs: Test envVarsService newline substitution behavior #17747 (review).
• .env file is currently not supported when running code in terminal: Use environment variables defined in .env file when running code in a terminal #944
• ${...} can not be used for process env variables when debugging: https://github.com/microsoft/debugpy/issues/821. Solution:
  • Debugging doesn't use environment variable substitution/resolution support in the extension. Expose

    vscode-python/src/client/common/variables/environmentVariablesProvider.ts

    Line 56 in 456ac06

    public async getEnvironmentVariables(resource?: Uri): Promise<EnvironmentVariables> {

    as an API and have the debugger use it to fetch env variables.
  • Follow the standard python-dotenv syntax to resolve process env variables in debugger: https://pypi.org/project/python-dotenv/, i.e add support to resolve using ${...} in addition to ${env:...}.

[starball5]

Related on Stack Overflow: Why does VSCode show inline comments as part of environment variables from .env file?

[ababak]

Any updates on this? The lack of support for newlines is problematic.

[eleanorjboyd]

@karthiknadig and @anthonykim1 - wondering if something along this lines should be a larger discussion across different features. I feel like I have seen a few of these regarding variable substitution and might be worthwhile to tackle?

[karthiknadig]

We can do this in the new environments extension. The plan is to have an implementation there that can handle broad set of cases.

The is still in ideation phase, and is a work in progress:
API: https://github.com/microsoft/vscode-python-environments/blob/0519050ee0d85baf0487bdb4a7519fbc06db60bc/src/api.ts#L1197-L1222

Implementation: https://github.com/microsoft/vscode-python-environments/blob/main/src/features/execution/envVariableManager.ts