Dynamic Index Settings (Cameron's Proposal) #778

da2ce7 · 2024-11-30T11:47:29Z

Related to torrust/torrust-index-gui#664

Current Situation

Torrust Index loads a read-only configuration upon startup; this configuration is static over the lifetime of the program.

Details

The Torrust Index reads its settings in TOML format from two places:

A file: /etc/torrust/index/index.toml, or a path overridden by the TORRUST_INDEX_CONFIG_TOML_PATH environment variable.
From the environment variable: TORRUST_INDEX_CONFIG_TOML.

If both configurations are present, the configuration supplied by the environment variable takes precedence.

The TOML format is overridden by configuration options supplied by the TORRUST_INDEX_CONFIG_OVERRIDE__ environment variable prefix.

This merged configuration is read-only for the lifetime of a torrust-index instance.

Configuration Types

There is a matrix of different types of configuration that the Torrust Index uses, for example:

	Immutable (Cannot be changed after initialization)	Mutable (Can be changed during runtime)
Startup-Defined	Startup-Immutable: Values known at startup, fixed for app lifetime (e.g., DB Connection String)	Startup-Mutable: Values known at startup, but can be modified later (e.g., Logging Level)
Runtime-Defined	Runtime-Immutable: Values determined during execution, fixed once set (e.g., API Socket)	Runtime-Mutable: Values determined during execution, can be changed dynamically (e.g., Index Policy)

There is also a matrix of different administration levels for the configuration:

	Private	Public	General
System Admin	Only System Admin can set and view	Only the System Admin can set; others may see	Override value by default or Service Admin
Service Admin	Cannot access	Can view	Can set
Application Defaults	Not applicable	Default value	Default value

Proposal

Split the Configuration into Four Separate Files Controlled by the System Administrator

The same settings can be supplied via environment variables, similar to the current implementation. If both a configuration file and the corresponding environment variable are present, the environment variable takes precedence.

1. Immutable-Private Configuration

/etc/torrust/index/private.toml or TORRUST_INDEX_CONFIG_PRIVATE

This file will contain settings that are set by the system administrator and should only be known by the system administrator.

Example: the DB Connection String.

2. Immutable-Public Configuration

/etc/torrust/index/public.toml or TORRUST_INDEX_CONFIG_PUBLIC

This file will contain settings that are set by the system administrator and may be shared with the service administrator (or more).

Examples:

Public socket of the service

Initial logging level of the service

3. General Forced

/etc/torrust/index/forced.toml or TORRUST_INDEX_CONFIG_FORCED

This file will contain settings that are overridden by the system administrator.

Examples:

Obligatory Privacy Policy

Runtime logging level of the service

4. General Default

/etc/torrust/index/default.toml or TORRUST_INDEX_CONFIG_DEFAULT

This file will contain default settings that are set by the system administrator.

Examples:

Default logo

Welcome message

Create an In-Database Settings File

As a record inside the database:

This record will contain a json_utf8 formatted entry with the settings supplied by the service administrator.

The following endpoints and actions will be created:

Endpoint	Action	Response
`/admin/current`	GET	200 OK: Returns the current configuration as supplied by the service administrator, or 204 No Content
`/admin/current`	PUT	200 OK: Sets new configuration; 400 Bad Request if the configuration is invalid
`/admin/merged`	GET	200 OK: Returns the current active configuration, including the public, overridden, current, and default settings, merged
`/admin/default`	GET	200 OK: Returns the default configuration, or 204 No Content
`/admin/override`	GET	200 OK: Returns any overrides for the configuration, or 204 No Content

The text was updated successfully, but these errors were encountered:

da2ce7 · 2024-11-30T12:34:08Z

Migration Plan

To ensure a smooth transition to the new configuration system without introducing breaking changes, the following migration plan is proposed:

Phase 1: Introduction of New Configuration Structure

Support Multiple Configuration Sources:
- Update the application to support reading configurations from the new files (private.toml, public.toml, override.toml, default.toml) and their corresponding environment variables:
  - TORRUST_INDEX_CONFIG_PRIVATE
  - TORRUST_INDEX_CONFIG_PUBLIC
  - TORRUST_INDEX_CONFIG_OVERRIDE
  - TORRUST_INDEX_CONFIG_DEFAULT
- Ensure that the application can merge configurations from the existing index.toml, the new configuration files, and environment variables.
Define Configuration Precedence:
1. Highest Priority: Environment variables with the TORRUST_INDEX_CONFIG_OVERRIDE_ prefix (individual overrides).
2. Next: Environment variables supplying the content of the new configuration files: private, public, override, and default, be if from a file, or supplied by an environmental variable.
3. Lowest Priority: Existing index.toml file or the path specified by TORRUST_INDEX_CONFIG_TOML_PATH.
Maintain Backward Compatibility:
- Retain full support for the existing index.toml configuration file to ensure current installations continue to function without modification.
Logging and Warnings:
- Implement logging to inform administrators when configurations are loaded from deprecated sources (index.toml), encouraging migration to the new configuration structure.

Phase 2: Gradual Migration of Settings

Deprecate Settings in index.toml:
- Identify settings that will be moved to the new configuration structure or to dynamic settings in the database.
- Mark these settings as deprecated in the documentation and output deprecation warnings in application logs when they are loaded from index.toml.
Administrator Guidance:
- Provide comprehensive documentation and migration guides to assist system administrators in transferring configurations from index.toml to the new files or environment variables.
- Offer examples and best practices for setting up the new configuration structure.
Optional Migration Tools:
- Develop scripts or utilities to help administrators convert index.toml configurations into the new format or appropriate environment variables.

Phase 3: Enable Dynamic Configuration Through Database

Implement In-Database Settings Management:
- Create the database record for dynamic settings (in json_utf8 format) to store configurations supplied by the service administrator.
- Deploy the new API endpoints to allow the service administrator to manage these settings at runtime.
Prioritize Dynamic Settings:
- Modify the configuration loading logic so that dynamic settings from the database take precedence over static configurations.
- Ensure that updates to the dynamic settings are applied without requiring a service restart.

Phase 4: Deprecation and Removal of `index.toml`

Announce Deprecation Timeline:
- Communicate to administrators that support for index.toml will be deprecated in a future major release (e.g., version 4.0.0).
- Provide a clear timeline and roadmap for the deprecation process.
Final Migration Steps:
- Encourage administrators to complete the migration to the new configuration structure before the deprecation date.
- Offer assistance and resources during this period to address any migration challenges.
Remove Support for index.toml:
- In the specified future release, remove the ability to load configurations from index.toml.
- Clean up associated code and resources related to the old configuration system.

Phase 5: Ongoing Support and Improvements

Monitoring and Support:
- Continue to support administrators post-migration by addressing any issues and providing updates.
- Gather feedback to improve the configuration system and administrative interfaces.
Iterative Enhancements:
- Explore potential enhancements such as web-based configuration editors or additional administrative tools.
- Consider further optimizations to configuration loading and management based on user feedback.

josecelano · 2024-12-03T10:39:59Z

Hi @da2ce7, I have some questions.

1. Difference between `Dynamic` and `Runtime` and between `Static` and `Instance`

Instance: The configuration is loaded when the application starts, and changing it requires
restarting it.
Runtime: The configuration is loaded when the application starts and reloaded when it changes (when it's updated in the database).

	Instance	Runtime
Static	DB Connection String	API Socket
Dynamic	Logging Level	Index Policy

What are Instance-Dynamic and Runtime-Static configurations?

From your examples, I would say:

Instance: Configuration that involves or affects external services (out-of-process).
Runtime: Configuration that only involves or affects internal services (in-process).

I don't know the difference between "Dynamic" and "Runtime".

2. `override.toml`

Why do you need the override.toml?
Why doesn't the System Administrator provide the correct values in private.toml and public.toml initially?

NOTE: We override values in index.toml with env vars. An extra layer of overriding could be very confusing if all toml files allow overriding their values with env vars.

3. `default.toml`

I assume this allows to override hardcoded defaults, right?

Keep `private.toml` and `public.toml` in the current one, `index.toml`?

I'm not sure if splitting the data using the permission is better than handling the permission in a different layer.

I assume the Service Administrator can only override the public.toml configuration.

Pros:

It's really clear what the System Administrator can see/edit.
Security should be easier to implement.

Cons:

Configuration permissions are hardcoded. For example, we can't allow the System Administrator to see or edit the "logging threshold", if we initially put it in the private.toml file.

4. Mandatory values

Where do we put the metadata information, such as the list of mandatory fields?

5. Versioning

Having four files instead of 1 makes it harder to keep consistency. I suppose we have to release new versions for all of them simultaneously.

6. Alternative 1: `metadata.toml`

Merge new files into a single one?

We could keep the current index.toml
Add a new file: metadata.toml
Allow to override some (non private) values by Service Administrator in the database.

The metadata.toml would contain:

[metadata]
schema_version = "2.0.0"

# List of mandatory fields
mandatory = [
  "database.connect_url"
]

# List of fields that only the System Administrator can see/edit
private = [
  "database.connect_url"
]

# Default values
[defaults.logging]
  threshold = "info"

We can also move this info to the metadata section of the index.toml file.

I guess the main advantage of this solution is flexibility. If we allow the system administrator to inject defautl values, these would be an extra level of indirection.

7. Alternative 2: New section in `index.toml`

We can also implement this proposal using the index.toml file with new sections:

# ...

[private]
# ...

[public]
# ...

[override]
# ...

[defaults]
# ...

Putting all the info in a single file would make it easier in the future to refactor becuase we only need to change code and not the way the configration is injected. That requires changes in containers, deployments etc.

[override]and [defaults] sections could be inside [metadata].

8. More questions

Should default values be stored in the database? I don't think so because there will be no way to be changed by the system administrator anymore.
I assume the System Administrator is not the root Service administrator.

da2ce7 · 2024-12-03T15:01:16Z

@josecelano Regarding Question One:

1. Difference between Dynamic and Runtime and between Static and Instance

I need to update the original posts to make the table more clear. But here is a better summary of the idea I have in mind:

Configuration Types

	Static	Dynamic
Instance	Values are known at application start and cannot be changed at runtime (e.g., DB Connection String)	Values are known at application start but can be changed at runtime (e.g., Logging Level)
Runtime	Values that may be determined at runtime and cannot be changed afterwards (e.g., API Socket)	Values are determined at runtime and can be changed dynamically (e.g., Index Policy)

In this matrix, we can see that:

Static configurations are those that cannot be changed at runtime, while Dynamic configurations can be changed at runtime.
Instance configurations are those whose values are known when the application starts, while Runtime configurations are those that may be determined during the application's execution.

By combining these two dimensions, we get four quadrants that help us understand the different types of configurations:

Static-Instance: Configurations that are known at application start and cannot be changed at runtime (e.g., DB Connection String).
Dynamic-Instance: Configurations that are known at application start but can be changed at runtime (e.g., Logging Level).
Static-Runtime: Configurations that may be determined at runtime and cannot be changed afterwards (e.g., API Socket).
Dynamic-Runtime: Configurations that may be determined at runtime and can be changed dynamically (e.g., Index Policy).

da2ce7 · 2024-12-03T15:16:17Z

@josecelano Regarding Question Two:

2. override.toml

The override.toml file serves as a way for the System Administrator to explicitly override dynamic values at runtime, forcing them to a specific value. To understand the role of the override.toml file, let's consider the following matrix:

	Static	Dynamic
No Override	Value is set by `public` or `private` file and cannot be changed	Value is set by `public` or `private` file, but can be changed at runtime
Override	Value is overridden, but not needed as it cannot be changed if set in the `public` or `private` file	Value is overridden by `override.toml` file and cannot be changed at runtime

In this matrix, we can see that:

For static values, the override.toml file is not needed, but it will still override the any value is still set by the public or private file.
For dynamic values, the override.toml file allows the System Administrator to override the value and prevent it from being changed at runtime.

By providing a separate override mechanism, the System Administrator can ensure that dynamic values are set to a specific value, even if they are changed at runtime. This is particularly useful for security or compliance reasons, where specific values need to be enforced.

The override.toml file provides a way to "pin" dynamic values to a specific value, preventing them from being changed at runtime. This is different from the public and private files, which set instance values that can be changed if they are dynamic.

da2ce7 · 2024-12-03T15:24:37Z

@josecelano Regarding Question Three:

3. default.toml

The default.toml file contains default settings that are set by the System Administrator. These default values override the default values that are included by us, the developers, in the application's code, and can be overridden by other configuration files or environment variables.

The purpose of the default.toml file is to allow the System Administrator to set their own default values, which may differ from the default values set by the application developers. This provides a way for the System Administrator to customize the application's default behavior without modifying the underlying code.

By defining default values in a separate file, the System Administrator can easily manage and update the default configuration without affecting other configuration files or the application code.

da2ce7 · 2024-12-03T15:32:47Z

@josecelano Regarding Question Four:

4. Mandatory values

The list of mandatory values is defined by the application code and is not something that can be changed through configuration files. It is the responsibility of the application to ensure that its configuration is coherent and valid. If the instance configuration is bad, the application will panic. If a proposed change to the dynamic configuration is invalid, the application will return an error.

We should ensure that the application is always in a consistent and valid state, and prevent users from accidentally or intentionally configuring the application in a way that could undefined behavior.

da2ce7 · 2024-12-03T15:38:50Z

@josecelano Regarding Question Five:

5. Versioning

To handle versioning of the configuration files, I suggest we should prefix all configuration files and environmental variables with the major version number. This will allow us to make significant changes to the configuration structure or syntax only when the major version is updated.

For example, if the current major version is 2, the configuration files would be named v2_private.toml, v2_public.toml, v2_override.toml, and v2_default.toml. Similarly, environmental variables would be include the V2 within their name.

Minor version updates will be used to add new features by adding more fields to the existing configuration files. This way, we can introduce new configuration options without breaking existing configurations.

Patches will be silent, meaning that minor bug fixes or tweaks will not require updates to the configuration files or environmental variables.

Edit:

I still think that the config files should have a [metadata] (or whatever) section that defines their namespace and exact version.

josecelano · 2024-12-03T16:15:59Z

@josecelano Regarding Question One:

1. Difference between Dynamic and Runtime and between Static and Instance

I need to update the original posts to make the table more clear. But here is a better summary of the idea I have in mind:

Configuration Types

Static Dynamic
Instance Values are known at application start and cannot be changed at runtime (e.g., DB Connection String) Values are known at application start but can be changed at runtime (e.g., Logging Level)
Runtime Values that may be determined at runtime and cannot be changed afterwards (e.g., API Socket) Values are determined at runtime and can be changed dynamically (e.g., Index Policy)
In this matrix, we can see that:

Static configurations are those that cannot be changed at runtime, while Dynamic configurations can be changed at runtime.

Instance configurations are those whose values are known when the application starts, while Runtime configurations are those that may be determined during the application's execution.

By combining these two dimensions, we get four quadrants that help us understand the different types of configurations:

Static-Instance: Configurations that are known at application start and cannot be changed at runtime (e.g., DB Connection String).

Dynamic-Instance: Configurations that are known at application start but can be changed at runtime (e.g., Logging Level).

Static-Runtime: Configurations that may be determined at runtime and cannot be changed afterwards (e.g., API Socket).

Dynamic-Runtime: Configurations that may be determined at runtime and can be changed dynamically (e.g., Index Policy).

I find it confusing to use "runtime" and "dynamic".

	Immutable (Cannot be changed after initialization)	Mutable (Can be changed during runtime)
Startup-Defined	Startup-Immutable: Values known at startup, fixed for app lifetime (e.g., DB Connection String)	Startup-Mutable: Values known at startup, but can be modified later (e.g., Logging Level)
Runtime-Defined	Runtime-Immutable: Values determined during execution, fixed once set (e.g., API Socket)	Runtime-Mutable: Values determined during execution, can be changed dynamically (e.g., Index Policy)

On the other hand, regarding API Socket, I suppose you mean you can specify "0.0.0.0:0" in the configuration, and the final value could be "0.0.0.0:37621". In the current implementation configuration, values do not change (after merging sources). I think that's a good thing. I would not update the original configuration value. I think configuration should always be an input. Are there other examples? I suppose you want to expose the real port in the configuration endpoints:

/admin/current
/admin/merged
/admin/default
/admin/override

I would add a new one to expose services or maybe in the root endpoint.

By the way, I would rename those endpoints to /config/current, ... or /config/config/current ...

josecelano · 2024-12-03T16:22:45Z

@josecelano Regarding Question Two:

2. override.toml

The override.toml file serves as a way for the System Administrator to explicitly override dynamic values at runtime, forcing them to a specific value. To understand the role of the override.toml file, let's consider the following matrix:

Static Dynamic
No Override Value is set by public or private file and cannot be changed Value is set by public or private file, but can be changed at runtime
Override Value is overridden, but not needed as it cannot be changed if set in the public or private file Value is overridden by override.toml file and cannot be changed at runtime
In this matrix, we can see that:

For static values, the override.toml file is not needed, but it will still override the any value is still set by the public or private file.

For dynamic values, the override.toml file allows the System Administrator to override the value and prevent it from being changed at runtime.

By providing a separate override mechanism, the System Administrator can ensure that dynamic values are set to a specific value, even if they are changed at runtime. This is particularly useful for security or compliance reasons, where specific values need to be enforced.

The override.toml file provides a way to "pin" dynamic values to a specific value, preventing them from being changed at runtime. This is different from the public and private files, which set instance values that can be changed if they are dynamic.

Ok, I understand now it's a way to enforce settings by the Systema Admin that is allowed to be changed dynamically by the Service Admin.

I think the name override.toml is too generaic becuase there are mnay ways to override values. What about forced_settings.toml or something like that.

Wouldn't having a list of fields that cannot be changed be easier? Then we can inform the Service Admin. We need to have it anyway; otherwise, it would be a surprise for the Service Admin to override some values without any effect.

da2ce7 · 2024-12-03T16:22:52Z

@josecelano Yes I'm open for renaming. I'm just trying to create words for different aspects in play...

A good example is the API endpoint.

The sys-admin sets a default API endpoint of 0.0.0.0:0, in default.toml. However after the DB is loaded, it sees that the endpoint has been set to 1.2.3.4:0 in the db, so it instead opens the API on ip 1.2.3.4...

But, if the sys-admin sets the API endpoint of 0.0.0.0:0 in public.toml, the app would give a warning and ignore any value that is included in the DB for the endpoint, and load on ip 0.0.0.0, because the API Endpoint is not a dynamic value.

da2ce7 · 2024-12-03T16:32:30Z

I think the name override.toml is too generaic becuase there are mnay ways to override values. What about forced_settings.toml or something like that.

I agree. even just forced.toml , could be more explicit that override.toml.

josecelano · 2024-12-03T17:02:08Z

On more question @da2ce7,

In-Database Settings

Can the Service Admin override public settings? Or DB settings are totally different from settings included in these four new files?

I have added a proposal for relocating values in the new files.

Current `index.toml`

[metadata]
app = "torrust-index"
purpose = "configuration"
schema_version = "2.0.0"

[logging]
threshold = "info"

[website]
name = "Torrust"

  [website.demo]
  warning = "⚠️ Please be aware: This demo resets all data weekly. Torrents not complying with our Usage Policies will be removed immediately without notice. We encourage the responsible use of this software in compliance with all legal requirements."

[website.terms.page]
title = "Usage Policies and Content Restrictions"
content = """

# Usage Policies and Content Restrictions

Our software is designed to support the distribution of legal, authorized content only. Users may only upload or share files that fall under the following categories:

- **Open-Source Licenses:** Content licensed under recognized open-source licenses, allowing for free distribution and modification.
- **Creative Commons Licenses:** Content released under Creative Commons licenses that permit sharing and distribution.
- **Public Domain:** Content that is free of copyright restrictions and available for public use.

**Prohibited Content:** Any content that infringes copyright, is subject to copyright protection, or is illegal under applicable laws is strictly prohibited. This includes but is not limited to copyrighted movies, music, software, books, and any other media.

**Enforcement:** We reserve the right to remove any content that does not comply with these policies without notice. We may also take additional steps, including reporting violations to the relevant authorities, if necessary.

"""

[website.terms.upload]
content_upload_agreement = "I confirm that the content I am uploading is authorized, and I have read and agree to the terms."

[tracker]
api_url = "http://localhost:1212/"
listed = false
private = false
token = "***"
token_valid_seconds = 7_257_600
url = "udp://localhost:6969"

[net]
bind_address = "0.0.0.0:0"

[auth]
user_claim_token_pepper = "***"

  [auth.password_constraints]
  max_password_length = 64
  min_password_length = 6

[database]
connect_url = "sqlite://data.db?mode=rwc"

[mail]
from = "[email protected]"
reply_to = "[email protected]"

  [mail.smtp]
  port = 25
  server = ""

    [mail.smtp.credentials]
    password = "***"
    username = ""

[image_cache]
capacity = 128_000_000
entry_size_limit = 4_000_000
max_request_timeout_ms = 1_000
user_quota_bytes = 64_000_000
user_quota_period_seconds = 3_600

[api]
default_torrent_page_size = 10
max_torrent_page_size = 30

[registration.email]
required = false
verification_required = false

[tracker_statistics_importer]
port = 3_002
torrent_info_update_interval = 3_600

New four files `private.toml`, `public.toml`, `default.toml`, `forced.toml`

`private.toml`

[metadata]
app = "torrust-index"
purpose = "private-configuration"
schema_version = "2.0.0"

[tracker]
token = "***"

[auth]
user_claim_token_pepper = "***"

[database]
connect_url = "sqlite://data.db?mode=rwc"

[mail.smtp.credentials]
password = "***"
username = ""

`public.toml`

[metadata]
app = "torrust-index"
purpose = "public-configuration"
schema_version = "2.0.0"

[logging]
threshold = "info"

[website]
name = "Torrust"

  [website.demo]
  warning = "⚠️ Please be aware: This demo resets all data weekly. Torrents not complying with our Usage Policies will be removed immediately without notice. We encourage the responsible use of this software in compliance with all legal requirements."

[website.terms.page]
title = "Usage Policies and Content Restrictions"
content = """

# Usage Policies and Content Restrictions

Our software is designed to support the distribution of legal, authorized content only. Users may only upload or share files that fall under the following categories:

- **Open-Source Licenses:** Content licensed under recognized open-source licenses, allowing for free distribution and modification.
- **Creative Commons Licenses:** Content released under Creative Commons licenses that permit sharing and distribution.
- **Public Domain:** Content that is free of copyright restrictions and available for public use.

**Prohibited Content:** Any content that infringes copyright, is subject to copyright protection, or is illegal under applicable laws is strictly prohibited. This includes but is not limited to copyrighted movies, music, software, books, and any other media.

**Enforcement:** We reserve the right to remove any content that does not comply with these policies without notice. We may also take additional steps, including reporting violations to the relevant authorities, if necessary.

"""

[website.terms.upload]
content_upload_agreement = "I confirm that the content I am uploading is authorized, and I have read and agree to the terms."

[tracker]
api_url = "http://localhost:1212/"
listed = false
private = false
token_valid_seconds = 7_257_600
url = "udp://localhost:6969"

[net]
bind_address = "0.0.0.0:0"

[auth]

[auth.password_constraints]
max_password_length = 64
min_password_length = 6

[mail]
from = "[email protected]"
reply_to = "[email protected]"

  [mail.smtp]
  port = 25
  server = ""

[image_cache]
capacity = 128_000_000
entry_size_limit = 4_000_000
max_request_timeout_ms = 1_000
user_quota_bytes = 64_000_000
user_quota_period_seconds = 3_600

[api]
default_torrent_page_size = 10
max_torrent_page_size = 30

[registration.email]
required = false
verification_required = false

[tracker_statistics_importer]
port = 3_002
torrent_info_update_interval = 3_600

`default.toml`

[metadata]
app = "torrust-index"
purpose = "default-configuration"
schema_version = "2.0.0"

# Empty if you want to use hardcoded default values

`forced.toml`

[metadata]
app = "torrust-index"
purpose = "forced-configuration"
schema_version = "2.0.0"

# Empty if you don't want to force some public values

josecelano · 2024-12-05T15:51:56Z

A list of properties for a single configuration value:

1. Mutability

Description: Whether the configuration value can be changed after the application starts.
Options:
- Immutable: Cannot be changed after startup.
- Mutable: Can be changed dynamically at runtime.
Representative Name: mutability

2. Setter Role

Description: Specifies who can set or modify the configuration value.
Options:
- SystemAdmin: Only the system administrator can set or edit the value.
- ServiceAdmin: Both the system administrator and service administrator can set or edit the value.
Representative Name: setter_role

3. Visibility

Description: Specifies who can view the configuration value.
Options:
- SystemAdminOnly: Only the system administrator can see the value.
- ServiceAdmin: Visible to both system administrators and service administrators.
- Public: Visible to all users.
Representative Name: visibility

4. Mandatory

Description: Whether the configuration value is required for the application to function.
Options:
- Mandatory: The value must be explicitly set; there is no default.
- Optional: The value is optional and uses a default if not provided.
Representative Name: mandatory

5. Runtime Immutability Reason

Description: Explains why a configuration cannot be changed at runtime.
Options:
- CodeConstraint: The code explicitly disallows changes to this configuration at runtime.
- StartupLoaded: The value is loaded at startup and is immutable for simplicity.
Representative Name: runtime_immutability_reason

6. Post-Initialization Modification

Description: Indicates whether the value can be altered or derived after it has been used or initialized.
Options:
- Direct: The value remains exactly as configured.
- Derived: The value may be manipulated or transformed by the application after being set (e.g., a port set to 0 resolves to a free port at runtime).
Representative Name: post_initialization_modification

Example JSON Representation for a Configuration Value

{
  "name": "bind_address",
  "mutability": "Immutable",
  "setter_role": "SystemAdmin",
  "visibility": "ServiceAdmin",
  "mandatory": "Optional",
  "runtime_immutability_reason": "CodeConstraint",
  "post_initialization_modification": "Derived"
}

Explanation with Examples

Using this structure, we can represent specific configuration values:

Database Connection String:

{
  "name": "db_connection_string",
  "mutability": "Immutable",
  "setter_role": "SystemAdmin",
  "visibility": "SystemAdminOnly",
  "mandatory": "Mandatory",
  "runtime_immutability_reason": "CodeConstraint",
  "post_initialization_modification": "Direct"
}

Immutable because it cannot change after startup.
Only the system administrator can set or view it.

Logging Level:

{
  "name": "logging_level",
  "mutability": "Mutable",
  "setter_role": "ServiceAdmin",
  "visibility": "ServiceAdmin",
  "mandatory": "Optional",
  "runtime_immutability_reason": null,
  "post_initialization_modification": "Direct"
}

Mutable because it can be changed at runtime.
Accessible and configurable by the service administrator.

Socket Address:

{
  "name": "socket_address",
  "mutability": "Immutable",
  "setter_role": "SystemAdmin",
  "visibility": "ServiceAdmin",
  "mandatory": "Mandatory",
  "runtime_immutability_reason": "StartupLoaded",
  "post_initialization_modification": "Derived"
}

Immutable because it is determined at startup.
Derived after binding if the port is set to 0.

josecelano · 2024-12-05T16:30:34Z

Hi @da2ce7 I have created a table to represent the current status for all values and properties:

I've just realized not all secrets are mandatory. I guess all of them should be. I'm going to build another table with what we would like to have at the end.

josecelano · 2024-12-05T16:37:48Z

And I think this is what we would like to have:

josecelano · 2024-12-05T17:03:56Z

NOTES:

SysAdmin cannot override. If SysAdmins need to override something to avoid restarting the service they can create a ServiceAdmnin account for themselves.
All secrets should be mandatory.
There are two types of secrets: in-process and out-of-process (connect to other services). I have doubts for some of them. Should the ServiceAdmin be able to change the tracker token? We are assuming the SysAdmin sets up the whole infrastructure (Index, Database, Mail server, Tracker). I'm not sure for the tracker. The ServiceAdmin could decide to switch to other Torrust Tracker instance.
Website settings should be moved to a CMS module or I18N module. I don't see the point to involve the SysAdmin there. In fact, that a thechnical debt. We decided to put it there because it was easier than building a new module for that.
In you proposal @da2ce7 you have 4 files. One of the advantages of having 4 files is the isolation in terms of user's privileges. However, if we merge them in memory (with Figment) we loose that isolation. Maybe the data structure we use to inject configuration should be the same to keep the data in memory.
The new dynamic settings we want to add are all of them settings that the ServiceAdmin can set and I would say most of them not relevant for the SysAdmin.
@da2ce7 can you provide some examples of dynamic (mutable) settings that the SysAdmin would like to enforce (overriding what the ServiceAdmin overrides). The only example I find is logging. Logging level could be something that the ServiceAdmin can define but in some cases the SysAdmin can enforce a certain level, for example, to debug infrastructure problems. But I think those should be different logs in the end.

josecelano · 2024-12-12T17:35:00Z

Today, in our weekly meeting, I mentioned that I would move website section to a minimal CMS, before doing this refactoring. But in the end, we agreed to leave the configuration as it is for now and allow the service admin to override some keys.

I will wait for @da2ce7 to give it a second thought, especially for the file names. I guess the final implementation will be:

Those 4 files, but maybe with different names.
A database table where we can dynamically override the keys for the public.yml file.

There will be some JSON schemas:

default and force could be the same (all keys)
private (subset)
public (subset)

I don't know if there is a way to represent that with JSON schemas without duplicating parts of the schemas (the subsets private and public):

https://json-schema.org/understanding-json-schema/structuring

Maybe we need to create the schema for the public keys, which is the only one that will be dynamically validated.

josecelano · 2024-12-13T16:46:15Z

Hi @da2ce7, since in this proposal we are going to include ALL keys in the files (unlike mine), I think this refactor is not necessary before adding the new settings I was planning to include:

Admins should be able to pause new registrations torrust-index-gui#665
Configure linking to Torrent Files Only or sharing Magnet Links only, or Both. torrust-index-gui#666
Customisable visible torrent data torrust-index-gui#670
Admins should be able to enable/disable external links in the torrent description torrust-index-gui#667

I mean, we can do it before or after adding the new settings.

I suggest starting to implement the new settings with the current format. In the meantime, we can refine this proposal. These features would be available earlier. The only problem is they would require the System Administrator to restart the service to change them until we implement this proposal. I don't think that's a problem because those options are options you generally don't want to change at all after starting the service. The only one that could require to change it dynamically is the one to pause registrations.

NOTE: I would still make them dynamic because one of the reasons is to move the responsibility for platform customization from the System Administrator to the Service Administrator.

I have created a proposal for the key names and the organization. These would be the new keys:

[website.gui]

[website.gui.torrenting]
enable_torrent_download = true # It hides torrent download buttons in GUI when false
enable_magnet_links = true     # It hides magnet links in GUI when false

[website.gui.torrent_view]
enable_external_links = false  # It does not escape external links in the torrent description when true
show_comment = false           # It hides torrent `comment` in GUI when false
show_creation_date = false     # It torrent `creation date` in GUI when false
show_created_by = false        # It hides torrent `created by ` in GUI when false
show_encoding = false          # It hides torrent `encoding` in GUI when false

[registration]
paused = false                 # It only disables registration temporarily

I'm using the gui namespace to differentiate this values from other values:

Translations: texts introduced by developers that could be in different languages in the future.
CMS: texts defined by the Service Admin (although they can have default values). These could be multilanguage in the future, too.

If you agree, I can remove the issue:

Setup scaffolding for dynamic global settings torrust-index-gui#664

From the EPI:

Index administration torrust-index-gui#657

Or move it to the end. I won't be a dependency any more. It would be done at the end.

da2ce7 mentioned this issue Nov 30, 2024

Setup scaffolding for dynamic global settings torrust/torrust-index-gui#664

Open

da2ce7 changed the title ~~Dynamic Index Settings~~ Dynamic Index Settings (Cameron's Proposal) Nov 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamic Index Settings (Cameron's Proposal) #778

Dynamic Index Settings (Cameron's Proposal) #778

da2ce7 commented Nov 30, 2024 •

edited

Loading

da2ce7 commented Nov 30, 2024 •

edited

Loading

josecelano commented Dec 3, 2024 •

edited

Loading

da2ce7 commented Dec 3, 2024 •

edited

Loading

da2ce7 commented Dec 3, 2024 •

edited

Loading

da2ce7 commented Dec 3, 2024

da2ce7 commented Dec 3, 2024

da2ce7 commented Dec 3, 2024 •

edited

Loading

josecelano commented Dec 3, 2024

josecelano commented Dec 3, 2024

da2ce7 commented Dec 3, 2024 •

edited

Loading

da2ce7 commented Dec 3, 2024

josecelano commented Dec 3, 2024 •

edited

Loading

josecelano commented Dec 5, 2024

josecelano commented Dec 5, 2024

josecelano commented Dec 5, 2024 •

edited

Loading

josecelano commented Dec 5, 2024 •

edited

Loading

josecelano commented Dec 12, 2024

josecelano commented Dec 13, 2024

Dynamic Index Settings (Cameron's Proposal) #778

Dynamic Index Settings (Cameron's Proposal) #778

Comments

da2ce7 commented Nov 30, 2024 • edited Loading

Current Situation

Details

Configuration Types

Proposal

Split the Configuration into Four Separate Files Controlled by the System Administrator

1. Immutable-Private Configuration

2. Immutable-Public Configuration

3. General Forced

4. General Default

Create an In-Database Settings File

da2ce7 commented Nov 30, 2024 • edited Loading

Migration Plan

Phase 1: Introduction of New Configuration Structure

Phase 2: Gradual Migration of Settings

Phase 3: Enable Dynamic Configuration Through Database

Phase 4: Deprecation and Removal of index.toml

Phase 5: Ongoing Support and Improvements

josecelano commented Dec 3, 2024 • edited Loading

1. Difference between Dynamic and Runtime and between Static and Instance

2. override.toml

3. default.toml

Keep private.toml and public.toml in the current one, index.toml?

4. Mandatory values

5. Versioning

6. Alternative 1: metadata.toml

7. Alternative 2: New section in index.toml

8. More questions

da2ce7 commented Dec 3, 2024 • edited Loading

da2ce7 commented Dec 3, 2024 • edited Loading

da2ce7 commented Dec 3, 2024

da2ce7 commented Dec 3, 2024

da2ce7 commented Dec 3, 2024 • edited Loading

josecelano commented Dec 3, 2024

josecelano commented Dec 3, 2024

da2ce7 commented Dec 3, 2024 • edited Loading

da2ce7 commented Dec 3, 2024

josecelano commented Dec 3, 2024 • edited Loading

Current index.toml

New four files private.toml, public.toml, default.toml, forced.toml

private.toml

public.toml

default.toml

forced.toml

josecelano commented Dec 5, 2024

1. Mutability

2. Setter Role

3. Visibility

4. Mandatory

5. Runtime Immutability Reason

6. Post-Initialization Modification

Example JSON Representation for a Configuration Value

Explanation with Examples

josecelano commented Dec 5, 2024

josecelano commented Dec 5, 2024 • edited Loading

josecelano commented Dec 5, 2024 • edited Loading

NOTES:

josecelano commented Dec 12, 2024

josecelano commented Dec 13, 2024

da2ce7 commented Nov 30, 2024 •

edited

Loading

da2ce7 commented Nov 30, 2024 •

edited

Loading

Phase 4: Deprecation and Removal of `index.toml`

josecelano commented Dec 3, 2024 •

edited

Loading

1. Difference between `Dynamic` and `Runtime` and between `Static` and `Instance`

2. `override.toml`

3. `default.toml`

Keep `private.toml` and `public.toml` in the current one, `index.toml`?

6. Alternative 1: `metadata.toml`

7. Alternative 2: New section in `index.toml`

da2ce7 commented Dec 3, 2024 •

edited

Loading

da2ce7 commented Dec 3, 2024 •

edited

Loading

da2ce7 commented Dec 3, 2024 •

edited

Loading

da2ce7 commented Dec 3, 2024 •

edited

Loading

josecelano commented Dec 3, 2024 •

edited

Loading

Current `index.toml`

New four files `private.toml`, `public.toml`, `default.toml`, `forced.toml`

`private.toml`

`public.toml`

`default.toml`

`forced.toml`

josecelano commented Dec 5, 2024 •

edited

Loading

josecelano commented Dec 5, 2024 •

edited

Loading