Mask the credentials when reading the stream config from streams API

[ye11ow]

The current benthos Streams API provide a good way to dynamically create and manage the streams. However, when reading an existing stream config via GET /streams/{id} endpoint, it would be better to mask all the credential fields or at least provide an option to mask them.

For example, currently when getting a stream with azure_cosmosdb as input, the response is like this:

{
  "active": true,
  "uptime": 222.438774375,
  "uptime_str": "3m42.438774667s",
  "config": {
    "input": {
      "azure_cosmosdb": {
        "account_key": "adfs",
        "args_mapping": "root = [\n  { \"Name\": \"@name\", \"Value\": \"benthos\" },\n]",
        "auto_replay_nacks": true,
        "batch_count": -1,
        "connection_string": "sdfa",
        "container": "testcontainer",
        "database": "testdb",
        "endpoint": "https://localhost:8081",
        "partition_keys_map": "root = \"blobfish\"",
        "query": "SELECT c.foo FROM testcontainer AS c WHERE c.bar = \"baz\" AND c.timestamp < @timestamp"
      }
    },
    "output": {
      "stdout": {
        "codec": "lines"
      }
    }
  }
}

Expected response would be:

{
  "active": true,
  "uptime": 222.438774375,
  "uptime_str": "3m42.438774667s",
  "config": {
    "input": {
      "azure_cosmosdb": {
        "account_key": "!!!SECRET_SCRUBBED!!!",
        "args_mapping": "root = [\n  { \"Name\": \"@name\", \"Value\": \"benthos\" },\n]",
        "auto_replay_nacks": true,
        "batch_count": -1,
        "connection_string": "!!!SECRET_SCRUBBED!!!",
        "container": "testcontainer",
        "database": "testdb",
        "endpoint": "https://localhost:8081",
        "partition_keys_map": "root = \"blobfish\"",
        "query": "SELECT c.foo FROM testcontainer AS c WHERE c.bar = \"baz\" AND c.timestamp < @timestamp"
      }
    },
    "output": {
      "stdout": {
        "codec": "lines"
      }
    }
  }
}

Both account_key and connection_string should be masked because they are marked as secret fields in the input definations.

[ye11ow]

@Jeffail @mihaitodor Any comments regards this new feature request? If it makes sense, I can work on it 😄

[Manognya-9491]

@ye11ow , I have a similar query regarding masking the secrets. I am trying to POST a config generated to benthos streams but I would prefer to send it via an https call instead of http using cert and key file fields in http_server input

[ye11ow]

I just realized that benthos used to mask the secret before. If I read the code correctly, this feature was removed in this commit 6435bb4#diff-ac44c8bf4b1d0d06feb56fbca87b0f82d82d809c9879e50b93a2469be9bf96e8L370 (since v4.25)

The Sanitised method was removed and it just called info.Config().GetRawSource() instead. Is this on purpose? Thanks!

cc @Manognya-9491 @Jeffail @mihaitodor

[Jeffail]

Hey @ye11ow, it's unintentional and probably because when the mechanism for extracting the source config as a structured value changed there was a blanket refactor. I believe there are accessible APIs for re-doing the sanitation on that data that we need to re-introduce, it'll look slightly different to the old code but functionally will be the same.

[sarwarbhuiyan]

Did we get to the bottom of the existing feature that had been removed?