aio rt trigger create fails to create a trigger with a feed from /whisk.system/alarms package

[dominik-kornas-wttech]

Describe the bug
Creating a trigger that has a feed (any of the alarms package feeds) fails with a Feed name is not valid error.

Example command failing with an error

$ aio rt trigger create my-trigger -f /whisk.system/alarms/once -p date "2022-10-26T11:00:00.000Z" -v
2022-10-26T10:25:23.920Z aio-cli-plugin-runtime {
  apiversion: 'v1',
  apihost: 'https://adobeioruntime.net',
  namespace: '******************',
  api_key: ''******************'
}
2022-10-26T10:25:23.920Z @adobe/aio-lib-runtime:RuntimeAPI:debug init options: {
  "apiversion": "v1",
  "apihost": "https://adobeioruntime.net",
  "namespace": "'******************",
  "api_key": "'******************"
}
2022-10-26T10:25:23.922Z @adobe/aio-lib-runtime:RuntimeAPI:debug proxy settings not found
2022-10-26T10:25:25.344Z aio-cli-plugin-runtime OpenWhiskError: PUT https://adobeioruntime.net/api/v1/namespaces/******************/triggers/my-trigger Returned HTTP 400 (Bad Request) --> "Feed name is not valid"
    at (...)
 »   Error: failed to create the trigger: feed name is not valid (400 Bad
 »   Request)
 »    specify --verbose flag for more information

namespace and APi key obfuscated

This looks like related to a leading slash, as the same command executed with wsk cli works:

$ wsk trigger create my-trigger -f whisk.system/alarms/once -p date "2022-10-26T11:00:00.000Z"
ok: invoked /whisk.system/alarms/once with id 3de991f748c94834a991f748c9783431
{
    "actionHost": "10.216.0.50",
    "activationId": "3de991f748c94834a991f748c9783431",
    "annotations": [
        {
            "key": "path",
            "value": "whisk.system/alarms/once"
        },
        {
            "key": "waitTime",
            "value": 128
        },
        {
            "key": "kind",
            "value": "nodejs:10"
        },
        {
            "key": "timeout",
            "value": false
        },
        {
            "key": "limits",
            "value": {
                "concurrency": 1,
                "logs": 10,
                "memory": 256,
                "timeout": 60000
            }
        },
        {
            "key": "initTime",
            "value": 158
        }
    ],
    "duration": 2277,
    "end": 1666781688181,
    "invokerInstanceId": {
        "instance": 4,
        "instanceType": "invoker",
        "uniqueName": "rt-invoker-4",
        "userMemory": "1500000000000000 B"
    },
    "logs": [],
    "name": "once",
    "namespace": "******************",
    "podName": "wskrt-invoker-44-370593-prewarm-nodejs10",
    "publish": false,
    "response": {
        "result": {
            "status": "success"
        },
        "size": 20,
        "status": "success",
        "success": true
    },
    "start": 1666781685904,
    "subject": "******************",
    "version": "0.0.245"
}
ok: created trigger my-trigger

Trying to use aio cli trigger create without the leading slash generates another error The requested resource does not exist

$ aio rt trigger create my-trigger -f whisk.system/alarms/once -p date "2022-10-26T11:00:00.000Z" -v
(...)
2022-10-26T10:40:22.974Z @adobe/aio-lib-runtime:RuntimeAPI:debug proxy settings not found
2022-10-26T10:40:26.448Z aio-cli-plugin-runtime OpenWhiskError: POST https://adobeioruntime.net/api/v1/namespaces/******************/actions/alarms/once Returned HTTP 404 (Not Found) --> "The requested resource does not exist."
    at (...)
 »   Error: failed to create the trigger: the requested resource does not
 »   exist. (404 Not Found)
 »    specify --verbose flag for more information

Note that in addition the working command does not align with the documentation which instructs to provide leading slash.
Documentation examples:

• creating trigger using wsk cli
• creating trigger using app builder manifest file

Expected behavior
The aio cli trigger create works as documented.
The App Builder app deployment (aio app run) works as well and allows to create triggers along actions during deployment.

Versions:

• OS: Windows 10 Pro
• aio cli: 9.1.0
• wsk cli: 1.2.0-windows-386
• node: 14.17.0

[aiojbot]

JIRA issue created: https://jira.corp.adobe.com/browse/ACNA-1851

[shazron]

Thank you for the report, we'll investigate. The leading slash is the correct way to reference the alarm package AFAIK.

[shazron]

Our daily e2e tests for aio-lib-runtime passed when creating a feed, but only tests for /whisk.system/alarms/alarm. Will continue probing and add a test for /whisk.system/alarms/once

[shazron]

Our e2e tests were run 12 hours ago, I just ran them manually and there are 2 failing tests with triggers and feeds - so it might be a service issue, not a cli issue. Will update this thread.

[dominik-kornas-wttech]

Thank you for quick response and looking into this! Looking forward to seeing results of your investigation.

[dominik-kornas-wttech]

If I may add, there is an odd behavior that I observed, which may be related to this issue.

When I accidentally tried to create a trigger with a date in the past, I have received an error (as expected) informing me about that. But it also resulted in one more error, with no details in the returned json (see below). Just a plain app error.

Since then, all triggers stopped firing as they were scheduled. I can still fire them manually from the command line, but they are not triggered on scheduled time or interval. Like some internal scheduler/cron has been broken by my attempt to set a trigger in the past.
What's also important, that issue fixes itself overnight (I've experienced that twice), presumably by some runtime maintenance that (I guess) restarts something in the service.

Details

This is how it appears on the activation list as:

 Datetime        Status    Kind      Version    Activation ID                    Start Wait Init Duration Entity
 ─────────────── ───────── ───────── ──────── ─ ──────────────────────────────── ───── ──── ──── ──────── ───────────────────────────────
 10/26 15:50:19  app error nodejs:10 0.0.245    fc47163f11a1433087163f11a1f330e5 warm  25   0    1001ms   whisk.system/alarms/once
 10/26 15:50:19  app error nodejs:10 0.0.245    d3114b406618461b914b406618b61bf7 warm  50   0    70ms     whisk.system/alarms/once

$ aio rt activation get d3114b406618461b914b406618b61bf7
{
  "actionHost": "10.216.0.31",
  "activationId": "d3114b406618461b914b406618b61bf7",
  "annotations": [
    {
      "key": "path",
      "value": "whisk.system/alarms/once"
    },
    {
      "key": "waitTime",
      "value": 50
    },
    {
      "key": "kind",
      "value": "nodejs:10"
    },
    {
      "key": "timeout",
      "value": false
    },
    {
      "key": "limits",
      "value": {
        "concurrency": 1,
        "logs": 10,
        "memory": 256,
        "timeout": 60000
      }
    }
  ],
  "duration": 70,
  "end": 1666792219086,
  "invokerInstanceId": {
    "instance": 4,
    "instanceType": "invoker",
    "uniqueName": "rt-invoker-4",
    "userMemory": "1500000000000000 B"
  },
  "logs": [],
  "name": "once",
  "namespace": "***************",
  "podName": "wskrt-invoker-44-376762-prewarm-nodejs10",
  "publish": false,
  "response": {
    "result": {
      "error": {
        "error": "date parameter '2022-10-26T13:16:00.000Z' must be in the future"
      }
    },
    "size": 85,
    "status": "application error",
    "success": false
  },
  "start": 1666792219016,
  "subject": "***************",
  "version": "0.0.245"
}

$ aio rt activation get fc47163f11a1433087163f11a1f330e5
{
  "actionHost": "10.216.0.31",
  "activationId": "fc47163f11a1433087163f11a1f330e5",
  "annotations": [
    {
      "key": "path",
      "value": "whisk.system/alarms/once"
    },
    {
      "key": "waitTime",
      "value": 25
    },
    {
      "key": "kind",
      "value": "nodejs:10"
    },
    {
      "key": "timeout",
      "value": false
    },
    {
      "key": "limits",
      "value": {
        "concurrency": 1,
        "logs": 10,
        "memory": 256,
        "timeout": 60000
      }
    }
  ],
  "duration": 1001,
  "end": 1666792220317,
  "invokerInstanceId": {
    "instance": 4,
    "instanceType": "invoker",
    "uniqueName": "rt-invoker-4",
    "userMemory": "1500000000000000 B"
  },
  "logs": [],
  "name": "once",
  "namespace": "***************",
  "podName": "wskrt-invoker-44-376762-prewarm-nodejs10",
  "publish": false,
  "response": {
    "result": {
      "error": {}
    },
    "size": 12,
    "status": "application error",
    "success": false
  },
  "start": 1666792219316,
  "subject": "***************",
  "version": "0.0.245"
}

[shazron]

The service issue turned out to be a red-herring. I tried creating an alarm via the aio cli and it was successful just now.

The next step is to get the value of the x-request-id header of the failing call.

• macOS/Linux: Prefix DEBUG=* before your command, then run it
• Windows Powershell: set: $env:DEBUG = '*', then run your command

[shazron]

If I may add, there is an odd behavior that I observed, which may be related to this issue.

When I accidentally tried to create a trigger with a date in the past, I have received an error (as expected) informing me about that. But it also resulted in one more error, with no details in the returned json (see below). Just a plain app error.

There are sometimes issues with the alarm provider's state. Right now the best course of action is to delete all rules and triggers and re-create them. Not ideal, but if you want to use a manifest that would be painless.

[dominik-kornas-wttech]

Hi @shazron, your mention of Windows Powershell actually struck me with the obvious question about the shell that's being used. I am mostly using git-bash and the failing command was executed from git-bash.
When I tried in Powershell and and then also in CMD it worked properly!

So apparently there is something incorrect with git-bash that I wasn't aware of.

If that's of any help, here's the failing git-bash command with --verbose that includes x-request-id.
I've removed some parts of the output to make it more readable.

$ wsk trigger create my-trigger -f /whisk.system/alarms/once -p date "2023-10-28T13:00:00.000Z" -p deleteAfterFire "rules" --verbose
REQUEST:
[PUT]   https://adobeioruntime.net/api/v1/namespaces/_/triggers/my-trigger?overwrite=false
Req Headers
{
  (...)
  "User-Agent": [
    "OpenWhisk-CLI/1.0 (2021-04-01T23:49:54.523+0000) windows 386"
  ]
}
Req Body
{"name":"my-trigger","annotations":[{"key":"feed","value":"/_/once"}]}

RESPONSE:Got response with code 200
Resp Headers
{
  "X-Azure-Ref": [
    "0QchbYwAAAACXtTcq3m8oS7aWYqgq/sCMV0FXMDFFREdFMDUxNwBiNzdiNjYzMy04ZjcwLTQyNWItOTJiZC03NmFlYmU5YWI5YzU="
  ],
  "X-Cache": [
    "CONFIG_NOCACHE"
  ],
  "X-Request-Id": [
    "OECo2knrvXfcI3z62I9vJnXAUONRvhpH"
  ]
}
Response body size is 200 bytes
Response body received:
{"annotations":[{"key":"feed","value":"/_/once"}],"limits":{},"name":"my-trigger","namespace":"53890-381goldflyingfish-stage","parameters":[],"publish":false,"updated":1666959425782,"version":"0.0.1"}
REQUEST:
[POST]  https://adobeioruntime.net/api/v1/namespaces/_/actions/./C:/Program%20Files/Git/whisk.system/alarms/once?blocking=true&result=false
Req Headers
(...)
Req Body
{"authKey":"********","date":"2023-10-28T13:00:00.000Z","deleteAfterFire":"rules","lifecycleEvent":"CREATE","triggerName":"/_/my-trigger"}

RESPONSE:Got response with code 400
Resp Headers
{
 (...)
  "X-Azure-Ref": [
  "0Q8hbYwAAAAASxLjiWO9ARZ2RO6eChBhaV0FXMDFFREdFMDUxNwBiNzdiNjYzMy04ZjcwLTQyNWItOTJiZC03NmFlYmU5YWI5YzU="
  ],
  "X-Cache": [
    "CONFIG_NOCACHE"
  ],
  "X-Request-Id": [
    "aCCr4BZHJSWLmb9ljXyoMvhQELZeLbvQ"
  ]
}
Response body size is 105 bytes
Response body received:
{"code":"aCCr4BZHJSWLmb9ljXyoMvhQELZeLbvQ","error":"The name of the entity contains illegal characters."}
REQUEST:
[GET]   https://adobeioruntime.net/api/v1/namespaces/_/triggers/my-trigger
Req Headers
(...)
RESPONSE:Got response with code 200
Resp Headers
{
  (...)
  "X-Azure-Ref": [
    "0Q8hbYwAAAAAjVmuQ+KGzS5Urp7XPGnliV0FXMDFFREdFMDUxNwBiNzdiNjYzMy04ZjcwLTQyNWItOTJiZC03NmFlYmU5YWI5YzU="
  ],
  "X-Cache": [
    "CONFIG_NOCACHE"
  ],
  "X-Request-Id": [
    "RTkPcuZQ7WvPgGN8hojo4rpWRCXpkGTJ"
  ]
}
Response body size is 200 bytes
Response body received:
{"annotations":[{"key":"feed","value":"/_/once"}],"limits":{},"name":"my-trigger","namespace":"*********","parameters":[],"publish":false,"updated":1666959425782,"version":"0.0.1"}
REQUEST:
[POST]  https://adobeioruntime.net/api/v1/namespaces/_/actions/once?blocking=true&result=false
Req Headers
(...)
Req Body
{"authKey":"********","date":"2023-10-28T13:00:00.000Z","deleteAfterFire":"rules","lifecycleEvent":"DELETE","triggerName":"/_/my-trigger"}

RESPONSE:Got response with code 404
Resp Headers
{
  (...)
  "X-Azure-Ref": [
    "0Q8hbYwAAAADlRMiIxz7CR5TeqMqGDYnkV0FXMDFFREdFMDUxNwBiNzdiNjYzMy04ZjcwLTQyNWItOTJiZC03NmFlYmU5YWI5YzU="
  ],
  "X-Cache": [
    "CONFIG_NOCACHE"
  ],
  "X-Request-Id": [
    "1Q4vHlh2cvtnXFO0IlZPG5QgbsqDM3da"
  ]
}
Response body size is 92 bytes
Response body received:
{"code":"1Q4vHlh2cvtnXFO0IlZPG5QgbsqDM3da","error":"The requested resource does not exist."}
REQUEST:
[DELETE]        https://adobeioruntime.net/api/v1/namespaces/_/triggers/my-trigger
Req Headers
(...)
RESPONSE:Got response with code 200
Resp Headers
{
 (...)
  "X-Azure-Ref": [
    "0Q8hbYwAAAADVvSKWEwegTacCFNRNvkqSV0FXMDFFREdFMDUxNwBiNzdiNjYzMy04ZjcwLTQyNWItOTJiZC03NmFlYmU5YWI5YzU="
  ],
  "X-Cache": [
    "CONFIG_NOCACHE"
  ],
  "X-Request-Id": [
    "FO2cKNeQriGaVGaWjipZ0rWca1D80I2F"
  ]
}
Response body size is 200 bytes
Response body received:
{"annotations":[{"key":"feed","value":"/_/once"}],"limits":{},"name":"my-trigger","namespace":"********","parameters":[],"publish":false,"updated":1666959425782,"version":"0.0.1"}
ok: deleted trigger my-trigger
error: Unable to create trigger 'my-trigger': Unable to configure feed '/_/once': The name of the entity contains illegal characters. (code aCCr4BZHJSWLmb9ljXyoMvhQELZeLbvQ)

---

There are sometimes issues with the alarm provider's state. Right now the best course of action is to delete all rules and triggers and re-create them. Not ideal, but if you want to use a manifest that would be painless.

I tried exactly this previously without success and had to wait until the next day when it fixed automatically. I will give it another try, though.

[dominik-kornas-wttech]

I found that setting the variable MSYS_NO_PATHCONV=1 (actually to any value) through export MSYS_NO_PATHCONV=1 or simply prepending it before the command, solves the issue and makes both following commands execute properly under git-bash.

MSYS_NO_PATHCONV=1 wsk trigger create my-trigger -f /whisk.system/alarms/once -p date "2023-10-28T13:30:00.000Z" -p deleteAfterFire "rules"
MSYS_NO_PATHCONV=1 aio rt create my-trigger -f /whisk.system/alarms/once -p date "2023-10-28T13:30:00.000Z" -p deleteAfterFire "rules"

So it appears to be an issue caused by my insufficient knowledge of git-bash.1

Running MSYS_NO_PATHCONV=1 aio app run in git-bash deploys all triggers with feeds included in the manifest properly.

---

However the breaking scheduler issue stays. I've created a trigger with date in the past through the manifest and since then all triggers stopped firing. Even when I removed all trigger and rules, and re-created them again.

[shazron]

Oh wow! I should have asked for the output of aio info as well to figure out your shell.
Does double quoting "/whisk.system/alarms/once" work, without the MSYS_NO_PATHCONV=1 env var?

I'll write up this path issue as a separate issue for other users' benefit (and to prime Google search).

However the breaking scheduler issue stays. I've created a trigger with date in the past through the manifest and since then all triggers stopped firing. Even when I removed all trigger and rules, and re-created them again.

Don't have an answer for you yet for this issue.

[shazron]

@dominik-kornas-wttech here's the advice I was given:

We still have the known issue when trying to delete and recreate the triggers of the same name. This requires a lot of rework in the current alarms and is a low priority for now -- the mitigation is to simply use unique names for the triggers to prevent reuse.

[dominik-kornas-wttech]

Hi!
I tried with double quotes, but the results is the same: an error prompt informing about feed name not being a valid one.

And regarding trigger names, I'm sure I tried with different ones and the issue persisted. From my point of view, this isn't really an issue with recreating triggers. The initial attempt to create a trigger fails because of the date in the past. Then, even if I create another trigger with a valid date and a different name, this and any other triggers aren't fired.

Though, I don't know the inner details of the alarms package so it still may be somehow related to the mentioned known issue, just the observation doesn't seem to be that much related.