Assertions do not lead to error in latest version of wdio-ui5-service

[FrankVisuals]

Describe the bug

We are currently using version 1.0.3 of wdio-ui5-service in our webdriver tests. For writing tests we are currently using the fe-testlib (https://ui5-community.github.io/wdi5/#/fe-testlib).

We recently tried to update our wdio-ui5-service version to 1.5.1. However, suddenly the assertions done in our facades do no longer lead to errors.

Here's an example test:

it("should open the app and check the page", async () => {
        await FioriElementsFacade.execute((_Given, When, Then) => {
            Then.onMyListReport.iSeeThisPage();
            Then.onMyObjectPage.iSeeThisPage();
        });

        expect(true).toBe(false);
});

Obviously, the call in the facade should fail - as it cannot be on both pages.

However, the error that shows is

Which is executed after the assertions that are done. If I remove the expect(true).toBe(false) assertion, the test runs green, which is also wrong.

If I roll back to 1.0.3 I see an expected error.

To Reproduce

Sadly this is a private repository, so I cannot share it.

Expected behavior

This code:

await FioriElementsFacade.execute((_Given, When, Then) => {
  Then.onMyListReport.iSeeThisPage();
  Then.onMyObjectPage.iSeeThisPage();
})

should lead to an error in the execution, log it accordingly and should let the test fail.

Logs/Console Output

wdi5: {
        logLevel: "verbose"
},

[0-0] [wdi5] successfully initialized wdio-ui5 bridge
[0-0] [wdi5] [test library] Seeing the page 'com.sap.<MYAPP>::PackagesList' - OK
[0-0] Error in "<MYAPP>"
Error: expect(received).toBe(expected) // Object.is equality

Expected: false
Received: true
    at /Users/MYAPP/my.test.ts:39:22
    at Generator.next (<anonymous>)
    at fulfilled (/Users/MYAPP/my.test.ts:5:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

if applicable, please copypasta code-fenced log output, e.g.

Runtime Env (please complete the following information):

• wdi5/wdio-ui5-service version: 1.5.2
• UI5 version: 1.108.4
• wdio-version (output of wdio --version): ^7.32.0
• node-version (output of node --version): 18.16.1
• OS: MacOS
• Browser + Version: Chrome 114.0.5735.198

    "@wdio/cli": "^7.20.5",
    "@wdio/devtools-service": "^7.20.0",
    "@wdio/local-runner": "^7.20.5",
    "@wdio/mocha-framework": "^7.20.3",
    "@wdio/sauce-service": "^7.20.5",
    "@wdio/selenium-standalone-service": "^7.20.3",
    "@wdio/spec-reporter": "^7.20.3",
    "@wdio/types": "^7.20.3",
    "wdio-ui5-service": "1.5.2",
    "webdriverio": "^7.32.0",

Additional context

[dominikfeininger]

Can you please help to track this issue down to root cause and find the exact version of wdio-ui5-service where this issue first occurred.

[FrankVisuals]

Sure! I just checked it - as far as I can see there was no release between version 1.4.0 and 1.5.0 - with version 1.4.0 everything works fine. The problem start occurring when consuming version 1.5.0

EDIT: I saw that there is a version 2.0.0-alpha.1 - the error also occurs in this version (just wanted to check if it might have been fixed there)

[FrankVisuals]

I'm pretty sure #506 will not fix this issue. The problem is not the message itself, but the test not stopping & failing when the assertion is wrong. The test should have stopped after

await FioriElementsFacade.execute((_Given, When, Then) => {
  Then.onMyListReport.iSeeThisPage();
  Then.onMyObjectPage.iSeeThisPage();
});

but it only fails after

expect(true).toBe(false);

Meaning that the first statement does not seem to execute the assertion as expected.

[Siolto]

Yes you are right, this PR will only improve the error messages. When I try to reproduce your issue with our tests and changed our first it step to:

it.only("should trigger search on ListReport page", async () => {
        await FioriElementsFacade.execute((Given, When, Then) => {
            Then.onTheMainPage.iSeeThisPage()
            Then.onTheDetailPage.iSeeThisPage()
        })
    })

an Error is correctly thrown. Only the error message is a little cumbersome which #506 will solve. Could you maybe try to reproduce this issue with our tests?

Thanks in advance!

[github-actions]

hey 👋 - silence for 30 days 🤐 ... anybody? 😀

[github-actions]

closed 📴 because silencio 🤫 since an additional 14 days after staleness 📠

[FrankVisuals]

Hey @Siolto, I apparently forgot to keep up with this issue. Sadly I am now back at writing wdi5/webdriver tests using the Fiori Test Facade and I am again facing these issues.

I'm using the latest versions of wdio and the wdio-ui5-service

"wdio-ui5-service": "2.0.2",
"webdriverio": "8.20.0"

You can also ping me directly so I can link the repository this is happening in.