How to pass test results to TestRail block wise from single test in cypress

[Shivani26Singh]

My cypress test is linked to multiple TestRail test cases. But if my test case fails in cypress, all my test results got failed. But in actual it's due to only one validation which is failing in cypress and not all. So is there any way, we can assign test case id block-wise in cypress test case, so that if validation passes then corresponding to that test case id gets passed in testrail & if any validation fails in the same cypress IT block, then test case id gets failed in testrail .

[boxblinkracer]

Hi there

aehm...I think I know what you mean, but it could also be that I might be confused :)

so you have something like this?

it('C1 C54 C33: this is my test() ....

it('C88: this is another test() ...

and then the first one fails...then C1, C54 and C33 should be marked as failed and C88 as passed?
maybe you can write a sample?

i think i dont get the block-wise thing :)

thank you

[tina-zeng]

Could it be caused by multiple it() containing the same case ID?

Have a notice saying "TestRun updated in TestRail: XXX " after each test file has run:

But after all, there is an error saying: "make: *** [cy-qa] Error 2", it is hard to tell what happened

And there aren't any tests in the test run.

[tina-zeng]

Could it be caused by multiple it() containing the same case ID?

Have a notice saying "TestRun updated in TestRail: XXX " after each test file has run: But after all, there is an error saying: "make: *** [cy-qa] Error 2", it is hard to tell what happened And there aren't any tests in the test run.

It's weird... Only run one case and the case passed, but there are no tests added to the run, seems it is not able to update the test result at all. I'm new to this tool.

[boxblinkracer]

Hi there

ok thank you for this
now I get it
so it says the run is updated, but not test cases are added to it
the error above of the makefile might just be the result of cypress, which returns as "failed", which is good..because tests are red. (i think make cy-ca)...so we can ignore this one

then let me reproduce it. maybe you can help me with the steps:

• i create a configuration with the mode to create new runs in testrail
• start a single green/passing test
• intgration shows testrun is updated, but no tests exist in testrail?

is that correct?
thank you

[Shivani26Singh]

No this is not my case. Problem I am facing is if it() block fails, then all three test cases C1 C54 C33 gets failed. Now in my case I have three validations in it() block corresponding to each test case id (that's why I linked three test cases of testrail with single cypress test.) & my 2 validations got passed and 1 got failed. So I want C1 and C54 should get passed and C33 should get failed. Is there any way to achieve this? Thanks, Shivani

…

On Tue, 10 Jan, 2023, 9:22 pm Christian, ***@***.***> wrote: Hi there aehm...I think I know what you mean, but it could also be that I might be confused :) so you have something like this? it('C1 C54 C33: this is my test() .... it('C88: this is another test() ... and then the first one fails...then C1, C54 and C33 should be marked as failed and C88 as passed? maybe you can write a sample? thank you — Reply to this email directly, view it on GitHub <#15 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD4UCU6NVEGWZQKOIARPGOTWRWAVRANCNFSM6AAAAAATW6PFVQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[boxblinkracer]

Hi
could it be that you have just 1 single cypress test with 3 validations IN it?

could you share your code? maybe anonymize it?

then I'm quite sure I get it :D
sorry

[tina-zeng]

Hi there

ok thank you for this now I get it so it says the run is updated, but not test cases are added to it the error above of the makefile might just be the result of cypress, which returns as "failed", which is good..because tests are red. (i think make cy-ca)...so we can ignore this one

then let me reproduce it. maybe you can help me with the steps:

• i create a configuration with the mode to create new runs in testrail
• start a single green/passing test
• intgration shows testrun is updated, but no tests exist in testrail?

is that correct? thank you

Yes, the same steps you said above. And I think my question might be a user experience issue:
I've found the root cause by calling testRail API directly and returning an error: 'Field :results contains one or more invalid results (case XXXX unknown or not part of the test run)' and the issue does not exist anymore.

[boxblinkracer]

aaaaaaa, now we have it
so because of the multiple Case IDs being assigned, the integration does try to add all 3 to the test run.
so either there is a bug that not all are assigned (could be) or not all case ids are existing in the TestRail instance?

so would all 3 case IDs exist in your project? then theres just the bug that i accidentally only add 1, which means later on when sending results, these are missing?

[tina-zeng]

As a user I would like to make some comments：

1. It should be able to handle these scenarios(The design of the case is not friendly to automated testing, but I have to map all the tests)：
   // multiple
   it(${ID1}:)
   it(${ID1}:)
   // single
   it(${ID2}:)
   // merged multiple/ single
   it(${ID3}:)
   it(${ID3},${ID4}:)
2. It should be able to run cases from specific sections in the same project(The team set all the tests in the same project, but those cases are actually written in 3 git repositories, so I have to run specific sections of the project for each git repository, this put me off using this tool for now)
3. All excepted tests should be added to the run when the run is created.
4. It should be able to handle unexpected failures and output the error message with the root cause reason.

[boxblinkracer]

Hi

to your questions

1. It's possible to all these scenarios
   1.1 you can have multiple tests with the same ID..then it would send 2 different tests in cyperss to the same test in TestRail...meaning 1 run gets 2 results...but does that make sense?

it('C1: ...)
it('C1: ...)

1.2 you can have a single assignment

it('C1: ...)

1.3 you can have multiple assignments
if more cases are mapped, then 1 result in Cypress will send a result to all mapped cases in TestRail.
in this case the result of this test leads to 3 results (for 3 cases) in TestRail

it('C1 C5 C6: ...)

2. running filtered tests and sections is not part of this framework. the integration is only for sending data.
   but i have also such a use cases (this is why i've created my cypress-filter repository...there are also others out there).
   then you can filter what suite to run.
   the integration itself will only send data from the tests that were executed.

and then if you send it against an existing run, the cases need to be part of it in testrail

...or if you create it dynamically, only the ones being executed will be added in testrail first, and then the result comes

3. this is happening. the only thing that might be a bug is, that I think it can be that when you have 1 cypress test with multiple case IDs + the config to create runs in TestRail, then not all 3 might be added in that run.

I think I forgot to test this. so this is actually my question from above, if that might be the problem?

4. not quite sure about that? TestRail API responses should already show what went wrong. if not, then there might be some improvement needed. one of the previous versions did already add new error handlers.
   but if you talk about unexpected failures inside cypress, then this is not part of this framework

i hope this explains it.
could you please give me either a "happening or not happening" regarding 3. from the list above?
because that might be a bug that I could think of

[tina-zeng]

Thanks for the clarification. Sorry for confusing you.
for #1 Actually, due to the division of labor, it is currently impossible for each case to have its own independent ID.
for #3 happening
for #4 Yes, I am talking about unexpected failures inside cypress

[boxblinkracer]

Hi there
ok so

1. if they have the same ID, then its sent multiple times, i cannot do something else. do you have an idea? or what would you think should happen with both C1 tests?
2. thx ill create a bug. ticket
3. maybe this helps you. you can hook into unexpected exception and avoid cypress being cancelled. https://github.com/mollie/Shopware6/blob/master/tests/Cypress/cypress/support/e2e.js