bug: Test failure in TestZeto_AnonNullifierBatch

[awrichar]

What happened?

Failure in TestZeto_AnonNullifierBatch

https://github.com/LF-Decentralized-Trust-labs/paladin/actions/runs/11726373055/job/32666690148?pr=406

2024-11-07T16:18:12.4693728Z time="2024-11-07T16:18:12.385Z" level=info msg="RPC[000000033] <-- pstate_queryContractNullifiers [200] OK (1.20ms)"
2024-11-07T16:18:12.4694966Z     e2e_test.go:172: 
2024-11-07T16:18:12.4696654Z         	Error Trace:	/home/runner/work/paladin/paladin/domains/zeto/integration-test/e2e_test.go:172
2024-11-07T16:18:12.4699225Z         	            				/home/runner/work/paladin/paladin/domains/zeto/integration-test/e2e_test.go:102
2024-11-07T16:18:12.4700732Z         	Error:      	"[0xc0010435e0 0xc001043650 0xc0010436c0]" should have 2 item(s), but has 3
2024-11-07T16:18:12.4701809Z         	Test:       	TestZetoDomainTestSuite/TestZeto_AnonNullifier
2024-11-07T16:18:12.4702607Z === RUN   TestZetoDomainTestSuite/TestZeto_AnonNullifierBatch

What did you expect to happen?

Test should pass

How can we reproduce it (as minimally and precisely as possible)?

See logs

Anything else we need to know?

No response

OS version

No response

[awrichar]

Happened again. https://github.com/LF-Decentralized-Trust-labs/paladin/actions/runs/11746268909/job/32725630177?pr=406

I see that the fix was adding a sleep... is there any way we can more definitively wait for the right result, vs sleeping and polling?

[jimthematrix]

Double checked that the true at the end of the parameters for the testbed_invoke call should get the rpc action to wait until all the event processing has been completed, which means after the return the new states and spent/confirmed states should all be committed to the states storage. So I don't have a theory on why sometimes the subsequent query failed to return the expected results. But added a commit in this PR #418 to turn on DB debug for the zeto e2e run, to see if we can get additional insight from the DB SQL command logs