Bug at startup where FireFly gives up creating listeners if cannot reach transaction manager

[EnriqueL8]

As part of the startup flow of FireFly, it will reconcile the intent from its database and call the relevant plugins. For example, it will make sure to activate the toke pools (listen to token pool events) or make sure the contract listeners are created in the transaction manager layer. The latter is the source of this bug and the the listener will never be created against the transaction manager. The worst part is that is fails silently, only when you retrieve the listener then in the status you will see that it cannot find the listener in the blockchain connector as such:

 "status": {
    "error": "FF10111: Error from ethereum connector: {\"error\":\"FF21046: Event listener '0190119e-0627-9252-799e-0c02208b939c' not found\"}"
  }

This will cause FireFly to never listen to those events and not let the user know that an issue has occurred. It's a bug in the way the Contract Manager handles errors from the transaction manager

[peterbroadhurst]

This will cause FireFly to never listen to those events and not let the user know that an issue has occurred. It's a bug in the way the Contract Manager handles errors from the transaction manager

I'm not convinced this is a full diagnosis here @EnriqueL8

If you have this level of detail worked out, I think logs of the success and failure cases would be helpful

[peterbroadhurst]

Note that I expect the testing, code changes and investigation of this one to overlap with the changes under discussion in #1531

[EnriqueL8]

Managed to replicate this one manually but unsure how this could occur at startup.

To replicate it, you can simply delete the listener in the blockchain connector and then query the listener on the FireFly API under /contract/listeners/<id> as part of that flow it will go and fetch the latest checkpoint information from the blockchain connector. If it cannot find it it will just give you:

 "status": {
    "error": "FF10111: Error from ethereum connector: {\"error\":\"FF21046: Event listener '01905405-6c96-0afe-9189-2796811ff49a' not found\"}"
  }

I does make me think that if we do not find the listener in the connector at query time we should created it.

If I delete the eventstream you get the same error, so I'm going to do a test of upgrading to 1.3 from 1.2.2 to see if I can replicate this issue

[peterbroadhurst]

Thanks @EnriqueL8 - going to take another look my side to understand the code better.

My naive reading of the code suggested that if err != nil in the below block, we would return err rather than dropping down to cm.blockchain.AddContractListener.

I'd coupled that to the "status": return above, because that also calls GetContractListenerStatus

firefly/internal/contracts/manager.go

Lines 797 to 802 in 16aec0b

	func (cm *contractManager) checkContractListenerExists(ctx context.Context, listener *core.ContractListener) error {
	found, _, _, err := cm.blockchain.GetContractListenerStatus(ctx, listener.Namespace, listener.BackendID, true)
	if err != nil {
	log.L(ctx).Errorf("Validating listener %s:%s (BackendID=%s) failed: %s", listener.Signature, listener.ID, listener.BackendID, err)
	return err
	}

[peterbroadhurst]

Ok - so the thing I missed here is the true being passed down for okNotFound, so the FF10111 error isn't actually passed to this code.

So I don't think we have a concrete issue to investigate there currently