fix: telemetry spans

[avilagaston9]

Fix Telemetry Spans

Motivation

We found that sometimes our Batcher tries to cancel batches that were actually included in the net, calling the batcherTaskCreationFailed endpoint, which finalizes the trace and prevents the Aggregator from registering its spans in the trace.

Description

• Stops finalizing the trace when a batcherTaskCreationFailed occurs.

Observations

On a real batcherTaskCreationFailed, the Aggregator won't receive the new task, and the trace will remain unfinished. Furthermore, the trace metadata won't be removed from the Telemetry server store. Despite that, we will be able to visualize the orphans spans with a warning that their parent ID is invalid.

#1477 was created to address this issue.

How To Test

1. Check that everything is working normally:

Run anvil, all Aligned components with one or more operators and start telemetry:

make telemetry_full_start

Go to jaeger and explore the generated traces:

2. Test the scenario addressed in this PR:

Change the Batcher create_new_task_retryable function in batcher/aligned-batcher/src/retry/batcher_retryables.rs:165 to return an error after receiving the receipt:

 // timeout to prevent a deadlock while waiting for the transaction to be included in a block.
    let _result = timeout(Duration::from_millis(transaction_wait_timeout), pending_tx)
        .await
        .map_err(|e| {
            warn!("Error while waiting for batch inclusion: {e}");
            RetryError::Permanent(BatcherError::ReceiptNotFoundError)
        })?
        .map_err(|e| {
            warn!("Error while waiting for batch inclusion: {e}");
            RetryError::Permanent(BatcherError::ReceiptNotFoundError)
        })?
        .ok_or(RetryError::Permanent(BatcherError::ReceiptNotFoundError));
    Err(RetryError::Permanent(BatcherError::ReceiptNotFoundError))

Then, start all components again and you should be able to see the Aggregator spans even when the Batcher sends Batcher - Task Creation Failed

3. Test a real cancel scenario:

Remove the hole content of the Batcher create_new_task_retryable function in batcher/aligned-batcher/src/retry/batcher_retryables.rs:105 and return an error without creating any task:

pub async fn create_new_task_retryable(
    batch_merkle_root: [u8; 32],
    batch_data_pointer: String,
    proofs_submitters: Vec<Address>,
    fee_params: CreateNewTaskFeeParams,
    transaction_wait_timeout: u64,
    payment_service: &BatcherPaymentService,
    payment_service_fallback: &BatcherPaymentService,
) -> Result<TransactionReceipt, RetryError<BatcherError>> {
    Err(RetryError::Permanent(BatcherError::ReceiptNotFoundError))
}

You should only be able to see the Batcher spans:

Type of change

• Bug fix

Checklist

• “Hotfix” to testnet, everything else to staging
• Linked to Github Issue
• This change depends on code or research by an external entity
  • Acknowledgements were updated to give credit
• Unit tests added
• This change requires new documentation.
  • Documentation has been added/updated.
• This change is an Optimization
  • Benchmarks added/run
• Has a known issue
  • Link to the open issue addressing it
• If your PR changes the Operator compatibility (Ex: Upgrade prover versions)
  • This PR adds compatibility for operator for both versions and do not change batcher/docs/examples
  • This PR updates batcher and docs/examples to the newer version. This requires the operator are already updated to be compatible

[JulianVentura]

Works as expected