LLD:SpeculosTesting
- Introduction
- Testing strategy
- Configuration and execution
- Reporting
- Workflow
- Contribution
- Limitations
This project is maintained by the QAA squad. Feel free to reach out if you have questions, thanks in advance for your contribution!
To enhance our testing framework and better align with user experience, we decided to create a new test suite leveraging Speculos.
Speculos is an emulator for Nano apps, allowing interaction with the blockchain, including verifying addresses, sending, swapping, and more.
Currently, we maintain over 70 UI end-to-end test scenarios executed using Speculos in conjunction with Playwright.
Spawning Speculos: For each cryptocurrency (coin), a dedicated Speculos instance is spawned. This instance emulates the specific Nano app needed for the tests.
Running Tests: The test scenarios are then executed, interacting with the Speculos instance to simulate real blockchain operations.
Terminating Speculos: Once the tests are completed, the Speculos instance is terminated to free up resources.
This strategy allows us to remove mocks and conduct tests that closely mirror real user interactions, providing more reliable test results.
To optimize the test execution time, especially during CI runs, we employ a sharding strategy. This involves splitting the test suite into smaller shards, which are run in parallel across multiple instances. This approach reduces the overall execution time and ensures faster feedback. Today we are using 3 shards.
A nightly run of the full test suite is scheduled to ensure ongoing stability. The result of this run is visible in the #live-repo-health slack channel.
You can also execute Speculos tests in the CI using the @Desktop - UI e2e - Test App workflow.
The workflow use specific inputs that can be configured before execution:
Input 1: Send notification on slack channel
Input 2: Send results to Xray:
Imput 3: Enable Broadcast input:
In case of test failures, the following steps should be taken to investigate:
Check Screenshots on allure report: Screenshots are automatically captured at the point of failure and can provide immediate insight into the issue.
Review Logs: Logs generated during test execution can help identify the root cause of the failure.
Rerun the Test: If needed, rerun the specific failed test in isolation to reproduce and analyze the issue.
All test results are collected and reported through Allure. The Allure dashboard provides detailed reports on each test case, including steps executed, screenshots, and logs. This enables easy tracking of test performance and quick identification of issues.
Test results are also integrated with Slack for real-time notifications. The results are posted in the #live-repo-health channel, including:
A direct link to the Allure report for detailed insights.
A link to the CI run where the tests were executed.
This ensures that the members of the channel are immediately informed of any issues that arise during test execution.
To send test results to Xray, ensure that the "Send tests results to Xray" option is checked in the workflow input fields. Additionally, you must provide a valid Jira Test Execution ticket ID in the "Test Execution ticket ID" field.
We welcome contributions to this project. Whether it's adding new test scenarios, or improving existing ones, your contributions are highly appreciated. Please follow the standard contribution guidelines and ensure all changes are tested thoroughly before submitting a pull request.
While this testing strategy offers significant benefits, there are some limitations to be aware of:
The primary limitation is that Speculos can only emulate the applications (apps) running on Nano devices, not the underlying operating system (OS) of the devices themselves. As a result, certain functionalities of Ledger Live cannot be fully covered by Speculos-based tests. For example:
- App Installation: It is not possible to simulate the downloading and installation of apps onto the Nano device.
- My Ledger Section: Operations that involve interacting with the "My Ledger" section, such as managing apps on the device, cannot be tested using Speculos.
We continue to explore ways to mitigate these limitations and enhance the overall coverage and reliability of our test suite.