Documents for setup CI lab physically in the US

[freedomtan]

1. host machines and all the supported devices...
2. air-conditioned?
3. manually intervention??

[freedomtan]

how "long" we are going to support older devices? we need to define it.

[anhappdev]

I think we will have many same questions as in the case of building a local server vs. using a cloud server?
But in the case of mobile device testing, we will have more issue to deal with since the software stack for it is not as mature as for server.

Hardware side:

• Stable power
• Stable internet connection
• Secure and safe place

Software side:

• How to deal with app crashes?
• How to deal with OS crashes?
• How to deal with OS update?
• How to deal with authentication / authorization?
• How to monitor or view test results and test logs?
• How to queue multiple tests, cancel tests?
• etc.

Most of the software issues may be solved with a framework for automated mobile testing. I did an internet search and found only this one http://appium.io but have no experience with it.

[freedomtan]

@anhappdev to add more descriptions, such current GitHub Workflow, here.

[anhappdev]

The current state of our CI/CD:

• On every new push to this GitHub repo, multiple workflows will be triggered via GitHub Actions to run different tests.
• One of the workflows will build the Android APK and send it to Firebase Test Lab to run integration tests on physical devices (currently Pixel 5 and 6). Firebase Test Lab will report back whether the tests passed or not.
• An APK with all backends can also be downloaded directly from GitHub on the Actions summary page.

---

It would be best if the new CI lab can be integrated into the current workflow. That means:

1. On every new push to this GitHub repo, an APK will be sent to the CI lab
2. The CI lab runs the tests on physical devices and report back the results (similar to how Firebase Test Lab works)

[nathanw-mlc]

Would the CI/CD results showing up in this public repo violate the terms of the CI/CD agreement with Arm, which states: "Arm will not share the results outside of MLCommons and Mobile Working Group."

[anhappdev]

Would the CI/CD results showing up in this public repo violate the terms of the CI/CD agreement with Arm, which states: "Arm will not share the results outside of MLCommons and Mobile Working Group."

In that case, the detailed results itself can be accessed on another private website. The Arm system just reports to the GitHub workflow whether the tests run successfully or not. This is also how Firebase Test Lab works.