PoC summary: results, issues found, next steps

[gctucker]

Once all the proof-of-concept phase is complete, we should put some comments here and maybe write a blog post about how it went: what it is doing now (running tests, producing results), which issues we've found and what the next steps might be to enable this in a production environment.

• Comments added to this issue
• If applicable, blog post on https://kernelci.org/blog/
• If applicable, email on [email protected] (or [email protected])

[gctucker]

Co-written with @alexandrasp:

• What is fstests on KernelCI doing now?
  • Running KVM tests directly on the kernelci2 server
  • Communicating with the staging API
  • Running "smoke" tests for every new kernel revision
  • The smoke tests are from xfstests-bld (ext4 and generic tests with fixed parameters)
  • Parsing the XML results and submitting them to the API
  • Results appear in email reports later on
• Limitations of the current PoC setup
  • Errors from the logs / results can't be sent to the API
• Any outstanding issues?
  • Top-level result should be "fail" if any test case failed, right now it's left as None
  • Some test cases are failing, issue reported to xfstests-bld (problem with "touch" command?)
    • Ted says the tests are passing for him
      Touch is generating unexpected output  tytso/xfstests-bld#34
• Next steps?
  • Adding test results for the kernel build steps
  • Making this production-ready
    • We need to be able to run VMs in parallel to scale
    • Typically we should consider using gce-xfstests to run the same tests in GCE rather than KVM
    • Add as a docker-compose service since we can in principle communicate with the API from a container
    • Discuss with Ted (and other maintainers) how this should be used with their workflow
      • Which branches? (LTS, next, fs subsystem specific branches etc.)
      • Which configurations? (fs types, block sizes, extra features etc.)
      • Which tests other than just the "smoke" tests?
      • Discuss example email report
  • Maybe later we could run on real hardware with LAVA, but that's another story

[gctucker]

Work on fstests has stopped so closing the issue now. New issues can be created if / when this work gets resumed in the future.