Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create unit tests for all Kokkos Tools libraries and utilities #191

Open
vlkale opened this issue Apr 28, 2023 · 1 comment
Open

Create unit tests for all Kokkos Tools libraries and utilities #191

vlkale opened this issue Apr 28, 2023 · 1 comment
Labels
testing Testing for Kokkos Tools functionality (unit tests and integration tests)

Comments

@vlkale
Copy link
Contributor

vlkale commented Apr 28, 2023

Complementing the example directory, a set of unit tests and a framework for testing needs to be created for all Kokkos Tools libraries. This includes in-house tooling libraries, e.g., kernel-logger, vendor connectors, e.g., roctx-connector, and utilities, e.g., Kokkos-sampler, and sophisticated tpls, apex Kokkos Tool library.

In the near-term and of priority: the focus will be on new tools integrated being into the Kokkos Tools repo, in particular the sampler utility and fixes in the nvtx-connnector, which is expected to have heavy use. (The rocprof-connector and vtune-connector should also have similar tests).

Longer-term: more tests, perhaps in part community-driven, will be incorporated in to the set of tests.

This may eventually involve a testing framework like Google Test and the testing would then happen within the new Kokkos Tools make build system and nightly CI builds of Kokkos Tools

@vlkale
Copy link
Contributor Author

vlkale commented May 5, 2023

I propose a process for developing these tests that is directly motivated by Kokkos applications using Kokkos Tools.

For now, let's define a unit as callback of a particular tool. Each unit test should verify (is the answer correct given the function's implementation), validate (is the answer correct based on requirements of function defined), and assess performance. These three aspects give the Kokkos Tools users an assessment of quality of the implementation.

The process for establishing a set of unit tests for the Kokkos Tools would be something like:


1st phase: Obtain simple example codes from applications and benchmarks, and ensure they work. An example of a test of the sampler utility's parallel_for tools callback applied for the simple kernel timer is shown in the file. outputForSampler with Atomic Benchmark

2nd phase: By stripping out functions and code from applications: Create tests of output of tool, with one test per unit, via assertions in code.

3rd phase: Use testing framework like ctest (and could be subsequently using Google Test framework) and integrate the tests under the framework into the Kokkos Tools cmake build system.


The 1st and 2nd phase would be stored in a separate repo and be not part of the Kokkos Tools repo. The code developed in the 3rd phase would be the official tests of the repo, which would be community-built and agreed upon via a community of Kokkos users.

This process could be followed or follow that of tests in Kokkos-core.

@vlkale vlkale added the testing Testing for Kokkos Tools functionality (unit tests and integration tests) label May 5, 2023
@vlkale vlkale changed the title Create unit tests for all Kokkos Tools connectors and utilities Create unit tests for all Kokkos Tools libraries and utilities Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
testing Testing for Kokkos Tools functionality (unit tests and integration tests)
Projects
None yet
Development

No branches or pull requests

1 participant