Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Support for Geekbench 6 Benchmark #5353

Open
wants to merge 13 commits into
base: master
Choose a base branch
from

Conversation

leyli16
Copy link

@leyli16 leyli16 commented Nov 8, 2024

This PR addresses the requirements outlined in [Issue #5238]. The following changes have been implemented:

Summary of Changes:

  1. Benchmarking Framework Setup
  • Created geekbench_benchmark.py files in both linux_benchmarks and windows_benchmarks directories.
  • Defined BENCHMARK_NAME and BENCHMARK_CONFIG constants, allowing PKB to recognize and locate the benchmark.
  • Added GetConfig, Prepare, Run, and Cleanup functions, each containing initial structure with pass/return statements, making the benchmark operational within PKB.
  1. Geekbench Installation Support
  • Developed geekbench.py package within linux_packages and windows_packages for installing Geekbench.
  • Implemented an Install function that takes a vm as input, executing necessary shell commands to install Geekbench.
  • Prioritized installation support for Ubuntu 24.04 and Windows. Added unit tests using mock to ensure the vm.RemoteCommand calls are correctly issued for installation.

61681731014318_ pic

  1. Running and Parsing Geekbench6
  • Manually ran Geekbench6 locally, producing output used for parsing and testing.

  • Stored Geekbench output in data/geekbench as a raw data file for testing the parsing function.

  • Developed ParseResults function in geekbench.py to process Geekbench output. The function produces a list of PKB Samples, with each sample representing a unique metric with:
    = metric_name, metric_value, metric_unit, and metric_metadata.

  • Added unit tests to verify accurate parsing and structured output for each metric from both CPU and GPU tests.
    WeChat963b91264433748770f5b2ad8a97dd76


Update from 11/29:

  1. Moved Geekbench test files to appropriate directories for better organization.
  2. Updated Prepare() to install Geekbench (vm.install('geekbench')).
  3. Added a TODO in Run() to trigger Geekbench execution.
  4. Logged informative messages for parsing failures in ParseResults while continuing execution.
  5. Refactored ParseResults() to use sample.Sample for consistent output.
Example Output:
        [
            Sample(
                metric="Single-Core Score",
                value=1795,
                unit="points",
                metadata={"category": "Single-Core"},
                timestamp=1699815932.123
            ),
            Sample(
                metric="Single-Core File Compression",
                value=269.3,
                unit="MB/sec",
                metadata={
                    "category": "Single-Core",
                    "test": "File Compression",
                    "score": 1875
                },
                timestamp=1699815932.123
            )
        ]
  1. Renamed references to geekbench_benchmark for naming consistency.

@leyli16 leyli16 changed the title Pull Request for Issue #5238: Add Support for Geekbench 6 Benchmark Add Support for Geekbench 6 Benchmark Nov 8, 2024
Copy link
Contributor

@jellyfishcake jellyfishcake left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice! Some small nits and questions.

@jellyfishcake
Copy link
Contributor

Hi leyli16, it has came to my attention that we already have a geekbench6 linux implementation, you cannot see it because it is not externalized here on github.
I am really sorry about the confusion. I can externalize our existing implementation so that we can build upon it to add the windows implementation.
Happy to work with you to contribute towards a benchmark that does not have a conflict. Feel free to reach out.

@leyli16
Copy link
Author

leyli16 commented Nov 29, 2024

Hi leyli16, it has came to my attention that we already have a geekbench6 linux implementation, you cannot see it because it is not externalized here on github.
I am really sorry about the confusion. I can externalize our existing implementation so that we can build upon it to add the windows implementation.
Happy to work with you to contribute towards a benchmark that does not have a conflict. Feel free to reach out.

Thank you for the clarification. I’ve already addressed the Windows implementation, and I’d be happy to collaborate further to ensure there’s no conflict. Let me know how I can assist!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants