Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(framework): add config and build capabilities to test framework #11

Merged
merged 9 commits into from
Oct 9, 2023
Merged

feat(framework): add config and build capabilities to test framework #11

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
ae1c283
feat(config): add parser for config.dts file
Diogo21Costa Sep 23, 2023
3d495f3
feat(build): add codegen integration
Diogo21Costa Sep 23, 2023
b1669ff
feat(build): add nix build integration
Diogo21Costa Sep 23, 2023
0f7dff8
feat(deploy): add test launching on target platform
Diogo21Costa Sep 23, 2023
ea5cb33
fix(pylint): rearrange library imports and add R1732 deviation
Diogo21Costa Sep 24, 2023
a497b6c
feat(config): allow custom path for bao-test/src directory
Diogo21Costa Sep 30, 2023
ca284f6
fix(framework): reorganize for cleaner structure
Diogo21Costa Oct 3, 2023
127bbae
update(readme): add instructions on how to run the framework
Diogo21Costa Oct 3, 2023
6d50c84
fix(format): include some format tweaks
Diogo21Costa Oct 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
183 changes: 55 additions & 128 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,137 +1,64 @@
# Table of contents
- [Test definition](#test-definition)
* [Unit test](#unit-test)
* [Test proprieties](#test-proprieties)
+ [Critical test](#critical-test)
+ [Long run test](#long-run-test)
+ [Test template](#test-template)
* [Test suit](#test-suit)
* [Test setup](#test-setup)
* [Setup test directory](#setup-test-directory)
## 0. Setup test directory

# Test definition
## Unit test
A unit test can be defined to evaluate a procedure (block of code).
The provided directory tree below illustrates an example of how to use the
framework:

## Test proprieties
For each test, the following parameters can be set:

### Critical test
- True - The test process should be stopped if the test fails
- False - The test process can continue even if the test fails

### Long run test
- True - The test duration is considered long (long-time run)
- False - The test is considered fast (short-time run

### Test template
A test definition must follow the outlined nomenclature.

```C
BAO_TEST(#suite_name, #test_name, #criticality, #long_run)
{
// test code here
}
```c
MUT
├── ci
├── src
├── tests
│ ├── configs
│ │ ├── test_cfg1.dts
│ │ ├── test_cfg2.dts
│ ├── src
│ │ ├── test_src1.c
│ │ ├── test_src2.c
│ ├── bao-tests (git repository)
│ ├── bao-nix (git repository)
```

## Test suit
The term "test suite" refers to a group of tests. Grouping the tests enables a more organized selection of tests, making running numerous tests easier.

<table class="tg">
<thead>
<tr>
<th class="tg-c3ow">Suit name</th>
<th class="tg-c3ow">Test name</th>
<th class="tg-c3ow">Critical test</th>
<th class="tg-c3ow">Long test</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tg-c3ow" rowspan="3">Suit_1<br></td>
<td class="tg-c3ow">Test_1</td>
<td class="tg-c3ow">T/F</td>
<td class="tg-c3ow">T/F</td>
</tr>
<tr>
<td class="tg-c3ow">Test_2</td>
<td class="tg-c3ow">T/F</td>
<td class="tg-c3ow">T/F</td>
</tr>
<tr>
<td class="tg-c3ow">Test_3</td>
<td class="tg-c3ow">T/F</td>
<td class="tg-c3ow">T/F</td>
</tr>
<tr>
<td class="tg-c3ow" rowspan="3">Suit_2<br></td>
<td class="tg-c3ow">Test_1</td>
<td class="tg-c3ow">T/F</td>
<td class="tg-c3ow">T/F</td>
</tr>
<tr>
<td class="tg-c3ow">Test_2</td>
<td class="tg-c3ow">T/F</td>
<td class="tg-c3ow">T/F</td>
</tr>
<tr>
<td class="tg-c3ow">Test_3</td>
<td class="tg-c3ow">T/F</td>
<td class="tg-c3ow">T/F</td>
</tr>
</tbody>
</table>

## Test setup
In order to use Bao Test Framework you have go throught the following steps:

1. Add the following lines to your linkerscript
```c
.testframework : {
__testframework_start = . ;
*(.testframework.PRIO* .testframework.prio*)
*(.testframework.*)
__testframework_end = . ;
}
```

2. Add the following lines to your makefile (use the variable TEST_DIR to indicate the directory where all the file containing tests are. The variable REPO_DIR should point to the base directory of this repository. Additionally, the variables names C_SRC, INC_DIRS, and CFLAGS may have to be modified according to your makefile.)

```c
TESTS_DIR:=$(SRC_DIR)/tests
REPO_DIR:=$(SRC_DIR)/../bao-tests
include bao-tests/src/bao-test.mk
C_SRC += $(BAO_TEST_SRCS)
INC_DIRS += $(BAO_TEST_INC_DIR)
CFLAGS += $(BAO_TEST_FLAGS)
```
(Note: If you are using the C library without the Python tool, make sure you add also delete the test build directory in your clean rule)

3. Add to your source the entry point (bao_test_entry()) to our framework, where you feel it is correct.
4. Create a .c file (or multiple) in the directory specified early, include bao_test.h and create tests according to the [test definition example](#Test template).
5. Build your system normally adding the following variable to your make command:
## 1. Get Bao-nix repository
```sh
git clone https://github.com/bao-project/bao-nix.git
```

```c
BAO_TEST=1 -> mandatory to use the framework, otherwise no test will be executed
SUITES="suite1 suite2" -> to specify which suites need to be executed
TESTS="test1 test2" -> to specify which tests are to bexecuted
```
## 2. Get Bao-tests repository
```sh
git clone https://github.com/bao-project/bao-tests.git
```

## Setup test directory
The test platform directory provides a template of a VMM, a VM, and a Guest. These templates are available in the BAO_Test/ directory. A test relating to a MUT should be placed in that module's directory. Tests must be grouped into suits (directory name equals to the suit name) inside the module directory.
## 3. How to configure
Configuration of the test framework is performed through a .dts file. This file allows you to define the test recipe, select the target platform, and specify
the tests to be run in a given setup. The configuration file follows this
template:
```dts
/dts-v1/;
/ {
platform = "test-target-platform";

test_config {
recipe_test {
nix_file = "test-recipe.nix";
suites = "list-of-suites";
tests = "list-of-tests";
log_level = "verbose-level";
};
};
};
```
## 4. How to use

```c
BAO_Test/
├─ VMM_template
├─ VM_template
├─ VM_Test/
│ ├─ Suit_1/
│ │ ├─ Test_1
│ │ ├─ Test_n
│ ├─ Suit_n/
│ │ ├─ Test_1
│ │ ├─ Test_n
├─ VMM_Test/
├─ Guest_Test/
After setting up the directory, run the following commands:
```sh
cd /path-to-MUT-dir/tests/bao-tests/framework/
python3 test_framework.py
```

Alternatively, you can launch the framework with the following arguments:
```sh
python3 test_framework.py \
-dts_path /path/to/config.dts \ # config.dts file
-bao_test_src_path /path/to/bao-tests/src \ # bao-tests repo (/src)
-tests_src_path /path/to/tests/src # tests to be executed
```
74 changes: 49 additions & 25 deletions codegen.py → framework/codegen.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,24 @@
# SPDX-License-Identifier: Apache-2.0
# Copyright (c) Bao Project and Contributors. All rights reserved
import sys
import argparse
import shutil
import os

"""
This script is used to generate Bao Project tests code.
It searches for C source files with 'BAO_TEST' markers and creates
corresponding test functions.
"""

import sys
import argparse
import shutil
import os

def parse_args():
"""
Function to parse command-line arguments for generating tests code.

Returns:
args (argparse.Namespace): Parsed command-line arguments.
"""
parser = argparse.ArgumentParser(description='Script to parse tests \
sourcesand generate tests code')

Expand All @@ -29,20 +35,38 @@ def parse_args():


def get_srcs_list(base_dir):
"""
Function to retrieve a list of C source files from the specified directory.

Args:
base_dir (str): Base directory to search for source files.

Returns:
list: List of C source file paths.
"""
c_srcs = []
for root, dirs, files in os.walk(base_dir):
for root, _, files in os.walk(base_dir):
for file in files:
if file.endswith(".c"):
c_srcs.append(os.path.join(root, file))
return c_srcs

def generate_code(base_dir):
"""
Function to generate code based on specified C source files.

Args:
base_dir (str): Base directory containing C source files.

Returns:
str: Generated code.
"""
c_files = get_srcs_list(base_dir)
tests_list = {}
code = ""
for file in c_files:
with open(file, "r") as f:
file_code = f.readlines()
with open(file, "r", encoding="utf8") as c_file:
file_code = c_file.readlines()

for line in file_code:
if "BAO_TEST" in line:
Expand All @@ -60,11 +84,11 @@ def generate_code(base_dir):
else:
tests_list[suite_name] = [test_name]

for suite in tests_list.keys():
for test in tests_list[suite]:
code += "\t#if defined " + test + " || " + suite + "\n"
code += "\tentry_test_" + suite + "_" + test + "();" + "\n"
code += "\t#endif" + "\n" + "\n"
for suite, tests in tests_list.items():
for test in tests:
code += f"\t#if defined {test} || {suite}\n"
code += f"\tentry_test_{suite}_{test}();\n"
code += "\t#endif\n\n"

return code[:-2]

Expand All @@ -75,29 +99,29 @@ def generate_code(base_dir):
tests_code = generate_code(tool_args.base_dir)

# Copy template to output directory
template_file = "./template.c"
if not os.path.isfile(template_file):
TEMPLATE_FILE = "../src/template.c"
if not os.path.isfile(TEMPLATE_FILE):
print("Template file missing!")
sys.exit()
shutil.copy(template_file, tool_args.out_code)
shutil.copy(TEMPLATE_FILE, tool_args.out_code)

# Read template
with open(tool_args.out_code, "r") as f:
code = f.readlines()
with open(tool_args.out_code, "r", encoding="utf8") as code_file:
read_code = code_file.readlines()

# Get codegen.py writable sections
code_sec_begin, code_sec_end = -1, -1
for index, line in enumerate(code):
if "// codegen.py section begin" in line:
for index, code_line in enumerate(read_code):
if "// codegen.py section begin" in code_line:
code_sec_begin = index

if "// codegen.py section end" in line:
if "// codegen.py section end" in code_line:
code_sec_end = index

# Write generated code to output file
out_code = ''.join(code[0:code_sec_begin+1])
out_code += tests_code
out_code += ''.join(code[code_sec_end:])
with open(tool_args.out_code, "w") as f:
f.writelines(out_code)
OUT_CODE = ''.join(read_code[0:code_sec_begin+1])
OUT_CODE += tests_code
OUT_CODE += ''.join(read_code[code_sec_end:])
with open(tool_args.out_code, "w", encoding="utf8") as out_file:
out_file.writelines(OUT_CODE)
print("Successfully generated bao tests code")
Loading
Loading