Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce CAN timing adjustments as snippet and rework complete test strategy #131

Merged
merged 12 commits into from
Sep 29, 2023
Merged

Introduce CAN timing adjustments as snippet and rework complete test strategy #131

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
651a808
dts: stm32f7(6/7)xxx: can2/3: deprecate sjw properties
rexut Sep 29, 2023
75c6026
snippets: add CAN timing adjustments for tests
rexut Sep 28, 2023
2ff8c26
shields: revert CAN timing adjustments for tests
rexut Sep 28, 2023
df336b9
tests: can: provide alternate configuration
rexut Sep 28, 2023
6a6b403
tests: watchdog: provide alternate configuration
rexut Sep 28, 2023
0779d45
tests: wdt_basic_api: revert watchdog API test clone
rexut Sep 28, 2023
72b75cb
tests: i2c: provide alternate configuration
rexut Sep 28, 2023
c4c98af
tests: i2c_target_api: revert I2C target API test clone
rexut Sep 28, 2023
a596a6f
tests: bridle: common: fix min flash size
rexut Sep 28, 2023
6d9cd78
tiac_magpie: tests: convert to use alt-config
rexut Sep 29, 2023
5b88911
ci: github: qa-shields: convert to use alt-config
rexut Sep 29, 2023
c844acf
release: update draft notes for upcoming 3.5.0
rexut Sep 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 17 additions & 13 deletions .github/workflows/qa-integration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -209,25 +209,29 @@ jobs:
# too high and most of the core tests are already performed by
# Zephyr in any case:
#
# --testsuite-root zephyr/tests/kernel \
# --testsuite-root zephyr/tests/arch/arm \
# --tag arm \
# --tag vector_relay \
# --tag kernel \
#
west twister --verbose --jobs 4 \
--retry-failed 5 --retry-interval 120 \
--outdir twister-out --no-clean --inline-logs \
--enable-size-report --platform-reports \
--device-testing --hardware-map ${HARDWARE_MAP} \
--extra-args SHIELD="loopback_test_tmph can_timing_adj" \
--testsuite-root bridle/tests/bridle \
--testsuite-root zephyr/tests/crypto/rand32 \
--testsuite-root zephyr/tests/drivers/entropy \
--testsuite-root bridle/tests/drivers/watchdog \
--testsuite-root zephyr/tests/drivers/watchdog \
--testsuite-root zephyr/tests/drivers/counter \
--testsuite-root zephyr/tests/drivers/hwinfo \
--testsuite-root zephyr/tests/drivers/gpio \
--testsuite-root zephyr/tests/drivers/spi \
--testsuite-root zephyr/tests/drivers/can
--extra-args SHIELD="loopback_test_tmph" \
--alt-config-root bridle/zephyr/alt-config \
--testsuite-root bridle/tests \
--testsuite-root zephyr/tests \
--tag bridle \
--tag hwinfo \
--tag random \
--tag entropy \
--tag watchdog \
--tag counter \
--tag gpio \
--tag spi \
--tag uart \
--tag can

- name: Upload integration test results
uses: actions/upload-artifact@v3
Expand Down
50 changes: 25 additions & 25 deletions boards/arm/tiac_magpie/doc/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -478,29 +478,31 @@ with a single call to Twister.

.. group-tab:: Running

.. attention:: This will take more than a quarter of an hour.

Build and run the tests on target as follows:

.. code-block:: console

$ ./zephyr/scripts/twister --jobs 4 \
$ west twister --jobs 4 \
--enable-size-report --platform-reports \
--device-testing --hardware-map map.yaml \
--extra-args SHIELD="loopback_test_tmph can_timing_adj" \
--testsuite-root bridle/tests/bridle \
--testsuite-root zephyr/tests/kernel \
--testsuite-root zephyr/tests/arch/arm \
--testsuite-root zephyr/tests/crypto/rand32 \
--testsuite-root zephyr/tests/drivers/entropy \
--testsuite-root zephyr/tests/drivers/watchdog \
--testsuite-root bridle/tests/drivers/watchdog \
--testsuite-root zephyr/tests/drivers/counter \
--testsuite-root zephyr/tests/drivers/hwinfo \
--testsuite-root zephyr/tests/drivers/uart \
--testsuite-root zephyr/tests/drivers/gpio \
--testsuite-root zephyr/tests/drivers/spi \
--testsuite-root zephyr/tests/drivers/can
--extra-args SHIELD="loopback_test_tmph" \
--alt-config-root bridle/zephyr/alt-config \
--testsuite-root bridle/tests \
--testsuite-root zephyr/tests \
--tag arm --tag vector_relay \
--tag kernel --exclude-tag security \
--tag bridle \
--tag hwinfo \
--tag random \
--tag entropy \
--tag watchdog \
--tag counter \
--tag gpio \
--tag spi \
--tag uart \
--tag can

.. attention:: This will take more than a quarter of an hour.

.. group-tab:: Results

Expand All @@ -517,19 +519,17 @@ with a single call to Twister.

INFO - Adding tasks to the queue...
INFO - Added initial list of jobs to queue

INFO - Total complete: :bgn:`248`/ :bgn:`248` 100% skipped: :byl:`125`, failed: :brd:`0`, error: :brd:`0`

INFO - 248 test scenarios (248 test instances) selected, 125 configurations skipped (105 by static filter, 20 at runtime).
INFO - :bgn:`123 of 248` test configurations passed (100.00%), :brd:`0` failed, :byl:`125` skipped with :bbk:`0` warnings in :bbk:`1852.06 seconds`
INFO - In total 1247 test cases were executed, 751 skipped on 1 out of total 580 platforms (0.17%)
INFO - :bgn:`123` test configurations executed on platforms, :brd:`0` test configurations were only built.
INFO - Total complete: :bgn:`1800/1800` 100% skipped: :byl:`1662`, failed: :bbk:`0`, error: :bbk:`0`
INFO - 2012 test scenarios (1800 test instances) selected, 1662 configurations skipped (1639 by static filter, 23 at runtime).
INFO - :bgn:`138 of 1800` test configurations passed (100.00%), :bbk:`0` failed, :bbk:`0` errored, :byl:`1662` skipped with :bbk:`0` warnings in :bbk:`1723.80 seconds`
INFO - In total 1327 test cases were executed, 9726 skipped on 1 out of total 634 platforms (0.16%)
INFO - :bgn:`138` test configurations executed on platforms, :brd:`0` test configurations were only built.

Hardware distribution summary:

\| Board \| ID \| Counter \|
\|-------------\|----------\|-----------\|
\| tiac_magpie \| DT04BNT1 \| 123 \|
\| tiac_magpie \| DT04BNT1 \| 138 \|

.. admonition:: Known but currently tolerated test cases with errors
:class: attention
Expand Down
53 changes: 27 additions & 26 deletions boards/arm/tiac_magpie/doc/tests/arch/arm.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@ Building and Running
--verbose --jobs 4 --inline-logs \
--enable-size-report --platform-reports \
--device-testing --hardware-map map.yaml \
--testsuite-root zephyr/tests/arch/arm
--alt-config-root bridle/zephyr/alt-config \
--testsuite-root zephyr/tests --tag arm --tag vector_relay

.. group-tab:: Results

Expand All @@ -49,31 +50,31 @@ Building and Running

INFO - Adding tasks to the queue...
INFO - Added initial list of jobs to queue

INFO - 3/22 tiac_magpie tests/arch/arm/arm_mem_protect/arch.arm.mem_protect.syscalls :byl:`SKIPPED` (runtime filter)
INFO - 4/22 tiac_magpie tests/arch/arm/arm_irq_advanced_features/arch.arm.irq_advanced_features.secure_fw :byl:`SKIPPED` (runtime filter)
INFO - 5/22 tiac_magpie tests/arch/arm/arm_irq_vector_table/arch.arm.irq_vector_table :bgn:`PASSED` (device 3.636s)
INFO - 6/22 tiac_magpie tests/arch/arm/arm_runtime_nmi/arch.interrupt.arm.nmi :bgn:`PASSED` (device 6.163s)
INFO - 7/22 tiac_magpie tests/arch/arm/arm_thread_swap_tz/arch.arm.swap.tz_off :byl:`SKIPPED` (runtime filter)
INFO - 8/22 tiac_magpie tests/arch/arm/arm_irq_advanced_features/arch.arm.irq_advanced_features :bgn:`PASSED` (device 2.346s)
INFO - 9/22 tiac_magpie tests/arch/arm/arm_tz_wrap_func/arch.arm.tz_wrap_func :bgn:`PASSED` (device 2.341s)
INFO - 10/22 tiac_magpie tests/arch/arm/arm_thread_swap_tz/arch.arm.swap.tz :byl:`SKIPPED` (runtime filter)
INFO - 11/22 tiac_magpie tests/arch/arm/arm_sw_vector_relay/arch.arm.sw_vector_relay :bgn:`PASSED` (device 2.480s)
INFO - 12/22 tiac_magpie tests/arch/arm/arm_thread_swap/arch.arm.swap.common.fpu_sharing.no_optimizations :bgn:`PASSED` (device 6.677s)
INFO - 13/22 tiac_magpie tests/arch/arm/arm_irq_zero_latency_levels/arch.arm.irq_zero_latency_levels.secure_fw :byl:`SKIPPED` (runtime filter)
INFO - 14/22 tiac_magpie tests/arch/arm/arm_thread_swap/arch.arm.swap.common.no_optimizations :bgn:`PASSED` (device 5.894s)
INFO - 15/22 tiac_magpie tests/arch/arm/arm_thread_swap/arch.arm.swap.common.fpu_sharing :bgn:`PASSED` (device 3.012s)
INFO - 16/22 tiac_magpie tests/arch/arm/arm_thread_swap/arch.arm.swap.common :bgn:`PASSED` (device 3.134s)
INFO - 17/22 tiac_magpie tests/arch/arm/arm_irq_zero_latency_levels/arch.arm.irq_zero_latency_levels :bgn:`PASSED` (device 4.351s)
INFO - 18/22 tiac_magpie tests/arch/arm/arm_interrupt/arch.interrupt.no_optimizations :bgn:`PASSED` (device 7.037s)
INFO - 19/22 tiac_magpie tests/arch/arm/arm_interrupt/arch.interrupt.extra_exception_info :bgn:`PASSED` (device 3.281s)
INFO - 20/22 tiac_magpie tests/arch/arm/arm_interrupt/arch.interrupt.arm :bgn:`PASSED` (device 3.147s)
INFO - 21/22 tiac_magpie tests/arch/arm/arm_ramfunc/arch.arm.ramfunc :bgn:`PASSED` (device 3.023s)
INFO - 22/22 tiac_magpie tests/arch/arm/arm_hardfault_validation/arch.interrupt.arm.hardfault_validation :bgn:`PASSED` (device 2.387s)

INFO - 22 test scenarios (22 test instances) selected, 7 configurations skipped (2 by static filter, 5 at runtime).
INFO - :bgn:`15 of 22` test configurations passed (100.00%), :bbk:`0` failed, :bbk:`0` errored, :byl:`7` skipped with :bbk:`0` warnings in :bbk:`191.27 seconds`
INFO - In total 34 test cases were executed, 9 skipped on 1 out of total 580 platforms (0.17%)
INFO - 1523/1543 tiac_magpie tests/arch/arm/arm_mem_protect/arch.arm.mem_protect.syscalls :byl:`SKIPPED` (runtime filter)
INFO - 1524/1543 tiac_magpie tests/arch/arm/arm_irq_advanced_features/arch.arm.irq_advanced_features.secure_fw :byl:`SKIPPED` (runtime filter)
INFO - 1525/1543 tiac_magpie tests/arch/arm/arm_irq_vector_table/arch.arm.irq_vector_table :bgn:`PASSED` (device: DT04BNT1, 2.398s)
INFO - 1526/1543 tiac_magpie tests/arch/arm/arm_runtime_nmi/arch.interrupt.arm.nmi :bgn:`PASSED` (device: DT04BNT1, 5.039s)
INFO - 1527/1543 tiac_magpie tests/arch/arm/arm_tz_wrap_func/arch.arm.tz_wrap_func :bgn:`PASSED` (device: DT04BNT1, 2.405s)
INFO - 1528/1543 tiac_magpie tests/arch/arm/arm_thread_swap_tz/arch.arm.swap.tz_off :byl:`SKIPPED` (runtime filter)
INFO - 1529/1543 tiac_magpie tests/arch/arm/arm_irq_advanced_features/arch.arm.irq_advanced_features :bgn:`PASSED` (device: DT04BNT1, 2.497s)
INFO - 1530/1543 tiac_magpie tests/arch/arm/arm_thread_swap_tz/arch.arm.swap.tz :byl:`SKIPPED` (runtime filter)
INFO - 1531/1543 tiac_magpie tests/arch/arm/arm_sw_vector_relay/arch.arm.sw_vector_relay :bgn:`PASSED` (device: DT04BNT1, 2.341s)
INFO - 1532/1543 tiac_magpie tests/arch/arm/arm_thread_swap/arch.arm.swap.common.fpu_sharing.no_optimizations :bgn:`PASSED` (device: DT04BNT1, 6.862s)
INFO - 1533/1543 tiac_magpie tests/arch/arm/arm_irq_zero_latency_levels/arch.arm.irq_zero_latency_levels.secure_fw :byl:`SKIPPED` (runtime filter)
INFO - 1534/1543 tiac_magpie tests/arch/arm/arm_thread_swap/arch.arm.swap.common.no_optimizations :bgn:`PASSED` (device: DT04BNT1, 5.969s)
INFO - 1535/1543 tiac_magpie tests/arch/arm/arm_thread_swap/arch.arm.swap.common.fpu_sharing :bgn:`PASSED` (device: DT04BNT1, 3.752s)
INFO - 1536/1543 tiac_magpie tests/arch/arm/arm_thread_swap/arch.arm.swap.common :bgn:`PASSED` (device: DT04BNT1, 3.073s)
INFO - 1537/1543 tiac_magpie tests/arch/arm/arm_irq_zero_latency_levels/arch.arm.irq_zero_latency_levels :bgn:`PASSED` (device: DT04BNT1, 2.658s)
INFO - 1538/1543 tiac_magpie tests/arch/arm/arm_interrupt/arch.interrupt.no_optimizations :bgn:`PASSED` (device: DT04BNT1, 6.192s)
INFO - 1539/1543 tiac_magpie tests/arch/arm/arm_interrupt/arch.interrupt.extra_exception_info :bgn:`PASSED` (device: DT04BNT1, 3.486s)
INFO - 1540/1543 tiac_magpie tests/arch/arm64/arm64_psci/arch.arm64.psci :byl:`SKIPPED` (runtime filter)
INFO - 1541/1543 tiac_magpie tests/arch/arm/arm_interrupt/arch.interrupt.arm :bgn:`PASSED` (device: DT04BNT1, 3.239s)
INFO - 1542/1543 tiac_magpie tests/arch/arm/arm_ramfunc/arch.arm.ramfunc :bgn:`PASSED` (device: DT04BNT1, 3.220s)
INFO - 1543/1543 tiac_magpie tests/arch/arm/arm_hardfault_validation/arch.interrupt.arm.hardfault_validation :bgn:`PASSED` (device: DT04BNT1, 2.414s)

INFO - 1755 test scenarios (1543 test instances) selected, 1528 configurations skipped (1522 by static filter, 6 at runtime).
INFO - :bgn:`15 of 1543` test configurations passed (100.00%), :bbk:`0` failed, :bbk:`0` errored, :byl:`1528` skipped with :bbk:`0` warnings in :bbk:`140.68 seconds`
INFO - In total 34 test cases were executed, 10775 skipped on 1 out of total 634 platforms (0.16%)
INFO - :bgn:`15` test configurations executed on platforms, :brd:`0` test configurations were only built.

Hardware distribution summary:
Expand Down
39 changes: 20 additions & 19 deletions boards/arm/tiac_magpie/doc/tests/drivers/adc.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,9 @@ Building and Running
--verbose --jobs 4 --inline-logs \
--enable-size-report --platform-reports \
--device-testing --hardware-map map.yaml \
--extra-args SHIELD=loopback_test_tmph \
--testsuite-root zephyr/tests/drivers/adc
--extra-args SHIELD="loopback_test_tmph" \
--alt-config-root bridle/zephyr/alt-config \
--testsuite-root zephyr/tests --tag adc

.. group-tab:: Results

Expand All @@ -56,13 +57,13 @@ Building and Running

INFO - Adding tasks to the queue...
INFO - Added initial list of jobs to queue
INFO - 3/3 tiac_magpie tests/drivers/adc/adc_api/drivers.adc :brd:`ERROR` Build failure (device)
INFO - see: :byl:`.../twister-out/tiac_magpie/tests/drivers/adc/adc_api/drivers.adc/build.log`
INFO - 1543/1543 tiac_magpie tests/drivers/adc/adc_api/drivers.adc :brd:`ERROR` Build failure (device)
INFO - :byl:`.../twister-out/tiac_magpie/tests/drivers/adc/adc_api/drivers.adc/build.log`
ERROR - Loading Zephyr default modules (Zephyr base).

INFO - 3 test scenarios (3 test instances) selected, 2 configurations skipped (2 by static filter, 0 at runtime).
INFO - :bgn:`0 of 3` test configurations passed (0.00%), :bbk:`0` failed, :brd:`1` errored, :byl:`2` skipped with :bbk:`0` warnings in :bbk:`8.19 seconds`
INFO - In total 6 test cases were executed, 15 skipped on 1 out of total 580 platforms (0.17%)
INFO - 1755 test scenarios (1543 test instances) selected, 1542 configurations skipped (1542 by static filter, 0 at runtime).
INFO - :bgn:`0 of 1543` test configurations passed (0.00%), :bbk:`0` failed, :brd:`1` errored, :byl:`1542` skipped with :bbk:`0` warnings in :bbk:`12.99 seconds`
INFO - In total 6 test cases were executed, 10803 skipped on 1 out of total 634 platforms (0.16%)
INFO - :bgn:`0` test configurations executed on platforms, :brd:`1` test configurations were only built.

Hardware distribution summary:
Expand All @@ -79,10 +80,10 @@ Building and Running
INFO - -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
INFO - The following issues were found (showing the top 10 items):
INFO - 1) tests/drivers/adc/adc_api/drivers.adc on tiac_magpie error (Build failure)
INFO -
INFO -
INFO - To rerun the tests, call twister using the following commandline:
INFO - west twister -p <PLATFORM> -s <TEST ID>, for example:
INFO -
INFO -
INFO - west twister -p tiac_magpie -s tests/drivers/adc/adc_api/drivers.adc
INFO - or with west:
INFO - west build -p -b tiac_magpie -T tests/drivers/adc/adc_api/drivers.adc
Expand All @@ -97,19 +98,19 @@ Open Issues

... ... ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:33:2: :brd:`error:` #error "Unsupported board."
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:38:30: :brd:`error:` :bbk:`'adc_channels'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:50:38: :brd:`error:` :bbk:`'adc_channels'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:52:29: :brd:`error:` :bbk:`'adc_channels_count'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:38:31: :brd:`error:` :bbk:`'adc_channels'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:50:39: :brd:`error:` :bbk:`'adc_channels'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:52:25: :brd:`error:` :bbk:`'adc_channels_count'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:94:37: :brd:`error:` :bbk:`'adc_channels'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:121:37: :brd:`error:` :bbk:`'adc_channels'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:123:29: :brd:`error:` :bbk:`'adc_channels_count'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:137:13: :brd:`error:` :bbk:`'adc_channels_count'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:169:37: :brd:`error:` :bbk:`'adc_channels'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:228:37: :brd:`error:` :bbk:`'adc_channels'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:258:31: :brd:`error:` :bbk:`'adc_channels_count'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:301:37: :brd:`error:` :bbk:`'adc_channels'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:303:13: :brd:`error:` :bbk:`'adc_channels_count'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:325:36: :brd:`error:` :bbk:`'adc_channels'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:140:13: :brd:`error:` :bbk:`'adc_channels_count'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:172:37: :brd:`error:` :bbk:`'adc_channels'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:231:37: :brd:`error:` :bbk:`'adc_channels'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:264:35: :brd:`error:` :bbk:`'adc_channels_count'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:307:37: :brd:`error:` :bbk:`'adc_channels'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:309:13: :brd:`error:` :bbk:`'adc_channels_count'` undeclared here ...
.../zephyr/tests/drivers/adc/adc_api/src/test_adc.c:334:36: :brd:`error:` :bbk:`'adc_channels'` undeclared here ...
... ... ...
... ... ...
... ... ...
Loading
Loading