Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IS-12 Test Suites #810

Open
wants to merge 191 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
191 commits
Select commit Hold shift + click to select a range
ec40a1d
[IS-12] Initial Commit for IS-12 Test Suite (#800)
jonathan-r-thorpe May 9, 2023
419ef85
[IS-12] Check Root Block and ClassManager (#814)
jonathan-r-thorpe Jun 21, 2023
c033475
[IS-12] Add auto tests for validating MS-05 standard class and dataty…
jonathan-r-thorpe Jul 18, 2023
dadf52b
Merge pull request #812 from AMWA-TV/zeroconf-marmoset
garethsb Jun 28, 2023
e36e6cc
[IS 12] Add Get/Set/Add/Remove/Find methods (#822)
jonathan-r-thorpe Aug 4, 2023
f841bb7
[IS-12] Specify explicit URL path for WebSocket endpoint (#827)
jonathan-r-thorpe Aug 8, 2023
f8c63b6
Check response from Find methods is an array
jonathan-r-thorpe Aug 10, 2023
c28ee82
Handle errors where control classes are not advertised by Class Manager
jonathan-r-thorpe Aug 23, 2023
d22005d
Fixed detection of authority keys in non-standard classes
jonathan-r-thorpe Aug 24, 2023
8073aa2
Handle missing device model properties
jonathan-r-thorpe Aug 24, 2023
2bbd542
Defend against missing properties or datatype definitions
jonathan-r-thorpe Aug 24, 2023
fa810fa
Improved error handling,
jonathan-r-thorpe Oct 6, 2023
34678c8
Bump Node 12.x to Node 16.x
garethsb Aug 3, 2023
db8c8cf
python-zeroconf 0.75.0 strict=False for _nmos-registration._tcp
garethsb Aug 14, 2023
6cb8f7c
Python 3.6 (and 3.7!) is End-Of-Life, and so is Node.js 14.x
garethsb Aug 18, 2023
b3008ad
Pass IP address of API under test to testssl.sh
garethsb Oct 5, 2023
353e746
Add MOCK_SERVICES_WARM_UP_DELAY
garethsb Oct 5, 2023
23110f1
Delay running tests until after MOCK_SERVICES_WARM_UP_DELAY
garethsb Oct 5, 2023
ab07908
Tweak message
garethsb Oct 5, 2023
049d674
Fixed send_command function as per suggestion from @alabou
jonathan-r-thorpe Oct 6, 2023
b165316
Add config option to bypass IS12 interactive testing
jonathan-r-thorpe Oct 24, 2023
19f71a8
Add constraints classes
jonathan-r-thorpe Oct 24, 2023
b73cd22
added xeger library
jonathan-r-thorpe Oct 24, 2023
ddc17d7
Add IS1202 test suite
jonathan-r-thorpe Oct 24, 2023
b081bb0
Add primitives to datatype models
jonathan-r-thorpe Oct 31, 2023
aa6faa9
Enhance test_35 (subscription and notification mechanism test) to all…
lo-simon Nov 6, 2023
d862805
Tweaked test_35
jonathan-r-thorpe Nov 6, 2023
6dd111d
Pass **kwargs to ControllerTest
jonathan-r-thorpe Nov 7, 2023
19d5b86
Pass **kwargs to ControllerTest
jonathan-r-thorpe Nov 7, 2023
e581283
Test sequences; remove non UTF-8 character from comment
jonathan-r-thorpe Nov 8, 2023
ce8afc0
Added constrained sequence tests
jonathan-r-thorpe Nov 8, 2023
a0937e1
Additional sequence checks
jonathan-r-thorpe Nov 8, 2023
7f84e40
Add runtime constraints to device model. Fix handling of primitive ty…
jonathan-r-thorpe Nov 10, 2023
a7af313
Added property and datatype constraints checking
jonathan-r-thorpe Nov 14, 2023
fe032c0
Removed disabled tests
jonathan-r-thorpe Nov 14, 2023
995e292
make datatype_schemas an object property
jonathan-r-thorpe Nov 14, 2023
cedacb8
Check for default value sequences when validating class descriptors
jonathan-r-thorpe Nov 14, 2023
1bb41bd
Flake8 formatting
jonathan-r-thorpe Nov 14, 2023
8409b8e
Cache class_manager. Simplify constraint checking.
jonathan-r-thorpe Nov 14, 2023
e0c6d7f
Merge remote-tracking branch 'nmos-testing-amwa/is-12' into is-12-inv…
jonathan-r-thorpe Nov 14, 2023
30d8059
Fix Block object construction. Fix Test_02
jonathan-r-thorpe Nov 15, 2023
72d070b
add missing primitive descriptor path
jonathan-r-thorpe Nov 15, 2023
9a0f6bd
reorder specs in NMOS Testing interface
jonathan-r-thorpe Nov 15, 2023
c5d43f7
Fix constraint violation error message
jonathan-r-thorpe Nov 15, 2023
83afc3b
Disable Testing Facade host and port if not interactively testing
jonathan-r-thorpe Nov 15, 2023
96aaaa9
Flake8 formatting
jonathan-r-thorpe Nov 15, 2023
c8caa23
Fix constraint violation error message
jonathan-r-thorpe Nov 15, 2023
049d62a
Refactor property testing
jonathan-r-thorpe Nov 16, 2023
5f2cb63
Handle property and method deprecated status codes
jonathan-r-thorpe Nov 16, 2023
7c93b49
Check deprecated properties
jonathan-r-thorpe Nov 17, 2023
12f556b
Report unchecked device model in tests
jonathan-r-thorpe Nov 17, 2023
738021f
Renumbered tests
jonathan-r-thorpe Nov 17, 2023
ef33d0f
updated spec URLs in comments
jonathan-r-thorpe Nov 17, 2023
23b3512
Handle property and method deprecated status codes
jonathan-r-thorpe Nov 16, 2023
0841000
Check deprecated properties
jonathan-r-thorpe Nov 17, 2023
63bafa7
Report unchecked device model in tests
jonathan-r-thorpe Nov 17, 2023
e673449
Renumbered tests
jonathan-r-thorpe Nov 17, 2023
dc2d30a
updated spec URLs in comments
jonathan-r-thorpe Nov 17, 2023
8234809
Merge remote-tracking branch 'origin/is-12-invasive' into is-12
jonathan-r-thorpe Nov 17, 2023
088b22e
Updated to conform merged branch
jonathan-r-thorpe Nov 17, 2023
af8d55e
Guard against missing class properties
jonathan-r-thorpe Dec 18, 2023
3640690
Add annotation to check properties test
jonathan-r-thorpe Dec 18, 2023
b53b562
Better error reporting for test_22 and test_23
jonathan-r-thorpe Dec 18, 2023
d5348e0
formatting
jonathan-r-thorpe Dec 18, 2023
3fa3fed
Fix spelling mistake in interactive testing config
jonathan-r-thorpe Jan 10, 2024
b1c91f4
Fixed Blocks with no members being set to None
jonathan-r-thorpe Jan 12, 2024
09f2d39
Fixed config spelling error
jonathan-r-thorpe Jan 12, 2024
b5205ac
Rename _execute_command to execute_command
jonathan-r-thorpe Jan 12, 2024
35cde03
Test discovered methods in Device Model
jonathan-r-thorpe Jan 12, 2024
fd8c9d3
Only test non-standard methods.
jonathan-r-thorpe Feb 7, 2024
95fe9e5
Moved some functions to IS12Utils
jonathan-r-thorpe Feb 14, 2024
245e3c6
Added more error reporting
jonathan-r-thorpe Feb 26, 2024
3483863
Guard against ControllerTest being run in non-interactive mode
jonathan-r-thorpe Feb 28, 2024
5a15d84
Added check constraints hierarchy test
jonathan-r-thorpe Mar 6, 2024
3260af4
Fixed do_check_property_test
jonathan-r-thorpe Mar 6, 2024
98b0a64
Add check read only properties and sequences
jonathan-r-thorpe Mar 6, 2024
3fad9a7
Made sequence tests tolerant to empty or null sequences
jonathan-r-thorpe Mar 6, 2024
a5350cd
Typo and formatting
jonathan-r-thorpe Mar 6, 2024
5ba312d
Initial add of IS-14-01 test suite
jonathan-r-thorpe Apr 23, 2024
af0c67a
Flake8 formatting
jonathan-r-thorpe Apr 26, 2024
7405325
Added Node API to IS-14 (for testing control endpoint)
jonathan-r-thorpe Apr 29, 2024
55eeb69
Initial refactoring
jonathan-r-thorpe Apr 29, 2024
7bb6a5f
Added additional specs to IS-14-01 test suite definition
jonathan-r-thorpe Apr 29, 2024
a9ed819
Move common functions from IS12Utils to MS05Utils
jonathan-r-thorpe Apr 29, 2024
03d411a
Delegate to MS0501Test for auto tests
jonathan-r-thorpe Apr 29, 2024
2fb54b2
API key now required when loading resources
jonathan-r-thorpe Apr 29, 2024
0062040
Delegate to MS0501Test for auto tests
jonathan-r-thorpe Apr 29, 2024
84959a6
flake8 formatting
jonathan-r-thorpe Apr 29, 2024
95a1eac
Move MS-05 tests into MS0501Test
jonathan-r-thorpe May 1, 2024
ee4888a
Fixed get_sequence_item
jonathan-r-thorpe May 1, 2024
223b29d
Refactor IS-14 API utility functions
jonathan-r-thorpe May 2, 2024
2dd9973
generalize get_control_class and get_datatype. Improve error handling.
jonathan-r-thorpe May 2, 2024
089a2b0
Implement IS-14 http request helper functions
jonathan-r-thorpe May 2, 2024
dc9608a
Formatting
jonathan-r-thorpe May 2, 2024
80dedb0
Moved error handling tests to MS0501Test
jonathan-r-thorpe May 3, 2024
36cec82
Add invoke method utility function.
jonathan-r-thorpe May 3, 2024
0e2f8f5
Do not cache spec_branch in construction
jonathan-r-thorpe May 3, 2024
ffc7ea0
Simplify utilities initialization
jonathan-r-thorpe May 3, 2024
1599e8d
Add reset so that state is reset on set_up_tests
jonathan-r-thorpe May 3, 2024
d5b5851
Initiailize IS-12 websocket on set_up_tests so failure is reported on…
jonathan-r-thorpe May 3, 2024
ddb1e7f
make base class generic
jonathan-r-thorpe May 3, 2024
ee57347
Updated comments
jonathan-r-thorpe May 3, 2024
9813fba
Changed " embedd3ed in f-strings to '
jonathan-r-thorpe May 8, 2024
e9aa570
Changed SET to PUT
jonathan-r-thorpe May 8, 2024
c3d58d9
Changed " embedded in f-strings to '
jonathan-r-thorpe May 8, 2024
37301ea
Fix IS-14 utility functions
jonathan-r-thorpe May 9, 2024
bad7453
Add missing sequence methods
jonathan-r-thorpe May 9, 2024
f9aa23a
Add role_path to get_sequence_length call
jonathan-r-thorpe May 9, 2024
5eda47f
Add device-configuration feature set to feature set tests
jonathan-r-thorpe Jun 3, 2024
e228800
Add additional validation of class descriptors
jonathan-r-thorpe Jun 3, 2024
80bcca4
Add is_block helper.
jonathan-r-thorpe Sep 9, 2024
013849d
Add initial IS-14 tests
jonathan-r-thorpe Sep 9, 2024
b32a401
Improve error reporting
jonathan-r-thorpe Sep 13, 2024
d96d424
Fix linting errors
jonathan-r-thorpe Oct 11, 2024
d276ec4
Add testing of enum ans struct sequences
jonathan-r-thorpe Oct 11, 2024
91cf7a5
Moved _get_datatyoe_schema into MS05Utils
jonathan-r-thorpe Oct 11, 2024
884d205
Make TestMetadata an inner class
jonathan-r-thorpe Oct 14, 2024
d9142a2
Use TestMetadata inner class. Remove unneeded data members.
jonathan-r-thorpe Oct 14, 2024
8beaa9a
Added NcDescriptor and NcPropertyDescriptor classes
jonathan-r-thorpe Oct 15, 2024
3e336bf
Refactor to use NcPropertyDescriptor class
jonathan-r-thorpe Oct 15, 2024
824678a
Refactored NcPropertyId and NcTouchpoint types
jonathan-r-thorpe Oct 15, 2024
dd33d4c
Refactor dictionaries into classes
jonathan-r-thorpe Oct 18, 2024
a5d9ebc
Update formatting of strings
jonathan-r-thorpe Oct 18, 2024
44b2724
Consistent quotes
jonathan-r-thorpe Oct 18, 2024
958d962
Renamed test_data directory from IS1201 to MS05
jonathan-r-thorpe Oct 18, 2024
9673baa
Refactored names of functions, improved consistency of strings
jonathan-r-thorpe Oct 18, 2024
798dcad
Create NcFieldDescriptor, NcParameterDescriptor and NcMethodDescripto…
jonathan-r-thorpe Oct 21, 2024
19008fc
Introduced constraint domain classes. Refactored test logic.
jonathan-r-thorpe Oct 22, 2024
5983e67
Handle 'any' datatype in method descriptors
jonathan-r-thorpe Oct 22, 2024
d94de38
Consistent formatting of strings. Refactor to use domain classes
jonathan-r-thorpe Oct 23, 2024
29310f8
Utility functions to format class ids and role paths. Create NcMethos…
jonathan-r-thorpe Oct 23, 2024
f5a5e42
Remove context parameter and use role_path instead
jonathan-r-thorpe Oct 24, 2024
34c312b
Remove MS-05-02 error handling from IS12Utils
jonathan-r-thorpe Nov 1, 2024
b36a657
Add NcMethodResult types
jonathan-r-thorpe Nov 1, 2024
ac99d1b
Remove redundant imports
jonathan-r-thorpe Nov 1, 2024
1184d46
Formatting
jonathan-r-thorpe Nov 1, 2024
4acb74e
Handle MS-05-02 errors
jonathan-r-thorpe Nov 1, 2024
dad1335
get_property returns NcMethodResult object
jonathan-r-thorpe Nov 1, 2024
12c6f3a
set_property returns NcMethodResult
jonathan-r-thorpe Nov 1, 2024
bd4099c
invoke_method returns NcMethodResult
jonathan-r-thorpe Nov 1, 2024
ebd79bc
get_sequence_item returns NcMethodResult
jonathan-r-thorpe Nov 1, 2024
cc3857a
get_sequence_length returns NcMethodResult
jonathan-r-thorpe Nov 2, 2024
3b80b6f
set_sequence_item returns NcMethodResult
jonathan-r-thorpe Nov 2, 2024
9dcd93d
add_sequence_item returns NcMethodResultId
jonathan-r-thorpe Nov 2, 2024
9424fc3
remove_sequence_item returns NcMethodResult
jonathan-r-thorpe Nov 2, 2024
e66fa76
get_member_descriptor returns NcMethodResult
jonathan-r-thorpe Nov 3, 2024
f297309
find_members_by_path returns NcMethodResult
jonathan-r-thorpe Nov 3, 2024
8ae7779
find_members_by_role returns NcMethodResult
jonathan-r-thorpe Nov 4, 2024
d061f4f
find_members_by_class_id returns NcMethodResult
jonathan-r-thorpe Nov 4, 2024
8dbff44
get_control_class returns NcMethodResult
jonathan-r-thorpe Nov 4, 2024
ff03fd3
get_datatype returns NcMethodResult
jonathan-r-thorpe Nov 4, 2024
33f499c
Use NcPropertyId and NcMethodId instead of raw dict
jonathan-r-thorpe Nov 4, 2024
9455fd1
Implement IS12Notification class
jonathan-r-thorpe Nov 4, 2024
82ebd13
Implement NcPropertyChangedEventData
jonathan-r-thorpe Nov 4, 2024
ec3c2a1
Implement IS12CommandResponse class
jonathan-r-thorpe Nov 4, 2024
c5b5d31
Check subscription response
jonathan-r-thorpe Nov 4, 2024
e397fde
Refactor checking of Block members
jonathan-r-thorpe Nov 5, 2024
edaa948
Formatting
jonathan-r-thorpe Nov 5, 2024
411d9c4
Implement IS12Error class. Formatting
jonathan-r-thorpe Nov 5, 2024
80ed30f
Refine IS14Utils error handling
jonathan-r-thorpe Nov 6, 2024
9148e63
Fix method invocation JSON
jonathan-r-thorpe Nov 7, 2024
df3bf37
Simplify property constraints checking
jonathan-r-thorpe Nov 7, 2024
d989b4b
Fix dicts embedded in f-strings
jonathan-r-thorpe Nov 7, 2024
0a0ab71
Add spec links to test failures
jonathan-r-thorpe Nov 7, 2024
f74eb31
Fix dicts embedded in f-strings
jonathan-r-thorpe Nov 7, 2024
2bac92d
Fix embedded code strings in f-strings
jonathan-r-thorpe Nov 7, 2024
e391efa
Fix embedded code strings in f-strings
jonathan-r-thorpe Nov 7, 2024
916336a
Format error messages
jonathan-r-thorpe Nov 7, 2024
f1261d1
Refactored constraints testing
jonathan-r-thorpe Nov 11, 2024
05f24f2
Improve notifications handling
alabou Nov 11, 2024
6777734
Flake8 formatting
jonathan-r-thorpe Nov 11, 2024
9d1d3b6
Rename context to error_msg_base
jonathan-r-thorpe Nov 12, 2024
ea7a5fa
consistent format of error messages
jonathan-r-thorpe Nov 12, 2024
592b4f6
Better handling of IS-12 errors
jonathan-r-thorpe Nov 13, 2024
2cbb4c0
Add robustness for missing classes and datatype definitions
jonathan-r-thorpe Nov 13, 2024
dbdc5df
Consistent formatting of error messages
jonathan-r-thorpe Nov 15, 2024
e53fa8e
Improve constraints testing
jonathan-r-thorpe Nov 19, 2024
5da7575
Add NcEventDescriptor
jonathan-r-thorpe Nov 19, 2024
6024577
Validate parameter constraints. Check for illegal non-standard datatypes
jonathan-r-thorpe Nov 19, 2024
0322fb7
Minor bug fixes
jonathan-r-thorpe Nov 19, 2024
2d49bfa
Merge branch 'master' into is-12
jonathan-r-thorpe Nov 19, 2024
a987151
Update comments
jonathan-r-thorpe Nov 20, 2024
becf193
Clearly indicate private helper functions
jonathan-r-thorpe Nov 20, 2024
edb91e4
Simplify constraints checking tests
jonathan-r-thorpe Nov 20, 2024
d140e05
Clearly indicate private helper functions
jonathan-r-thorpe Nov 20, 2024
88ebc67
Improve error message
jonathan-r-thorpe Nov 20, 2024
3d85279
Allow calling test to explicitly set test method name reported in Tes…
jonathan-r-thorpe Nov 21, 2024
3e6c263
Fix interaction with Testing Facade
jonathan-r-thorpe Nov 21, 2024
1c6ee65
Format error message
jonathan-r-thorpe Nov 21, 2024
089a33e
Remove IS-14 test suites
jonathan-r-thorpe Nov 21, 2024
d1403e2
Improve descriptor validation error reporting
jonathan-r-thorpe Nov 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions nmostesting/Config.py
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,37 @@
}
}
},
"is-12": {
"repo": "is-12",
"versions": ["v1.0"],
"default_version": "v1.0",
"apis": {
"ncp": {
"name": "Control Protocol"
},
}
},
"ms-05-02": {
"repo": "ms-05-02",
"versions": ["v1.0"],
"default_version": "v1.0",
"apis": {
"controlframework": {
"name": "Control Framework"
},
}
},
"nmos-control-feature-sets": {
"repo": "nmos-control-feature-sets",
"versions": ["main"],
"default_version": "main",
"apis": {
"featuresets": {
"name": "Control Feature Sets",
"repo_paths": ["identification", "monitoring"]
}
}
},
"bcp-002-01": {
"repo": "bcp-002-01",
"versions": ["v1.0"],
Expand Down
Loading