Low performance on TG initialization in tests

[anton7811]

Currently, each test does a connection to TG and port assignment. Those operations take more than 20 seconds.
When there will be about 300 tests time wastage will be about 2 hours.

Existing methods tgen_utils_traffic_generator_connect and tgen_utils_create_devices_and_connect expect protocols configuration which is a part of the test case scenario.

The idea is to preserve connection and port assignment on a session level. So each test will only configure/clear protocols and/or streams. For that, we have to decouple the connection from any TG configuration.

[SerhiyBoikoPLV]

After a long test run when reusing Ixia sessions, some tests might start timing out when trying to access tgen api (start protocols, assign ports, etc.). This is most often seen with the older 10/100/1000 LSM XMVDC16 card, but can also happen on the 10G card.

The easiest solution that we have is to delete the current session at the end of every test (or every test group). This will ensure that there is no persistent configuration left in the session that might cause this issue. Unfortunately, this will significantly increase the test run duration (the last successful run was ~25hrs long, as opposed to ~18hrs when reusing sessions), but it will make the test runs more consistent.

Ethernet card:

INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Connected to Linux Gateway Session ID: 91
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Reserving test ports, and may take a minute...
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Connection to Ixia REST API Server Established
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Assigning ports
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Changing port: 10.36.118.199:2:5 media mode copper
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Changing port: 10.36.118.199:2:6 media mode copper
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Changing port: 10.36.118.199:2:7 media mode copper
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Changing port: 10.36.118.199:2:8 media mode copper
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding interface on ixia port 10.36.118.199:2:5 swp swp33
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding device {'name': '10.36.118.199:2:5_1.1.1.2/24', 'count': 1, 'ip': '1.1.1.2', 'gw': '1.1.1.1', 'plen': 24, 'vlan': None, 'version': 'ipv4', 'type': 'port'}
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding interface on ixia port 10.36.118.199:2:6 swp swp34
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding device {'name': '10.36.118.199:2:6_1.1.1.3/24', 'count': 1, 'ip': '1.1.1.3', 'gw': '1.1.1.1', 'plen': 24, 'vlan': None, 'version': 'ipv4', 'type': 'port'}
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding interface on ixia port 10.36.118.199:2:7 swp swp35
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding device {'name': '10.36.118.199:2:7_1.1.1.4/24', 'count': 1, 'ip': '1.1.1.4', 'gw': '1.1.1.1', 'plen': 24, 'vlan': None, 'version': 'ipv4', 'type': 'port'}
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding interface on ixia port 10.36.118.199:2:8 swp swp36
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding device {'name': '10.36.118.199:2:8_1.1.1.5/24', 'count': 1, 'ip': '1.1.1.5', 'gw': '1.1.1.1', 'plen': 24, 'vlan': None, 'version': 'ipv4', 'type': 'port'}
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] [{'ixia': {'command': 'connect', 'rc': 0, 'result': 'Connected!'}}]
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Setting up Tgen traffic for ['10.36.118.199:2:6_1.1.1.3/24', '10.36.118.199:2:7_1.1.1.4/24', '10.36.118.199:2:8_1.1.1.5/24'] -> Broadcast
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Creating raw traffic stream
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding endpoint swp33 to swp34
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding endpoint swp33 to swp35
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding endpoint swp33 to swp36
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] [{'ixia': {'command': 'set_traffic', 'rc': 0, 'result': ''}}]
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Saving Tgen config file None
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Starting Protocols
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Starting All Protocols
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] [{'ixia': {'command': 'start_protocols', 'rc': -1, 'result': ' Following error(s) occurred: Remote call exceeds timeout: 3600000 ms\n\n'}}]
 -----------------------------Captured log teardown------------------------------ 

All subsequent tests that are using tgen start to fail as well:

INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Connected to Linux Gateway Session ID: 91
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Reserving test ports, and may take a minute...
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Connection to Ixia REST API Server Established
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Exception Caught: Couldn't set port 10.36.118.199;2;5 to /vport:6. It has already been assigned to /vport:1
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] <class 'ixnetwork_restpy.errors.BadRequestError'>, ixnetwork_ixia_client_impl.py:110
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] [{'ixia': {'command': 'connect', 'rc': -1, 'result': 'Error!'}}]
 -----------------------------Captured log teardown------------------------------ 

10G card:

INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Connected to Linux Gateway Session ID: 84
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Reserving test ports, and may take a minute...
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Connection to Ixia REST API Server Established
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Assigning ports
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Changing port: 10.36.118.199:1:5 media mode fiber
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Changing port: 10.36.118.199:1:6 media mode fiber
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Changing port: 10.36.118.199:1:7 media mode fiber
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Changing port: 10.36.118.199:1:8 media mode fiber
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding interface on ixia port 10.36.118.199:1:5 swp swp1
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding device {'name': '10.36.118.199:1:5_1.1.1.2/24', 'count': 1, 'ip': '1.1.1.2', 'gw': '1.1.1.1', 'plen': 24, 'vlan': None, 'version': 'ipv4', 'type': 'port'}
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding interface on ixia port 10.36.118.199:1:6 swp swp2
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding device {'name': '10.36.118.199:1:6_1.1.1.3/24', 'count': 1, 'ip': '1.1.1.3', 'gw': '1.1.1.1', 'plen': 24, 'vlan': None, 'version': 'ipv4', 'type': 'port'}
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding interface on ixia port 10.36.118.199:1:7 swp swp3
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding device {'name': '10.36.118.199:1:7_1.1.1.4/24', 'count': 1, 'ip': '1.1.1.4', 'gw': '1.1.1.1', 'plen': 24, 'vlan': None, 'version': 'ipv4', 'type': 'port'}
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding interface on ixia port 10.36.118.199:1:8 swp swp4
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding device {'name': '10.36.118.199:1:8_1.1.1.5/24', 'count': 1, 'ip': '1.1.1.5', 'gw': '1.1.1.1', 'plen': 24, 'vlan': None, 'version': 'ipv4', 'type': 'port'}
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] [{'ixia': {'command': 'connect', 'rc': 0, 'result': 'Connected!'}}]
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Setting up Tgen traffic for stream_1
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Creating raw traffic stream
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding endpoint swp1 to swp2
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] [{'ixia': {'command': 'set_traffic', 'rc': 0, 'result': ''}}]
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Setting up Tgen traffic for stream_2
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Creating raw traffic stream
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding endpoint swp1 to swp3
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] [{'ixia': {'command': 'set_traffic', 'rc': 0, 'result': ''}}]
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Setting up Tgen traffic for stream_3
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Creating raw traffic stream
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Adding endpoint swp1 to swp4
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] [{'ixia': {'command': 'set_traffic', 'rc': 0, 'result': ''}}]
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Saving Tgen config file None
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Starting Protocols
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] Starting All Protocols
INFO     DENT:Logger.py:84 [Ixia Traffic Generator] [{'ixia': {'command': 'start_protocols', 'rc': -1, 'result': 'Expecting value: line 1 column 1 (char 0)'}}]