AWX Tasks hanging on ansible.utils.cli_parse >= 5.1.0

[bewing]

SUMMARY

AWX Tasks running ansible.utils.cli_parse are hanging

ISSUE TYPE

• Bug Report

COMPONENT NAME

ansible.utils.cli_parse

ANSIBLE VERSION

ansible [core 2.17.5]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.11/site-packages/ansible
  ansible collection location = /runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.10 (main, Sep  9 2024, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-2)] (/usr/bin/python3.11)
  jinja version = 3.1.4
  libyaml = True

COLLECTION VERSION

# /usr/share/ansible/collections/ansible_collections
Collection    Version
------------- -------
ansible.utils 5.1.0

CONFIGURATION

CONFIG_FILE() = /etc/ansible/ansible.cfg
DISPLAY_SKIPPED_HOSTS(/etc/ansible/ansible.cfg) = False

OS / ENVIRONMENT

awx-operator 2.19.1
AWX 24.6.1

STEPS TO REPRODUCE

We have several extremely large AWX jobs that collect information from the CLI and push new configs to switch devices. After upgrading from ansible.utils 5.0.0 to 5.1.0 and 5.1.2, jobs occasionally hang indefinitely, not producing anymore job output. The last log output of every hung job contains "task_action": "ansible.utils.cli_parse"

Producing the same AWX EE image, but with ansible.utils==5.0.0, no jobs hang.

I will see if I can produce a publishable test case, but it may take some time. I did want to get this recorded in case any other operators are seeing similar issues.