Expose a way to exclude dependencies (#664)

* Adds functionality to exclude Python or system requirements from included collections.
* Drop requirements-parser in favor of pep508
* Better comment handling to not lose URL anchors, pass thru non-pep508 data
* Removes requirement sanitization (no longer consolidate duplicate requirement entries).

---------

Co-authored-by: Matt Martz <[email protected]>

docs/collection_metadata.rst

@@ -35,7 +35,7 @@ If the ``meta/execution-environment.yml`` file is not present, by default, Ansib
 Dependency introspection
 ========================
 
-If any dependencies are given, the introspection is run by Ansible Builder so that the requirements are found and sanitized (deduped) before container image assembly.
+If any dependencies are given, the introspection is run by Ansible Builder so that the requirements are found before container image assembly.
 
 A user can see the introspection output during
 the builder intermediate phase using the ``build -v3`` option.
@@ -63,13 +63,11 @@ Run the ``introspect`` command against your collection path:
 
 ::
 
-    ansible-builder introspect --sanitize COLLECTION_PATH
+    ansible-builder introspect COLLECTION_PATH
 
 The default collection path used by the ``ansible-galaxy`` command is ``~/.ansible/collections/``.
 Read more about collection paths in the `Ansible configuration settings <https://docs.ansible.com/ansible/latest/reference_appendices/config.html#collections-paths>`_ guide.
 
-The ``--sanitize`` option reviews all of the collection requirements and removes duplicates. It also removes any Python requirements that should normally be excluded (see :ref:`python_deps` below).
-
 .. note::
     Use the ``-v3`` option to ``introspect`` to see logging messages about requirements that are being excluded.
 
@@ -94,20 +92,28 @@ Then, if the ``ansible_collection`` directory is in your home directory, you can
 
 ::
 
-  ansible-builder introspect --sanitize ~/
+  ansible-builder introspect ~/
 
 .. _python_deps:
 
 Python Dependencies
 ^^^^^^^^^^^^^^^^^^^
 
-Ansible Builder combines all the Python requirements files from all collections into a single file using the ``requirements-parser`` library. This library supports complex syntax, including references to other files.
+Ansible Builder combines all the Python requirements files from all collections into a single file.
+
+Certain package names are specifically *ignored* by ``ansible-builder``, meaning that Ansible Builder
+does not include them in the combined file of Python dependencies, even if a collection lists them as
+dependencies. These include test packages and packages that provide Ansible itself. The full list can
+be found in ``EXCLUDE_REQUIREMENTS`` in ``src/ansible_builder/_target_scripts/introspect.py``.
 
-If multiple collections require the same *package name*, Ansible Builder combines them into a single entry and combines the constraints.
+If you need to include one of these ignored package names, use the ``--user-pip`` option of the
+``introspect`` command to list it in the user requirements file. Packages supplied this way are
+not processed against the list of excluded Python packages.
 
-Certain package names are specifically *ignored* by ``ansible-builder``, meaning that Ansible Builder does not include them in the combined file of Python dependencies, even if a collection lists them as dependencies. These include test packages and packages that provide Ansible itself. The full list can be found in ``EXCLUDE_REQUIREMENTS`` in ``src/ansible_builder/_target_scripts/introspect.py``.
+.. note::
 
-If you need to include one of these ignored package names, use the ``--user-pip`` option of the ``introspect`` command to list it in the user requirements file. Packages supplied this way are not processed against the list of excluded Python packages.
+  These dependencies are subject to the same `PEP 508 <https://peps.python.org/pep-0508/>`_ format
+  restrictions described for Python requirements in the :ref:`EE definition specification <python-pep508>`.
 
 System-level Dependencies
 ^^^^^^^^^^^^^^^^^^^^^^^^^

docs/definition.rst

@@ -50,6 +50,11 @@ Here is a sample version 3 EE file. To use Ansible Builder 3.x, you must specify
         - six
         - psutil
       system: bindep.txt
+      exclude:
+        python:
+          - docker
+        system:
+          - python3-Cython
 
     images:
       base_image:
@@ -232,10 +237,20 @@ The following keys are valid for this section:
       ``requirements.yml`` file (see below for examples). Read more about
       the requirements file format in the `Galaxy user guide <https://docs.ansible.com/ansible/latest/galaxy/user_guide.html#install-multiple-collections-with-a-requirements-file>`_.
 
+    .. _python-pep508:
+
     ``python``
       The Python installation requirements. This may either be a filename, or a
       list of requirements (see below for an example).
 
+      .. note::
+
+        Python requirement specifications are expected to be limited to features defined by
+        `PEP 508 <https://peps.python.org/pep-0508/>`_. Hash tag comments will always be allowed.
+        Any deviation from this specification will be passed through to pip unverified and unaltered,
+        although this is considered undefined and unsupported behavior. It is not recommended that
+        you depend on this behavior.
+
     ``python_interpreter``
       A dictionary that defines the Python system package name to be installed by
       ``dnf`` (``package_system``) and/or a path to the Python interpreter to be used
@@ -245,6 +260,57 @@ The following keys are valid for this section:
       The system packages to be installed, in bindep format. This may either
       be a filename, or a list of requirements (see below for an example).
 
+    ``exclude``
+      A dictionary defining the Python or system requirements to be excluded from the top-level dependency
+      requirements of referenced collections. These exclusions will not apply to the user supplied Python or
+      system dependencies, nor will they apply to dependencies of dependencies (top-level only).
+
+      The following keys are valid for this section:
+
+        * ``python`` - A list of Python dependencies to be excluded.
+        * ``system`` - A list of system dependencies to be excluded.
+        * ``all_from_collections`` - If you want to exclude *all* Python and system dependencies from one or
+          more collections, supply a list of collection names under this key.
+
+      The exclusion feature supports two forms of matching:
+
+        * Simple name matching.
+        * Advanced name matching using regular expressions.
+
+      For simple name matching, you need only supply the name of the requirement/collection to match.
+      All values will be compared in a case-insensitive manner.
+
+      For advanced name matching, begin the exclusion string with the tilde (``~``) character to
+      indicate that the remaining portion of the string is a regular expression to be used to match
+      a requirement/collection name. The regex should be considered case-insensitive.
+
+      .. note::
+        The regular expression must match the full requirement/collection name. For example, ``~foo.``
+        does not fully match the name ``foobar``, but ``~foo.+`` does.
+
+      With both forms of matching, the exclusion string will be compared against the *simple* name of
+      any Python or system requirement. For example, if you need to exclude the system requirement that
+      appears as ``foo [!platform:gentoo]`` within an included collection, then your exclusion string should be
+      ``foo``. To exclude the Python requirement ``bar == 1.0.0``, your exclusion string would be ``bar``.
+
+      Example using both simple and advanced matching:
+
+      .. code:: yaml
+
+        dependencies:
+            exclude:
+                python:
+                    - docker
+                system:
+                    - python3-Cython
+                all_from_collections:
+                    # Regular expression to exclude all from community collections
+                    - ~community\..+
+
+      .. note::
+        The ``exclude`` option requires ``ansible-builder`` version ``3.1`` or newer.
+
+
 The following example uses filenames that contain various dependencies:
 
 .. code:: yaml

setup.cfg

@@ -36,10 +36,9 @@ max-line-length=160
 include_package_data = true
 install_requires =
     PyYAML
-    requirements_parser
     bindep
     jsonschema
-    setuptools; python_version >= "3.12"
+    packaging
 python_requires = >=3.9