Add kafka producer to send task status messages #5405
Conversation
|
@kahowell We would like to see this work move forward. This PR needs to be rebased. The documentation now lives in In order for one of us to test this, it would be great if you could add a guide called "Integrate with Kafka" here: https://github.com/pulp/pulpcore/tree/main/staging_docs/admin/guides This guide should provide instructions on how to configure Pulp and create some kind of consumer. I don't have any experience in this area, so I will defer to you on what should be included in the guide. |
|
Actually, the runtime environment is provided in: https://github.com/pulp/oci_env/pull/159/files. Can we also copy these parts into the official pulpcore documentation? |
|
@dkliban I rebased and moved the documentation as requested, and then retested to make sure things still work as intended. A quick note, likely unrelated to this PR: I had troubles this time getting my oci_env to play nice until I manually removed pulp_cli_deb from the environment and maybe some other dep things, I lost track 😄. |
|
There are some checks failing (e.g. this work needs a changelog entry in CHANGES/) but other than that, this LGTM. |
To try locally, you can use the oci-env kafka profile from pulp/oci_env#159. Set up the oci-env to use the kafka profile: ``` COMPOSE_PROFILE=kafka ``` From a fresh oci-env pulp instance, try: ```shell export REPO_NAME=$(head /dev/urandom | tr -dc a-z | head -c5) export REMOTE_NAME=$(head /dev/urandom | tr -dc a-z | head -c5) oci-env pulp file repository create --name $REPO_NAME oci-env pulp file remote create --name $REMOTE_NAME \ --url 'https://fixtures.pulpproject.org/file/PULP_MANIFEST' oci-env pulp file repository sync --name $REPO_NAME --remote $REMOTE_NAME ``` Then inspect the kafka message that is produced via: ```shell oci-env exec -s kafka \ /opt/kafka/bin/kafka-console-consumer.sh \ --bootstrap-server=localhost:9092 \ --offset earliest \ --partition 0 \ --topic pulpcore.tasking.status \ --max-messages 1 ``` Closes pulp#5337
To try locally, you can use the oci-env kafka profile from pulp/oci_env#159.
Set up the oci-env to use the kafka profile:
From a fresh oci-env pulp instance, try:
Then inspect the kafka message that is produced via:
oci-env exec -s kafka \ /opt/kafka/bin/kafka-console-consumer.sh \ --bootstrap-server=localhost:9092 \ --offset earliest \ --partition 0 \ --topic pulpcore.tasking.status \ --max-messages 1Closes #5337