Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Kafka #91

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Add Kafka #91

wants to merge 14 commits into from

Conversation

elisalimli
Copy link
Contributor

@elisalimli elisalimli commented Mar 11, 2024

What does this PR do?

image

Fixes #39 #53

Steps to test

  1. Run docker-compose up -d to spin up kafka, kafka-ui and redis
  2. Start python server
  3. Run cd ./service/kafka/ & python3 ./consume.py

@elisalimli elisalimli requested a review from homanp March 11, 2024 16:29
@elisalimli elisalimli marked this pull request as ready for review March 11, 2024 16:29
@teocns
Copy link

teocns commented Mar 11, 2024

Hey @elisalimli, I am just taking a look at ray which can simplify the integration

Also worth reading distributed kafka consumers with ray

@teocns
Copy link

teocns commented Mar 11, 2024

I am testing this in a Kubernetes cluster and provide feedback along the way;

@elisalimli
Copy link
Contributor Author

Hey @elisalimli, I am just taking a look at ray which can simplify the integration

Also worth reading distributed kafka consumers with ray

@teocns I haven't looked at it yet. What would be the benefit of using Ray instead of just Kafka?

@elisalimli
Copy link
Contributor Author

@homanp we can deploy this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Offload ingestion to background tasks
2 participants