Add data loader service and update documentation for local development #190
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Introduce new data_loader service for sample data ingestion - Configure data_loader to use Ubuntu 20.04 base image - Install kafkacat and tar utilities in data_loader container - Set up volume mounting for load_data.sh script and data directory - Configure data_loader to run with 'load-data' profile for optional execution - Ensure data_loader depends on Kafka service - Set environment variables for Kafka broker, topic, and data file location
- Use Ubuntu 20.04 base image - Install kafkacat and tar - Set /app as working directory - Set entrypoint to /bin/bash
- Extract tarball of sample data - Wait for Kafka broker to be ready - Create topic if not exists - Upload JSON data to Kafka topic
- Add details on docker-compose services - Include steps for running tests, loading sample data - Clarify build and run instructions with feature flags
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a data loader service to facilitate sample data ingestion and updates the documentation for a smoother local development experience.
Key changes: