Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add ClickHouse connection #57

Merged
merged 60 commits into from
Dec 30, 2024
Merged

Add ClickHouse connection #57

merged 60 commits into from
Dec 30, 2024

Conversation

lamr02n
Copy link
Collaborator

@lamr02n lamr02n commented Dec 6, 2024

All the connectors for ClickHouse as well as the insert operations were added. The database now receives information regarding timestamps, messages and more about Loglines and Batches.

lamr02n added 30 commits November 20, 2024 11:52
@lamr02n
Add monitoring related files
9bd5bdf
@lamr02n
Merge branch 'refs/heads/main' into add-clickhouse-connection
0baa73b 
# Conflicts:
#	config.yaml
#	tests/test_server.py
@lamr02n
Make LogServer write to ClickHouse
1e25801
@lamr02n
Add requirements.monitoring.txt
2986ea9
@lamr02n
Make Collector write to ClickHouse
04199b1
@lamr02n
Add forbidden field names to logline_handler.py
150fe7d
@lamr02n
Fix dns_loglines insertion
a501ba2
@lamr02n
Add dns_loglines insertion in collector.py
63746d7
@lamr02n
Add logline_status insertion in collector.py
393cc01
@lamr02n
Add logline_timestamps insertion in collector.py
db16775
@lamr02n
Add development query script
61776ff
@lamr02n
Update logline_timestamps insertion in collector.py
a321ebf
@lamr02n
Bug fixing
9203bc2
@lamr02n
Add batch_id to Batch object and make Prefilter use Batch
6b0e27c
@lamr02n
Add logline_timestamps insertion in batch_handler.py
386494a
@lamr02n
Move Batch datatype to its own file
56787c5
@lamr02n
Rename clickhouse_batch.py to clickhouse_batch_sender.py
a333511
@lamr02n
Use Marshmallow to transmit data to MonitoringAgent
e20f441
@lamr02n
Make BufferedBatch store Batch IDs
360ff50
@lamr02n
Add database insertion in BufferedBatch
0318a92
@lamr02n
Switch status field in connectors by is_active
63dacf9
@lamr02n
Use correct transactional_id in batch_handler.py
13aab6b
@lamr02n
Update tests for batch_handler.py
eabde81
@lamr02n
Update tests and fix bugs
133320f
@lamr02n
Add batch_timestamps insertion in batch_handler.py
abadc20
@lamr02n
Update test_inspector.py
468803d
@lamr02n
Update status field for Collector database insertion
b181280
@lamr02n
Update Prefilter and add database insertion
7f9655c
@lamr02n
Bug fix
e6711db
@lamr02n
Update Inspector and add database insertion
1a2f2b9
stefanDeveloper
stefanDeveloper approved these changes Dec 13, 2024
View reviewed changes
Copy link
Owner

@stefanDeveloper stefanDeveloper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me :)

@stefanDeveloper
Copy link
Owner

Do you know why windows tests are still failing?

@lamr02n
Copy link
Collaborator Author

lamr02n commented Dec 13, 2024

Do you know why windows tests are still failing?

Not exactly, but it might have to do with caches. We could try to clear those.

@lamr02n
Copy link
Collaborator Author

lamr02n commented Dec 13, 2024

I will do a couple of manual integration tests before merging to ensure everything still works as it should.

@stefanDeveloper
Copy link
Owner

Do you know why windows tests are still failing?

Not exactly, but it might have to do with caches. We could try to clear those.

I fixed the problem with the mingw versions.

@stefanDeveloper stefanDeveloper added this to the First heiDGAF version milestone Dec 13, 2024
@stefanDeveloper
Copy link
Owner

I will do a couple of manual integration tests before merging to ensure everything still works as it should.

After your integration tests, please put the merge request in ready to merge. I will merge it accordingly.

@stefanDeveloper stefanDeveloper mentioned this pull request Dec 13, 2024
Closed
This was linked to issues Dec 13, 2024
Add table showing and explaining configuration values #31
Open
Implement proper data classes #43
Closed
@lamr02n lamr02n marked this pull request as ready for review December 13, 2024 15:22
@lamr02n lamr02n linked an issue Dec 23, 2024 that may be closed by this pull request
Add timestamps between stages for traceability #11
Closed
@lamr02n lamr02n removed a link to an issue Dec 23, 2024
Add table showing and explaining configuration values #31
Open
@stefanDeveloper stefanDeveloper merged commit 23409c8 into main Dec 30, 2024
17 checks passed
@lamr02n lamr02n deleted the add-clickhouse-connection branch December 30, 2024 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stefanDeveloper stefanDeveloper stefanDeveloper approved these changes

Assignees

@lamr02n lamr02n

Labels
enhancement New feature or request
Projects
None yet
Milestone
First heiDGAF version
Development

Successfully merging this pull request may close these issues.

Implement proper data classes Add timestamps between stages for traceability
2 participants
@lamr02n @stefanDeveloper