Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the Arcsight module and the modules framework #16794

Merged
merged 7 commits into from
Dec 19, 2024

Conversation

kaisecheng
Copy link
Contributor

@kaisecheng kaisecheng commented Dec 13, 2024

Release notes

Remove Arcsight module

What does this PR do?

Remove all module related code

  • remove arcsight module
  • remove module framework
  • remove module tests
  • remove module configs

The removal of doc is addressed in #16796

Why is it important/What is the impact to the user?

All modules are removed in v9.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files (and/or docker env variables)
  • I have added tests that prove my fix is effective or that my feature works

Author's Checklist

  • passed exhaustive test
  • tested legacy monitoring
  • tested central pipeline management
  • bin/logstash --modules arcsight --setup gives ERROR: Unrecognised option '--modules'
  • Config module in logstash.yml gives Your settings are invalid. Reason: Setting "modules" doesn't exist
    modules:
    - name: arcsight
    

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@kaisecheng kaisecheng linked an issue Dec 13, 2024 that may be closed by this pull request
@kaisecheng kaisecheng marked this pull request as ready for review December 13, 2024 15:00
@@ -126,8 +126,6 @@ var validSettings = []string{
"xpack.management.elasticsearch.ssl.cipher_suites",
"xpack.geoip.download.endpoint",
"xpack.geoip.downloader.enabled",
"cloud.id",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these "cloud" settings related to the module being removed?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see these are removed in i18n etc too.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is cloud.id here only relevant to modules, or is this where it is also set for central management and internal collection?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cloud.id is only used for modules
CPM uses xpack.management.elasticsearch.cloud_id. Legacy monitoring uses xpack.monitoring.elasticsearch.cloud_id

Copy link
Member

@donoghuc donoghuc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a clean removal. +28 −7,374 is pretty nice!

Copy link
Member

@robbavey robbavey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some questions about cloud.id

docker/data/logstash/env2yaml/env2yaml.go Show resolved Hide resolved
# var.PLUGIN_TYPE.PLUGIN_NAME.KEY
#
# modules:
#
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Cloud Settings below still refer to module specific settings var.elasticsearch.hosts and var.kibana.host, etc.
If these settings are not relevant at all, we should remove them. If they are relevant to central management and internal collection, we should move and rework them

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deleted

@@ -126,8 +126,6 @@ var validSettings = []string{
"xpack.management.elasticsearch.ssl.cipher_suites",
"xpack.geoip.download.endpoint",
"xpack.geoip.downloader.enabled",
"cloud.id",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is cloud.id here only relevant to modules, or is this where it is also set for central management and internal collection?

@robbavey
Copy link
Member

I think there are some additional removals -

I don't believe we use this ElasticsearchClient outside of Modules code, despite it living outside of the main modules code area...
Once that has gone, I believe this can go too, along with
https://github.com/elastic/logstash/blob/main/logstash-core/lib/logstash/util/modules_setting_array.rb

Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

@kaisecheng
Copy link
Contributor Author

@robbavey this is ready for a second look
cloud.id is for modules only. I have removed the mentioned dangling code.

@kaisecheng kaisecheng requested a review from robbavey December 17, 2024 15:32
Copy link
Member

@robbavey robbavey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is so awesome! I do have a couple of questions before approval:

  • Do we know why those two require statements suddenly needed to be added? Was it related to this PR, or a general issue?
  • It didn't look like it from reading through the code, but did we confirm whether or not there are dependencies that we import that we no longer need to after this PR?

@@ -21,6 +21,8 @@
require "spec_helper"
require "open-uri"
require "webmock/rspec"
require "faraday"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, do you know what changed that forced this change? Is it part of this change? Or something outside of it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is caused by the removal of elasticsearch_client.rb and the import in runner_spec.rb. The elasticsearch related classes somehow include Faraday.

The import in other spec files affect the global namespace for the entire test suite. You can add require "faraday" in runner_spec and get a green run of ./gradlew :logstash-core:rubyTests, but it will fail when running a single rspec file as webserver_spec itself doesn't link faraday.

@kaisecheng kaisecheng requested a review from robbavey December 19, 2024 12:57
Copy link
Member

@robbavey robbavey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@kaisecheng kaisecheng merged commit 0578974 into elastic:main Dec 19, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Remove the Modules framework code from the main branch of Logstash. Remove the Arcsight module from x-pack
4 participants