Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the Arcsight module and the modules framework #16794

Merged
merged 7 commits into from
Dec 19, 2024
Merged

Remove the Arcsight module and the modules framework #16794

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c2e15ad
remove modules framework
kaisecheng Dec 13, 2024
74002f7
remove env2yaml
kaisecheng Dec 13, 2024
094092f
remove i18n
kaisecheng Dec 13, 2024
ecd7c1d
remove cloud.id config in yml
kaisecheng Dec 17, 2024
a3f35e1
remove modules_setting_array
kaisecheng Dec 17, 2024
b47188a
remove elasticsearch_client and ssl helper
kaisecheng Dec 17, 2024
e74fd14
fix import
kaisecheng Dec 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 0 additions & 4 deletions build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -333,7 +333,6 @@ tasks.register("assembleTarDistribution") {
inputs.files fileTree("${projectDir}/bin")
inputs.files fileTree("${projectDir}/config")
inputs.files fileTree("${projectDir}/lib")
inputs.files fileTree("${projectDir}/modules")
inputs.files fileTree("${projectDir}/logstash-core-plugin-api")
inputs.files fileTree("${projectDir}/logstash-core/lib")
inputs.files fileTree("${projectDir}/logstash-core/src")
Expand All @@ -350,7 +349,6 @@ tasks.register("assembleOssTarDistribution") {
inputs.files fileTree("${projectDir}/bin")
inputs.files fileTree("${projectDir}/config")
inputs.files fileTree("${projectDir}/lib")
inputs.files fileTree("${projectDir}/modules")
inputs.files fileTree("${projectDir}/logstash-core-plugin-api")
inputs.files fileTree("${projectDir}/logstash-core/lib")
inputs.files fileTree("${projectDir}/logstash-core/src")
Expand All @@ -365,7 +363,6 @@ tasks.register("assembleZipDistribution") {
inputs.files fileTree("${projectDir}/bin")
inputs.files fileTree("${projectDir}/config")
inputs.files fileTree("${projectDir}/lib")
inputs.files fileTree("${projectDir}/modules")
inputs.files fileTree("${projectDir}/logstash-core-plugin-api")
inputs.files fileTree("${projectDir}/logstash-core/lib")
inputs.files fileTree("${projectDir}/logstash-core/src")
Expand All @@ -382,7 +379,6 @@ tasks.register("assembleOssZipDistribution") {
inputs.files fileTree("${projectDir}/bin")
inputs.files fileTree("${projectDir}/config")
inputs.files fileTree("${projectDir}/lib")
inputs.files fileTree("${projectDir}/modules")
inputs.files fileTree("${projectDir}/logstash-core-plugin-api")
inputs.files fileTree("${projectDir}/logstash-core/lib")
inputs.files fileTree("${projectDir}/logstash-core/src")
Expand Down
19 changes: 0 additions & 19 deletions config/logstash.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,25 +181,6 @@
#
# api.auth.basic.password_policy.mode: WARN
#
# ------------ Module Settings ---------------
# Define modules here. Modules definitions must be defined as an array.
# The simple way to see this is to prepend each `name` with a `-`, and keep
# all associated variables under the `name` they are associated with, and
# above the next, like this:
#
# modules:
# - name: MODULE_NAME
# var.PLUGINTYPE1.PLUGINNAME1.KEY1: VALUE
# var.PLUGINTYPE1.PLUGINNAME1.KEY2: VALUE
# var.PLUGINTYPE2.PLUGINNAME1.KEY1: VALUE
# var.PLUGINTYPE3.PLUGINNAME3.KEY1: VALUE
#
# Module variable names must be in the format of
#
# var.PLUGIN_TYPE.PLUGIN_NAME.KEY
#
# modules:
#
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Cloud Settings below still refer to module specific settings var.elasticsearch.hosts and var.kibana.host, etc.
If these settings are not relevant at all, we should remove them. If they are relevant to central management and internal collection, we should move and rework them

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deleted

# ------------ Cloud Settings ---------------
# Define Elastic Cloud settings here.
# Format of cloud.id is a base64 value e.g. dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRub3RhcmVhbCRpZGVudGlmaWVy
Expand Down
6 changes: 2 additions & 4 deletions docker/data/logstash/env2yaml/env2yaml.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,12 @@ package main
import (
"errors"
"fmt"
"gopkg.in/yaml.v2"
donoghuc marked this conversation as resolved.
Show resolved Hide resolved
"io/ioutil"
"log"
"os"
"strings"

"gopkg.in/yaml.v2"
)

var validSettings = []string{
Expand Down Expand Up @@ -67,7 +68,6 @@ var validSettings = []string{
"log.level",
"log.format",
"log.format.json.fix_duplicate_message_fields",
"modules",
"metric.collect",
"path.logs",
"path.plugins",
Expand Down Expand Up @@ -126,8 +126,6 @@ var validSettings = []string{
"xpack.management.elasticsearch.ssl.cipher_suites",
"xpack.geoip.download.endpoint",
"xpack.geoip.downloader.enabled",
"cloud.id",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these "cloud" settings related to the module being removed?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see these are removed in i18n etc too.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is cloud.id here only relevant to modules, or is this where it is also set for central management and internal collection?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cloud.id is only used for modules
CPM uses xpack.management.elasticsearch.cloud_id. Legacy monitoring uses xpack.monitoring.elasticsearch.cloud_id

"cloud.auth",
}

// Given a setting name, return a downcased version with delimiters removed.
Expand Down
130 changes: 0 additions & 130 deletions logstash-core/lib/logstash/config/modules_common.rb
View file
Open in desktop

This file was deleted.

32 changes: 0 additions & 32 deletions logstash-core/lib/logstash/config/source/base.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,37 +71,5 @@ def config_path
def config_path?
!(config_path.nil? || config_path.empty?)
end

def modules_cli_setting
@settings.get_setting("modules.cli")
end

def modules_cli
modules_cli_setting.value
end

def modules_cli?
!(modules_cli.nil? || modules_cli.empty?)
end

def modules_setting
@settings.get_setting("modules")
end

def modules
modules_setting.value
end

def modules?
!(modules.nil? || modules.empty?)
end

def both_module_configs?
modules_cli? && modules?
end

def modules_defined?
modules_cli? || modules?
end
end
end end end
2 changes: 1 addition & 1 deletion logstash-core/lib/logstash/config/source/local.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ def pipeline_configs

def match?
# see basic settings predicates and getters defined in the base class
(config_string? || config_path?) && !(modules_cli? || modules?) && !automatic_reload_with_config_string?
(config_string? || config_path?) && !automatic_reload_with_config_string?
end

def config_conflict?
Expand Down
68 changes: 0 additions & 68 deletions logstash-core/lib/logstash/config/source/modules.rb
View file
Open in desktop

This file was deleted.

6 changes: 3 additions & 3 deletions logstash-core/lib/logstash/config/source/multi_local.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ def pipeline_configs
end

def match?
if modules_cli? || modules? || config_string? || config_path?
if config_string? || config_path?
return false
end
detect_pipelines if !@detect_pipelines_called
Expand All @@ -62,7 +62,7 @@ def invalid_pipelines_detected?
def config_conflict?
@conflict_messages.clear
# are there any auto-reload conflicts?
if !(modules_cli? || modules? || config_string? || config_path?)
if !(config_string? || config_path?)
detect_pipelines if !@detect_pipelines_called
if @detected_marker.nil?
@conflict_messages << I18n.t("logstash.runner.config-pipelines-failed-read", :path => pipelines_yaml_location)
Expand All @@ -74,7 +74,7 @@ def config_conflict?
@conflict_messages << @detected_marker.message
end
else
do_warning? && logger.warn("Ignoring the 'pipelines.yml' file because modules or command line options are specified")
do_warning? && logger.warn("Ignoring the 'pipelines.yml' file because command line options are specified")
end
@conflict_messages.any?
end
Expand Down
1 change: 0 additions & 1 deletion logstash-core/lib/logstash/config/source_loader.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@
# under the License.

require "logstash/config/source/local"
require "logstash/config/source/modules"
require "logstash/config/source/multi_local"
require "thread"
require "set"
Expand Down
8 changes: 0 additions & 8 deletions logstash-core/lib/logstash/environment.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@
require "logstash/settings"
require "logstash/util/cloud_setting_id"
require "logstash/util/cloud_setting_auth"
require "logstash/util/modules_setting_array"
require "socket"
require "stud/temporary"

Expand All @@ -39,13 +38,6 @@ module Environment
Setting::NullableString.new("path.config", nil, false),
Setting::WritableDirectory.new("path.data", ::File.join(LogStash::Environment::LOGSTASH_HOME, "data")),
Setting::NullableString.new("config.string", nil, false),
Setting::Modules.new("modules.cli", LogStash::Util::ModulesSettingArray, []),
Setting::Modules.new("modules", LogStash::Util::ModulesSettingArray, []),
Setting.new("modules_list", Array, []),
Setting.new("modules_variable_list", Array, []),
Setting::Modules.new("cloud.id", LogStash::Util::CloudSettingId),
Setting::Modules.new("cloud.auth", LogStash::Util::CloudSettingAuth),
Setting::Boolean.new("modules_setup", false),
Setting::Boolean.new("config.test_and_exit", false),
Setting::Boolean.new("config.reload.automatic", false),
Setting::TimeValue.new("config.reload.interval", "3s"), # in seconds
Expand Down
Loading
Loading