From 4292ef91793e805dc39bc649f655b901754c39ad Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue, 5 Dec 2023 12:58:48 +0100 Subject: [PATCH 01/11] Shutdown DLQ segments flusher only if it has been started (#15649) (#15656) In DLQ unit testing sometime the DLQ writer is started explicitly without starting the segments flushers. In such cases the test 's logs contains exceptions which could lead to think that the test fails silently. Avoid to invoke scheduledFlusher's shutdown when it's not started (such behaviour is present only in tests). (cherry picked from commit eddd91454f35a996ea80185399897004d43a8771) Co-authored-by: Andrea Selva --- .../java/org/logstash/common/io/DeadLetterQueueWriter.java | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/logstash-core/src/main/java/org/logstash/common/io/DeadLetterQueueWriter.java b/logstash-core/src/main/java/org/logstash/common/io/DeadLetterQueueWriter.java index 342c3ce751e..b321005b49b 100644 --- a/logstash-core/src/main/java/org/logstash/common/io/DeadLetterQueueWriter.java +++ b/logstash-core/src/main/java/org/logstash/common/io/DeadLetterQueueWriter.java @@ -254,7 +254,10 @@ public void close() { } try { - flushScheduler.shutdown(); + // flushScheduler is null only if it's not explicitly started, which happens only in tests. + if (flushScheduler != null) { + flushScheduler.shutdown(); + } } catch (Exception e) { logger.warn("Unable shutdown flush scheduler, ignoring", e); } From 44788ccfc15fc3cb739f4ac255b2f45edec80345 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 6 Dec 2023 17:23:19 -0500 Subject: [PATCH 02/11] Doc: Update Logstash intro and security overview for serverless (#15313) (#15664) (cherry picked from commit 0954a687f0599cc57cbeee08b9056c9235347082) Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> --- docs/index.asciidoc | 20 +++++++++++++++ docs/static/security/es-security.asciidoc | 30 +++++++++++++++++------ 2 files changed, 42 insertions(+), 8 deletions(-) diff --git a/docs/index.asciidoc b/docs/index.asciidoc index 41eedb50105..2b623e59bb6 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -34,6 +34,26 @@ type of event can be enriched and transformed with a broad array of input, filte native codecs further simplifying the ingestion process. Logstash accelerates your insights by harnessing a greater volume and variety of data. + +[serverless] +.Logstash to {serverless-full} +**** +You'll use the {ls} <> to send data to {serverless-full}. +Note these differences between {es-serverless} and both {ess} and self-managed {es}: + +* Use *API keys* to access {serverless-full} from {ls}. +Any user-based security settings in your in your <> configuration are ignored and may cause errors. +* {serverless-full} uses *data streams* and {ref}/data-stream-lifecycle.html[{dlm} ({dlm-init})] instead of {ilm} ({ilm-init}). +Any {ilm-init} settings in your <> configuration are ignored and may cause errors. +* *{ls} monitoring* is available through the https://github.com/elastic/integrations/blob/main/packages/logstash/_dev/build/docs/README.md[{ls} Integration] in {serverless-docs}/observability/what-is-observability-serverless[Elastic Observability] on {serverless-full}. + + +.Known issue for {ls} to {es-serverless}. + +The logstash-output-elasticsearch `hosts` setting defaults to port :9200. Set the value to port :443 instead. +**** + + // The pass blocks here point to the correct repository for the edit links in the guide. // Introduction diff --git a/docs/static/security/es-security.asciidoc b/docs/static/security/es-security.asciidoc index 0c985aa1f4b..5df08ec12e6 100644 --- a/docs/static/security/es-security.asciidoc +++ b/docs/static/security/es-security.asciidoc @@ -11,21 +11,35 @@ See {ref}/configuring-stack-security.html[Starting the Elastic Stack with securi {ess} uses certificates signed by standard publicly trusted certificate authorities, and therefore setting a cacert is not necessary. -.Hosted {ess} simplifies security +.Security to {serverless-full} [[serverless]] [NOTE] ===== -Our hosted {ess} on Elastic Cloud simplifies safe, secure communication between {ls} and {es}. -When you configure the elasticsearch output plugin to use <> with either the <> or the <>, no additional SSL configuration steps are needed. -Examples: +{es-serverless} simplifies safe, secure communication between {ls} and {es}. -* `output {elasticsearch { cloud_id => "" cloud_auth => "" } }` -* `output {elasticsearch { cloud_id => "" api_key => "" } }`` +Configure the <> to use <> and an <> to establish safe, secure communication between {ls} and {es-serverless}. +No additional SSL configuration steps are needed. + +Configuration example: -For more details, check out the -{logstash-ref}/connecting-to-cloud.html[Logstash-to-Cloud documentation]. +* `output {elasticsearch { cloud_id => "" api_key => "" } }` +For more details, check out <>. +===== + +.Security to hosted {ess} [[hosted-ess]] +[NOTE] +===== +Our hosted {ess} on Elastic Cloud simplifies safe, secure communication between {ls} and {es}. +When you configure the <> to use <> with either the <> or the <>, no additional SSL configuration steps are needed. {ess-leadin-short} + +Configuration example: + +* `output {elasticsearch { cloud_id => "" cloud_auth => "" } }` +* `output {elasticsearch { cloud_id => "" api_key => "" } }` + +For more details, check out <> or <>. ===== [discrete] From 4e3e09c5d6e0b06ac138a1f05668142375c99704 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 7 Dec 2023 11:45:11 +0000 Subject: [PATCH 03/11] Release notes for 8.11.2 (#15660) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: logstashmachine <43502315+logstashmachine@users.noreply.github.com> Co-authored-by: João Duarte --- docs/static/releasenotes.asciidoc | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/docs/static/releasenotes.asciidoc b/docs/static/releasenotes.asciidoc index 2ee2ec8b9ee..1419411b085 100644 --- a/docs/static/releasenotes.asciidoc +++ b/docs/static/releasenotes.asciidoc @@ -3,6 +3,7 @@ This section summarizes the changes in the following releases: +* <> * <> * <> * <> @@ -49,6 +50,32 @@ This section summarizes the changes in the following releases: * <> +[[logstash-8-11-2]] +=== Logstash 8.11.2 Release Notes + +* Added missing method of logger wrapper for puma https://github.com/elastic/logstash/pull/15642[#15642] +* Prevent calling shutdown on the DLQ segments flusher if it hasn't been started yet https://github.com/elastic/logstash/pull/15656[#15656] +* Remove dependency on jackson-dataformat-yaml https://github.com/elastic/logstash/pull/15599[#15599] + +==== Plugins + +*Mutate Filter - 3.5.8* + +* Fix "Can't modify frozen string" error when converting boolean to `string` https://github.com/logstash-plugins/logstash-filter-mutate/pull/171[#171] + +*Beats Input - 6.7.2* + +* Restore Lumberjack event parsing source code https://github.com/logstash-plugins/logstash-input-beats/pull/486[#486] + +*Elastic_serverless_forwarder Input - 0.1.4* + +* [DOC] Adds tips for using the logstash-input-elastic_serverless_forwarder plugin with the Elasticsearch output plugin https://github.com/logstash-plugins/logstash-input-elastic_serverless_forwarder/pull/7[#7] + +*Validator_support Mixin - 1.1.1* + +* Allow single-word host names such as "localhost". This addresses the inability to set "hosts" to "localhost" in the logstash-filter-elastic_integration plugin. https://github.com/logstash-plugins/logstash-mixin-validator_support/pull/7[#7] + + [[logstash-8-11-1]] === Logstash 8.11.1 Release Notes From 3d22fb395acf0991d557824ede3683488da315d8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Duarte?= Date: Thu, 7 Dec 2023 15:29:51 +0000 Subject: [PATCH 04/11] bump to 8.11.3 (#15669) --- Gemfile.jruby-3.1.lock.release | 4 ++-- versions.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Gemfile.jruby-3.1.lock.release b/Gemfile.jruby-3.1.lock.release index de45509fa03..de1e2f41218 100644 --- a/Gemfile.jruby-3.1.lock.release +++ b/Gemfile.jruby-3.1.lock.release @@ -2,12 +2,12 @@ PATH remote: logstash-core-plugin-api specs: logstash-core-plugin-api (2.1.16-java) - logstash-core (= 8.11.2) + logstash-core (= 8.11.3) PATH remote: logstash-core specs: - logstash-core (8.11.2-java) + logstash-core (8.11.3-java) cgi (~> 0.3.6) clamp (~> 1) concurrent-ruby (~> 1, < 1.1.10) diff --git a/versions.yml b/versions.yml index a3eecb7566c..9333bd35dad 100644 --- a/versions.yml +++ b/versions.yml @@ -1,7 +1,7 @@ --- # alpha and beta qualifiers are now added via VERSION_QUALIFIER environment var -logstash: 8.11.2 -logstash-core: 8.11.2 +logstash: 8.11.3 +logstash-core: 8.11.3 logstash-core-plugin-api: 2.1.16 bundled_jdk: From 45f6dce3e8aa9919e6daa0043a882e12ba3ca97a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Duarte?= Date: Thu, 7 Dec 2023 18:41:20 +0000 Subject: [PATCH 05/11] Update JRuby to 9.4.5.0 (#15531) (#15670) * update lockfile template's BUNDLED WITH to match jruby's bundler * upgrade jruby-core in rubyUtils.gradle --- Gemfile.jruby-3.1.lock.release | 2 +- rubyUtils.gradle | 2 +- versions.yml | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Gemfile.jruby-3.1.lock.release b/Gemfile.jruby-3.1.lock.release index de1e2f41218..e13dbc7891c 100644 --- a/Gemfile.jruby-3.1.lock.release +++ b/Gemfile.jruby-3.1.lock.release @@ -1019,4 +1019,4 @@ DEPENDENCIES webmock (~> 3) BUNDLED WITH - 2.3.25 + 2.3.26 diff --git a/rubyUtils.gradle b/rubyUtils.gradle index 67b3474763b..1a803e9e2a1 100644 --- a/rubyUtils.gradle +++ b/rubyUtils.gradle @@ -25,7 +25,7 @@ buildscript { dependencies { classpath "org.yaml:snakeyaml:${snakeYamlVersion}" classpath "de.undercouch:gradle-download-task:4.0.4" - classpath "org.jruby:jruby-core:9.4.2.0" + classpath "org.jruby:jruby-core:9.4.5.0" } } diff --git a/versions.yml b/versions.yml index 9333bd35dad..b0774f6f401 100644 --- a/versions.yml +++ b/versions.yml @@ -13,8 +13,8 @@ bundled_jdk: # jruby must reference a *released* version of jruby which can be downloaded from the official download url # *and* for which jars artifacts are published for compile-time jruby: - version: 9.4.2.0 - sha1: c338c1d3846e51b651e31e248097fdee4920056a + version: 9.4.5.0 + sha1: 9f2d039c975659a641d5002d0999aec73b963260 # jruby-runtime-override, if specified, will override the jruby version installed in vendor/jruby #jruby-runtime-override: # url: https://oss.sonatype.org/content/repositories/snapshots/org/jruby/jruby-dist/9.3.0.0-SNAPSHOT/jruby-dist-9.3.0.0-20210723.214927-259-bin.tar.gz From 7e9f9e079dca38f5689de515902f290e283ba912 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 7 Dec 2023 15:00:07 -0500 Subject: [PATCH 06/11] Doc: Add docs for extending integrations with filter-elastic_integrations (#15518) (#15675) (cherry picked from commit 6c2e123a2409dcf78ef2103ed4f568380debea18) Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> --- docs/index.asciidoc | 4 ++ docs/static/ea-integrations.asciidoc | 79 ++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+) create mode 100644 docs/static/ea-integrations.asciidoc diff --git a/docs/index.asciidoc b/docs/index.asciidoc index 2b623e59bb6..d6905d8ff9e 100644 --- a/docs/index.asciidoc +++ b/docs/index.asciidoc @@ -133,6 +133,10 @@ include::static/config-management.asciidoc[] include::static/management/configuring-centralized-pipelines.asciidoc[] +// EA Integrations to Logstash +// (Planting near module content for now. Will likely move it up in info architecture.) +include::static/ea-integrations.asciidoc[] + // Working with Logstash Modules include::static/modules.asciidoc[] diff --git a/docs/static/ea-integrations.asciidoc b/docs/static/ea-integrations.asciidoc new file mode 100644 index 00000000000..4d4340c3acb --- /dev/null +++ b/docs/static/ea-integrations.asciidoc @@ -0,0 +1,79 @@ +[[ea-integrations]] +== Using {ls} with Elastic {integrations} (Beta) + +You can take advantage of the extensive, built-in capabilities of Elastic {integrations}--such as managing data collection, transformation, and visualization--and then use {ls} for additional data processing and output options. +{ls} can further expand capabilities for use cases where you need additional processing, or if you need your data delivered to multiple destinations. + +[discrete] +[[integrations-value]] +=== Elastic {integrations}: ingesting to visualizing + +https://docs.elastic.co/integrations[Elastic {integrations}] provide quick, end-to-end solutions for: + +* ingesting data from a variety of data sources +* getting the data into the {stack}, and +* visualizing it with purpose-built dashboards. + +{integrations} are available for https://docs.elastic.co/integrations/all_integrations[popular services and platforms], such as Nginx, AWS, and MongoDB, as well as many generic input types like log files. +Each integration includes pre-packaged assets to help reduce the time between ingest and insights. + +To see available integrations, go to the {kib} home page, and click **Add {integrations}**. +You can use the query bar to search for integrations you may want to use. +When you find an integration for your data source, the UI walks you through adding and configuring it. + +[discrete] +[[integrations-and-ls]] +=== Extend {integrations} with {ls} (Beta) + +Logstash can run the ingest pipeline component of your integration when you use the Logstash filter-elastic_integration plugin. + +.How to + +**** +Create a {ls} pipeline that uses the <> plugin, and the https://github.com/elastic/logstash-filter-elastic_integration[logstash-filter-elastic_integration] plugin as the _first_ filter in your {ls} pipeline. +You can add more filters for additional processing, but they must come after the `logstash-filter-elastic_integration` plugin in your configuration. +Add an output plugin to complete your pipeline. +**** + + +**Sample pipeline configuration** + +[source,ruby] +----- +input { + elastic_agent { + port => 5044 + } +} + +filter { + elastic_integration{ <1> + cloud_id => "" + cloud_auth => " + source => "[http][host]" + target => "[@metadata][tenant]" + dictionary_path => "/etc/conf.d/logstash/tenants.yml" + } +} + +output { <3> + if [@metadata][tenant] == "tenant01" { + elasticsearch { + cloud_id => "" + api_key => "" + } + } else if [@metadata][tenant] == "tenant02" { + elasticsearch { + cloud_id => "" + api_key => "" + } + } +} +----- + +<1> Use `filter-elastic_agent` as the first filter in your pipeline +<2> You can use additional filters as long as they follow `filter-elastic_agent` +<3> Sample config to output data to multiple destinations From 53d154c54fb9dd68e2f53d47a45301441552a2d5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Duarte?= Date: Tue, 12 Dec 2023 10:09:08 +0000 Subject: [PATCH 07/11] bump version to 8.11.4 (#15682) --- Gemfile.jruby-3.1.lock.release | 4 ++-- versions.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Gemfile.jruby-3.1.lock.release b/Gemfile.jruby-3.1.lock.release index e13dbc7891c..fc1806f3533 100644 --- a/Gemfile.jruby-3.1.lock.release +++ b/Gemfile.jruby-3.1.lock.release @@ -2,12 +2,12 @@ PATH remote: logstash-core-plugin-api specs: logstash-core-plugin-api (2.1.16-java) - logstash-core (= 8.11.3) + logstash-core (= 8.11.4) PATH remote: logstash-core specs: - logstash-core (8.11.3-java) + logstash-core (8.11.4-java) cgi (~> 0.3.6) clamp (~> 1) concurrent-ruby (~> 1, < 1.1.10) diff --git a/versions.yml b/versions.yml index b0774f6f401..6b145f405b5 100644 --- a/versions.yml +++ b/versions.yml @@ -1,7 +1,7 @@ --- # alpha and beta qualifiers are now added via VERSION_QUALIFIER environment var -logstash: 8.11.3 -logstash-core: 8.11.3 +logstash: 8.11.4 +logstash-core: 8.11.4 logstash-core-plugin-api: 2.1.16 bundled_jdk: From a45349ea474085899b4a19bc164794395631c010 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Tue, 12 Dec 2023 09:17:46 -0500 Subject: [PATCH 08/11] Release notes for 8.11.3 (#15681) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: João Duarte Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com> --- docs/static/releasenotes.asciidoc | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/static/releasenotes.asciidoc b/docs/static/releasenotes.asciidoc index 1419411b085..8432c13239a 100644 --- a/docs/static/releasenotes.asciidoc +++ b/docs/static/releasenotes.asciidoc @@ -3,6 +3,7 @@ This section summarizes the changes in the following releases: +* <> * <> * <> * <> @@ -50,6 +51,19 @@ This section summarizes the changes in the following releases: * <> +[[logstash-8-11-3]] +=== Logstash 8.11.3 Release Notes + +[[docs-8.11.3]] +==== Documentation enhancements + +* Document how to further transform events processed by the filter-elastic_integration plugin https://github.com/elastic/logstash/pull/15675[#15675] + +[[dependencies-8.11.3]] +==== Updates to dependencies + +* Update JRuby to 9.4.5.0 https://github.com/elastic/logstash/pull/15670[#15670] + [[logstash-8-11-2]] === Logstash 8.11.2 Release Notes From 7c0fc32f83caedc16231e7019d5e37fea52b117d Mon Sep 17 00:00:00 2001 From: Rob Bavey Date: Tue, 19 Dec 2023 05:37:33 -0500 Subject: [PATCH 09/11] Remove temporary dependencies in logstash-core.gemspec (#15694) When the initial 8.11.0 release was created, a late downgrade in jruby from `9.4.3.0` to `9.4.2.0` was required to avoid a silent crash during `Concurrent::TimerTask`. However, as part of the downgrade, we also added some cherry-picked dependencies to logstash-core to still include some of the stlib fixes from `9.4.3.0` in the Logstash release. We recently upgraded the version of jruby from `9.4.2.0` to `9.4.5.0`, but did not remove these cherry-picked dependencies, some of which are now outdated, leading to issues such as #15687. This commit removes those cherry-picked dependencies and updates the lockfile --- Gemfile.jruby-3.1.lock.release | 41 +++++++---------------------- logstash-core/logstash-core.gemspec | 12 --------- 2 files changed, 10 insertions(+), 43 deletions(-) diff --git a/Gemfile.jruby-3.1.lock.release b/Gemfile.jruby-3.1.lock.release index fc1806f3533..7d404b2f982 100644 --- a/Gemfile.jruby-3.1.lock.release +++ b/Gemfile.jruby-3.1.lock.release @@ -8,14 +8,10 @@ PATH remote: logstash-core specs: logstash-core (8.11.4-java) - cgi (~> 0.3.6) clamp (~> 1) concurrent-ruby (~> 1, < 1.1.10) - date (~> 3.3.3) down (~> 5.2.0) elasticsearch (~> 7) - ffi (~> 1.15.5) - ffi-binary-libfixposix (~> 0.5.1.1) filesize (~> 0.2) gems (~> 1) i18n (~> 1) @@ -23,27 +19,21 @@ PATH jruby-openssl (~> 0.14.1) manticore (~> 0.6) minitar (~> 0.8) - net-http (~> 0.3.0) - net-protocol (~> 0.1.2) pry (~> 0.12) puma (~> 6.3, >= 6.3.1) rack (~> 2) - reline (~> 0.3.5) rubyzip (~> 1) sinatra (~> 2) stud (~> 0.0.19) thread_safe (~> 0.3.6) thwait - time (~> 0.2.2) - timeout (~> 0.3.2) treetop (~> 1) tzinfo-data - uri (~> 0.12.1) GEM remote: https://rubygems.org/ specs: - addressable (2.8.5) + addressable (2.8.6) public_suffix (>= 2.0.2, < 6.0) amazing_print (1.5.0) arr-pm (0.0.12) @@ -89,12 +79,11 @@ GEM belzebuth (0.2.3) childprocess benchmark-ips (2.12.0) - bigdecimal (3.1.4-java) + bigdecimal (3.1.5-java) bindata (2.4.15) buftok (0.2.0) builder (3.2.4) cabin (0.9.0) - cgi (0.3.6-java) childprocess (4.1.0) ci_reporter (2.1.0) builder (>= 2.1.2) @@ -157,9 +146,7 @@ GEM faraday-patron (1.0.0) faraday-rack (1.0.0) faraday-retry (1.0.3) - ffi (1.15.5-java) - ffi-binary-libfixposix (0.5.1.1-java) - ffi (~> 1.0) + ffi (1.16.3-java) filesize (0.2.0) fivemat (1.3.7) flores (0.0.8) @@ -193,7 +180,6 @@ GEM i18n (1.14.1) concurrent-ruby (~> 1.0) insist (1.0.0) - io-console (0.6.0-java) jar-dependencies (0.4.1) jls-grok (0.11.5) cabin (>= 0.6.0) @@ -281,7 +267,7 @@ GEM logstash-mixin-ecs_compatibility_support (~> 1.3) logstash-mixin-event_support (~> 1.0) logstash-patterns-core - logstash-codec-netflow (4.3.0) + logstash-codec-netflow (4.3.1) bindata (>= 1.5.0) logstash-core-plugin-api (~> 2.0) logstash-mixin-event_support (~> 1.0) @@ -458,7 +444,7 @@ GEM logstash-mixin-ecs_compatibility_support (~> 1.3) logstash-mixin-scheduler (~> 1.0) stud (~> 0.0.22) - logstash-input-file (4.4.5) + logstash-input-file (4.4.6) addressable concurrent-ruby (~> 1.0) logstash-codec-multiline (~> 3.0) @@ -565,13 +551,13 @@ GEM logstash-core-plugin-api (>= 1.60, <= 2.99) logstash-mixin-ecs_compatibility_support (~> 1.2) logstash-mixin-normalize_config_support (~> 1.0) - logstash-input-twitter (4.1.0) + logstash-input-twitter (4.1.1) http-form_data (~> 2) logstash-core-plugin-api (>= 1.60, <= 2.99) logstash-mixin-ecs_compatibility_support (~> 1.3) logstash-mixin-event_support (~> 1.0) logstash-mixin-validator_support (~> 1.0) - public_suffix (~> 3) + public_suffix (> 4, < 6) stud (>= 0.0.22, < 0.1) twitter (= 6.2.0) logstash-input-udp (3.5.0) @@ -747,9 +733,7 @@ GEM mustermann (2.0.2) ruby2_keywords (~> 0.0.1) naught (1.1.0) - net-http (0.3.2) - uri - net-imap (0.4.6) + net-imap (0.4.8) date net-protocol net-pop (0.1.2) @@ -784,7 +768,7 @@ GEM spoon (~> 0.0) psych (5.1.1.1-java) jar-dependencies (>= 0.1.7) - public_suffix (3.1.1) + public_suffix (5.0.4) puma (6.4.0-java) nio4r (~> 2.0) raabro (1.4.0) @@ -797,9 +781,7 @@ GEM rainbow (3.1.1) rake (13.0.6) redis (4.8.1) - regexp_parser (2.8.2) - reline (0.3.9) - io-console (~> 0.5) + regexp_parser (2.8.3) rexml (3.2.6) rspec (3.12.0) rspec-core (~> 3.12.0) @@ -855,8 +837,6 @@ GEM thwait (0.2.0) e2mmap tilt (2.3.0) - time (0.2.2) - date timeout (0.3.2) treetop (1.6.12) polyglot (~> 0.3) @@ -877,7 +857,6 @@ GEM tzinfo (>= 1.0.0) unf (0.1.4-java) unicode-display_width (2.5.0) - uri (0.12.2) webhdfs (0.10.2) addressable webmock (3.19.1) diff --git a/logstash-core/logstash-core.gemspec b/logstash-core/logstash-core.gemspec index 00aa9fb1a93..442e013b95e 100644 --- a/logstash-core/logstash-core.gemspec +++ b/logstash-core/logstash-core.gemspec @@ -85,16 +85,4 @@ Gem::Specification.new do |gem| # Using the scheduler directly might lead to issues e.g. when join-ing, see: # https://github.com/logstash-plugins/logstash-mixin-scheduler/blob/v1.0.1/lib/logstash/plugin_mixins/scheduler/rufus_impl.rb#L85= # and https://github.com/elastic/logstash/issues/13773 - - # TEMPORARY: delta between JRuby 9.4.2.0 and 9.4.3.0 - gem.add_runtime_dependency "cgi", "~> 0.3.6" - gem.add_runtime_dependency "date", "~> 3.3.3" - gem.add_runtime_dependency "ffi", "~> 1.15.5" - gem.add_runtime_dependency "net-http", "~> 0.3.0" - gem.add_runtime_dependency "net-protocol", "~> 0.1.2" - gem.add_runtime_dependency "reline", "~> 0.3.5" - gem.add_runtime_dependency "ffi-binary-libfixposix", "~> 0.5.1.1" - gem.add_runtime_dependency "time", "~> 0.2.2" - gem.add_runtime_dependency "timeout", "~> 0.3.2" - gem.add_runtime_dependency "uri", "~> 0.12.1" end From 86ea3206792dcdc60e47ccbea6432da31312831a Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 20 Dec 2023 18:50:17 +0200 Subject: [PATCH 10/11] [ci] Reusable unit + IT test steps for Buildkite (#15708) (#15714) This commit is a pre-requisite for adding unit + IT tests in a dedicated phase of the Exhaustive tests pipeline. It refactors the tests currently used by PR jobs, so that they become reusable. (cherry picked from commit 03d7b59f2aec85e0a0224ddd6edaa3ba54bf5217) Co-authored-by: Dimitrios Liappis --- .buildkite/pull_request_pipeline.yml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.buildkite/pull_request_pipeline.yml b/.buildkite/pull_request_pipeline.yml index 19fa8b78927..524cad810dd 100644 --- a/.buildkite/pull_request_pipeline.yml +++ b/.buildkite/pull_request_pipeline.yml @@ -38,9 +38,14 @@ steps: ephemeralStorage: "100Gi" command: | set -euo pipefail + if [[ $BUILDKITE_PULL_REQUEST == "false" ]]; then + # https://github.com/elastic/logstash/pull/15486 for background + export ENABLE_SONARQUBE="false" + else + source .buildkite/scripts/pull-requests/sonar-env.sh + fi source .buildkite/scripts/common/container-agent.sh - source .buildkite/scripts/pull-requests/sonar-env.sh ci/unit_tests.sh java - label: ":lab_coat: Integration Tests / part 1" From 84b4fec2dac6726129f567e1fa598475083f9007 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Thu, 21 Dec 2023 12:42:47 +0200 Subject: [PATCH 11/11] [ci] Add testing phase to exhaustive tests suite (#15711) (#15717) This is the second part of the migration of the exhaustive/main Jenkins Job to Buildkite. So far we've migrated the "compatibility phase" and this commit adds the "testing phase"[^1], which is essentially the same amount of tests that we ran on PR jobs. Relates https://github.com/elastic/ingest-dev/issues/1722 Depends https://github.com/elastic/logstash/pull/15708 [^1]: For more details, refer to the sequence diagram in https://github.com/elastic/ingest-dev/issues/1722#issuecomment-1824378635 (cherry picked from commit 9538338abbd282395d758bf807f91246b597d53e) Co-authored-by: Dimitrios Liappis --- .../scripts/exhaustive-tests/generate-steps.py | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/.buildkite/scripts/exhaustive-tests/generate-steps.py b/.buildkite/scripts/exhaustive-tests/generate-steps.py index 07a302c13aa..03ad4ceb4ea 100644 --- a/.buildkite/scripts/exhaustive-tests/generate-steps.py +++ b/.buildkite/scripts/exhaustive-tests/generate-steps.py @@ -9,6 +9,7 @@ VM_IMAGES_FILE = ".buildkite/scripts/common/vm-images.json" VM_IMAGE_PREFIX = "platform-ingest-logstash-multi-jdk-" +CUR_PATH = os.path.dirname(os.path.abspath(__file__)) def slugify_bk_key(key: str) -> str: """ @@ -20,6 +21,10 @@ def slugify_bk_key(key: str) -> str: return key.translate(mapping_table) +def testing_phase_steps() -> typing.Dict[str, typing.List[typing.Any]]: + with open(os.path.join(CUR_PATH, "..", "..", "pull_request_pipeline.yml")) as fp: + return YAML().load(fp) + def compat_linux_step(imagesuffix: str) -> dict[str, typing.Any]: linux_command = LiteralScalarString("""#!/usr/bin/env bash set -eo pipefail @@ -89,15 +94,23 @@ def randomized_windows_os() -> str: structure = {"steps": []} + structure["steps"].append({ + "group": "Testing Phase", + "key": "testing-phase", + **testing_phase_steps(), + }) + structure["steps"].append({ "group": "Compatibility / Linux", "key": "compatibility-linux", + "depends_on": "testing-phase", "steps": compat_linux_steps, }) structure["steps"].append({ "group": "Compatibility / Windows", "key": "compatibility-windows", + "depends_on": "testing-phase", "steps": [compat_windows_step(imagesuffix=windows_test_os)], })