LS-to-LS - Add Integration Testing

[roaksoax]

Description

Core level integration tests

The goal is to make sure every core release has a validated/working version of the plugin.
To provide some internal details, we have multiple integration tests under the qa/integration package under the core repo and they validate the plugins such as es-output, kafka-input, etc.. plugins to be sure they work with current releasing version.
Similarly, we need to add an integration test so that should prove the logstash-integration-logstash plugin is not broken.

Plugin level integration tests

The goal here, we need to make sure every change made has to be validated before the merge.
As we are doing with some plugins, such as Elasticsearch output plugin, we need to spin up the docker environment. For this case, Logstash docker env by installing the logstash-integration-logstash current changes and desired pipeline configs. We probably also need to directly update the Gemfile since we use the source or another point is to directly install and run the docker as I have tested on the local. Then, we should be able to run another docker env to run the integration tests.
Integration tests should cover:

• basic plain data transfer
• ECS compatibility
• SSL configurations
• Setting up username and password would be also beneficial
• retry: this requires host(s) to be down or got backpressure, thinking how we can achieve the retry scenarious

# create a logstash-container docker container
docker create --name logstash-container --network elastic-package-stack_default \
  -h logstash-host \
  -p 9600:9600 -p 9800:9800/tcp \
  docker.elastic.co/logstash/logstash:"${ELASTIC_STACK_VERSION}"

# copy ES certificates
docker cp tmp/certs logstash-container:/usr/share/logstash/config

# run logstash-container, copy locally built plugin to container and install plugin
docker start logstash-container &
sleep 10
docker exec -it logstash-container sh -c "mkdir plugins"
cd .. && cd .. && cd ..
docker cp logstash-integration-logstash logstash-container:/usr/share/logstash/plugins/
cd logstash-integration-logstash/.buildkite/scripts

# TODO: Replace if plugin is embedded, otherwise append at tail
docker exec -it logstash-container sh -c "echo 'gem \"logstash-integration-logstash\", :path=>\"/usr/share/logstash/plugins/logstash-integration-logstash\"' >> /usr/share/logstash/Gemfile"

# Install the plugin
docker exec -it logstash-container sh -c "bin/logstash-plugin install --no-verify"

# copy config files to container and rerun logstash-container
docker cp e2e-framework/configs/logstash.conf logstash-container:/usr/share/logstash/pipeline
docker cp e2e-framework/configs/logstash.yml logstash-container:/usr/share/logstash/config

# restarting Logstash after installing the plugin
docker stop logstash-container
sleep 5
docker start logstash-container
sleep 5

Tasks

Preview Give feedback

1. Add integration testing for Logstash core (Simple logstash to logstash)
2. Add integration testing for Plugin per-pr (one logstash to multiple outputs w/SSL)