If condition regex issue with fields containing newlines

[JustinHendersonSMAPPER]

Logstash information:

Please include the following information:

1. Logstash version (e.g. bin/logstash --version)

Testing with multiple version of Logstash 8 such as 8.2.0 and 8.8.1.

2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker)
   Docker
3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)
   Docker

Plugins installed: (bin/logstash-plugin list --verbose)
Default plus logstash-output-opensearch

JVM (e.g. java -version):
Default docker JVMs

OS version (uname -a if on a Unix-like system):
Tested on Windows, CentOS, and Ubuntu 22.04 including x64 and ARM

For ARM I was testing on:
Linux stl-ancillary01 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:29:36 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Description of the problem including expected versus actual behavior:
When trying to do an if condition using a regex expression that starts with (^) and ends with ($), the regex does not work as expected if the field contains newlines. When there are newlines in a field, it seems the behavior is to analyze each line of the field's value and then return True if any single line matches the passed regexp. Either the default Logstash regex does not work as expected or this behavior should have more documentation for conditional events when using regexp (https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html).

Steps to reproduce:
For example:

Test Log:

{"message":"The forest has not been registered with Azure. Password policies cannot be downloaded from Azure unless this is corrected.\n \n Resolution steps: an administrator must run the Register-AzureADPasswordProtectionForest cmdlet which is installed as part of the Azure AD Password Protection Proxy software.\n \n Additional information may be available at https://aka.ms/AzureADPasswordProtection ","winlog":{"task":"None","provider_guid":"{FCE041B2-EACD-48A2-8E09-4D5D43C0FF69}","user":{"identifier":"S-1-5-18","type":"User","domain":"NT AUTHORITY","name":"SYSTEM"},"api":"wineventlog","record_id":87466,"channel":"Microsoft-AzureADPasswordProtection-DCAgent/Admin","computer_name":"computer1.test.int","provider_name":"Microsoft-AzureADPasswordProtection-DCAgent","event_id":"30016","opcode":"Info","activity_id":"{87561BD6-8854-4A3A-B9F6-1660030AF393}","process":{"thread":{"id":8468},"pid":4116}},"@timestamp":"2023-10-18T19:30:05.896Z","log":{"level":"warning"},"@version":"1","ecs":{"version":"8.0.0"},"agent":{"type":"winlogbeat","version":"8.7.1","ephemeral_id":"3cb270df-8f79-4721-9a92-f8d85c6b920f","name":"computer1"},"host":{"name":"computer1","hostname":"computer1","architecture":"x86_64","os":{"name":"Windows Server 2012 R2 Standard","kernel":"6.3.9600.21620 (winblue_ltsb_escrow.230929-1158)","build":"9600.21620","version":"6.3","type":"windows","family":"windows","platform":"windows"},"event":{"action":"None","provider":"Microsoft-AzureADPasswordProtection-DCAgent","code":"30016","kind":"event","created":"2023-10-18T19:30:07.478Z"},"tags":["beats_input_codec_plain_applied"]}

Or a simplified test log:

{"message":"This is\n  \n a test"}

If you use an if condition such as the =~ condition in the filter below, both logs above will get dropped even though the intended regex is to delete any logs that have a message field beginning with or ending with only whitespace.

filter {
  if [message] {
    if [message] =~ "^\s*$" {
      drop {}
    }
  }

If the Logstash configuration is updated to below with the /g flag, it works as intended.

filter {
  if [message] {
    if [message] =~ "^\s*$/g" {
      drop {}
    }
  }