testDLQWriterFlusherRemovesExpiredSegmentWhenCurrentHeadSegmentIsEmpty fails on Windows 2022/2019 using Buildkite

[dliappis]

While transitioning JDK matrix jobs from Jenkins to Buildkite we've spotted a strange failure with testDLQWriterFlusherRemovesExpiredSegmentWhenCurrentHeadSegmentIsEmpty. It happens on Windows 2022, 2019 and occasionally on 2019 agents. Interestingly enough it doesn't fail using the same VM image / user / CSP account / instance type when running it in an interactive UI session over RDP.

Example failure:

org.logstash.common.io.DeadLetterQueueWriterAgeRetentionTest > testDLQWriterFlusherRemovesExpiredSegmentWhenCurrentHeadSegmentIsEmpty FAILED
--
  | org.awaitility.core.ConditionTimeoutException: Condition was evaluated in 986 milliseconds which is earlier than expected minimum timeout 1 seconds
  | at app//org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:172)
  | at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:78)
  | at app//org.awaitility.core.CallableCondition.await(CallableCondition.java:26)
  | at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
  | at app//org.awaitility.core.ConditionFactory.until(ConditionFactory.java:954)
  | at app//org.logstash.common.io.DeadLetterQueueWriterAgeRetentionTest.testDLQWriterFlusherRemovesExpiredSegmentWhenCurrentHeadSegmentIsEmpty(DeadLetterQueueWriterAgeRetentionTest.java:345)

By looking at the test, we tested the following scenarios:

1. Increase flushInterval from 1s to 2s

This results in success in Buildkite.

2. Introduce a delay between writing a DLQ entry and starting the Awaitility test i.e. like so:

diff --git a/logstash-core/src/test/java/org/logstash/common/io/DeadLetterQueueWriterAgeRetentionTest.java b/logstash-core/src/test/java/org/logstash/common/io/DeadLetterQueueWriterAgeRetentionTest.java
index 6c9bb5a02..69cfe3f1a 100644
--- a/logstash-core/src/test/java/org/logstash/common/io/DeadLetterQueueWriterAgeRetentionTest.java
+++ b/logstash-core/src/test/java/org/logstash/common/io/DeadLetterQueueWriterAgeRetentionTest.java
@@ -337,6 +337,7 @@ public class DeadLetterQueueWriterAgeRetentionTest {

             DLQEntry entry = new DLQEntry(event, "", "", "00001", DeadLetterQueueReaderTest.constantSerializationLengthTimestamp(fakeClock));
             writeManager.writeEntry(entry);
+            Thread.sleep(50);

             // wait the flush interval so that the current head segment is sealed
             Awaitility.await("After the flush interval head segment is sealed and a fresh empty head is created")

makes the test fail, on an interactive Windows 2019 session where the default test always succeeds.

This indicates that, if there is a delay (e.g. due to an operating system context switch) between starting to write the DLQ file and the Awaitility test, it may delay starting the test and its timer such that the condition that the file doesn't become available <1s fails. Therefore, there is potential for this test to fail, regardless of the OS, if cycles get stolen between the two actions.

[andsel]

This is greatly due to the fact that we are testing about time and this make the test fragile.
At first we should remove the invocations of atLeast, the original intent of it was to give the scheduled task inside the DQL writer enough time to run its sealer job.

Add a TODO to refactor the scheduler inside DLQ writer so that it can abstract from time, and respond to events like "schedule period elapsed". In this way the test could easily test conditions like "after a schedule period is elapsed, and after the sealer jab executed verify that the set of segment files is what we expect".

---

This is addressed by #15594

[andsel]

Fixed by #15680

[andsel]

Follow up issue for BK failure not related to the way we used Awaitility in this context: #15767