Monolith OSS - Performance problems on ingest beyond 72 hours

[MaduMitha-Ravi]

Problem 1 - High Mean CPU usage on Query load (small and large)

• Continuous Ingestion of data on OSS on the software without the limit of 72h
• Query load simulated after 72h of Ingest is observed to have high CPU usage compared to the same query load before the 72h (around 26h)
  • Mean CPU jumps from 46% to 84%
  • Max CPU (on before 72h) is known from LIMIT query

Problem 2 - Increase in Latency for 5m look back

• Continuous Ingestion of data on OSS on the software without the limit of 72h
• Query load simulated after 72h of Ingest is observed to have higher query latency compared to the same query load before the 72h (around 26h) even for a small query look such as 5m
  • Mean Query latency (p95) goes from 32ms to 111ms for 5m query look back, about 3X times