palssave 504 timeouts at /dashboard/works

[aprilrieger]

Story

Client reported error: I'm getting a 504 error when I attempted to access this page in the dashboard. https://palsave.hykucommons.org/dashboard/works?locale=en

Convo in slack link: https://assaydepot.slack.com/archives/C0313NKC08L/p1731946496472579

Went to ingress-nginx logs with stern: stern ^ingress-nginx-controller- -n ingress-nginx | grep --line-buffered "palni-palci-production-" | grep --line-buffered "https://palsave.hykucommons.org/dashboard/works?locale=en"

Found logs:

ingress-nginx-controller-5dcb699bfd-zst4c controller 10.0.6.79 - - [19/Nov/2024:16:09:40 +0000] "GET /dashboard/works?locale=en HTTP/2.0" 499 0 "https://palsave.hykucommons.org/dashboard/works?locale=en" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" 138 41.807 [palni-palci-production-palni-palci-production-nginx-80] [] 10.0.6.220:8080 0 41.807 - 237dcd94eace9d50c9904c937484de95
ingress-nginx-controller-5dcb699bfd-zst4c controller 10.0.6.79 - - [19/Nov/2024:16:10:40 +0000] "GET /dashboard/my/collections?locale=en HTTP/2.0" 504 562 "https://palsave.hykucommons.org/dashboard/works?locale=en" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" 82 60.004 [palni-palci-production-palni-palci-production-nginx-80] [] 10.0.6.220:8080 562 60.005 504 5f1dc801fbdbf6527e944eefe9cf8fbd
ingress-nginx-controller-5dcb699bfd-zst4c controller 10.0.5.43 - - [19/Nov/2024:16:15:03 +0000] "GET /dashboard/works?locale=en HTTP/2.0" 499 0 "https://palsave.hykucommons.org/dashboard/works?locale=en" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" 1248 3.020 [palni-palci-production-palni-palci-production-nginx-80] [] 10.0.6.220:8080 0 2.966 - 46b2deae43ad0c70d8a271716f4e1511
ingress-nginx-controller-5dcb699bfd-zst4c controller 10.0.5.43 - - [19/Nov/2024:16:16:03 +0000] "GET /dashboard/works?locale=en HTTP/2.0" 499 0 "https://palsave.hykucommons.org/dashboard/works?locale=en" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" 46 60.017 [palni-palci-production-palni-palci-production-nginx-80] [] 10.0.6.220:8080 0 59.989 - 3936b53076067a24c590fc2bd793ed79
ingress-nginx-controller-5dcb699bfd-zst4c controller 10.0.5.43 - - [19/Nov/2024:16:17:03 +0000] "GET /dashboard/works?locale=en HTTP/2.0" 504 562 "https://palsave.hykucommons.org/dashboard/works?locale=en" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" 201 60.030 [palni-palci-production-palni-palci-production-nginx-80] [] 10.0.6.220:8080 562 60.002 504 9f2ac67b45bdd6ba10341ff2e9053ee7
ingress-nginx-controller-5dcb699bfd-zst4c controller 10.0.6.94 - - [19/Nov/2024:16:31:07 +0000] "GET /assets/application-082e0c591d730d693244f9d2526032d39593d1edd9ef4236bb52ba4f8a6ac486.css HTTP/2.0" 200 75272 "https://palsave.hykucommons.org/dashboard/works?locale=en" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36" 621 0.005 [palni-palci-production-palni-palci-production-nginx-80] [] 10.0.6.220:8080 75272 0.005 200 7186fe3e4b60c0f1eb2ffc1112d8f8f8
ingress-nginx-controller-5dcb699bfd-zst4c controller 10.0.6.94 - - [19/Nov/2024:16:31:07 +0000] "GET /system/logo_images/1/original/PALSave_reversed_repository__1_.png HTTP/2.0" 200 176591 "https://palsave.hykucommons.org/dashboard/works?locale=en" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36" 132 0.021 [palni-palci-production-palni-palci-production-nginx-80] [] 10.0.6.220:8080 176591 0.021 200 87c2ab89a30444406735790c807a1523
ingress-nginx-controller-5dcb699bfd-zst4c controller 10.0.6.94 - - [19/Nov/2024:16:31:07 +0000] "GET /assets/application-0a080b94471b04104915f4fcf97fe84ff2f6379cc4121f38b1beae8c6ab1a7b3.js HTTP/2.0" 200 592389 "https://palsave.hykucommons.org/dashboard/works?locale=en" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36" 98 0.027 [palni-palci-production-palni-palci-production-nginx-80] [] 10.0.6.220:8080 592389 0.027 200 c9d189e30ec31584fc51353707b094c1
ingress-nginx-controller-5dcb699bfd-zst4c controller 10.0.6.79 - - [19/Nov/2024:16:41:35 +0000] "GET /dashboard/my/works?locale=en HTTP/2.0" 200 5392 "https://palsave.hykucommons.org/dashboard/works?locale=en" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36" 707 1.075 [palni-palci-production-palni-palci-production-nginx-80] [] 10.0.6.220:8080 5411 1.047 200 97bd4cb4a5d51111606e6cd685b88136
ingress-nginx-controller-5dcb699bfd-zst4c controller 10.0.6.79 - - [19/Nov/2024:16:43:57 +0000] "GET /notifications?locale=en HTTP/2.0" 499 0 "https://palsave.hykucommons.org/dashboard/works?locale=en" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36" 496 36.158 [palni-palci-production-palni-palci-production-nginx-80] [] 10.0.6.220:8080 0 36.130 - 0a5e80f1788c628537635a2270c7efb2
ingress-nginx-controller-5dcb699bfd-zst4c controller 10.0.6.79 - - [19/Nov/2024:16:45:02 +0000] "GET /dashboard/works?locale=en HTTP/2.0" 504 562 "https://palsave.hykucommons.org/dashboard/works?locale=en" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36" 189 60.033 [palni-palci-production-palni-palci-production-nginx-80] [] 10.0.6.220:8080 562 60.003 504 5cd6378d5808b2a3d3daad54a8862cb8
ingress-nginx-controller-5dcb699bfd-zst4c controller 10.0.6.79 - - [19/Nov/2024:16:45:02 +0000] "GET /favicon.ico HTTP/2.0" 200 0 "https://palsave.hykucommons.org/dashboard/works?locale=en" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36" 73 0.030 [palni-palci-production-palni-palci-production-nginx-80] [] 10.0.6.220:8080 0 0.003 200 a96346a4442be6b14203ddb7c5b61d55

Key Observations

• Requests to /dashboard/works?locale=en are timing out consistently after 60 seconds.
• The client closed the connection before the server could respond, possibly due to long wait times.
• The logs show the upstream service (10.0.6.220:8080) taking significant time to respond, e.g., 60.033 seconds or 41.807 seconds. (Service palsfcrepo:8080)

I reviewed the logs in service palsfcrepo:8080 and saw 58 occurrences of:

WARN (S3AbortableInputStream) Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection.

I went to track down the bucket information and found the bucket info here in rancher: https://r.notch8.cloud/dashboard/c/c-w9925/explorer/secret/palni-palci-production/palsfcrepo?mode=edit#data

And went to the SoftServ AWS account in the besties-fcrepo bucket and cannot see any logging setup, and logging for this service will cost extra -- need to get permission to add logging or find another way to follow the trail.

Acceptance Criteria

• Client can log in an a Super Admin and navigate to https://palsave.hykucommons.org/dashboard/works?locale=en without receiving a 504 timeout error

[aprilrieger]

Ok I have the page loading finally and the site speed is back up to a tolerable level. I had to block Googlebot, which I know client didnt want.

I tried to setup a WAF (Security Firewall) rule to rate-limit and verified bot that accesses the expensive queries and that didnt work.

So this is a bandaid for now, I will update the ticket and the client and we will have to punt to a time when I can get some time or grab someone’s attention