[bitnami/discourse] Intermittent slow page load times for Discourse

[manning-ncsa]

My Discourse site (only a few dozen users and very little activity so far) exhibits intermittent slow page/component load times that can be anywhere from 3-20 seconds. This occurs approximately once every ten page loads (as I repeatedly press keys G, J to cycle through posts, for example). During these events, the spinner spins and the user has to simply wait.

I have been watching the browser dev console and the server log as I navigate around trying to identify the cause of these slow load times, and I have identified at least two of the culprits: /timings and 123.json?track_visit=true&forceLoad=true (where 123 is the topic ID).

Changing the "flush timings secs" site setting does not seem to change the behavior.

Does anyone have advice how to troubleshoot this problem? A few seconds here and there would be tolerable, but I fear that these frequent 5-20 second load times will stifle use of the platform.

Our instance is deployed on Kubernetes using the Bitnami Helm chart, Docker image docker.io/bitnami/discourse:2.7.8-debian-10-r0. I tried asking for help on the Discourse forum but this is considered an unsupported deployment method.

[carrodher]

Please, note the Bitnami team is not actively monitoring GitHub Issues for this Helm chart. This issue will stay open for 20 days to allow the community to contribute, after 15 days without activity the issue will be marked as stale being closed after 5 days.

If you would like to contribute to this Helm chart by fixing a bug or adding a new feature, feel free to create a PR, the Bitnami team will be happy to review it and provide feedback.

[manning-ncsa]

Thank you @carrodher. It seemed inappropriate to create a GitHub issue like this, but I only did it because the https://community.bitnami.com/ Discourse post editor would not let me create a topic there where I assumed it would receive more feedback.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[leventov]

We have exactly the same problem. @manning-ncsa was you able to hunt down this issue?

[manning-ncsa]

@leventov Unfortunately, I have had no luck identifying the cause or a solution. The effort it would take to start hacking the source code to debug on my own has been a prohibitive barrier for me so far.

[aqua-pro]

Hello, have you found a solution to this issue? The same thing is happening with my light (15 people) community. The server machine has 8c/16t at 3.7 GHz, 64GB RAM, 2x 512 GB SSD and 1 GBPS dl/ul. Thank you

[deprecastor]

Hello, we face the same issue, does you drop the app or found a solution?
EDIT: We can see that there is "your.url.tld/something/poll" called and is canceled by the browser after five dot something seconds.
EDIT2: @leventov @aqua-pro @manning-ncsa plz follow the issue: 172

[manning-ncsa]

@deprecastor Thanks for the link to that other issue. Following the conversation and links to other issues, it appears that this is actually caused by some component choices by Bitnami, namely the use of Passenger. One solution mentioned in this comment is to use the official Docker image instead of the Bitnami image. It would be easier to disable long polling as described here but I do not actually see that option in my admin settings! (We are running discourse-7.3.7.)

[deprecastor]

@manning-ncsa yes we also miss this option. This is kind of weird. The app Version is 2.8.4, so I would have expected this "disable polling" thingy. Well as @carrodher already mentioned, we have to fix that on our own. Well did you already tried out the bitnami docker image? Will have some time for that tomorrow, maybe :-D

[deprecastor]

@manning-ncsa There is kind of a solution. Please read this
There the msg bus for passenger:
location /message-bus { passenger_app_group_name foo_websocket; passenger_force_max_concurrent_requests_per_process 0; }
We added this to the "nginx.conf" in the /tmp/ folder, something with "passenger-standalone". Then restart the nginx.. like this: /opt/bitnami/discourse/vendor/bundle/ruby/2.7.0/gems/passenger-6.0.14/buildout/support-binaries/nginx-1.20.2 -c /tmp/passenger-standalone.27uc9x/nginx.conf -p /tmp/passenger-standalone.27uc9x -s reload
The result is promising and we think this could work. But my coworkers and I have no clue how to bring this into code. This is just a tmp fix and if the POD moves it is gone.

[aqua-pro]

Thank you so much @deprecastor! That last message of yours works perfectly ❤️

[karpfediem]

Thanks for digging this up @deprecastor !
It is a rough workaround, but atleast it works.
For anyone stumbling across, i made a small (dirty) script to automate these steps.

Feel free to adjust to your deployment and improve, while we can wait for bitnami/containers#2355

#!/bin/env bash
set -x

NAMESPACE="discourse"
POD_NAME=$(kubectl get pods -n $NAMESPACE | grep 'discourse' | head -n 1 | awk '{print $1}')
TMP_NAME=$(kubectl exec -i -t -n $NAMESPACE $POD_NAME -c discourse -- sh -c "find /tmp -type d -name 'passenger-standalone*'" 2>/dev/null)
TMP_NAME=$(echo $TMP_NAME | sed -e 's/[$\r]//g')

function pod_exec {
  kubectl exec -i -t -n $NAMESPACE $POD_NAME -c discourse -- sh -c "$@" 2>/dev/null
}

function configureMessageBus {
  pod_exec "sed -i 's/        ### END your own configuration options ###/        location \/message-bus {\\
          passenger_app_group_name foo_websocket;\\
          passenger_force_max_concurrent_requests_per_process 0;\\
        }\\
        ### END your own configuration options ###/g' ${TMP_NAME}/nginx.conf"
}

function reloadPassenger {
  pod_exec "/opt/bitnami/discourse/vendor/bundle/ruby/2.7.0/gems/passenger-6.0.14/buildout/support-binaries/nginx-1.20.2 -c $TMP_NAME/nginx.conf -p $TMP_NAME/ -s reload"
}


configureMessageBus

reloadPassenger

[deprecastor]

Hello @karpfediem, sry I forgotten to show the final solution to go around this here. In the PR that I opened and then closed, you can read a commend from the User @alemorcuq. You add your custom nginx config by using ".Values.discourse.extraVolumeMounts" and then use ".Values.discourse.extraEnvVars" field to set the option to load it like in my comment. You must create 'DISCOURSE_PASSENGER_EXTRA_FLAGS' because it is used by the docker image in use. More about the nginx config template -> https://www.phusionpassenger.com/library/config/standalone/reference/#--nginx-config-template-nginx_config_template

[karpfediem]

You must create 'DISCOURSE_PASSENGER_EXTRA_FLAGS' because it is used by the docker image in use.

Ah that's possibly even better then. I saw your PR a while ago and thought it tried expose it in the chart because it was not usable before. But that comment is good news to me 😄 . So thanks for letting me know, i will definitely try that option!

The remaining issue would still be the passenger nginx.conf template not being up to date at some point.
However since discourse does not even use passenger officially anymore i doubt it would change very often, if at all.

[deprecastor]

Yeah would like to see a patch in the code to have not work around in the first place. But I am not familiar with this to do this. Sadly no time atm. Maybe you can create a PR to solve this. Would be great.

[karpfediem]

Managed to use this, thanks again. I'm not deep enough into bitnami charts + containers to do this, or have the time either. But as mentioned a PR is already being worked on that would replace Passenger with Unicorn (which is now Discourse's default choice), making this workaround obsolete.
Here are the relevant adjustments for the "permanent" fix for reference:

values.yaml

extraVolumes:
  - name: passenger-nginx-volume
    configMap:
      name: discourse-passenger-nginx
discourse:
  extraEnvVars:
    - name: DISCOURSE_PASSENGER_EXTRA_FLAGS
      value: "--nginx-config-template /tmp/nginx.conf.erb"
  extraVolumeMounts:
    - mountPath: /tmp/nginx.conf.erb
      name: passenger-nginx-volume
      subPath: nginx.conf.erb

templates/configmap.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: discourse-passenger-nginx
  namespace: {{ .Values.namespace }}
data:
  nginx.conf.erb: |-
{{ .Files.Get "nginx.conf.erb" | indent 4 }}

nginx.conf.erb

##########################################################################
#  Passenger Standalone is built on the same technology that powers
#  Passenger for Nginx, so any configuration option supported by Passenger
#  for Nginx can be applied to Passenger Standalone as well. You can do
#  this by direct editing the Nginx configuration template that is used by
#  Passenger Standalone.
#
#  This file is the original template. DO NOT EDIT THIS FILE DIRECTLY.
#  Instead, make a copy of this file and pass the `--nginx-config-template`
#  parameter to Passenger Standalone.
#
#  Learn more about using the Nginx configuration template at:
#  https://www.phusionpassenger.com/library/config/standalone/intro.html#nginx-configuration-template
#
#  *** NOTE ***
#  If you customize the template file, make sure you keep an eye on the
#  original template file and merge any changes. New Phusion Passenger
#  features may require changes to the template file.
##############################################################

<%= include_passenger_internal_template('global.erb') %>

worker_processes 1;
events {
    worker_connections 4096;
}

http {
    <%= include_passenger_internal_template('http.erb', 4) %>

    ### BEGIN your own configuration options ###
    # This is a good place to put your own config
    # options. Note that your options must not
    # conflict with the ones Passenger already sets.
    # Learn more at:
    # https://www.phusionpassenger.com/library/config/standalone/intro.html#nginx-configuration-template

    ### END your own configuration options ###

    default_type application/octet-stream;
    types_hash_max_size 2048;
    server_names_hash_bucket_size 64;
    client_max_body_size 1024m;
    access_log off;
    keepalive_timeout 60;
    underscores_in_headers on;
    gzip on;
    gzip_comp_level 3;
    gzip_min_length 150;
    gzip_proxied any;
    gzip_types text/plain text/css text/json text/javascript
        application/javascript application/x-javascript application/json
        application/rss+xml application/vnd.ms-fontobject application/x-font-ttf
        application/xml font/opentype image/svg+xml text/xml;

    <% if @app_finder.multi_mode? %>
        # Default server entry for mass deployment mode.
        server {
            <%= include_passenger_internal_template('mass_deployment_default_server.erb', 12) %>
        }
    <% end %>

    <% for app in @apps %>
    server {
        <%= include_passenger_internal_template('server.erb', 8, true, binding) %>
        <%= include_passenger_internal_template('rails_asset_pipeline.erb', 8, false) %>

        ### BEGIN your own configuration options ###
        # This is a good place to put your own config
        # options. Note that your options must not
        # conflict with the ones Passenger already sets.
        # Learn more at:
        # https://www.phusionpassenger.com/library/config/standalone/intro.html#nginx-configuration-template

        location /message-bus {
          passenger_app_group_name foo_websocket;
          passenger_force_max_concurrent_requests_per_process 0;
        }
        ### END your own configuration options ###
    }
    passenger_pre_start <%= listen_url(app) %>;
    <% end %>

    <%= include_passenger_internal_template('footer.erb', 4) %>
}

Note the nginx.conf.erb template might change in the future, i found it in /opt/bitnami/discourse/vendor/bundle/ruby/2.7.0/gems/passenger-6.0.14/resources/templates/standalone/config.erb

[ph0]

I get this issue too. When I go from main page to users or back and forth from there it is either instant or it takes many seconds and this is a new install with 2 inactive users.

[thanhdxuan]

I get this issue too. When I go from main page to users or back and forth from there it is either instant or it takes many seconds and this is a new install with 2 inactive users.

Are u found out the solution for this issue? I get this issue too. I ran Docker container in Ubuntu 22.08 with Passenger Standalone.