Cinstance N+1 issues clean-up #3889

akostadinov · 2024-09-13T22:08:02Z

No description provided.

``` Bullet::Notification::UnoptimizedQueryError: user: localuser GET http://admin.foo.3scale.localhost:39073/p/admin/applications/16 AVOID eager loading detected ApplicationPlan => [:plan_metrics] Remove from your query: .includes([:plan_metrics]) ```

Fixes issue with keys.feature:Regenerate provider key Bullet::Notification::UnoptimizedQueryError: user: avalon PUT http://master-account.3scale.localhost:42285/p/admin/applications/4/change_user_key AVOID eager loading detected ApplicationPlan => [:original] Remove from your query: .includes([:original])

``` Bullet::Notification::UnoptimizedQueryError: user: default GET /admin/api/applications.json?provider_key=***&inactive_since=2014-05-05 AVOID eager loading detected Service => [:default_application_plan] Remove from your query: .includes([:default_application_plan]) ```

The `#where_values_hash` method does not support joins and sub-queries. Originally the `account.provided_cinstances` part was ignored because it was JOINs. With the update to sub-queries, it turned into `plan_id: nil` which is incorrect and broke the logic. So now we keep logic as previously by resorting only to the `Cinstance.permitted_for` part of the query. This relies on the fact that `Cinstance.plan.issuer` is set as `Cinstance.service` when that issuer is a service. Also relies on the fact that `User.member_permission_service_ids` will not set to ids of services that don't belong to the account. Which may not be ideal but allows for permission checking without extra database queries.

Historically it was a conscious decision to optimize access to cintance -> service by denormalizing the database. So we can access service directly and not through the plan. We also keep these two in sync with model callbacks. So this commit further simplifies the queries to fully rely on this fact.

jlledom · 2024-10-29T13:58:35Z

app/concerns/new_application_form.rb

@@ -27,7 +27,7 @@ def buyers
  end

  def products
-    paginated_products.map { |p| ServicePresenter.new(p).new_application_data.as_json }
+    paginated_products.includes(:default_application_plan).map { |p| ServicePresenter.new(p).new_application_data.as_json }


How do you identify the N+1 scenarios? From Bullet logs?

Bullet errors are enabled in testing but many are whitelisted to avoid breaking test suite. See the environment file.

jlledom · 2024-10-29T14:04:28Z

app/controllers/admin/api/accounts_controller.rb

@@ -110,6 +115,12 @@ def buyer_account
    @buyer_account ||= buyer_accounts.find(params[:id])
  end

+  def to_present(accounts)
+    # ActiveRecord::Associations::Preloader.new(records: Array(accounts), associations: [:annotations, {bought_plans: %i[original]}]).call # Rails 7.x
+    ActiveRecord::Associations::Preloader.new.preload(Array(accounts), [:annotations, {bought_plans: %i[original]}])


Not sure how this Preloader class works. Is the data kept in the memory forever? or only for this controller instance life?

the associated objects are preloaded into the array of objects. Similar to how #includes on associations works. So you can say it is in memory until normal garbage collection takes place.

jlledom · 2024-10-29T14:12:30Z

app/controllers/admin/api/accounts_controller.rb

@@ -110,6 +115,12 @@ def buyer_account
    @buyer_account ||= buyer_accounts.find(params[:id])
  end

+  def to_present(accounts)


Maybe the method should be called preload!? Calling it to_present makes me think of a conversion that doesn't modify the original given object.

this has changed in the merged PR.

same one as in the other comments #3845

jlledom · 2024-10-29T14:12:55Z

app/controllers/admin/api/applications_controller.rb

@@ -7,7 +7,7 @@ class Admin::Api::ApplicationsController < Admin::Api::BaseController
  # GET /admin/api/applications.xml
  def index
    apps = applications.scope_search(search)
-           .serialization_preloading.paginate(:page => current_page, :per_page => per_page)
+           .serialization_preloading(request.format).paginate(:page => current_page, :per_page => per_page)


What's this for?

different preloading rules for different requested format (xml vs json). So a parameter had to be added. But this is already upstream, need to rebase.

jlledom · 2024-10-29T14:14:46Z

app/lib/backend_api_logic/routing_policy.rb

@@ -18,7 +18,7 @@ def policy_chain
    end

    def with_subpaths?
-      backend_api_configs.with_subpath.any?
+      backend_api_configs.any?(&:with_subpath?)


Is not the original better? It scopes the results. Your modified version always returns all configs and then calls :with_subpath? for each one.

This was already merged with #3845

If you look from the perspective of taking a service and calling this method on it, what you say makes sense. I fixed a number of N+1 though with the aforementioned PR and I don't remember which test it was related to. Also I looked at a couple of requests and optimized them to a maximum least number of queries.

I think in this case, this change is based on the premises that we have the backend_api_configs already preloaded for the service(s) we deal with. So calling with_subpath? on each is more effective than performing a new database query. Applying a scope results in a new query.

In the other PR I have added active_record_query_trace gem and I assume it showed to me that a query came from this line where I didn't expect a query at this point.

I think even if backend_api_configs is not preloaded, it wouldn't be a huge deal because I don't expect too many backends in services. And it will still be one query, although with more data returned than the original code. It will also load the backend_api_configs into the respective service in case they are further needed.

If you have spotted a particular call that is less efficient this way, we may think about it. But I think it might likely be an edge case when a single service is involved. But with original code I don't see how we can avoid N+1 when many services are loaded at once and we want to preload everything needed for presentation. As the original code will still try to perform a new query for each service.

Hope explanation makes sense. But better comment further on the original (merged) PR because I will be rebasing this one to avoid all the extra commits already in master.

Nice, so data is preloaded and you save a query. Fine.

jlledom · 2024-10-29T14:17:38Z

app/models/service.rb

+  sifter :of_account do |account|
+    account_id == account.id
+  end
+
+  scope :of_account, ->(account) { where.has { sift(:of_account, account) } }


Why is sifter better than a regular scope?

sifters can be applied to other models, scopes only to associations of same model

jlledom · 2024-10-29T15:34:26Z

config/environments/test.rb

-      Bullet.add_safelist class_name: "Cinstance", type: :unused_eager_loading, association: :plan
-      Bullet.add_safelist class_name: "Cinstance", type: :unused_eager_loading, association: :user_account
-      Bullet.add_safelist class_name: "Cinstance", type: :unused_eager_loading, association: :user_account
-      Bullet.add_safelist class_name: "Cinstance", type: :unused_eager_loading, association: :service


I didn't know about this list. Nice to remove items from it

https://issues.redhat.com/browse/THREESCALE-9973

akostadinov added 13 commits September 11, 2024 23:25

THREESCALE-11139 simplify applications query

c16f40d

optimize application queries further

8ed62c9

debugging info on failure

081a8c4

remove some n+1 issues

aaea00d

allow tracing SQL queries

82e5492

fix N+1s for NginxesController

01d602c

N+1 in signup controller

53d0222

fix cinstances N+1 bullets

62b07d9

github-actions bot added the Stale label Oct 14, 2024

akostadinov removed the Stale label Oct 14, 2024

3scale deleted a comment from github-actions bot Oct 14, 2024

jlledom reviewed Oct 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cinstance N+1 issues clean-up #3889

Cinstance N+1 issues clean-up #3889

akostadinov commented Sep 13, 2024

jlledom Oct 29, 2024

akostadinov Oct 29, 2024

jlledom Oct 29, 2024

akostadinov Oct 29, 2024

jlledom Oct 29, 2024

akostadinov Oct 29, 2024

jlledom Oct 30, 2024

akostadinov Oct 30, 2024

jlledom Oct 29, 2024

akostadinov Oct 29, 2024

jlledom Oct 29, 2024

akostadinov Oct 29, 2024

jlledom Oct 30, 2024

jlledom Oct 29, 2024

akostadinov Oct 29, 2024

jlledom Oct 29, 2024

akostadinov Oct 29, 2024

Cinstance N+1 issues clean-up #3889

Are you sure you want to change the base?

Cinstance N+1 issues clean-up #3889

Conversation

akostadinov commented Sep 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment