-
Notifications
You must be signed in to change notification settings - Fork 74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cinstance N+1 issues clean-up #3889
base: master
Are you sure you want to change the base?
Changes from all commits
c16f40d
8ed62c9
f788d8d
634e6ec
081a8c4
37a6b21
4e95548
ef6760b
aaea00d
82e5492
01d602c
53d0222
62b07d9
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -22,22 +22,24 @@ def index | |
def find | ||
buyer_account = find_buyer_account | ||
authorize! :read, buyer_account | ||
respond_with(buyer_account) | ||
respond_with(to_present(buyer_account)) | ||
end | ||
|
||
# Account Read | ||
# GET /admin/api/accounts/{id}.xml | ||
def show | ||
authorize! :read, buyer_account | ||
|
||
respond_with(buyer_account) | ||
respond_with(to_present(buyer_account)) | ||
end | ||
|
||
# Account Update | ||
# PUT /admin/api/accounts/{id}.xml | ||
def update | ||
authorize! :update, buyer_account | ||
|
||
to_present(buyer_account) | ||
|
||
buyer_account.vat_rate = params[:vat_rate].to_f if params[:vat_rate] | ||
buyer_account.settings.attributes = billing_params | ||
buyer_account.update_with_flattened_attributes(flat_params) | ||
|
@@ -71,6 +73,7 @@ def change_plan | |
def approve | ||
authorize! :approve, buyer_account | ||
|
||
to_present buyer_account | ||
buyer_account.approve | ||
|
||
respond_with(buyer_account) | ||
|
@@ -81,6 +84,7 @@ def approve | |
def reject | ||
authorize! :reject, buyer_account | ||
|
||
to_present buyer_account | ||
buyer_account.reject | ||
|
||
respond_with(buyer_account) | ||
|
@@ -91,6 +95,7 @@ def reject | |
def make_pending | ||
authorize! :update, buyer_account | ||
|
||
to_present buyer_account | ||
buyer_account.make_pending | ||
|
||
respond_with(buyer_account) | ||
|
@@ -110,6 +115,12 @@ def buyer_account | |
@buyer_account ||= buyer_accounts.find(params[:id]) | ||
end | ||
|
||
def to_present(accounts) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe the method should be called There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this has changed in the merged PR. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Which PR? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. same one as in the other comments #3845 |
||
# ActiveRecord::Associations::Preloader.new(records: Array(accounts), associations: [:annotations, {bought_plans: %i[original]}]).call # Rails 7.x | ||
ActiveRecord::Associations::Preloader.new.preload(Array(accounts), [:annotations, {bought_plans: %i[original]}]) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not sure how this There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. the associated objects are preloaded into the array of objects. Similar to how |
||
accounts | ||
end | ||
|
||
def buyer_users | ||
@buyer_users ||= current_account.buyer_users | ||
end | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -7,7 +7,7 @@ class Admin::Api::ApplicationsController < Admin::Api::BaseController | |
# GET /admin/api/applications.xml | ||
def index | ||
apps = applications.scope_search(search) | ||
.serialization_preloading.paginate(:page => current_page, :per_page => per_page) | ||
.serialization_preloading(request.format).paginate(:page => current_page, :per_page => per_page) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What's this for? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. different preloading rules for different requested format (xml vs json). So a parameter had to be added. But this is already upstream, need to rebase. |
||
respond_with(apps) | ||
end | ||
|
||
|
@@ -24,25 +24,19 @@ def application_filter_params | |
end | ||
|
||
def applications | ||
@applications ||= begin | ||
cinstances = current_account.provided_cinstances.where(service: accessible_services) | ||
if (service_id = params[:service_id]) | ||
cinstances = cinstances.where(service_id: service_id) | ||
end | ||
cinstances | ||
end | ||
@applications ||= current_account.provided_cinstances.merge(accessible_services) | ||
end | ||
|
||
def application | ||
@application ||= case | ||
|
||
when user_key = params[:user_key] | ||
when param_key = params[:user_key] | ||
# TODO: these scopes should be in model layer | ||
# but there is scope named by_user_key already | ||
applications.joins(:service).where("(services.backend_version = '1' AND cinstances.user_key = ?)", user_key).first! | ||
applications.where.has { (service.backend_version == '1') & (user_key == param_key) }.first! | ||
|
||
when app_id = params[:app_id] | ||
applications.joins(:service).where("(services.backend_version <> '1' AND cinstances.application_id = ?)", app_id).first! | ||
applications.where.has { (service.backend_version != '1') & (application_id == app_id) }.first! | ||
|
||
else | ||
applications.find(params[:application_id] || params[:id]) | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -18,7 +18,7 @@ def policy_chain | |
end | ||
|
||
def with_subpaths? | ||
backend_api_configs.with_subpath.any? | ||
backend_api_configs.any?(&:with_subpath?) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is not the original better? It scopes the results. Your modified version always returns all configs and then calls There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This was already merged with #3845 If you look from the perspective of taking a service and calling this method on it, what you say makes sense. I fixed a number of N+1 though with the aforementioned PR and I don't remember which test it was related to. Also I looked at a couple of requests and optimized them to a maximum least number of queries. I think in this case, this change is based on the premises that we have the backend_api_configs already preloaded for the service(s) we deal with. So calling In the other PR I have added I think even if backend_api_configs is not preloaded, it wouldn't be a huge deal because I don't expect too many backends in services. And it will still be one query, although with more data returned than the original code. It will also load the backend_api_configs into the respective service in case they are further needed. If you have spotted a particular call that is less efficient this way, we may think about it. But I think it might likely be an edge case when a single service is involved. But with original code I don't see how we can avoid N+1 when many services are loaded at once and we want to preload everything needed for presentation. As the original code will still try to perform a new query for each service. Hope explanation makes sense. But better comment further on the original (merged) PR because I will be rebasing this one to avoid all the extra commits already in master. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nice, so data is preloaded and you save a query. Fine. |
||
end | ||
|
||
class Builder | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -60,7 +60,11 @@ class Service < ApplicationRecord # rubocop:disable Metrics/ClassLength | |
service.has_many :api_docs_services, class_name: 'ApiDocs::Service' | ||
end | ||
|
||
scope :of_account, ->(account) { where.has { account_id == account.id } } | ||
sifter :of_account do |account| | ||
account_id == account.id | ||
end | ||
|
||
scope :of_account, ->(account) { where.has { sift(:of_account, account) } } | ||
Comment on lines
+63
to
+67
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why is There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. sifters can be applied to other models, scopes only to associations of same model |
||
|
||
has_one :proxy, dependent: :destroy, inverse_of: :service, autosave: true | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -120,8 +120,6 @@ | |
Bullet.add_safelist class_name: "Alert", type: :n_plus_one_query, association: :cinstance | ||
Bullet.add_safelist class_name: "ApiDocs::Service", type: :unused_eager_loading, association: :service | ||
Bullet.add_safelist class_name: "ApplicationPlan", type: :n_plus_one_query, association: :customizations | ||
Bullet.add_safelist class_name: "ApplicationPlan", type: :n_plus_one_query, association: :issuer | ||
Bullet.add_safelist class_name: "ApplicationPlan", type: :n_plus_one_query, association: :original | ||
Bullet.add_safelist class_name: "ApplicationPlan", type: :n_plus_one_query, association: :pricing_rules | ||
Bullet.add_safelist class_name: "ApplicationPlan", type: :n_plus_one_query, association: :usage_limits | ||
Bullet.add_safelist class_name: "ApplicationPlan", type: :unused_eager_loading, association: :issuer | ||
|
@@ -133,17 +131,6 @@ | |
Bullet.add_safelist class_name: "CMS::File", type: :n_plus_one_query, association: :provider | ||
Bullet.add_safelist class_name: "CMS::Page", type: :n_plus_one_query, association: :provider | ||
Bullet.add_safelist class_name: "CMS::Page", type: :n_plus_one_query, association: :section | ||
Bullet.add_safelist class_name: "Cinstance", type: :n_plus_one_query, association: :plan | ||
Bullet.add_safelist class_name: "Cinstance", type: :n_plus_one_query, association: :service | ||
Bullet.add_safelist class_name: "Cinstance", type: :n_plus_one_query, association: :user_account | ||
Bullet.add_safelist class_name: "Cinstance", type: :unused_eager_loading, association: :plan | ||
Bullet.add_safelist class_name: "Cinstance", type: :unused_eager_loading, association: :service | ||
Bullet.add_safelist class_name: "Cinstance", type: :unused_eager_loading, association: :service | ||
Bullet.add_safelist class_name: "Cinstance", type: :unused_eager_loading, association: :service | ||
Bullet.add_safelist class_name: "Cinstance", type: :unused_eager_loading, association: :plan | ||
Bullet.add_safelist class_name: "Cinstance", type: :unused_eager_loading, association: :user_account | ||
Bullet.add_safelist class_name: "Cinstance", type: :unused_eager_loading, association: :user_account | ||
Bullet.add_safelist class_name: "Cinstance", type: :unused_eager_loading, association: :service | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I didn't know about this list. Nice to remove items from it There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
Bullet.add_safelist class_name: "Invoice", type: :counter_cache, association: :payment_transactions | ||
Bullet.add_safelist class_name: "Invoice", type: :n_plus_one_query, association: :buyer_account | ||
Bullet.add_safelist class_name: "Invoice", type: :n_plus_one_query, association: :provider_account | ||
|
@@ -170,10 +157,8 @@ | |
Bullet.add_safelist class_name: "Service", type: :counter_cache, association: :backend_api_configs | ||
Bullet.add_safelist class_name: "Service", type: :counter_cache, association: :cinstances | ||
Bullet.add_safelist class_name: "Service", type: :n_plus_one_query, association: :account | ||
Bullet.add_safelist class_name: "Service", type: :n_plus_one_query, association: :default_application_plan | ||
Bullet.add_safelist class_name: "Service", type: :n_plus_one_query, association: :default_service_plan | ||
Bullet.add_safelist class_name: "Service", type: :n_plus_one_query, association: :metrics | ||
Bullet.add_safelist class_name: "Service", type: :n_plus_one_query, association: :proxy | ||
Bullet.add_safelist class_name: "Service", type: :unused_eager_loading, association: :application_plans | ||
Bullet.add_safelist class_name: "ServiceContract", type: :n_plus_one_query, association: :plan | ||
Bullet.add_safelist class_name: "ServiceContract", type: :n_plus_one_query, association: :user_account | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
if (Rails.env.development? || Rails.env.test?) && ENV["TRACE_SQL"] == "1" | ||
ActiveRecordQueryTrace.enabled = true | ||
ActiveRecordQueryTrace.level = :app | ||
Rails.application.config.log_level = :debug | ||
end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How do you identify the N+1 scenarios? From Bullet logs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bullet errors are enabled in testing but many are whitelisted to avoid breaking test suite. See the environment file.