BigW Consortium Gitlab
Instead of applying WHERE on a UNION, apply the WHERE on each of the seperate SELECT statements, and do UNION on that. Local tests with about 2_000_000 projects: - 1_500_000 private projects - 40_000 internal projects - 400_000 public projects For the API endpoint `/api/v4/projects?visibility=private` the slowest query was: ```sql SELECT "projects".* FROM "projects" WHERE ... ``` The original query took 1073.8ms. The query refactored to UNION of SELECT/WHERE took 2.3ms. The original query was: ```sql SELECT "projects".* FROM "projects" WHERE "projects"."pending_delete" = $1 AND (projects.id IN (SELECT "projects"."id" FROM "projects" INNER JOIN "project_authorizations" ON "projects"."id" = "project_authorizations"."project_id" WHERE "projects"."pending_delete" = 'f' AND "project_authorizations"."user_id" = 23 UNION SELECT "projects"."id" FROM "projects" WHERE "projects"."visibility_level" IN (20, 10))) AND "projects"."visibility_level" = $2 AND "projects"."archived" = $3 ORDER BY "projects"."created_at" DESC LIMIT 20 OFFSET 0 [["pending_delete", "f"], ["visibility_level", 0], ["archived", "f"]] ``` The refactored query: ```sql SELECT "projects".* FROM "projects" WHERE "projects"."pending_delete" = $1 AND (projects.id IN (SELECT "projects"."id" FROM "projects" INNER JOIN "project_authorizations" ON "projects"."id" = "project_authorizations"."project_id" WHERE "projects"."pending_delete" = 'f' AND "project_authorizations"."user_id" = 23 AND "projects"."visibility_level" = 0 AND "projects"."archived" = 'f' UNION SELECT "projects"."id" FROM "projects" WHERE "projects"."visibility_level" IN (20, 10) AND "projects"."visibility_level" = 0 AND "projects"."archived" = 'f')) ORDER BY "projects"."created_at" DESC LIMIT 20 OFFSET 0 [["pending_delete", "f"]] ```
Name |
Last commit
|
Last update |
---|---|---|
.github | Loading commit data... | |
.gitlab | Loading commit data... | |
app | Loading commit data... | |
bin | Loading commit data... | |
builds | Loading commit data... | |
changelogs | Loading commit data... | |
config | Loading commit data... | |
db | Loading commit data... | |
doc | Loading commit data... | |
docker | Loading commit data... | |
features | Loading commit data... | |
fixtures/emojis | Loading commit data... | |
generator_templates | Loading commit data... | |
lib | Loading commit data... | |
locale | Loading commit data... | |
log | Loading commit data... | |
public | Loading commit data... | |
qa | Loading commit data... | |
rubocop | Loading commit data... | |
scripts | Loading commit data... | |
shared | Loading commit data... | |
spec | Loading commit data... | |
tmp | Loading commit data... | |
vendor | Loading commit data... | |
.babelrc | Loading commit data... | |
.codeclimate.yml | Loading commit data... | |
.csscomb.json | Loading commit data... | |
.eslintignore | Loading commit data... | |
.eslintrc | Loading commit data... | |
.flayignore | Loading commit data... | |
.foreman | Loading commit data... | |
.gitignore | Loading commit data... | |
.gitlab-ci.yml | Loading commit data... | |
.haml-lint.yml | Loading commit data... | |
.mailmap | Loading commit data... | |
.pkgr.yml | Loading commit data... | |
.rspec | Loading commit data... | |
.rubocop.yml | Loading commit data... | |
.rubocop_todo.yml | Loading commit data... | |
.ruby-version | Loading commit data... | |
.scss-lint.yml | Loading commit data... | |
CHANGELOG.md | Loading commit data... | |
CONTRIBUTING.md | Loading commit data... | |
GITALY_SERVER_VERSION | Loading commit data... | |
GITLAB_PAGES_VERSION | Loading commit data... | |
GITLAB_SHELL_VERSION | Loading commit data... | |
GITLAB_WORKHORSE_VERSION | Loading commit data... | |
Gemfile | Loading commit data... | |
Gemfile.lock | Loading commit data... | |
LICENSE | Loading commit data... | |
MAINTENANCE.md | Loading commit data... | |
PROCESS.md | Loading commit data... | |
Procfile | Loading commit data... | |
README.md | Loading commit data... | |
Rakefile | Loading commit data... | |
VERSION | Loading commit data... | |
config.ru | Loading commit data... | |
doc_styleguide.md | Loading commit data... | |
docker-compose.yml | Loading commit data... | |
package.json | Loading commit data... | |
yarn.lock | Loading commit data... |