BigW Consortium Gitlab

  1. 01 Mar, 2018 1 commit
  2. 28 Feb, 2018 1 commit
  3. 23 Feb, 2018 1 commit
  4. 22 Feb, 2018 1 commit
    • Optimise searching for users using short queries · 41bfe82b
      Yorick Peterse authored
      This optimises searching for users when using queries consisting out of
      one or two characters such as "ab". We optimise such cases by searching
      for `LOWER(name)` and `LOWER(username)` instead of using `ILIKE`. Using
      `LOWER` produces a _much_ better performing query.
      
      For example, when searching for all users matching the term "a" we'd
      produce the following plan:
      
           Limit  (cost=637.69..637.74 rows=20 width=805) (actual time=41.983..41.995 rows=20 loops=1)
             Buffers: shared hit=8330
             ->  Sort  (cost=637.69..638.61 rows=368 width=805) (actual time=41.982..41.990 rows=20 loops=1)
                   Sort Key: (CASE WHEN ((name)::text = 'a'::text) THEN 0 WHEN ((username)::text = 'a'::text) THEN 1 WHEN ((email)::text = 'a'::text) THEN 2 ELSE 3 END), name
                   Sort Method: top-N heapsort  Memory: 35kB
                   Buffers: shared hit=8330
                   ->  Bitmap Heap Scan on users  (cost=75.47..627.89 rows=368 width=805) (actual time=9.452..41.305 rows=277 loops=1)
                         Recheck Cond: (((name)::text ~~* 'a'::text) OR ((username)::text ~~* 'a'::text) OR ((email)::text = 'a'::text))
                         Rows Removed by Index Recheck: 7601
                         Heap Blocks: exact=7636
                         Buffers: shared hit=8327
                         ->  BitmapOr  (cost=75.47..75.47 rows=368 width=0) (actual time=8.290..8.290 rows=0 loops=1)
                               Buffers: shared hit=691
                               ->  Bitmap Index Scan on index_users_on_name_trigram  (cost=0.00..38.85 rows=180 width=0) (actual time=4.369..4.369 rows=4071 loops=1)
                                     Index Cond: ((name)::text ~~* 'a'::text)
                                     Buffers: shared hit=360
                               ->  Bitmap Index Scan on index_users_on_username_trigram  (cost=0.00..34.41 rows=188 width=0) (actual time=3.896..3.896 rows=4140 loops=1)
                                     Index Cond: ((username)::text ~~* 'a'::text)
                                     Buffers: shared hit=328
                               ->  Bitmap Index Scan on users_email_key  (cost=0.00..1.94 rows=1 width=0) (actual time=0.022..0.022 rows=0 loops=1)
                                     Index Cond: ((email)::text = 'a'::text)
                                     Buffers: shared hit=3
           Planning time: 3.912 ms
           Execution time: 42.171 ms
      
      With the changes in this commit we now produce the following plan
      instead:
      
           Limit  (cost=13257.48..13257.53 rows=20 width=805) (actual time=1.567..1.579 rows=20 loops=1)
             Buffers: shared hit=287
             ->  Sort  (cost=13257.48..13280.93 rows=9379 width=805) (actual time=1.567..1.572 rows=20 loops=1)
                   Sort Key: (CASE WHEN ((name)::text = 'a'::text) THEN 0 WHEN ((username)::text = 'a'::text) THEN 1 WHEN ((email)::text = 'a'::text) THEN 2 ELSE 3 END), name
                   Sort Method: top-N heapsort  Memory: 35kB
                   Buffers: shared hit=287
                   ->  Bitmap Heap Scan on users  (cost=135.66..13007.91 rows=9379 width=805) (actual time=0.194..1.107 rows=277 loops=1)
                         Recheck Cond: ((lower((name)::text) = 'a'::text) OR (lower((username)::text) = 'a'::text) OR ((email)::text = 'a'::text))
                         Heap Blocks: exact=277
                         Buffers: shared hit=287
                         ->  BitmapOr  (cost=135.66..135.66 rows=9379 width=0) (actual time=0.152..0.152 rows=0 loops=1)
                               Buffers: shared hit=10
                               ->  Bitmap Index Scan on yorick_test_users  (cost=0.00..124.75 rows=9377 width=0) (actual time=0.101..0.101 rows=277 loops=1)
                                     Index Cond: (lower((name)::text) = 'a'::text)
                                     Buffers: shared hit=4
                               ->  Bitmap Index Scan on index_on_users_lower_username  (cost=0.00..1.94 rows=1 width=0) (actual time=0.035..0.035 rows=1 loops=1)
                                     Index Cond: (lower((username)::text) = 'a'::text)
                                     Buffers: shared hit=3
                               ->  Bitmap Index Scan on users_email_key  (cost=0.00..1.94 rows=1 width=0) (actual time=0.014..0.014 rows=0 loops=1)
                                     Index Cond: ((email)::text = 'a'::text)
                                     Buffers: shared hit=3
           Planning time: 0.303 ms
           Execution time: 1.687 ms
      
      Here we can see the new query is 25 times faster compared to the old
      query.
  5. 20 Feb, 2018 3 commits
  6. 16 Feb, 2018 2 commits
  7. 15 Feb, 2018 1 commit
  8. 12 Feb, 2018 1 commit
  9. 08 Feb, 2018 2 commits
    • Add indexes and change SQL for expired artifacts to deal with artifacts migration efficiently · 271e7a32
      Greg Stark authored
      Artifacts are in the middle of being migrated from ci_builds to
      ci_job_artifacts. The expiration date is currently visible in both of
      these tables and the test for whether an expired artifact is present
      for a job is complex as it requires checking both the of the tables.
      
      Add two new indexes, one on ci_builds.artifacts_expire_at and one on
      ci_job_artifacts.expire_at to enable finding expired artifacts
      efficiently.
      
      And until the migration is finished, replace the SQL for finding
      expired and non-expired artifacts with a hand-crafted UNION ALL based
      query instead of using OR. This overcomes a database optimizer
      limitation that prevents it from using these indexes.
      
      When the migration is finished the next version should remove this
      query and replace it with a much simpler query on just
      ci_job_artifacts. See
      https://gitlab.com/gitlab-org/gitlab-ce/issues/42561 for followup.
    • Early migrations populating fork-networks: no-op · 7e00adcc
      Bob Van Landuyt authored
      Since populating the fork networks was scheduled multiple times
      because of bugs that needed to be fixed:
      
      - https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15595/
        Creating fork networks for projects that were missing the root of
        the fork network.
      
      - https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15709
        The API allowed creating forked_project_links without
        fork_network_members.
      
      Scheduling this migration multiple times would case it to run
      concurrently. Which in turn would try to insert the same data into
      `fork_network_members` causing duplicate key errors.
      
      This avoids running the migration multiple times.
  10. 07 Feb, 2018 1 commit
  11. 06 Feb, 2018 1 commit
  12. 05 Feb, 2018 2 commits
  13. 02 Feb, 2018 6 commits
  14. 01 Feb, 2018 4 commits
  15. 26 Jan, 2018 1 commit
  16. 23 Jan, 2018 1 commit
    • Use limit for search count queries · 090ca9c3
      Jan Provaznik authored
      Search query is especially slow if a user searches a generic string
      which matches many records, in such case search can take tens of
      seconds or time out. To speed up the search query, we search only for
      first 1000 records, if there is >1000 matching records we just display
      "1000+" instead of precise total count supposing that with such amount
      the exact count is not so important for the user.
      
      Because for issues even limited search was not fast enough, 2-phase
      approach is used for issues: first we use simpler/faster query to get
      all public issues, if this exceeds the limit, we just return the limit.
      If the amount of matching results is lower than limit, we re-run more
      complex search query (which includes also confidential issues).
      Re-running the complex query should be fast enough in such case because the
      amount of matching issues is lower than limit.
      
      Because exact total_count is now limited, this patch also switches to
      to "prev/next" pagination.
      
      Related #40540
  17. 22 Jan, 2018 1 commit
  18. 18 Jan, 2018 2 commits
  19. 17 Jan, 2018 2 commits
  20. 15 Jan, 2018 1 commit
  21. 12 Jan, 2018 1 commit
  22. 11 Jan, 2018 1 commit
  23. 10 Jan, 2018 1 commit
    • Denormalize commits count for merge request diffs · e6a1db6d
      Jan Provaznik authored
      For each MR diff an extra 'SELECT COUNT()' is executed
      to get number of commits for the diff. Overall time to get counts for
      all MR diffs may be quite expensive. To speed up loading of MR info,
      information about number of commits is stored in a MR diff's extra column.
      
      Closes #38068
  24. 08 Jan, 2018 2 commits
    • Avoid adding index if already exists · d2f4e8f9
      Paco Guzman authored
    • Backport authorized_keys_enabled defaults to true' · 797fe0a6
      Michael Kozono authored
      Originally from branch 'fix-authorized-keys-enabled-default-2738' via merge request https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2240
      
      Removed background migrations which were intended to fix state after using Gitlab
      without a default having been set
      
      Squashed commits:
      Locally, if Spring was not restarted, `current_application_settings` was still cached, which prevented the migration from editing the file. This will also ensure that any app server somehow hitting old cache data will properly default this setting regardless.
      Retroactively fix migration
        This allows us to identify customers who ran the broken migration. Their `authorized_keys_enabled` column does not have a default at this point.
        We will fix the column after we fix the `authorized_keys` file.
      Fix authorized_keys file if needed
      Add default to authorized_keys_enabled setting
        Reminder: The original migration was fixed retroactively a few commits ago, so people who did not ever run GitLab 9.3.0 already have a column that defaults to true and disallows nulls. I have tested on PostgreSQL and MySQL that it is safe to run this migration regardless.
        Affected customers who did run 9.3.0 are the ones who need this migration to fix the authorized_keys_enabled column.
        The reason for the retroactive fix plus this migration is that it allows us to run a migration in between to fix the authorized_keys file only for those who ran 9.3.0.
      Tweaks to address feedback
      Extract work into background migration
      Move batch-add-logic to background migration
        Do the work synchronously to avoid multiple workers attempting to add batches of keys at the same time.
        Also, make the delete portion wait until after adding is done.
      Do read and delete work in background migration
      Fix Rubocop offenses
      Add changelog entry
      Inform the user of actions taken or not taken
      Prevent unnecessary `select`s and `remove_key`s
      Add logs for action taken
      Fix optimization
      Reuse `Gitlab::ShellAdapter`
      Guarantee the earliest key
      Fix migration spec for MySQL