BigW Consortium Gitlab

  1. 15 Jun, 2016 40 commits
    • Merge branch 'fair-usage-of-shared-runners' into 'master' · 919e3ed2
      Stan Hu authored
      Fair usage of Shared Runners
      
      ## What does this MR do?
      
      Introduces a fair usage scheduler for shared runners.
      
      It tries to assign builds to shared runner from projects that have the lowest number of builds currently running on shared runners.
      
      **Example 1**:
      ```
      We have following builds in queue:
      build 1 for project 1
      build 2 for project 1
      build 3 for project 1
      build 4 for project 2
      build 5 for project 2
      build 6 for project 3
      
      With the new algorithm we will assign builds in following order:
      - We choose build 1, because project 1 doesn't run currently any builds and has the lowest build number from projects that doesn't run builds,
      - We choose build 4, because project 2 doesn't run currently any builds and has the lowest build number from projects that doesn't run builds,
      - We choose build 6, because project 3 doesn't run currently any builds and has the lowest build number from projects that doesn't run builds,
      - We choose build 2, because project 1 as other it runs 1 build,
      - We choose build 5, because project 2 runs 1 build, where project 1 runs 2 builds now,
      - We choose build 3, because project 1 and runs 2 builds.
      ```
      
       
      **Example 2**:
      ```
      We have following builds in queue:
      build 1 for project 1
      build 2 for project 1
      build 3 for project 1
      build 4 for project 2
      build 5 for project 2
      build 6 for project 3
      
      With the new algorithm we will assign builds in following order:
      - We choose build 1, because project 1 doesn't run currently any builds and has the lowest build number from projects that doesn't run builds,
      - We finish build 1,
      - We choose build 2, because project 1 doesn't run currently any builds and has the lowest build number from projects that doesn't run builds,
      - We choose build 4, because project 2 doesn't run currently any builds and has the lowest build number from projects that doesn't run builds,
      - We finish build 4,
      - We choose build 5, because project 2 doesn't run currently any builds and has the lowest build number from projects that doesn't run builds,
      - We choose build 6, because project 3 doesn't run currently any builds,
      - We choose build 3, because project 1, 2 and 3 runs exactly one build now,
      ```
      
      ## Why was this MR needed?
      
      Currently, we are scheduling builds using FIFO. This is catastrophic if there are projects that create a 100-300 jobs, this basically eats most of available shared runners.
      
      ## Performance
      
      All this logic is implemented with the help of SQL queries, because this is the fastest way to process 1k-2k pending builds in queue.
      It's not the fastest SQL query, because it sorts based on number of running_builds, and this forces to calculate a number of running builds for all dependent projects. However, since we have one/two shared runners that asks every few seconds for builds this should have minimal impact on DB performance.
      
      ```
       explain analyze SELECT "ci_builds".* FROM "ci_builds" JOIN (SELECT "ci_builds"."gl_project_id", count(case when status = 'running' AND runner_id = (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."is_shared" = 't') then 1 end) as running_builds FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."gl_project_id" AND "projects"."pending_delete" = 'f' WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."status" IN ('running', 'pending') AND "projects"."builds_enabled" = 't' AND "projects"."shared_runners_enabled" = 't' GROUP BY "ci_builds"."gl_project_id") AS projects ON ci_builds.gl_project_id=projects.gl_project_id WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."status" = 'pending' AND "ci_builds"."runner_id" IS NULL  ORDER BY projects.running_builds ASC, ci_builds.id ASC;
                                                                                        QUERY PLAN                                                                        
                 
      --------------------------------------------------------------------------------------------------------------------------------------------------------------------
      -----------
       Sort  (cost=64777.28..64777.29 rows=1 width=1010) (actual time=301.794..302.535 rows=1537 loops=1)
         Sort Key: (count(CASE WHEN (((public.ci_builds.status)::text = 'running'::text) AND (public.ci_builds.runner_id = $0)) THEN 1 ELSE NULL::integer END)), public.ci
      _builds.id
         Sort Method: quicksort  Memory: 1423kB
         ->  Nested Loop  (cost=63279.78..64777.27 rows=1 width=1010) (actual time=66.384..298.724 rows=1537 loops=1)
               ->  HashAggregate  (cost=63177.15..63177.30 rows=15 width=15) (actual time=65.641..65.851 rows=187 loops=1)
                     InitPlan 1 (returns $0)
                       ->  Seq Scan on ci_runners  (cost=0.00..26963.66 rows=1 width=4) (actual time=1.145..34.381 rows=1 loops=1)
                             Filter: is_shared
                             Rows Removed by Filter: 6965
                     ->  Nested Loop  (cost=0.00..36186.34 rows=2715 width=15) (actual time=0.065..29.717 rows=1710 loops=1)
                           ->  Index Scan using index_ci_builds_on_status on ci_builds  (cost=0.00..8913.95 rows=3577 width=15) (actual time=0.051..12.012 rows=2583 loops
      =1)
                                 Index Cond: ((status)::text = ANY ('{running,pending}'::text[]))
                                 Filter: ((type)::text = 'Ci::Build'::text)
                                 Rows Removed by Filter: 1219
                           ->  Index Scan using projects_pkey on projects  (cost=0.00..7.61 rows=1 width=4) (actual time=0.003..0.004 rows=1 loops=2583)
                                 Index Cond: (id = public.ci_builds.gl_project_id)
                                 Filter: ((NOT pending_delete) AND builds_enabled AND shared_runners_enabled)
                                 Rows Removed by Filter: 0
               ->  Bitmap Heap Scan on ci_builds  (cost=102.63..106.64 rows=1 width=1002) (actual time=1.216..1.231 rows=8 loops=187)
                     Recheck Cond: ((gl_project_id = public.ci_builds.gl_project_id) AND ((status)::text = 'pending'::text))
                     Filter: ((runner_id IS NULL) AND ((type)::text = 'Ci::Build'::text))
                     ->  BitmapAnd  (cost=102.63..102.63 rows=1 width=0) (actual time=1.201..1.201 rows=0 loops=187)
                           ->  Bitmap Index Scan on index_ci_builds_on_gl_project_id  (cost=0.00..10.52 rows=241 width=0) (actual time=0.406..0.406 rows=1944 loops=187)
                                 Index Cond: (gl_project_id = public.ci_builds.gl_project_id)
                           ->  Bitmap Index Scan on index_ci_builds_on_status  (cost=0.00..91.78 rows=3089 width=0) (actual time=0.652..0.652 rows=3362 loops=187)
                                 Index Cond: ((status)::text = 'pending'::text)
       Total runtime: 303.832 ms
      ```
      
      ## Specific runners
      
      It doesn't affect the specific runners which still serve builds FIFO.
      
      @stanhu @markpundsack @yorickpeterse What do you think?
      
      
      See merge request !4634
    • Merge branch… · 2cfbf3e8
      Robert Speicher authored
      Merge branch '18131-pressing-merge-when-build-succeeds-at-the-same-exact-time-that-a-build-is-failing-will-merge-the-failed-build' into 'master'
      
      Fix race condition on auto merge
      
      Resolves #18131
      
      See merge request !4443
    • Merge branch 'project-services-inverse-of' into 'master' · 32a5ff70
      Rémy Coutable authored
      Set inverse_of for Project/Services relation
      
      ## What does this MR do?
      
      This MR adds the `inverse_of:` option to two associations to reduce the number of queries when running code such as ` project.gitlab_issue_tracker_service.project`.
      
      ## Are there points in the code the reviewer needs to double check?
      
      No.
      
      ## Why was this MR needed?
      
      In !4410 it was revealed code such as the above is used and would run SQL queries when the root object (usually a Project) was already present. By using `inverse_of` Rails can just re-use those Project instances.
      
      ## What are the relevant issue numbers?
      
      None.
      
      ## Does this MR meet the acceptance criteria?
      
      - [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
      - [x] ~~[Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)~~
      - [x] ~~API support added~~
      - [x] Tests
        - [x] ~~Added for this feature/bug~~
        - [x] All builds are passing
      - [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
      - [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
      - [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
      
      See merge request !4674
    • Rename ci_commit to pipeline · 17ad286e
      Z.J. van de Weg authored
    • Fix race condition on auto merge · 3fe4a2f5
      Zeger-Jan van de Weg authored
    • Set inverse_of for Project/Services relation · 8a9164bf
      Yorick Peterse authored
      This ensures that code such as this don't run needless SQL queries:
      
          project.gitlab_issue_tracker_service.project
      
      This also means that if the root `project` eager loads any associations
      the Service object will be able to re-use those.
    • Merge branch 'remove-shell-env' into 'master' · af493ccf
      Douwe Maan authored
      Get rid of Gitlab::ShellEnv
      
      Remove obsolete code we once needed for Grack and satellites.
      
      See merge request !4673
    • Merge branch '18338-change-admin-navigation-to-match-new-ui' into 'master' · e09d573e
      Dmitriy Zaporozhets authored
      Resolve "Change admin navigation to match new UI"
      
      ## What does this MR do?
      Moves admin navigation to layout nav to match Project, Groups, and Profile navigation
      
      ## Are there points in the code the reviewer needs to double check?
      
      ## Why was this MR needed?
      
      ## What are the relevant issue numbers?
      Closes #18338
      
      ## Screenshots (if relevant)
      ![Screen_Shot_2016-06-14_at_10.26.40_AM](/uploads/f0b8c8b259da16d929be2b36e8eeafb8/Screen_Shot_2016-06-14_at_10.26.40_AM.png)
      ![Screen_Shot_2016-06-14_at_10.26.44_AM](/uploads/07d9ece2063dfcfec1f0f2647d8ee782/Screen_Shot_2016-06-14_at_10.26.44_AM.png)
      ## Does this MR meet the acceptance criteria?
      
      - [ ] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
      - [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
      - [ ] API support added
      - [ ] Tests
        - [ ] Added for this feature/bug
        - [ ] All builds are passing
      - [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
      - [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
      - [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
      
      See merge request !4650
    • Merge branch 'preventdefault-disabled-buttons' into 'master' · b2247398
      Jacob Schatz authored
      Prevent default disabled buttons and links.
      
      ## What does this MR do?
      Prevents default action for disabled buttons and links. If the element has `.btn` and `.disabled` at the same time, its default action will be prevented.
      
      ## Are there points in the code the reviewer needs to double check?
      Yes. Is there a better way to do that?
      
      ## Why was this MR needed?
      Right now we can click disabled links and it cause some troubles like in #18079.
      
      ## What are the relevant issue numbers?
      Fixes #18079
      
      ## Screenshots (if relevant)
      ![disabled-click](/uploads/48b58ce130f843e530e62632bcc27436/disabled-click.gif)
      
      ## Does this MR meet the acceptance criteria?
      
      - [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
      - [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
      - [ ] API support added
      - [x] Tests
        - [x] Added for this feature/bug
        - [x] All builds are passing
      - [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
      - [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
      - [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
      
      See merge request !4658
    • Merge branch 'show_number_of_mrs_in_milestone_page' into 'master' · a69cd321
      Jacob Schatz authored
      Show number of processed MRs in milestone page
      
      closes #14025 
      
      ![Screen_Shot_2016-05-02_at_6.20.34_PM](/uploads/9208e58c0e0c77133bd73332ee3b3342/Screen_Shot_2016-05-02_at_6.20.34_PM.png)
      
      See merge request !4006
    • Fix admin hooks spec · bcbe9b4d
      Dmitriy Zaporozhets authored
      Signed-off-by: 's avatarDmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>
    • Prevent default disabled buttons and links. · 415b032b
      Fatih Acet authored
    • Merge branch 'add-index-on-requested_at-to-members' into 'master' · 33b68f87
      Yorick Peterse authored
      Add index on `requested_at` to the `members` table
      
      See merge request !4679
    • Merge branch 'mr-buttons-update' into 'master' · 93a496d2
      Jacob Schatz authored
      Fixed issue with MR buttons being in a group
      
      ## What does this MR do?
      
      Fixes design issue with the buttons in the merge request widget
      
      ## Screenshots (if relevant)
      
      ![Screen_Shot_2016-06-09_at_14.13.32](/uploads/44ac4175f5a97a8d7baf97be6eebc684/Screen_Shot_2016-06-09_at_14.13.32.png)
      
      
      See merge request !4562
    • Merge branch '18225-avoid-full-table-users-count' into 'master' · 2f459a03
      Jacob Vosmaer (GitLab) authored
      Check if the Users table has exactly one user limiting the whole set
      
      ## What does this MR do?
      
      Limit the query set so about a full scan for all the rows on the users table (only scan to records)
      
      #18225 
      
      See merge request !4492
    • Merge branch 'docker-registry-manifest-v1' into 'master' · a8b46301
      Rémy Coutable authored
      Add support for Docker Registry manifest v1
      
      ## What does this MR do?
      
      Adds support for Manifest V1 generated by older versions of Docker (before 1.10).
      
      ## What are the relevant issue numbers?
      
      Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/18609
      
      
      See merge request !4669
    • Merge branch 'issuable-sidebar-dropdown-deselect-fix' into 'master' · e27de526
      Jacob Schatz authored
      Fixed issue with de-selecting dropdown option in issue sidebar
      
      ## What does this MR do?
      
      When de-selecting either an assignee or milestone, we try to send `undefined` which jQuery removes from the request causing a routing error. Instead this MR sends `null` which jQuery still sends and then correctly removes either the milestone or assignee. 
      
      ## What are the relevant issue numbers?
      
      Closes #18641 
      
      ## Screenshots (if relevant)
      
      ![assignee](/uploads/6eaca416fb6e31eabf4c038967160c07/assignee.gif)
      
      ## Does this MR meet the acceptance criteria?
      
      - [ ] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
      - [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
      - [ ] API support added
      - [ ] Tests
        - [ ] Added for this feature/bug
        - [ ] All builds are passing
      - [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
      - [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
      - [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
      
      See merge request !4671
    • Merge branch 'tooltip-change' into 'master' · d1f0ad45
      Jacob Schatz authored
      Update project star/unstar tooltip
      
      ## What does this MR do?
      Fixes project toggle star button tooltip to show actual tooltip for its state.
      
      ## Are there points in the code the reviewer needs to double check?
      No.
      
      ## Why was this MR needed?
      - When you star a project tooltip wasn't changing.
      - When you star a project and refresh the page, tooltip was still showing `Star Project`
      
      ## What are the relevant issue numbers?
      Fixes #18143
      
      ## Screenshots (if relevant)
      
      ### Before 
      ![Screen_Shot_2016-06-15_at_15.31.57](/uploads/e005951c23eeed7c2761a0ec1911f3ce/Screen_Shot_2016-06-15_at_15.31.57.png)
      
      ### After
      ![tooltip-change](/uploads/4caec99eac6ffcc7331947fdfdb9e051/tooltip-change.gif)
      
      ## Does this MR meet the acceptance criteria?
      
      - [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
      - [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
      - [ ] API support added
      - [ ] Tests
        - [ ] Added for this feature/bug
        - [ ] All builds are passing
      - [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
      - [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
      - [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
      
      See merge request !4672
    • Rearrange order of tabs · 6d10d825
      Annabel Dunstone authored
    • Show created_at in table column · 3213023d
      Kamil Trzcinski authored
    • Nest li elements directly under ul · f1245bde
      Annabel Dunstone authored
    • Move builds tab to admin overview · d3b6c185
      Annabel Dunstone authored
    • Add monitoring link with subtabs · d1c3f3d8
      Annabel Dunstone authored
    • Add sub links to overview · 922a164d
      Annabel Dunstone authored
    • Add counter for abuse reports · 736ba42b
      Annabel Dunstone authored
    • Remove admin layout-nav counters · 58c8661c
      Annabel Dunstone authored
    • Move admin nav to horizontal layout nav · 10ae4a8e
      Annabel Dunstone authored
    • Merge branch 'eager-loading-issue-parser' into 'master' · 8bfbafbb
      Douwe Maan authored
      Eager load project relations in IssueParser
      
      ## What does this MR do?
      
      This changes the ReferenceParser class to eager load various associations. This in turn results in the permissions checking code (e.g. the `Ability` model) to _not_ run dozens if not hundreds of extra SQL queries depending on the amount of references involved (in a single document).
      
      ## Are there points in the code the reviewer needs to double check?
      
      No.
      
      ## Why was this MR needed?
      
      In !4410 it was revealed a _lot_ of a queries came from the `Ability` model and the code it would call. In many cases this was because the code would simply get a project, then get the owners; or get a group, then get some association of that. Eager loading these associations is a fairly simple solution and greatly cuts down the number of queries.
      
      ## What are the relevant issue numbers?
      
      None.
      
      ## Does this MR meet the acceptance criteria?
      
      - [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
      - [x] ~~[Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)~~
      - [x] ~~API support added~~
      - [ ] Tests
        - [x] ~~Added for this feature/bug~~
        - [ ] All builds are passing
      - [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
      - [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
      - [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
      
      See merge request !4675
    • Merge branch 'list-items' into 'master' · a6042a2d
      Jacob Schatz authored
      Remove div between ul and li
      
      ## What does this MR do?
      Adds `container_class`to `ul` instead of `div` for valid HTML
      
      ## What are the relevant issue numbers?
      https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4592#note_12420843
      
      
      
      See merge request !4633
    • Merge branch 'zj-system-notes-not-awardable' into 'master' · 84138691
      Douwe Maan authored
      Award Emoji can't be awarded on system notes backend
      
      
      
      See merge request !4668
    • Merge branch 'environments-and-deployments' into 'master' · be098459
      Rémy Coutable authored
      Add environments and deployments
      
      This MR is a continuation of https://gitlab.com/gitlab-org/gitlab-ce/issues/17009.
      
      The current implementation is as follow:
      1. We have two new tables: `environments` and `deployments`.
      2. We have a new tab: `Environments` under `Pipelines` where you can see all you environments and add a new one.
      3. We add a new option to `.gitlab-ci.yml` to track where we should create a deployment for environment.
      4. If environment in `.gitlab-ci.yml` is specified it will create a deployment. **If environment does not exist it will be created.** (this got changed)
      5. The deployment is always successful and shows the time of the action, in that case a build that presumably should do deployment. In the future we could extend deployment with statuses: success, failure. We could extend deployments with information that this is partial or full deployment.
      6. User have to create environments that he will track first.
      7. User can remove environments.
      8. User can retry/rollback past deployment (in that case we retry past build). The new build when succeeds it will create a new deployment.
      9. Currently environment have only one parameter: `name`. In the future it should have: `variables`, `credentials` and possibly `runners` and maybe other resources.
      10. Currently deployment have this parameters: `sha`, `ref`, `deployable (in this case a build)`, `user (who triggered a deployment)`, `created_at`.
      
      The `.gitlab-ci.yml`:
      ```
      deploy to production:
        stage: deploy
        script: dpl travis...
        environment: production
      ```
      
      What needs to be done:
      - [x] Write initial implementation
      - [x] Improve implementation (@ayufan)
      - [x] Write tests (@ayufan)
      - [x] Improve UX of the forms (cc @markpundsack) - reviewed by @markpundsack
      - [x] Improve implementation of the views (cc @jschatz1) - done by @iamphill 
      - [x] Write .gitlab-ci.yml documentation for `environments` - done by @ayufan
      - [ ] Write user documentation (@ayufan and @markpundsack)
      
      See merge request !4605
    • Eager load project relations in IssueParser · fce675d7
      Yorick Peterse authored
      By eager loading these associations we can greatly cut down the number
      of SQL queries executed when processing documents with lots of
      references, especially in cases where there are references belonging to
      the same project.
      
      Since these associations are so specific to the reference parsing
      process and the permissions checking process that follows it I opted to
      include them directly in IssueParser instead of using something like a
      scope. Once we have a need for it we can move this code to a scope or
      method.
    • Get rid of Gitlab::ShellEnv · c32e6125
      Jacob Vosmaer authored
    • Merge branch 'seed-fu-award-emoji' into 'master' · a4a85c26
      Douwe Maan authored
      Seed Award Emoji while seeding the database
      
      ## What does this MR do?
      
      Lets `rake dev:setup` create award emoji on _some_ awardables. Seemed overkill to give all that privilege. 
      
      See merge request !4555