Merge remote-tracking branch 'upstream/master' into pipeline-notifications

CHANGELOG.md

@@ -3,6 +3,10 @@ Please view this file on the master branch, on stable branches it's out of date.
 ## 8.14.0 (2016-11-22)
   - Adds user project membership expired event to clarify why user was removed (Callum Dryden)
   - Add CI notifications. Who triggered a pipeline would receive an email after the pipeline is succeeded or failed. Users could also update notification settings accordingly. !6342
+  - Trim leading and trailing whitespace on project_path (Linus Thiel)
+  - Fix HipChat notifications rendering (airatshigapov, eisnerd)
+  - Simpler arguments passed to named_route on toggle_award_url helper method
+  - Fix: Backup restore doesn't clear cache
 
 ## 8.13.0 (2016-10-22)
 
@@ -22,11 +26,13 @@ Please view this file on the master branch, on stable branches it's out of date.
   - Add `/projects/visible` API endpoint (Ben Boeckel)
   - Fix centering of custom header logos (Ashley Dumaine)
   - Keep around commits only pipeline creation as pipeline data doesn't change over time
+  - Update duration at the end of pipeline
   - ExpireBuildArtifactsWorker query builds table without ordering enqueuing one job per build to cleanup
   - Add group level labels. (!6425)
   - Add an example for testing a phoenix application with Gitlab CI in the docs (Manthan Mallikarjun)
   - Cancelled pipelines could be retried. !6927
   - Updating verbiage on git basics to be more intuitive
+  - Fix project_feature record not generated on project creation
   - Clarify documentation for Runners API (Gennady Trafimenkov)
   - The instrumentation for Banzai::Renderer has been restored
   - Change user & group landing page routing from /u/:username to /:username
@@ -35,11 +41,14 @@ Please view this file on the master branch, on stable branches it's out of date.
   - AbstractReferenceFilter caches project_refs on RequestStore when active
   - Replaced the check sign to arrow in the show build view. !6501
   - Add a /wip slash command to toggle the Work In Progress status of a merge request. !6259 (tbalthazar)
+  - ProjectCacheWorker updates caches at most once per 15 minutes per project
   - Fix Error 500 when viewing old merge requests with bad diff data
   - Create a new /templates namespace for the /licenses, /gitignores and /gitlab_ci_ymls API endpoints. !5717 (tbalthazar)
+  - Fix viewing merged MRs when the source project has been removed !6991
   - Speed-up group milestones show page
   - Fix inconsistent options dropdown caret on mobile viewports (ClemMakesApps)
   - Extract project#update_merge_requests and SystemHooks to its own worker from GitPushService
+  - Fix discussion thread from emails for merge requests. !7010
   - Don't include archived projects when creating group milestones. !4940 (Jeroen Jacobs)
   - Add tag shortcut from the Commit page. !6543
   - Keep refs for each deployment
@@ -135,6 +144,7 @@ Please view this file on the master branch, on stable branches it's out of date.
   - Delete dynamic environments
   - Fix buggy iOS tooltip layering behavior.
   - Make guests unable to view MRs on private projects
+  - Fix broken Project API docs (Takuya Noguchi)
 
 ## 8.12.7
 

app/assets/javascripts/protected_branches/protected_branches_bundle.js

+/*= require_tree . */

app/assets/stylesheets/pages/login.scss

@@ -53,6 +53,7 @@
       margin: 0 0 10px;
     }
 
+
     .login-footer {
       margin-top: 10px;
 
@@ -246,3 +247,19 @@
     padding: 65px; // height of footer + bottom padding of email confirmation link
   }
 }
+
+// For sign in pane only, to improve tab order, the following removes the submit button from
+// normal document flow and pins it to the bottom of the form. For context, see !6867 & !6928
+
+.login-box {
+  .new_user {
+    position: relative;
+    padding-bottom: 35px;
+  }
+
+  .move-submit-down {
+    position: absolute;
+    width: 100%;
+    bottom: 0;
+  }
+}

app/assets/stylesheets/pages/pipelines.scss

@@ -369,10 +369,6 @@
 
     &:hover {
       background-color: $gray-lighter;
-
-      .dropdown-menu-toggle {
-        background-color: transparent;
-      }
     }
 
     &.playable {
@@ -402,6 +398,15 @@
         }
       }
 
+      .tooltip {
+        white-space: nowrap;
+
+        .tooltip-inner {
+          overflow: hidden;
+          text-overflow: ellipsis;
+        }
+      }
+
       .ci-status-text {
         width: 135px;
         white-space: nowrap;
@@ -419,6 +424,7 @@
       }
 
       .dropdown-menu-toggle {
+        background-color: transparent;
         border: none;
         width: auto;
         padding: 0;

app/controllers/projects/commits_controller.rb

@@ -13,7 +13,7 @@ class Projects::CommitsController < Projects::ApplicationController
 
     @commits =
       if search.present?
-        @repository.find_commits_by_message(search, @ref, @path, @limit, @offset).compact
+        @repository.find_commits_by_message(search, @ref, @path, @limit, @offset)
       else
         @repository.commits(@ref, path: @path, limit: @limit, offset: @offset)
       end

app/controllers/projects/merge_requests_controller.rb

@@ -398,7 +398,7 @@ class Projects::MergeRequestsController < Projects::ApplicationController
 
       status ||= "preparing"
     else
-      ci_service = @merge_request.source_project.ci_service
+      ci_service = @merge_request.source_project.try(:ci_service)
       status = ci_service.commit_status(merge_request.diff_head_sha, merge_request.source_branch) if ci_service
 
       if ci_service.respond_to?(:commit_coverage)
@@ -554,7 +554,7 @@ class Projects::MergeRequestsController < Projects::ApplicationController
   def define_pipelines_vars
     @pipelines = @merge_request.all_pipelines
 
-    if @pipelines.any?
+    if @pipelines.present?
       @pipeline = @pipelines.first
       @statuses = @pipeline.statuses.relevant
     end

app/controllers/projects/project_members_controller.rb

@@ -32,21 +32,6 @@ class Projects::ProjectMembersController < Projects::ApplicationController
       current_user: current_user
     )
 
-    if params[:group_ids].present?
-      group_ids = params[:group_ids].split(',')
-      groups = Group.where(id: group_ids)
-
-      groups.each do |group|
-        next unless can?(current_user, :read_group, group)
-
-        project.project_group_links.create(
-          group: group,
-          group_access: params[:access_level],
-          expires_at: params[:expires_at]
-        )
-      end
-    end
-
     redirect_to namespace_project_project_members_path(@project.namespace, @project)
   end
 

app/helpers/award_emoji_helper.rb

@@ -3,8 +3,8 @@ module AwardEmojiHelper
     return url_for([:toggle_award_emoji, awardable]) unless @project
 
     if awardable.is_a?(Note)
-      # We render a list of notes very frequently and calling the specific method is a lot faster than the generic one (6.5x)
-      toggle_award_emoji_namespace_project_note_url(namespace_id: @project.namespace, project_id: @project, id: awardable.id)
+      # We render a list of notes very frequently and calling the specific method is a lot faster than the generic one (4.5x)
+      toggle_award_emoji_namespace_project_note_url(@project.namespace, @project, awardable.id)
     else
       url_for([:toggle_award_emoji, @project.namespace.becomes(Namespace), @project, awardable])
     end

app/helpers/merge_requests_helper.rb

@@ -86,11 +86,15 @@ module MergeRequestsHelper
   end
 
   def source_branch_with_namespace(merge_request)
-    branch = link_to(merge_request.source_branch, namespace_project_commits_path(merge_request.source_project.namespace, merge_request.source_project, merge_request.source_branch))
+    namespace = merge_request.source_project_namespace
+    branch = merge_request.source_branch
+
+    if merge_request.source_branch_exists?
+      namespace = link_to(namespace, project_path(merge_request.source_project))
+      branch = link_to(branch, namespace_project_commits_path(merge_request.source_project.namespace, merge_request.source_project, merge_request.source_branch))
+    end
 
     if merge_request.for_fork?
-      namespace = link_to(merge_request.source_project_namespace,
-        project_path(merge_request.source_project))
       namespace + ":" + branch
     else
       branch

app/models/ci/pipeline.rb

@@ -59,9 +59,6 @@ module Ci
 
       before_transition any => [:success, :failed, :canceled] do |pipeline|
         pipeline.finished_at = Time.now
-      end
-
-      before_transition do |pipeline|
         pipeline.update_duration
       end
 

app/models/deployment.rb

@@ -11,7 +11,7 @@ class Deployment < ActiveRecord::Base
 
   delegate :name, to: :environment, prefix: true
 
-  after_save :create_ref
+  after_create :create_ref
 
   def commit
     project.commit(sha)
@@ -102,6 +102,6 @@ class Deployment < ActiveRecord::Base
   private
 
   def ref_path
-    File.join(environment.ref_path, 'deployments', id.to_s)
+    File.join(environment.ref_path, 'deployments', iid.to_s)
   end
 end

app/models/environment.rb

@@ -71,8 +71,8 @@ class Environment < ActiveRecord::Base
 
     return nil unless ref
 
-    deployment_id = ref.split('/').last
-    deployments.find(deployment_id)
+    deployment_iid = ref.split('/').last
+    deployments.find_by(iid: deployment_iid)
   end
 
   def ref_path

app/models/merge_request.rb

@@ -326,21 +326,17 @@ class MergeRequest < ActiveRecord::Base
   def validate_fork
     return true unless target_project && source_project
     return true if target_project == source_project
-    return true unless forked_source_project_missing?
+    return true unless source_project_missing?
 
     errors.add :validate_fork,
                'Source project is not a fork of the target project'
   end
 
   def closed_without_fork?
-    closed? && forked_source_project_missing?
+    closed? && source_project_missing?
   end
 
-  def closed_without_source_project?
-    closed? && !source_project
-  end
-
-  def forked_source_project_missing?
+  def source_project_missing?
     return false unless for_fork?
     return true unless source_project
 
@@ -348,9 +344,7 @@ class MergeRequest < ActiveRecord::Base
   end
 
   def reopenable?
-    return false if closed_without_fork? || closed_without_source_project? || merged?
-
-    closed?
+    closed? && !source_project_missing? && source_branch_exists?
   end
 
   def ensure_merge_request_diff
@@ -662,7 +656,7 @@ class MergeRequest < ActiveRecord::Base
   end
 
   def has_ci?
-    source_project.ci_service && commits.any?
+    source_project.try(:ci_service) && commits.any?
   end
 
   def branch_missing?
@@ -694,12 +688,9 @@ class MergeRequest < ActiveRecord::Base
 
     @environments ||=
       begin
-        environments = source_project.environments_for(
-          source_branch, diff_head_commit)
-        environments += target_project.environments_for(
-          target_branch, diff_head_commit, with_tags: true)
-
-        environments.uniq
+        envs = target_project.environments_for(target_branch, diff_head_commit, with_tags: true)
+        envs.concat(source_project.environments_for(source_branch, diff_head_commit)) if source_project
+        envs.uniq
       end
   end
 

app/models/project.rb

@@ -32,8 +32,8 @@ class Project < ActiveRecord::Base
   default_value_for(:shared_runners_enabled) { current_application_settings.shared_runners_enabled }
 
   after_create :ensure_dir_exist
+  after_create :create_project_feature, unless: :project_feature
   after_save :ensure_dir_exist, if: :namespace_id_changed?
-  after_initialize :setup_project_feature
 
   # set last_activity_at to the same as created_at
   after_create :set_last_activity_at
@@ -1310,11 +1310,6 @@ class Project < ActiveRecord::Base
     "projects/#{id}/pushes_since_gc"
   end
 
-  # Prevents the creation of project_feature record for every project
-  def setup_project_feature
-    build_project_feature unless project_feature
-  end
-
   def default_branch_protected?
     current_application_settings.default_branch_protection == Gitlab::Access::PROTECTION_FULL ||
       current_application_settings.default_branch_protection == Gitlab::Access::PROTECTION_DEV_CAN_MERGE
@@ -1344,6 +1339,13 @@ class Project < ActiveRecord::Base
     shared_projects.any?
   end
 
+  # Similar to the normal callbacks that hook into the life cycle of an
+  # Active Record object, you can also define callbacks that get triggered
+  # when you add an object to an association collection. If any of these
+  # callbacks throw an exception, the object will not be added to the
+  # collection. Before you add a new board to the boards collection if you
+  # already have 1, 2, or n it will fail, but it if you have 0 that is lower
+  # than the number of permitted boards per project it won't fail.
   def validate_board_limit(board)
     raise BoardLimitExceeded, 'Number of permitted boards exceeded' if boards.size >= NUMBER_OF_PERMITTED_BOARDS
   end

app/models/project_services/hipchat_service.rb

 class HipchatService < Service
+  include ActionView::Helpers::SanitizeHelper
+
   MAX_COMMITS = 3
+  HIPCHAT_ALLOWED_TAGS = %w[
+    a b i strong em br img pre code
+    table th tr td caption colgroup col thead tbody tfoot
+    ul ol li dl dt dd
+  ]
 
   prop_accessor :token, :room, :server, :notify, :color, :api_version
   boolean_accessor :notify_only_broken_builds
@@ -88,6 +95,10 @@ class HipchatService < Service
     end
   end
 
+  def render_line(text)
+    markdown(text.lines.first.chomp, pipeline: :single_line) if text
+  end
+
   def create_push_message(push)
     ref_type = Gitlab::Git.tag_ref?(push[:ref]) ? 'tag' : 'branch'
     ref = Gitlab::Git.ref_name(push[:ref])
@@ -110,7 +121,7 @@ class HipchatService < Service
       message << "(<a href=\"#{project.web_url}/compare/#{before}...#{after}\">Compare changes</a>)"
 
       push[:commits].take(MAX_COMMITS).each do |commit|
-        message << "<br /> - #{commit[:message].lines.first} (<a href=\"#{commit[:url]}\">#{commit[:id][0..5]}</a>)"
+        message << "<br /> - #{render_line(commit[:message])} (<a href=\"#{commit[:url]}\">#{commit[:id][0..5]}</a>)"
       end
 
       if push[:commits].count > MAX_COMMITS
@@ -121,12 +132,22 @@ class HipchatService < Service
     message
   end
 
-  def format_body(body)
-    if body
-      body = body.truncate(200, separator: ' ', omission: '...')
-    end
+  def markdown(text, options = {})
+    return "" unless text
+
+    context = {
+      project: project,
+      pipeline: :email
+    }
+
+    Banzai.render(text, context)
 
-    "<pre>#{body}</pre>"
+    context.merge!(options)
+
+    html = Banzai.post_process(Banzai.render(text, context), context)
+    sanitized_html = sanitize(html, tags: HIPCHAT_ALLOWED_TAGS, attributes: %w[href title alt])
+
+    sanitized_html.truncate(200, separator: ' ', omission: '...')
   end
 
   def create_issue_message(data)
@@ -134,7 +155,7 @@ class HipchatService < Service
 
     obj_attr = data[:object_attributes]
     obj_attr = HashWithIndifferentAccess.new(obj_attr)
-    title = obj_attr[:title]
+    title = render_line(obj_attr[:title])
     state = obj_attr[:state]
     issue_iid = obj_attr[:iid]
     issue_url = obj_attr[:url]
@@ -143,10 +164,7 @@ class HipchatService < Service
     issue_link = "<a href=\"#{issue_url}\">issue ##{issue_iid}</a>"
     message = "#{user_name} #{state} #{issue_link} in #{project_link}: <b>#{title}</b>"
 
-    if description
-      description = format_body(description)
-      message << description
-    end
+    message << "<pre>#{markdown(description)}</pre>"
 
     message
   end
@@ -159,23 +177,20 @@ class HipchatService < Service
     merge_request_id = obj_attr[:iid]
     state = obj_attr[:state]
     description = obj_attr[:description]
-    title = obj_attr[:title]
+    title = render_line(obj_attr[:title])
 
     merge_request_url = "#{project_url}/merge_requests/#{merge_request_id}"
     merge_request_link = "<a href=\"#{merge_request_url}\">merge request !#{merge_request_id}</a>"
     message = "#{user_name} #{state} #{merge_request_link} in " \
       "#{project_link}: <b>#{title}</b>"
 
-    if description
-      description = format_body(description)
-      message << description
-    end
+    message << "<pre>#{markdown(description)}</pre>"
 
     message
   end
 
   def format_title(title)
-    "<b>" + title.lines.first.chomp + "</b>"
+    "<b>#{render_line(title)}</b>"
   end
 
   def create_note_message(data)
@@ -186,11 +201,13 @@ class HipchatService < Service
     note = obj_attr[:note]
     note_url = obj_attr[:url]
     noteable_type = obj_attr[:noteable_type]
+    commit_id = nil
 
     case noteable_type
     when "Commit"
       commit_attr = HashWithIndifferentAccess.new(data[:commit])
-      subject_desc = commit_attr[:id]
+      commit_id = commit_attr[:id]
+      subject_desc = commit_id
       subject_desc = Commit.truncate_sha(subject_desc)
       subject_type = "commit"
       title = format_title(commit_attr[:message])
@@ -218,10 +235,7 @@ class HipchatService < Service
     message = "#{user_name} commented on #{subject_html} in #{project_link}: "
     message << title
 
-    if note
-      note = format_body(note)
-      message << note
-    end
+    message << "<pre>#{markdown(note, ref: commit_id)}</pre>"
 
     message
   end

app/models/repository.rb

@@ -109,6 +109,10 @@ class Repository
   end
 
   def find_commits_by_message(query, ref = nil, path = nil, limit = 1000, offset = 0)
+    unless exists? && has_visible_content? && query.present?
+      return []
+    end
+
     ref ||= root_ref
 
     args = %W(
@@ -117,9 +121,8 @@ class Repository
     )
     args = args.concat(%W(-- #{path})) if path.present?
 
-    git_log_results = Gitlab::Popen.popen(args, path_to_repo).first.lines.map(&:chomp)
-    commits = git_log_results.map { |c| commit(c) }
-    commits
+    git_log_results = Gitlab::Popen.popen(args, path_to_repo).first.lines
+    git_log_results.map { |c| commit(c.chomp) }.compact
   end
 
   def find_branch(name, fresh_repo: true)

app/views/devise/sessions/_new_base.html.haml

@@ -5,6 +5,8 @@
   %div.form-group
     = f.label :password
     = f.password_field :password, class: "form-control bottom", required: true, title: "This field is required."
+  %div.submit-container.move-submit-down
+    = f.submit "Sign in", class: "btn btn-save"
   - if devise_mapping.rememberable?
     .remember-me.checkbox
       %label{for: "user_remember_me"}
@@ -12,5 +14,3 @@
         %span Remember me
       .pull-right
         = link_to "Forgot your password?", new_password_path(resource_name)
-  %div.submit-container
-    = f.submit "Sign in", class: "btn btn-save"

app/views/projects/ci/builds/_build_pipeline.html.haml

 - is_playable = subject.playable? && can?(current_user, :update_build, @project)
 - if is_playable
-  = link_to play_namespace_project_build_path(subject.project.namespace, subject.project, subject, return_to: request.original_url), method: :post, title: 'Play' do
+  = link_to play_namespace_project_build_path(subject.project.namespace, subject.project, subject, return_to: request.original_url), method: :post, data: { toggle: 'tooltip', title: "#{subject.name} - play", container: '.pipeline-graph', placement: 'bottom' } do
     = render_status_with_link('build', 'play')
     .ci-status-text= subject.name
 - elsif can?(current_user, :read_build, @project)
-  = link_to namespace_project_build_path(subject.project.namespace, subject.project, subject) do
+  = link_to namespace_project_build_path(subject.project.namespace, subject.project, subject), data: { toggle: 'tooltip', title: "#{subject.name} - #{subject.status}", container: '.pipeline-graph', placement: 'bottom' } do
     %span.ci-status-icon
       = render_status_with_link('build', subject.status)
     .ci-status-text= subject.name

app/views/projects/commit/_pipeline_status_group.html.haml

 - group_status = CommitStatus.where(id: subject).status
-%button.dropdown-menu-toggle{ type: 'button', data: { toggle: 'dropdown' } }
+%button.dropdown-menu-toggle.has-tooltip{ type: 'button', data: { toggle: 'dropdown', title: "#{name} - #{group_status}" } }
   %span.ci-status-icon
     = render_status_with_link('build', group_status)
   %span.ci-status-text

app/views/projects/generic_commit_statuses/_generic_commit_status_pipeline.html.haml

-- if subject.target_url
-  = link_to subject.target_url do
+%a{ data: { toggle: 'tooltip', title: "#{subject.name} - #{subject.status}", container: '.pipeline-graph', placement: 'bottom' } }
+  - if subject.target_url
+    = link_to subject.target_url do
+      %span.ci-status-icon
+        = render_status_with_link('commit status', subject.status)
+      %span.ci-status-text= subject.name
+  - else
     %span.ci-status-icon
       = render_status_with_link('commit status', subject.status)
     %span.ci-status-text= subject.name
-- else
-  %span.ci-status-icon
-    = render_status_with_link('commit status', subject.status)
-  %span.ci-status-text= subject.name

app/views/projects/merge_requests/_show.html.haml

@@ -26,19 +26,19 @@
             %ul.dropdown-menu.dropdown-menu-align-right
               %li= link_to "Email Patches", merge_request_path(@merge_request, format: :patch)
               %li= link_to "Plain Diff",    merge_request_path(@merge_request, format: :diff)
-      - unless @merge_request.closed_without_fork?
-        .normal
-          %span Request to merge
-          %span.label-branch= source_branch_with_namespace(@merge_request)
-          %span into
-          %span.label-branch
-            = link_to @merge_request.target_branch, namespace_project_commits_path(@project.namespace, @project, @merge_request.target_branch)
-          - if @merge_request.open? && @merge_request.diverged_from_target_branch?
-            %span (#{pluralize(@merge_request.diverged_commits_count, 'commit')} behind)
+      .normal
+        %span Request to merge
+        %span.label-branch= source_branch_with_namespace(@merge_request)
+        %span into
+        %span.label-branch
+          = link_to @merge_request.target_branch, namespace_project_commits_path(@project.namespace, @project, @merge_request.target_branch)
+        - if @merge_request.open? && @merge_request.diverged_from_target_branch?
+          %span (#{pluralize(@merge_request.diverged_commits_count, 'commit')} behind)
 
-    - unless @merge_request.closed_without_source_project?
+    - if @merge_request.source_branch_exists?
       = render "projects/merge_requests/show/how_to_merge"
-      = render "projects/merge_requests/widget/show.html.haml"
+
+    = render "projects/merge_requests/widget/show.html.haml"
 
     - if @merge_request.source_branch_exists? && @merge_request.mergeable? && @merge_request.can_be_merged_by?(current_user)
       .light.prepend-top-default.append-bottom-default
@@ -52,7 +52,7 @@
           = link_to namespace_project_merge_request_path(@project.namespace, @project, @merge_request), data: { target: 'div#notes', action: 'notes', toggle: 'tab' } do
             Discussion
             %span.badge= @merge_request.mr_and_commit_notes.user.count
-        - unless @merge_request.closed_without_source_project?
+        - if @merge_request.source_project
           %li.commits-tab
             = link_to commits_namespace_project_merge_request_path(@project.namespace, @project, @merge_request), data: { target: 'div#commits', action: 'commits', toggle: 'tab' } do
               Commits

app/views/projects/new.html.haml

@@ -27,6 +27,7 @@
                 - else
                   .input-group-addon.static-namespace
                     #{root_url}#{current_user.username}/
+                  = f.hidden_field :namespace_id, value: current_user.namespace_id
           .form-group.col-xs-12.col-sm-6.project-path
             = f.label :namespace_id, class: 'label-light' do
               %span
@@ -126,6 +127,11 @@
     }
   });
 
+  $('#new_project').submit(function(){
+    var $path = $('#project_path');
+    $path.val($path.val().trim());
+  });
+
   $('#project_path').keyup(function(){
     if($(this).val().length !=0) {
       $('.btn_import_gitlab_project').attr('disabled', false);

app/views/projects/protected_branches/index.html.haml

 - page_title "Protected branches"
+- content_for :page_specific_javascripts do
+  = page_specific_javascript_tag('protected_branches/protected_branches_bundle.js')
 
 .row.prepend-top-default.append-bottom-default
   .col-lg-3

app/workers/project_cache_worker.rb

+# Worker for updating any project specific caches.
+#
+# This worker runs at most once every 15 minutes per project. This is to ensure
+# that multiple instances of jobs for this worker don't hammer the underlying
+# storage engine as much.
 class ProjectCacheWorker
   include Sidekiq::Worker
 
   sidekiq_options queue: :default
 
+  LEASE_TIMEOUT = 15.minutes.to_i
+
   def perform(project_id)
+    if try_obtain_lease_for(project_id)
+      Rails.logger.
+        info("Obtained ProjectCacheWorker lease for project #{project_id}")
+    else
+      Rails.logger.
+        info("Could not obtain ProjectCacheWorker lease for project #{project_id}")
+
+      return
+    end
+
+    update_caches(project_id)
+  end
+
+  def update_caches(project_id)
     project = Project.find(project_id)
 
     return unless project.repository.exists?
@@ -15,4 +36,10 @@ class ProjectCacheWorker
       project.repository.build_cache
     end
   end
+
+  def try_obtain_lease_for(project_id)
+    Gitlab::ExclusiveLease.
+      new("project_cache_worker:#{project_id}", timeout: LEASE_TIMEOUT).
+      try_obtain
+  end
 end

config/application.rb

@@ -87,6 +87,7 @@ module Gitlab
     config.assets.precompile << "users/users_bundle.js"
     config.assets.precompile << "network/network_bundle.js"
     config.assets.precompile << "profile/profile_bundle.js"
+    config.assets.precompile << "protected_branches/protected_branches_bundle.js"
     config.assets.precompile << "diff_notes/diff_notes_bundle.js"
     config.assets.precompile << "boards/boards_bundle.js"
     config.assets.precompile << "merge_conflicts/merge_conflicts_bundle.js"

db/migrate/20161019213545_generate_project_feature_for_projects.rb

+class GenerateProjectFeatureForProjects < ActiveRecord::Migration
+  DOWNTIME = true
+
+  DOWNTIME_REASON = <<-HEREDOC
+    Application was eager loading project_feature for all projects generating an extra query
+    everytime a project was fetched. We removed that behavior to avoid the extra query, this migration
+    makes sure all projects have a project_feature record associated.
+  HEREDOC
+
+  def up
+    # Generate enabled values for each project feature 20, 20, 20, 20, 20
+    # All features are enabled by default
+    enabled_values = [ProjectFeature::ENABLED] * 5
+
+    execute <<-EOF.strip_heredoc
+      INSERT INTO project_features
+      (project_id, merge_requests_access_level, builds_access_level,
+      issues_access_level, snippets_access_level, wiki_access_level)
+      (SELECT projects.id, #{enabled_values.join(',')} FROM projects LEFT OUTER JOIN project_features
+      ON project_features.project_id = projects.id
+      WHERE project_features.id IS NULL)
+    EOF
+  end
+
+  def down
+    "Not needed"
+  end
+end

db/schema.rb

@@ -11,7 +11,7 @@
 #
 # It's strongly recommended that you check this file into your version control system.
 
-ActiveRecord::Schema.define(version: 20161018024550) do
+ActiveRecord::Schema.define(version: 20161019213545) do
 
   # These are extensions that must be enabled in order to support this database
   enable_extension "plpgsql"

doc/api/projects.md

@@ -1333,8 +1333,6 @@ Parameters:
 
 | Attribute | Type | Required | Description |
 | --------- | ---- | -------- | ----------- |
-| `query` (required) - A string contained in the project name
-| `per_page` (optional) - number of projects to return per page
-| `page` (optional) - the page to retrieve
-| `order_by` (optional) - Return requests ordered by `id`, `name`, `created_at` or `last_activity_at` fields
+| `query` | string | yes | A string contained in the project name |
+| `order_by` | string | no | Return requests ordered by `id`, `name`, `created_at` or `last_activity_at` fields |
 | `sort` | string | no | Return requests sorted in `asc` or `desc` order |

features/support/capybara.rb

@@ -20,5 +20,5 @@ unless ENV['CI'] || ENV['CI_SERVER']
 end
 
 Spinach.hooks.before_run do
-  TestEnv.warm_asset_cache
+  TestEnv.warm_asset_cache unless ENV['CI'] || ENV['CI_SERVER']
 end

lib/api/commit_statuses.rb

@@ -67,9 +67,14 @@ module API
         pipeline = @project.ensure_pipeline(ref, commit.sha, current_user)
 
         status = GenericCommitStatus.running_or_pending.find_or_initialize_by(
-          project: @project, pipeline: pipeline,
-          user: current_user, name: name, ref: ref)
-        status.attributes = declared(params).slice(:target_url, :description)
+          project: @project,
+          pipeline: pipeline,
+          user: current_user,
+          name: name,
+          ref: ref,
+          target_url: params[:target_url],
+          description: params[:description]
+        )
 
         begin
           case params[:state].to_s

lib/gitlab/email/handler/create_note_handler.rb

@@ -46,7 +46,9 @@ module Gitlab
             noteable_type:  sent_notification.noteable_type,
             noteable_id:    sent_notification.noteable_id,
             commit_id:      sent_notification.commit_id,
-            line_code:      sent_notification.line_code
+            line_code:      sent_notification.line_code,
+            position:       sent_notification.position,
+            type:           sent_notification.note_type
           ).execute
         end
       end

lib/gitlab/project_search_results.rb

@@ -73,11 +73,7 @@ module Gitlab
     end
 
     def commits
-      if project.empty_repo? || query.blank?
-        []
-      else
-        project.repository.find_commits_by_message(query).compact
-      end
+      project.repository.find_commits_by_message(query)
     end
 
     def project_ids_relation

lib/tasks/gitlab/backup.rake

@@ -51,6 +51,7 @@ namespace :gitlab do
         $progress.puts 'done'.color(:green)
         Rake::Task['gitlab:backup:db:restore'].invoke
       end
+
       Rake::Task['gitlab:backup:repo:restore'].invoke unless backup.skipped?('repositories')
       Rake::Task['gitlab:backup:uploads:restore'].invoke unless backup.skipped?('uploads')
       Rake::Task['gitlab:backup:builds:restore'].invoke unless backup.skipped?('builds')
@@ -58,6 +59,7 @@ namespace :gitlab do
       Rake::Task['gitlab:backup:lfs:restore'].invoke unless backup.skipped?('lfs')
       Rake::Task['gitlab:backup:registry:restore'].invoke unless backup.skipped?('registry')
       Rake::Task['gitlab:shell:setup'].invoke
+      Rake::Task['cache:clear'].invoke
 
       backup.cleanup
     end

spec/controllers/projects/merge_requests_controller_spec.rb

@@ -913,7 +913,7 @@ describe Projects::MergeRequestsController do
   end
 
   describe 'GET ci_environments_status' do
-    context 'when the environment is from a forked project' do
+    context 'the environment is from a forked project' do
       let!(:forked)       { create(:project) }
       let!(:environment)  { create(:environment, project: forked) }
       let!(:deployment)   { create(:deployment, environment: environment, sha: forked.commit.id, ref: 'master') }

spec/features/merge_requests/created_from_fork_spec.rb

@@ -25,6 +25,20 @@ feature 'Merge request created from fork' do
     expect(page).to have_content 'Test merge request'
   end
 
+  context 'source project is deleted' do
+    background do
+      MergeRequests::MergeService.new(project, user).execute(merge_request)
+      fork_project.destroy!
+    end
+
+    scenario 'user can access merge request' do
+      visit_merge_request(merge_request)
+
+      expect(page).to have_content 'Test merge request'
+      expect(page).to have_content "(removed):#{merge_request.source_branch}"
+    end
+  end
+
   context 'pipeline present in source project' do
     include WaitForAjax
 

spec/features/projects/import_export/import_file_spec.rb

@@ -3,13 +3,8 @@ require 'spec_helper'
 feature 'Import/Export - project import integration test', feature: true, js: true do
   include Select2Helper
 
-  let(:admin) { create(:admin) }
-  let(:normal_user) { create(:user) }
-  let!(:namespace) { create(:namespace, name: "asd", owner: admin) }
   let(:file) { File.join(Rails.root, 'spec', 'features', 'projects', 'import_export', 'test_project_export.tar.gz') }
   let(:export_path) { "#{Dir::tmpdir}/import_file_spec" }
-  let(:project) { Project.last }
-  let(:project_hook) { Gitlab::Git::Hook.new('post-receive', project.repository.path) }
 
   background do
     allow_any_instance_of(Gitlab::ImportExport).to receive(:storage_path).and_return(export_path)
@@ -19,41 +14,43 @@ feature 'Import/Export - project import integration test', feature: true, js: tr
     FileUtils.rm_rf(export_path, secure: true)
   end
 
-  context 'admin user' do
+  context 'when selecting the namespace' do
+    let(:user) { create(:admin) }
+    let!(:namespace) { create(:namespace, name: "asd", owner: user) }
+
     before do
-      login_as(admin)
+      login_as(user)
     end
 
     scenario 'user imports an exported project successfully' do
-      expect(Project.all.count).to be_zero
-
       visit new_project_path
 
-      select2('2', from: '#project_namespace_id')
+      select2(namespace.id, from: '#project_namespace_id')
       fill_in :project_path, with: 'test-project-path', visible: true
       click_link 'GitLab export'
 
       expect(page).to have_content('GitLab project export')
-      expect(URI.parse(current_url).query).to eq('namespace_id=2&path=test-project-path')
+      expect(URI.parse(current_url).query).to eq("namespace_id=#{namespace.id}&path=test-project-path")
 
       attach_file('file', file)
 
-      click_on 'Import project' # import starts
+      expect { click_on 'Import project' }.to change { Project.count }.from(0).to(1)
 
+      project = Project.last
       expect(project).not_to be_nil
       expect(project.issues).not_to be_empty
       expect(project.merge_requests).not_to be_empty
-      expect(project_hook).to exist
-      expect(wiki_exists?).to be true
+      expect(project_hook_exists?(project)).to be true
+      expect(wiki_exists?(project)).to be true
       expect(project.import_status).to eq('finished')
     end
 
     scenario 'invalid project' do
-      project = create(:project, namespace_id: 2)
+      project = create(:project, namespace: namespace)
 
       visit new_project_path
 
-      select2('2', from: '#project_namespace_id')
+      select2(namespace.id, from: '#project_namespace_id')
       fill_in :project_path, with: project.name, visible: true
       click_link 'GitLab export'
 
@@ -66,11 +63,11 @@ feature 'Import/Export - project import integration test', feature: true, js: tr
     end
 
     scenario 'project with no name' do
-      create(:project, namespace_id: 2)
+      create(:project, namespace: namespace)
 
       visit new_project_path
 
-      select2('2', from: '#project_namespace_id')
+      select2(namespace.id, from: '#project_namespace_id')
 
       # click on disabled element
       find(:link, 'GitLab export').trigger('click')
@@ -81,24 +78,30 @@ feature 'Import/Export - project import integration test', feature: true, js: tr
     end
   end
 
-  context 'normal user' do
+  context 'when limited to the default user namespace' do
+    let(:user) { create(:user) }
     before do
-      login_as(normal_user)
+      login_as(user)
     end
 
-    scenario 'non-admin user is allowed to import a project' do
-      expect(Project.all.count).to be_zero
-
+    scenario 'passes correct namespace ID in the URL' do
       visit new_project_path
 
       fill_in :project_path, with: 'test-project-path', visible: true
 
-      expect(page).to have_content('GitLab export')
+      click_link 'GitLab export'
+
+      expect(page).to have_content('GitLab project export')
+      expect(URI.parse(current_url).query).to eq("namespace_id=#{user.namespace.id}&path=test-project-path")
     end
   end
 
-  def wiki_exists?
+  def wiki_exists?(project)
     wiki = ProjectWiki.new(project)
     File.exist?(wiki.repository.path_to_repo) && !wiki.repository.empty?
   end
+
+  def project_hook_exists?(project)
+    Gitlab::Git::Hook.new('post-receive', project.repository.path).exists?
+  end
 end

spec/fixtures/emails/commands_in_reply.eml

@@ -23,8 +23,6 @@ Cool!
 
 /close
 /todo
-/due tomorrow
-
 
 On Sun, Jun 9, 2013 at 1:39 PM, eviltrout via Discourse Meta
 <reply+59d8df8370b7e95c5a49fbf86aeb2c93@appmail.adventuretime.ooo> wrote:

spec/fixtures/emails/commands_only_reply.eml

@@ -21,8 +21,6 @@ X-Scanned-By: MIMEDefang 2.69 on IPv6:2001:470:1d:165::1
 
 /close
 /todo
-/due tomorrow
-
 
 On Sun, Jun 9, 2013 at 1:39 PM, eviltrout via Discourse Meta
 <reply+59d8df8370b7e95c5a49fbf86aeb2c93@appmail.adventuretime.ooo> wrote:

spec/lib/gitlab/email/handler/create_note_handler_spec.rb

@@ -12,10 +12,13 @@ describe Gitlab::Email::Handler::CreateNoteHandler, lib: true do
 
   let(:email_raw) { fixture_file('emails/valid_reply.eml') }
   let(:project)   { create(:project, :public) }
-  let(:noteable)  { create(:issue, project: project) }
   let(:user)      { create(:user) }
+  let(:note)      { create(:diff_note_on_merge_request, project: project) }
+  let(:noteable)  { note.noteable }
 
-  let!(:sent_notification) { SentNotification.record(noteable, user.id, mail_key) }
+  let!(:sent_notification) do
+    SentNotification.record_note(note, user.id, mail_key)
+  end
 
   context "when the recipient address doesn't include a mail key" do
     let(:email_raw) { fixture_file('emails/valid_reply.eml').gsub(mail_key, "") }
@@ -82,7 +85,6 @@ describe Gitlab::Email::Handler::CreateNoteHandler, lib: true do
           expect { receiver.execute }.to change { noteable.notes.count }.by(1)
 
           expect(noteable.reload).to be_closed
-          expect(noteable.due_date).to eq(Date.tomorrow)
           expect(TodoService.new.todo_exist?(noteable, user)).to be_truthy
         end
       end
@@ -100,7 +102,6 @@ describe Gitlab::Email::Handler::CreateNoteHandler, lib: true do
         expect { receiver.execute }.to change { noteable.notes.count }.by(1)
 
         expect(noteable.reload).to be_open
-        expect(noteable.due_date).to be_nil
         expect(TodoService.new.todo_exist?(noteable, user)).to be_falsy
       end
     end
@@ -117,7 +118,6 @@ describe Gitlab::Email::Handler::CreateNoteHandler, lib: true do
         expect { receiver.execute }.to change { noteable.notes.count }.by(2)
 
         expect(noteable.reload).to be_closed
-        expect(noteable.due_date).to eq(Date.tomorrow)
         expect(TodoService.new.todo_exist?(noteable, user)).to be_truthy
       end
     end
@@ -138,10 +138,11 @@ describe Gitlab::Email::Handler::CreateNoteHandler, lib: true do
 
     it "creates a comment" do
       expect { receiver.execute }.to change { noteable.notes.count }.by(1)
-      note = noteable.notes.last
+      new_note = noteable.notes.last
 
-      expect(note.author).to eq(sent_notification.recipient)
-      expect(note.note).to include("I could not disagree more.")
+      expect(new_note.author).to eq(sent_notification.recipient)
+      expect(new_note.position).to eq(note.position)
+      expect(new_note.note).to include("I could not disagree more.")
     end
 
     it "adds all attachments" do
@@ -160,10 +161,11 @@ describe Gitlab::Email::Handler::CreateNoteHandler, lib: true do
       shared_examples 'an email that contains a mail key' do |header|
         it "fetches the mail key from the #{header} header and creates a comment" do
           expect { receiver.execute }.to change { noteable.notes.count }.by(1)
-          note = noteable.notes.last
+          new_note = noteable.notes.last
 
-          expect(note.author).to eq(sent_notification.recipient)
-          expect(note.note).to include('I could not disagree more.')
+          expect(new_note.author).to eq(sent_notification.recipient)
+          expect(new_note.position).to eq(note.position)
+          expect(new_note.note).to include('I could not disagree more.')
         end
       end
 

spec/models/merge_request_spec.rb

@@ -1198,7 +1198,7 @@ describe MergeRequest, models: true do
     end
   end
 
-  describe "#forked_source_project_missing?" do
+  describe "#source_project_missing?" do
     let(:project)      { create(:project) }
     let(:fork_project) { create(:project, forked_from_project: project) }
     let(:user)         { create(:user) }
@@ -1211,13 +1211,13 @@ describe MergeRequest, models: true do
           target_project: project)
       end
 
-      it { expect(merge_request.forked_source_project_missing?).to be_falsey }
+      it { expect(merge_request.source_project_missing?).to be_falsey }
     end
 
     context "when the source project is the same as the target project" do
       let(:merge_request) { create(:merge_request, source_project: project) }
 
-      it { expect(merge_request.forked_source_project_missing?).to be_falsey }
+      it { expect(merge_request.source_project_missing?).to be_falsey }
     end
 
     context "when the fork does not exist" do
@@ -1231,7 +1231,7 @@ describe MergeRequest, models: true do
         unlink_project.execute
         merge_request.reload
 
-        expect(merge_request.forked_source_project_missing?).to be_truthy
+        expect(merge_request.source_project_missing?).to be_truthy
       end
     end
   end
@@ -1274,38 +1274,6 @@ describe MergeRequest, models: true do
     end
   end
 
-  describe '#closed_without_source_project?' do
-    let(:project)      { create(:project) }
-    let(:user)         { create(:user) }
-    let(:fork_project) { create(:project, forked_from_project: project, namespace: user.namespace) }
-    let(:destroy_service) { Projects::DestroyService.new(fork_project, user) }
-
-    context 'when the merge request is closed' do
-      let(:closed_merge_request) do
-        create(:closed_merge_request,
-          source_project: fork_project,
-          target_project: project)
-      end
-
-      it 'returns false if the source project exists' do
-        expect(closed_merge_request.closed_without_source_project?).to be_falsey
-      end
-
-      it 'returns true if the source project does not exist' do
-        destroy_service.execute
-        closed_merge_request.reload
-
-        expect(closed_merge_request.closed_without_source_project?).to be_truthy
-      end
-    end
-
-    context 'when the merge request is open' do
-      it 'returns false' do
-        expect(subject.closed_without_source_project?).to be_falsey
-      end
-    end
-  end
-
   describe '#reopenable?' do
     context 'when the merge request is closed' do
       it 'returns true' do

spec/models/project_services/hipchat_service_spec.rb

@@ -117,7 +117,7 @@ describe HipchatService, models: true do
     end
 
     context 'issue events' do
-      let(:issue) { create(:issue, title: 'Awesome issue', description: 'please fix') }
+      let(:issue) { create(:issue, title: 'Awesome issue', description: '**please** fix') }
       let(:issue_service) { Issues::CreateService.new(project, user) }
       let(:issues_sample_data) { issue_service.hook_data(issue, 'open') }
 
@@ -135,12 +135,12 @@ describe HipchatService, models: true do
             "<a href=\"#{obj_attr[:url]}\">issue ##{obj_attr["iid"]}</a> in " \
             "<a href=\"#{project.web_url}\">#{project_name}</a>: " \
             "<b>Awesome issue</b>" \
-            "<pre>please fix</pre>")
+            "<pre><strong>please</strong> fix</pre>")
       end
     end
 
     context 'merge request events' do
-      let(:merge_request) { create(:merge_request, description: 'please fix', title: 'Awesome merge request', target_project: project, source_project: project) }
+      let(:merge_request) { create(:merge_request, description: '**please** fix', title: 'Awesome merge request', target_project: project, source_project: project) }
       let(:merge_service) { MergeRequests::CreateService.new(project, user) }
       let(:merge_sample_data) { merge_service.hook_data(merge_request, 'open') }
 
@@ -159,7 +159,7 @@ describe HipchatService, models: true do
             "<a href=\"#{obj_attr[:url]}\">merge request !#{obj_attr["iid"]}</a> in " \
             "<a href=\"#{project.web_url}\">#{project_name}</a>: " \
             "<b>Awesome merge request</b>" \
-            "<pre>please fix</pre>")
+            "<pre><strong>please</strong> fix</pre>")
       end
     end
 
@@ -203,7 +203,7 @@ describe HipchatService, models: true do
         let(:merge_request_note) do
           create(:note_on_merge_request, noteable: merge_request,
                                          project: project,
-                                         note: "merge request note")
+                                         note: "merge request **note**")
         end
 
         it "calls Hipchat API for merge request comment events" do
@@ -222,7 +222,7 @@ describe HipchatService, models: true do
               "<a href=\"#{obj_attr[:url]}\">merge request !#{merge_id}</a> in " \
               "<a href=\"#{project.web_url}\">#{project_name}</a>: " \
               "<b>#{title}</b>" \
-              "<pre>merge request note</pre>")
+              "<pre>merge request <strong>note</strong></pre>")
         end
       end
 
@@ -230,7 +230,7 @@ describe HipchatService, models: true do
         let(:issue) { create(:issue, project: project) }
         let(:issue_note) do
           create(:note_on_issue, noteable: issue, project: project,
-                                 note: "issue note")
+                                 note: "issue **note**")
         end
 
         it "calls Hipchat API for issue comment events" do
@@ -247,7 +247,7 @@ describe HipchatService, models: true do
               "<a href=\"#{obj_attr[:url]}\">issue ##{issue_id}</a> in " \
               "<a href=\"#{project.web_url}\">#{project_name}</a>: " \
               "<b>#{title}</b>" \
-              "<pre>issue note</pre>")
+              "<pre>issue <strong>note</strong></pre>")
         end
       end
 

spec/models/project_spec.rb

@@ -67,6 +67,14 @@ describe Project, models: true do
     it { is_expected.to have_many(:notification_settings).dependent(:destroy) }
     it { is_expected.to have_many(:forks).through(:forked_project_links) }
 
+    context 'after create' do
+      it "creates project feature" do
+        project = FactoryGirl.build(:project)
+
+        expect { project.save }.to change{ project.project_feature.present? }.from(false).to(true)
+      end
+    end
+
     describe '#members & #requesters' do
       let(:project) { create(:project, :public) }
       let(:requester) { create(:user) }
@@ -531,9 +539,9 @@ describe Project, models: true do
   end
 
   describe '#has_wiki?' do
-    let(:no_wiki_project) { build(:project, wiki_enabled: false, has_external_wiki: false) }
-    let(:wiki_enabled_project) { build(:project) }
-    let(:external_wiki_project) { build(:project, has_external_wiki: true) }
+    let(:no_wiki_project)       { create(:project, wiki_access_level: ProjectFeature::DISABLED, has_external_wiki: false) }
+    let(:wiki_enabled_project)  { create(:project) }
+    let(:external_wiki_project) { create(:project, has_external_wiki: true) }
 
     it 'returns true if project is wiki enabled or has external wiki' do
       expect(wiki_enabled_project).to have_wiki

spec/requests/api/labels_spec.rb

@@ -22,8 +22,7 @@ describe API::API, api: true  do
       expect(response).to have_http_status(200)
       expect(json_response).to be_an Array
       expect(json_response.size).to eq(2)
-      expect(json_response.first['name']).to eq(group_label.name)
-      expect(json_response.second['name']).to eq(label1.name)
+      expect(json_response.map { |l| l['name'] }).to match_array([group_label.name, label1.name])
     end
   end
 

spec/workers/project_cache_worker_spec.rb

@@ -6,21 +6,39 @@ describe ProjectCacheWorker do
   subject { described_class.new }
 
   describe '#perform' do
-    it 'updates project cache data' do
-      expect_any_instance_of(Repository).to receive(:size)
-      expect_any_instance_of(Repository).to receive(:commit_count)
+    context 'when an exclusive lease can be obtained' do
+      before do
+        allow(subject).to receive(:try_obtain_lease_for).with(project.id).
+          and_return(true)
+      end
 
-      expect_any_instance_of(Project).to receive(:update_repository_size)
-      expect_any_instance_of(Project).to receive(:update_commit_count)
+      it 'updates project cache data' do
+        expect_any_instance_of(Repository).to receive(:size)
+        expect_any_instance_of(Repository).to receive(:commit_count)
 
-      subject.perform(project.id)
+        expect_any_instance_of(Project).to receive(:update_repository_size)
+        expect_any_instance_of(Project).to receive(:update_commit_count)
+
+        subject.perform(project.id)
+      end
+
+      it 'handles missing repository data' do
+        expect_any_instance_of(Repository).to receive(:exists?).and_return(false)
+        expect_any_instance_of(Repository).not_to receive(:size)
+
+        subject.perform(project.id)
+      end
     end
 
-    it 'handles missing repository data' do
-      expect_any_instance_of(Repository).to receive(:exists?).and_return(false)
-      expect_any_instance_of(Repository).not_to receive(:size)
+    context 'when an exclusive lease can not be obtained' do
+      it 'does nothing' do
+        allow(subject).to receive(:try_obtain_lease_for).with(project.id).
+          and_return(false)
+
+        expect(subject).not_to receive(:update_caches)
 
-      subject.perform(project.id)
+        subject.perform(project.id)
+      end
     end
   end
 end