Merge branch…

changelogs/unreleased/22719-provide-a-new-gitlab-workhorse-install-rake-task-similar-to-gitlab-shell-install.yml

+---
+title: New `gitlab:workhorse:install` rake task
+merge_request: 6574
+author: 

doc/install/installation.md

@@ -175,7 +175,7 @@ We recommend using a PostgreSQL database. For MySQL check the
     ```bash
     sudo -u postgres psql -d template1 -c "CREATE USER git CREATEDB;"
     ```
-    
+
 1. Create the `pg_trgm` extension (required for GitLab 8.6+):
 
     ```bash
@@ -396,15 +396,25 @@ GitLab Shell is an SSH access and repository management software developed speci
 
 ### Install gitlab-workhorse
 
-GitLab-Workhorse uses [GNU Make](https://www.gnu.org/software/make/).
-If you are not using Linux you may have to run `gmake` instead of
-`make` below.
+GitLab-Workhorse uses [GNU Make](https://www.gnu.org/software/make/). The
+following command-line will install GitLab-Workhorse in `/home/git/gitlab-workhorse`
+which is the recommended location.
 
-    cd /home/git
-    sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-workhorse.git
-    cd gitlab-workhorse
-    sudo -u git -H git checkout v1.0.1
-    sudo -u git -H make
+    cd /home/git/gitlab
+
+    sudo -u git -H bundle exec rake "gitlab:workhorse:install[/home/git/gitlab-workhorse]" RAILS_ENV=production
+
+You can specify a different Git repository by providing `GITLAB_WORKHORSE_REPO`:
+
+    cd /home/git/gitlab
+
+    sudo -u git -H bundle exec rake "gitlab:workhorse:install[/home/git/gitlab-workhorse]" GITLAB_WORKHORSE_REPO=https://example.com/gitlab-workhorse.git RAILS_ENV=production
+
+You can specify a different version to use by providing `GITLAB_WORKHORSE_VERSION`:
+
+    cd /home/git/gitlab
+
+    sudo -u git -H bundle exec rake "gitlab:workhorse:install[/home/git/gitlab-workhorse]" GITLAB_WORKHORSE_VERSION=0.8.1 RAILS_ENV=production
 
 ### Initialize Database and Activate Advanced Features
 

doc/update/8.12-to-8.13.md

@@ -166,7 +166,7 @@ See [smtp_settings.rb.sample] as an example.
 Ensure you're still up-to-date with the latest init script changes:
 
     sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab
-    
+
 For Ubuntu 16.04.1 LTS:
 
     sudo systemctl daemon-reload

doc/update/8.14-to-8.15.md

+# From 8.14 to 8.15
+
+Make sure you view this update guide from the tag (version) of GitLab you would
+like to install. In most cases this should be the highest numbered production
+tag (without rc in it). You can select the tag in the version dropdown at the
+top left corner of GitLab (below the menu bar).
+
+If the highest number stable branch is unclear please check the
+[GitLab Blog](https://about.gitlab.com/blog/archives.html) for installation
+guide links by version.
+
+### 1. Stop server
+
+    sudo service gitlab stop
+
+### 2. Backup
+
+```bash
+cd /home/git/gitlab
+sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
+```
+
+### 3. Update Ruby
+
+We will continue supporting Ruby < 2.3 for the time being but we recommend you
+upgrade to Ruby 2.3 if you're running a source installation, as this is the same
+version that ships with our Omnibus package.
+
+You can check which version you are running with `ruby -v`.
+
+Download and compile Ruby:
+
+```bash
+mkdir /tmp/ruby && cd /tmp/ruby
+curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.1.tar.gz
+echo 'c39b4001f7acb4e334cb60a0f4df72d434bef711  ruby-2.3.1.tar.gz' | shasum --check - && tar xzf ruby-2.3.1.tar.gz
+cd ruby-2.3.1
+./configure --disable-install-rdoc
+make
+sudo make install
+```
+
+Install Bundler:
+
+```bash
+sudo gem install bundler --no-ri --no-rdoc
+```
+
+### 4. Get latest code
+
+```bash
+sudo -u git -H git fetch --all
+sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically
+```
+
+For GitLab Community Edition:
+
+```bash
+sudo -u git -H git checkout 8-15-stable
+```
+
+OR
+
+For GitLab Enterprise Edition:
+
+```bash
+sudo -u git -H git checkout 8-15-stable-ee
+```
+
+### 5. Update gitlab-shell
+
+```bash
+cd /home/git/gitlab-shell
+sudo -u git -H git fetch --all --tags
+sudo -u git -H git checkout v4.0.0
+```
+
+### 6. Update gitlab-workhorse
+
+Install and compile gitlab-workhorse. This requires
+[Go 1.5](https://golang.org/dl) which should already be on your system from
+GitLab 8.1.
+
+```bash
+sudo -u git -H bundle exec rake "gitlab:workhorse:install[/home/git/gitlab-workhorse]" RAILS_ENV=production
+```
+
+### 7. Install libs, migrations, etc.
+
+```bash
+cd /home/git/gitlab
+
+# MySQL installations (note: the line below states '--without postgres')
+sudo -u git -H bundle install --without postgres development test --deployment
+
+# PostgreSQL installations (note: the line below states '--without mysql')
+sudo -u git -H bundle install --without mysql development test --deployment
+
+# Optional: clean up old gems
+sudo -u git -H bundle clean
+
+# Run database migrations
+sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production
+
+# Clean up assets and cache
+sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production
+```
+
+### 8. Update configuration files
+
+#### New configuration options for `gitlab.yml`
+
+There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example). View them with the command below and apply them manually to your current `gitlab.yml`:
+
+```sh
+git diff origin/8-13-stable:config/gitlab.yml.example origin/8-15-stable:config/gitlab.yml.example
+```
+
+#### Git configuration
+
+Configure Git to generate packfile bitmaps (introduced in Git 2.0) on
+the GitLab server during `git gc`.
+
+```sh
+sudo -u git -H git config --global repack.writeBitmaps true
+```
+
+#### Nginx configuration
+
+Ensure you're still up-to-date with the latest NGINX configuration changes:
+
+```sh
+# For HTTPS configurations
+git diff origin/8-13-stable:lib/support/nginx/gitlab-ssl origin/8-15-stable:lib/support/nginx/gitlab-ssl
+
+# For HTTP configurations
+git diff origin/8-13-stable:lib/support/nginx/gitlab origin/8-15-stable:lib/support/nginx/gitlab
+```
+
+If you are using Apache instead of NGINX please see the updated [Apache templates].
+Also note that because Apache does not support upstreams behind Unix sockets you
+will need to let gitlab-workhorse listen on a TCP port. You can do this
+via [/etc/default/gitlab].
+
+[Apache templates]: https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache
+[/etc/default/gitlab]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-15-stable/lib/support/init.d/gitlab.default.example#L38
+
+#### SMTP configuration
+
+If you're installing from source and use SMTP to deliver mail, you will need to add the following line
+to config/initializers/smtp_settings.rb:
+
+```ruby
+ActionMailer::Base.delivery_method = :smtp
+```
+
+See [smtp_settings.rb.sample] as an example.
+
+[smtp_settings.rb.sample]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-15-stable/config/initializers/smtp_settings.rb.sample#L13
+
+#### Init script
+
+Ensure you're still up-to-date with the latest init script changes:
+
+    sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab
+
+For Ubuntu 16.04.1 LTS:
+
+    sudo systemctl daemon-reload
+
+### 9. Start application
+
+    sudo service gitlab start
+    sudo service nginx restart
+
+### 10. Check application status
+
+Check if GitLab and its environment are configured correctly:
+
+    sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
+
+To make sure you didn't miss anything run a more thorough check:
+
+    sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production
+
+If all items are green, then congratulations, the upgrade is complete!
+
+## Things went south? Revert to previous version (8.14)
+
+### 1. Revert the code to the previous version
+
+Follow the [upgrade guide from 8.13 to 8.14](8.13-to-8.14.md), except for the
+database migration (the backup is already migrated to the previous version).
+
+### 2. Restore from the backup
+
+```bash
+cd /home/git/gitlab
+sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production
+```
+
+If you have more than one backup `*.tar` file(s) please add `BACKUP=timestamp_of_backup` to the command above.

lib/tasks/gitlab/helpers.rake

+require 'tasks/gitlab/task_helpers'
+
+# Prevent StateMachine warnings from outputting during a cron task
+StateMachines::Machine.ignore_method_conflicts = true if ENV['CRON']
+
+namespace :gitlab do
+  include Gitlab::TaskHelpers
+end

lib/tasks/gitlab/shell.rake

@@ -5,42 +5,23 @@ namespace :gitlab do
       warn_user_is_not_gitlab
 
       default_version = Gitlab::Shell.version_required
-      default_version_tag = 'v' + default_version
-      args.with_defaults(tag: default_version_tag, repo: "https://gitlab.com/gitlab-org/gitlab-shell.git")
+      default_version_tag = "v#{default_version}"
+      args.with_defaults(tag: default_version_tag, repo: 'https://gitlab.com/gitlab-org/gitlab-shell.git')
 
-      user = Gitlab.config.gitlab.user
-      home_dir = Rails.env.test? ? Rails.root.join('tmp/tests') : Gitlab.config.gitlab.user_home
       gitlab_url = Gitlab.config.gitlab.url
       # gitlab-shell requires a / at the end of the url
       gitlab_url += '/' unless gitlab_url.end_with?('/')
       target_dir = Gitlab.config.gitlab_shell.path
 
-      # Clone if needed
-      if File.directory?(target_dir)
-        Dir.chdir(target_dir) do
-          system(*%W(Gitlab.config.git.bin_path} fetch --tags --quiet))
-          system(*%W(Gitlab.config.git.bin_path} checkout --quiet #{default_version_tag}))
-        end
-      else
-        system(*%W(#{Gitlab.config.git.bin_path} clone -- #{args.repo} #{target_dir}))
-      end
+      checkout_or_clone_tag(tag: default_version_tag, repo: args.repo, target_dir: target_dir)
 
       # Make sure we're on the right tag
       Dir.chdir(target_dir) do
-        # First try to checkout without fetching
-        # to avoid stalling tests if the Internet is down.
-        reseted = reset_to_commit(args)
-
-        unless reseted
-          system(*%W(#{Gitlab.config.git.bin_path} fetch origin))
-          reset_to_commit(args)
-        end
-
         config = {
-          user: user,
+          user: Gitlab.config.gitlab.user,
           gitlab_url: gitlab_url,
           http_settings: {self_signed_cert: false}.stringify_keys,
-          auth_file: File.join(home_dir, ".ssh", "authorized_keys"),
+          auth_file: File.join(user_home, ".ssh", "authorized_keys"),
           redis: {
             bin: %x{which redis-cli}.chomp,
             namespace: "resque:gitlab"
@@ -74,7 +55,7 @@ namespace :gitlab do
       # be an issue since it is more than likely that there are no "normal"
       # user accounts on a gitlab server). The alternative is for the admin to
       # install a ruby (1.9.3+) in the global path.
-      File.open(File.join(home_dir, ".ssh", "environment"), "w+") do |f|
+      File.open(File.join(user_home, ".ssh", "environment"), "w+") do |f|
         f.puts "PATH=#{ENV['PATH']}"
       end
 
@@ -142,15 +123,4 @@ namespace :gitlab do
     puts "Quitting...".color(:red)
     exit 1
   end
-
-  def reset_to_commit(args)
-    tag, status = Gitlab::Popen.popen(%W(#{Gitlab.config.git.bin_path} describe -- #{args.tag}))
-
-    unless status.zero?
-      tag, status = Gitlab::Popen.popen(%W(#{Gitlab.config.git.bin_path} describe -- origin/#{args.tag}))
-    end
-
-    tag = tag.strip
-    system(*%W(#{Gitlab.config.git.bin_path} reset --hard #{tag}))
-  end
 end

lib/tasks/gitlab/workhorse.rake

+namespace :gitlab do
+  namespace :workhorse do
+    desc "GitLab | Install or upgrade gitlab-workhorse"
+    task :install, [:dir] => :environment do |t, args|
+      warn_user_is_not_gitlab
+      unless args.dir.present?
+        abort %(Please specify the directory where you want to install gitlab-workhorse:\n  rake "gitlab:workhorse:install[/home/git/gitlab-workhorse]")
+      end
+
+      tag = "v#{ENV['GITLAB_WORKHORSE_VERSION'] || Gitlab::Workhorse.version}"
+      repo = ENV['GITLAB_WORKHORSE_REPO'] || 'https://gitlab.com/gitlab-org/gitlab-workhorse.git'
+
+      checkout_or_clone_tag(tag: tag, repo: repo, target_dir: args.dir)
+
+      _, status = Gitlab::Popen.popen(%w[which gmake])
+      command = status.zero? ? 'gmake' : 'make'
+
+      Dir.chdir(args.dir) do
+        run_command!([command])
+      end
+    end
+  end
+end

spec/rake_helper.rb

@@ -8,7 +8,7 @@ RSpec.configure do |config|
   config.before(:all) do
     $stdout = StringIO.new
 
-    Rake.application.rake_require 'tasks/gitlab/task_helpers'
+    Rake.application.rake_require 'tasks/gitlab/helpers'
     Rake::Task.define_task :environment
   end
 

spec/support/rake_helpers.rb

 module RakeHelpers
-  def run_rake_task(task_name)
+  def run_rake_task(task_name, *args)
     Rake::Task[task_name].reenable
-    Rake.application.invoke_task task_name
+    Rake.application.invoke_task("#{task_name}[#{args.join(',')}]")
   end
 
   def stub_warn_user_is_not_gitlab

spec/tasks/gitlab/backup_rake_spec.rb

@@ -5,7 +5,7 @@ describe 'gitlab:app namespace rake task' do
   let(:enable_registry) { true }
 
   before :all do
-    Rake.application.rake_require 'tasks/gitlab/task_helpers'
+    Rake.application.rake_require 'tasks/gitlab/helpers'
     Rake.application.rake_require 'tasks/gitlab/backup'
     Rake.application.rake_require 'tasks/gitlab/shell'
     Rake.application.rake_require 'tasks/gitlab/db'

spec/tasks/gitlab/mail_google_schema_whitelisting.rb

@@ -3,7 +3,7 @@ require 'rake'
 
 describe 'gitlab:mail_google_schema_whitelisting rake task' do
   before :all do
-    Rake.application.rake_require "tasks/gitlab/task_helpers"
+    Rake.application.rake_require "tasks/gitlab/helpers"
     Rake.application.rake_require "tasks/gitlab/mail_google_schema_whitelisting"
     # empty task as env is already loaded
     Rake::Task.define_task :environment

spec/tasks/gitlab/task_helpers_spec.rb

+require 'spec_helper'
+require 'tasks/gitlab/task_helpers'
+
+class TestHelpersTest
+  include Gitlab::TaskHelpers
+end
+
+describe Gitlab::TaskHelpers do
+  subject { TestHelpersTest.new }
+
+  let(:repo) { 'https://gitlab.com/gitlab-org/gitlab-test.git' }
+  let(:clone_path) { Rails.root.join('tmp/tests/task_helpers_tests').to_s }
+  let(:tag) { 'v1.1.0' }
+
+  describe '#checkout_or_clone_tag' do
+    before do
+      allow(subject).to receive(:run_command!)
+      expect(subject).to receive(:reset_to_tag).with(tag, clone_path)
+    end
+
+    context 'target_dir does not exist' do
+      it 'clones the repo, retrieve the tag from origin, and checkout the tag' do
+        expect(subject).to receive(:clone_repo).with(repo, clone_path)
+
+        subject.checkout_or_clone_tag(tag: tag, repo: repo, target_dir: clone_path)
+      end
+    end
+
+    context 'target_dir exists' do
+      before do
+        expect(Dir).to receive(:exist?).and_return(true)
+      end
+
+      it 'fetch and checkout the tag' do
+        expect(subject).to receive(:checkout_tag).with(tag, clone_path)
+
+        subject.checkout_or_clone_tag(tag: tag, repo: repo, target_dir: clone_path)
+      end
+    end
+  end
+
+  describe '#clone_repo' do
+    it 'clones the repo in the target dir' do
+      expect(subject).
+        to receive(:run_command!).with(%W[#{Gitlab.config.git.bin_path} clone -- #{repo} #{clone_path}])
+
+      subject.clone_repo(repo, clone_path)
+    end
+  end
+
+  describe '#checkout_tag' do
+    it 'clones the repo in the target dir' do
+      expect(subject).
+        to receive(:run_command!).with(%W[#{Gitlab.config.git.bin_path} -C #{clone_path} fetch --tags --quiet])
+      expect(subject).
+        to receive(:run_command!).with(%W[#{Gitlab.config.git.bin_path} -C #{clone_path} checkout --quiet #{tag}])
+
+      subject.checkout_tag(tag, clone_path)
+    end
+  end
+
+  describe '#reset_to_tag' do
+    let(:tag) { 'v1.1.0' }
+    before do
+      expect(subject).
+        to receive(:run_command!).with(%W[#{Gitlab.config.git.bin_path} -C #{clone_path} reset --hard #{tag}])
+    end
+
+    context 'when the tag is not checked out locally' do
+      before do
+        expect(subject).
+          to receive(:run_command!).with(%W[#{Gitlab.config.git.bin_path} -C #{clone_path} describe -- #{tag}]).and_raise(Gitlab::TaskFailedError)
+      end
+
+      it 'fetch origin, ensure the tag exists, and resets --hard to the given tag' do
+        expect(subject).
+          to receive(:run_command!).with(%W[#{Gitlab.config.git.bin_path} -C #{clone_path} fetch origin])
+        expect(subject).
+          to receive(:run_command!).with(%W[#{Gitlab.config.git.bin_path} -C #{clone_path} describe -- origin/#{tag}]).and_return(tag)
+
+        subject.reset_to_tag(tag, clone_path)
+      end
+    end
+
+    context 'when the tag is checked out locally' do
+      before do
+        expect(subject).
+          to receive(:run_command!).with(%W[#{Gitlab.config.git.bin_path} -C #{clone_path} describe -- #{tag}]).and_return(tag)
+      end
+
+      it 'resets --hard to the given tag' do
+        subject.reset_to_tag(tag, clone_path)
+      end
+    end
+  end
+end

spec/tasks/gitlab/users_rake_spec.rb

@@ -5,7 +5,7 @@ describe 'gitlab:users namespace rake task' do
   let(:enable_registry) { true }
 
   before :all do
-    Rake.application.rake_require 'tasks/gitlab/task_helpers'
+    Rake.application.rake_require 'tasks/gitlab/helpers'
     Rake.application.rake_require 'tasks/gitlab/users'
 
     # empty task as env is already loaded

spec/tasks/gitlab/workhorse_rake_spec.rb

+require 'rake_helper'
+
+describe 'gitlab:workhorse namespace rake task' do
+  before :all do
+    Rake.application.rake_require 'tasks/gitlab/workhorse'
+  end
+
+  describe 'install' do
+    let(:repo) { 'https://gitlab.com/gitlab-org/gitlab-workhorse.git' }
+    let(:clone_path) { Rails.root.join('tmp/tests/gitlab-workhorse').to_s }
+    let(:tag) { "v#{File.read(Rails.root.join(Gitlab::Workhorse::VERSION_FILE)).chomp}" }
+    before do
+      allow(ENV).to receive(:[])
+    end
+
+    context 'no dir given' do
+      it 'aborts and display a help message' do
+        # avoid writing task output to spec progress
+        allow($stderr).to receive :write
+        expect { run_rake_task('gitlab:workhorse:install') }.to raise_error /Please specify the directory where you want to install gitlab-workhorse/
+      end
+    end
+
+    context 'when an underlying Git command fail' do
+      it 'aborts and display a help message' do
+        expect_any_instance_of(Object).
+          to receive(:checkout_or_clone_tag).and_raise 'Git error'
+
+        expect { run_rake_task('gitlab:workhorse:install', clone_path) }.to raise_error 'Git error'
+      end
+    end
+
+    describe 'checkout or clone' do
+      before do
+        expect(Dir).to receive(:chdir).with(clone_path)
+      end
+
+      it 'calls checkout_or_clone_tag with the right arguments' do
+        expect_any_instance_of(Object).
+          to receive(:checkout_or_clone_tag).with(tag: tag, repo: repo, target_dir: clone_path)
+
+        run_rake_task('gitlab:workhorse:install', clone_path)
+      end
+
+      context 'given a specific repo' do
+        before do
+          expect(ENV).to receive(:[]).with('GITLAB_WORKHORSE_REPO').and_return('https://gitlab.com/user1/gitlab-workhorse.git')
+        end
+
+        it 'calls checkout_or_clone_tag with the given repo' do
+          expect_any_instance_of(Object).
+            to receive(:checkout_or_clone_tag).with(tag: tag, repo: 'https://gitlab.com/user1/gitlab-workhorse.git', target_dir: clone_path)
+
+          run_rake_task('gitlab:workhorse:install', clone_path)
+        end
+      end
+
+      context 'given a specific version' do
+        before do
+          allow(ENV).to receive(:[]).with('GITLAB_WORKHORSE_VERSION').and_return('42.42.0')
+        end
+
+        it 'calls checkout_or_clone_tag with the given repo' do
+          expect_any_instance_of(Object).
+            to receive(:checkout_or_clone_tag).with(tag: 'v42.42.0', repo: repo, target_dir: clone_path)
+
+          run_rake_task('gitlab:workhorse:install', clone_path)
+        end
+      end
+    end
+
+    describe 'gmake/make' do
+      before do
+        FileUtils.mkdir_p(clone_path)
+        expect(Dir).to receive(:chdir).with(clone_path).and_call_original
+      end
+
+      context 'gmake is available' do
+        before do
+          expect_any_instance_of(Object).to receive(:checkout_or_clone_tag)
+          allow_any_instance_of(Object).to receive(:run_command!).with(['gmake']).and_return(true)
+        end
+
+        it 'calls gmake in the gitlab-workhorse directory' do
+          expect(Gitlab::Popen).to receive(:popen).with(%w[which gmake]).and_return(['/usr/bin/gmake', 0])
+          expect_any_instance_of(Object).to receive(:run_command!).with(['gmake']).and_return(true)
+
+          run_rake_task('gitlab:workhorse:install', clone_path)
+        end
+      end
+
+      context 'gmake is not available' do
+        before do
+          expect_any_instance_of(Object).to receive(:checkout_or_clone_tag)
+          allow_any_instance_of(Object).to receive(:run_command!).with(['make']).and_return(true)
+        end
+
+        it 'calls make in the gitlab-workhorse directory' do
+          expect(Gitlab::Popen).to receive(:popen).with(%w[which gmake]).and_return(['', 42])
+          expect_any_instance_of(Object).to receive(:run_command!).with(['make']).and_return(true)
+
+          run_rake_task('gitlab:workhorse:install', clone_path)
+        end
+      end
+    end
+  end
+end