Prepare for zero downtime migrations

doc/development/migration_style_guide.md

@@ -4,28 +4,53 @@ When writing migrations for GitLab, you have to take into account that
 these will be ran by hundreds of thousands of organizations of all sizes, some with
 many years of data in their database.
 
-In addition, having to take a server offline for a an upgrade small or big is
-a big burden for most organizations. For this reason it is important that your
-migrations are written carefully, can be applied online and adhere to the style guide below.
+In addition, having to take a server offline for a a upgrade small or big is a
+big burden for most organizations. For this reason it is important that your
+migrations are written carefully, can be applied online and adhere to the style
+guide below.
 
-Migrations should not require GitLab installations to be taken offline unless
-_absolutely_ necessary - see the ["What Requires Downtime?"](what_requires_downtime.md)
-page. If a migration requires downtime, this should be clearly mentioned during
-the review process, as well as being documented in the monthly release post. For
-more information, see the "Downtime Tagging" section below.
+Migrations are **not** allowed to require GitLab installations to be taken
+offline unless _absolutely necessary_. Downtime assumptions should be based on
+the behaviour of a migration when performed using PostgreSQL, as various
+operations in MySQL may require downtime without there being alternatives.
+
+When downtime is necessary the migration has to be approved by:
+
+1. The VP of Engineering
+1. A Backend Lead
+1. A Database Specialist
+
+An up-to-date list of people holding these titles can be found at
+<https://about.gitlab.com/team/>.
+
+The document ["What Requires Downtime?"](what_requires_downtime.md) specifies
+various database operations, whether they require downtime and how to
+work around that whenever possible.
 
 When writing your migrations, also consider that databases might have stale data
-or inconsistencies and guard for that. Try to make as little assumptions as possible
-about the state of the database.
+or inconsistencies and guard for that. Try to make as few assumptions as
+possible about the state of the database.
+
+Please don't depend on GitLab-specific code since it can change in future
+versions. If needed copy-paste GitLab code into the migration to make it forward
+compatible.
+
+## Commit Guidelines
 
-Please don't depend on GitLab specific code since it can change in future versions.
-If needed copy-paste GitLab code into the migration to make it forward compatible.
+Each migration **must** be added in its own commit with a descriptive commit
+message. If a commit adds a migration it _should only_ include the migration and
+any corresponding changes to `db/schema.rb`. This makes it easy to revert a
+database migration without accidentally reverting other changes.
 
 ## Downtime Tagging
 
 Every migration must specify if it requires downtime or not, and if it should
-require downtime it must also specify a reason for this. To do so, add the
-following two constants to the migration class' body:
+require downtime it must also specify a reason for this. This is required even
+if 99% of the migrations won't require downtime as this makes it easier to find
+the migrations that _do_ require downtime.
+
+To tag a migration, add the following two constants to the migration class'
+body:
 
 * `DOWNTIME`: a boolean that when set to `true` indicates the migration requires
   downtime.
@@ -50,12 +75,53 @@ from a migration class.
 
 ## Reversibility
 
-Your migration should be reversible. This is very important, as it should
+Your migration **must be** reversible. This is very important, as it should
 be possible to downgrade in case of a vulnerability or bugs.
 
 In your migration, add a comment describing how the reversibility of the
 migration was tested.
 
+## Multi Threading
+
+Sometimes a migration might need to use multiple Ruby threads to speed up a
+migration. For this to work your migration needs to include the module
+`Gitlab::Database::MultiThreadedMigration`:
+
+```ruby
+class MyMigration < ActiveRecord::Migration
+  include Gitlab::Database::MigrationHelpers
+  include Gitlab::Database::MultiThreadedMigration
+end
+```
+
+You can then use the method `with_multiple_threads` to perform work in separate
+threads. For example:
+
+```ruby
+class MyMigration < ActiveRecord::Migration
+  include Gitlab::Database::MigrationHelpers
+  include Gitlab::Database::MultiThreadedMigration
+
+  def up
+    with_multiple_threads(4) do
+      disable_statement_timeout
+
+      # ...
+    end
+  end
+end
+```
+
+Here the call to `disable_statement_timeout` will use the connection local to
+the `with_multiple_threads` block, instead of re-using the global connection
+pool.  This ensures each thread has its own connection object, and won't time
+out when trying to obtain one.
+
+**NOTE:** PostgreSQL has a maximum amount of connections that it allows. This
+limit can vary from installation to installation. As a result it's recommended
+you do not use more than 32 threads in a single migration. Usually 4-8 threads
+should be more than enough.
+
 ## Removing indices
 
 When removing an index make sure to use the method `remove_concurrent_index` instead
@@ -78,7 +144,10 @@ end
 
 ## Adding indices
 
-If you need to add an unique index please keep in mind there is possibility of existing duplicates. If it is possible write a separate migration for handling this situation. It can be just removing or removing with overwriting all references to these duplicates depend on situation.
+If you need to add a unique index please keep in mind there is the possibility
+of existing duplicates being present in the database. This means that should
+always _first_ add a migration that removes any duplicates, before adding the
+unique index.
 
 When adding an index make sure to use the method `add_concurrent_index` instead
 of the regular `add_index` method. The `add_concurrent_index` method
@@ -90,17 +159,22 @@ so:
 ```ruby
 class MyMigration < ActiveRecord::Migration
   include Gitlab::Database::MigrationHelpers
+
   disable_ddl_transaction!
 
-  def change
+  def up
+    add_concurrent_index :table, :column
+  end
 
+  def down
+    remove_index :table, :column if index_exists?(:table, :column)
   end
 end
 ```
 
 ## Adding Columns With Default Values
 
-When adding columns with default values you should use the method
+When adding columns with default values you must use the method
 `add_column_with_default`. This method ensures the table is updated without
 requiring downtime. This method is not reversible so you must manually define
 the `up` and `down` methods in your migration class.
@@ -123,6 +197,9 @@ class MyMigration < ActiveRecord::Migration
 end
 ```
 
+Keep in mind that this operation can easily take 10-15 minutes to complete on
+larger installations (e.g. GitLab.com). As a result you should only add default
+values if absolutely necessary.
 
 ## Integer column type
 
@@ -147,13 +224,15 @@ add_column(:projects, :foo, :integer, default: 10, limit: 8)
 
 ## Testing
 
-Make sure that your migration works with MySQL and PostgreSQL with data. An empty database does not guarantee that your migration is correct.
+Make sure that your migration works with MySQL and PostgreSQL with data. An
+empty database does not guarantee that your migration is correct.
 
 Make sure your migration can be reversed.
 
 ## Data migration
 
-Please prefer Arel and plain SQL over usual ActiveRecord syntax. In case of using plain SQL you need to quote all input manually with `quote_string` helper.
+Please prefer Arel and plain SQL over usual ActiveRecord syntax. In case of
+using plain SQL you need to quote all input manually with `quote_string` helper.
 
 Example with Arel:
 
@@ -177,3 +256,17 @@ select_all("SELECT name, COUNT(id) as cnt FROM tags GROUP BY name HAVING COUNT(i
   execute("DELETE FROM tags WHERE id IN(#{duplicate_ids.join(",")})")
 end
 ```
+
+If you need more complex logic you can define and use models local to a
+migration. For example:
+
+```ruby
+class MyMigration < ActiveRecord::Migration
+  class Project < ActiveRecord::Base
+    self.table_name = 'projects'
+  end
+end
+```
+
+When doing so be sure to explicitly set the model's table name so it's not
+derived from the class name or namespace.

doc/update/README.md

@@ -48,6 +48,23 @@ GitLab provides official Docker images for both Community and Enterprise
 editions. They are based on the Omnibus package and instructions on how to
 update them are in [a separate document][omnidocker].
 
+## Upgrading without downtime
+
+Starting with GitLab 9.1.0 it's possible to upgrade to a newer version of GitLab
+without having to take your GitLab instance offline. However, for this to work
+there are the following requirements:
+
+1. You can only upgrade 1 release at a time. For example, if 9.1.15 is the last
+   release of 9.1 then you can safely upgrade from that version to 9.2.0.
+   However, if you are running 9.1.14 you first need to upgrade to 9.1.15.
+2. You have to use [post-deployment
+   migrations](../development/post_deployment_migrations.md).
+3. You are using PostgreSQL. If you are using MySQL you will still need downtime
+   when upgrading.
+
+This applies to major, minor, and patch releases unless stated otherwise in a
+release post.
+
 ## Upgrading between editions
 
 GitLab comes in two flavors: [Community Edition][ce] which is MIT licensed,

lib/gitlab/database/migration_helpers.rb

@@ -89,7 +89,8 @@ module Gitlab
         ADD CONSTRAINT #{key_name}
         FOREIGN KEY (#{column})
         REFERENCES #{target} (id)
-        ON DELETE #{on_delete} NOT VALID;
+        #{on_delete ? "ON DELETE #{on_delete}" : ''}
+        NOT VALID;
         EOF
 
         # Validate the existing constraint. This can potentially take a very
@@ -250,6 +251,245 @@ module Gitlab
           raise error
         end
       end
+
+      # Renames a column without requiring downtime.
+      #
+      # Concurrent renames work by using database triggers to ensure both the
+      # old and new column are in sync. However, this method will _not_ remove
+      # the triggers or the old column automatically; this needs to be done
+      # manually in a post-deployment migration. This can be done using the
+      # method `cleanup_concurrent_column_rename`.
+      #
+      # table - The name of the database table containing the column.
+      # old - The old column name.
+      # new - The new column name.
+      # type - The type of the new column. If no type is given the old column's
+      #        type is used.
+      def rename_column_concurrently(table, old, new, type: nil)
+        if transaction_open?
+          raise 'rename_column_concurrently can not be run inside a transaction'
+        end
+
+        trigger_name = rename_trigger_name(table, old, new)
+        quoted_table = quote_table_name(table)
+        quoted_old = quote_column_name(old)
+        quoted_new = quote_column_name(new)
+
+        if Database.postgresql?
+          install_rename_triggers_for_postgresql(trigger_name, quoted_table,
+                                                 quoted_old, quoted_new)
+        else
+          install_rename_triggers_for_mysql(trigger_name, quoted_table,
+                                            quoted_old, quoted_new)
+        end
+
+        old_col = column_for(table, old)
+        new_type = type || old_col.type
+
+        add_column(table, new, new_type,
+                   limit: old_col.limit,
+                   default: old_col.default,
+                   null: old_col.null,
+                   precision: old_col.precision,
+                   scale: old_col.scale)
+
+        update_column_in_batches(table, new, Arel::Table.new(table)[old])
+
+        copy_indexes(table, old, new)
+        copy_foreign_keys(table, old, new)
+      end
+
+      # Changes the type of a column concurrently.
+      #
+      # table - The table containing the column.
+      # column - The name of the column to change.
+      # new_type - The new column type.
+      def change_column_type_concurrently(table, column, new_type)
+        temp_column = "#{column}_for_type_change"
+
+        rename_column_concurrently(table, column, temp_column, type: new_type)
+      end
+
+      # Performs cleanup of a concurrent type change.
+      #
+      # table - The table containing the column.
+      # column - The name of the column to change.
+      # new_type - The new column type.
+      def cleanup_concurrent_column_type_change(table, column)
+        temp_column = "#{column}_for_type_change"
+
+        transaction do
+          # This has to be performed in a transaction as otherwise we might have
+          # inconsistent data.
+          cleanup_concurrent_column_rename(table, column, temp_column)
+          rename_column(table, temp_column, column)
+        end
+      end
+
+      # Cleans up a concurrent column name.
+      #
+      # This method takes care of removing previously installed triggers as well
+      # as removing the old column.
+      #
+      # table - The name of the database table.
+      # old - The name of the old column.
+      # new - The name of the new column.
+      def cleanup_concurrent_column_rename(table, old, new)
+        trigger_name = rename_trigger_name(table, old, new)
+
+        if Database.postgresql?
+          remove_rename_triggers_for_postgresql(table, trigger_name)
+        else
+          remove_rename_triggers_for_mysql(trigger_name)
+        end
+
+        remove_column(table, old)
+      end
+
+      # Performs a concurrent column rename when using PostgreSQL.
+      def install_rename_triggers_for_postgresql(trigger, table, old, new)
+        execute <<-EOF.strip_heredoc
+        CREATE OR REPLACE FUNCTION #{trigger}()
+        RETURNS trigger AS
+        $BODY$
+        BEGIN
+          NEW.#{new} := NEW.#{old};
+          RETURN NEW;
+        END;
+        $BODY$
+        LANGUAGE 'plpgsql'
+        VOLATILE
+        EOF
+
+        execute <<-EOF.strip_heredoc
+        CREATE TRIGGER #{trigger}
+        BEFORE INSERT OR UPDATE
+        ON #{table}
+        FOR EACH ROW
+        EXECUTE PROCEDURE #{trigger}()
+        EOF
+      end
+
+      # Installs the triggers necessary to perform a concurrent column rename on
+      # MySQL.
+      def install_rename_triggers_for_mysql(trigger, table, old, new)
+        execute <<-EOF.strip_heredoc
+        CREATE TRIGGER #{trigger}_insert
+        BEFORE INSERT
+        ON #{table}
+        FOR EACH ROW
+        SET NEW.#{new} = NEW.#{old}
+        EOF
+
+        execute <<-EOF.strip_heredoc
+        CREATE TRIGGER #{trigger}_update
+        BEFORE UPDATE
+        ON #{table}
+        FOR EACH ROW
+        SET NEW.#{new} = NEW.#{old}
+        EOF
+      end
+
+      # Removes the triggers used for renaming a PostgreSQL column concurrently.
+      def remove_rename_triggers_for_postgresql(table, trigger)
+        execute("DROP TRIGGER #{trigger} ON #{table}")
+        execute("DROP FUNCTION #{trigger}()")
+      end
+
+      # Removes the triggers used for renaming a MySQL column concurrently.
+      def remove_rename_triggers_for_mysql(trigger)
+        execute("DROP TRIGGER #{trigger}_insert")
+        execute("DROP TRIGGER #{trigger}_update")
+      end
+
+      # Returns the (base) name to use for triggers when renaming columns.
+      def rename_trigger_name(table, old, new)
+        'trigger_' + Digest::SHA256.hexdigest("#{table}_#{old}_#{new}").first(12)
+      end
+
+      # Returns an Array containing the indexes for the given column
+      def indexes_for(table, column)
+        column = column.to_s
+
+        indexes(table).select { |index| index.columns.include?(column) }
+      end
+
+      # Returns an Array containing the foreign keys for the given column.
+      def foreign_keys_for(table, column)
+        column = column.to_s
+
+        foreign_keys(table).select { |fk| fk.column == column }
+      end
+
+      # Copies all indexes for the old column to a new column.
+      #
+      # table - The table containing the columns and indexes.
+      # old - The old column.
+      # new - The new column.
+      def copy_indexes(table, old, new)
+        old = old.to_s
+        new = new.to_s
+
+        indexes_for(table, old).each do |index|
+          new_columns = index.columns.map do |column|
+            column == old ? new : column
+          end
+
+          # This is necessary as we can't properly rename indexes such as
+          # "ci_taggings_idx".
+          unless index.name.include?(old)
+            raise "The index #{index.name} can not be copied as it does not "\
+              "mention the old column. You have to rename this index manually first."
+          end
+
+          name = index.name.gsub(old, new)
+
+          options = {
+            unique: index.unique,
+            name: name,
+            length: index.lengths,
+            order: index.orders
+          }
+
+          # These options are not supported by MySQL, so we only add them if
+          # they were previously set.
+          options[:using] = index.using if index.using
+          options[:where] = index.where if index.where
+
+          unless index.opclasses.blank?
+            opclasses = index.opclasses.dup
+
+            # Copy the operator classes for the old column (if any) to the new
+            # column.
+            opclasses[new] = opclasses.delete(old) if opclasses[old]
+
+            options[:opclasses] = opclasses
+          end
+
+          add_concurrent_index(table, new_columns, options)
+        end
+      end
+
+      # Copies all foreign keys for the old column to the new column.
+      #
+      # table - The table containing the columns and indexes.
+      # old - The old column.
+      # new - The new column.
+      def copy_foreign_keys(table, old, new)
+        foreign_keys_for(table, old).each do |fk|
+          add_concurrent_foreign_key(fk.from_table,
+                                     fk.to_table,
+                                     column: new,
+                                     on_delete: fk.on_delete)
+        end
+      end
+
+      # Returns the column for the given table and column name.
+      def column_for(table, name)
+        name = name.to_s
+
+        columns(table).find { |column| column.name == name }
+      end
     end
   end
 end

lib/gitlab/database/multi_threaded_migration.rb

+module Gitlab
+  module Database
+    module MultiThreadedMigration
+      MULTI_THREAD_AR_CONNECTION = :thread_local_ar_connection
+
+      # This overwrites the default connection method so that every thread can
+      # use a thread-local connection, while still supporting all of Rails'
+      # migration methods.
+      def connection
+        Thread.current[MULTI_THREAD_AR_CONNECTION] ||
+          ActiveRecord::Base.connection
+      end
+
+      # Starts a thread-pool for N threads, along with N threads each using a
+      # single connection. The provided block is yielded from inside each
+      # thread.
+      #
+      # Example:
+      #
+      #     with_multiple_threads(4) do
+      #       execute('SELECT ...')
+      #     end
+      #
+      # thread_count - The number of threads to start.
+      #
+      # join - When set to true this method will join the threads, blocking the
+      #        caller until all threads have finished running.
+      #
+      # Returns an Array containing the started threads.
+      def with_multiple_threads(thread_count, join: true)
+        pool = Gitlab::Database.create_connection_pool(thread_count)
+
+        threads = Array.new(thread_count) do
+          Thread.new do
+            pool.with_connection do |connection|
+              begin
+                Thread.current[MULTI_THREAD_AR_CONNECTION] = connection
+                yield
+              ensure
+                Thread.current[MULTI_THREAD_AR_CONNECTION] = nil
+              end
+            end
+          end
+        end
+
+        threads.each(&:join) if join
+
+        threads
+      end
+    end
+  end
+end

spec/lib/gitlab/database/multi_threaded_migration_spec.rb

+require 'spec_helper'
+
+describe Gitlab::Database::MultiThreadedMigration do
+  let(:migration) do
+    Class.new { include Gitlab::Database::MultiThreadedMigration }.new
+  end
+
+  describe '#connection' do
+    after do
+      Thread.current[described_class::MULTI_THREAD_AR_CONNECTION] = nil
+    end
+
+    it 'returns the thread-local connection if present' do
+      Thread.current[described_class::MULTI_THREAD_AR_CONNECTION] = 10
+
+      expect(migration.connection).to eq(10)
+    end
+
+    it 'returns the global connection if no thread-local connection was set' do
+      expect(migration.connection).to eq(ActiveRecord::Base.connection)
+    end
+  end
+
+  describe '#with_multiple_threads' do
+    it 'starts multiple threads and yields the supplied block in every thread' do
+      output = Queue.new
+
+      migration.with_multiple_threads(2) do
+        output << migration.connection.execute('SELECT 1')
+      end
+
+      expect(output.size).to eq(2)
+    end
+
+    it 'joins the threads when the join parameter is set' do
+      expect_any_instance_of(Thread).to receive(:join).and_call_original
+
+      migration.with_multiple_threads(1) { }
+    end
+  end
+end

spec/models/concerns/ignorable_column_spec.rb

+require 'spec_helper'
+
+describe IgnorableColumn do
+  let :base_class do
+    Class.new do
+      def self.columns
+        # This method does not have access to "double"
+        [Struct.new(:name).new('id'), Struct.new(:name).new('title')]
+      end
+    end
+  end
+
+  let :model do
+    Class.new(base_class) do
+      include IgnorableColumn
+    end
+  end
+
+  describe '.columns' do
+    it 'returns the columns, excluding the ignored ones' do
+      model.ignore_column(:title)
+
+      expect(model.columns.map(&:name)).to eq(%w(id))
+    end
+  end
+
+  describe '.ignored_columns' do
+    it 'returns a Set' do
+      expect(model.ignored_columns).to be_an_instance_of(Set)
+    end
+
+    it 'returns the names of the ignored columns' do
+      model.ignore_column(:title)
+
+      expect(model.ignored_columns).to eq(Set.new(%w(title)))
+    end
+  end
+end