Add documentation for running Gitaly on another server

changelogs/unreleased/doc-gitaly-network.yml

+---
+title: Add option to run Gitaly on a remote server
+merge_request: 12381
+author:

doc/administration/gitaly/index.md

@@ -33,6 +33,144 @@ prometheus_listen_addr = "localhost:9236"
 Changes to `/home/git/gitaly/config.toml` are applied when you run `service
 gitlab restart`.
 
+## Running Gitaly on its own server
+
+> This is an optional way to deploy Gitaly which can benefit GitLab
+installations that are larger than a single machine. Most
+installations will be better served with the default configuration
+used by Omnibus and the GitLab source installation guide.
+
+Starting with GitLab 9.4 it is possible to run Gitaly on a different
+server from the rest of the application. This can improve performance
+when running GitLab with its repositories stored on an NFS server.
+
+At the moment (GitLab 9.4) Gitaly is not yet a replacement for NFS
+because some parts of GitLab still bypass Gitaly when accessing Git
+repositories. If you choose to deploy Gitaly on your NFS server you
+must still also mount your Git shares on your GitLab application
+servers.
+
+Gitaly network traffic is unencrypted so you should use a firewall to
+restrict access to your Gitaly server.
+
+Below we describe how to configure a Gitaly server at address
+`gitaly.internal:9999` with secret token `abc123secret`. We assume
+your GitLab installation has two repository storages, `default` and
+`storage1`.
+
+### Client side token configuration
+
+Start by configuring a token on the client side.
+
+Omnibus installations:
+
+```ruby
+# /etc/gitlab/gitlab.rb
+gitlab_rails['gitaly_token'] = 'abc123secret'
+```
+
+Source installations:
+
+```yaml
+# /home/git/gitlab/config/gitlab.yml
+gitlab:
+  gitaly:
+    token: 'abc123secret'
+```
+
+You need to reconfigure (Omnibus) or restart (source) for these
+changes to be picked up.
+
+### Gitaly server configuration
+
+Next, on the Gitaly server, we need to configure storage paths, enable
+the network listener and configure the token.
+
+Note: if you want to reduce the risk of downtime when you enable
+authentication you can temporarily disable enforcement, see
+https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md#authentication
+.
+
+In most or all cases the storage paths below end in `/repositories`. Check the
+directory layout on your Gitaly server to be sure.
+
+Omnibus installations:
+
+```ruby
+# /etc/gitlab/gitlab.rb
+gitaly['listen_addr'] = '0.0.0.0:9999'
+gitaly['auth_token'] = 'abc123secret'
+gitaly['storage'] = [
+  { 'name' => 'default', 'path' => '/path/to/default/repositories' },
+  { 'name' => 'storage1', 'path' => '/path/to/storage1/repositories' },
+]
+```
+
+Source installations:
+
+```toml
+# /home/git/gitaly/config.toml
+listen_addr = '0.0.0.0:9999'
+
+[auth]
+token = 'abc123secret'
+
+[[storage]
+name = 'default'
+path = '/path/to/default/repositories'
+
+[[storage]]
+name = 'storage1'
+path = '/path/to/storage1/repositories'
+```
+
+Again, reconfigure (Omnibus) or restart (source).
+
+### Converting clients to use the Gitaly server
+
+Now as the final step update the client machines to switch from using
+their local Gitaly service to the new Gitaly server you just
+configured. This is a risky step because if there is any sort of
+network, firewall, or name resolution problem preventing your GitLab
+server from reaching the Gitaly server then all Gitaly requests will
+fail.
+
+We assume that your Gitaly server can be reached at
+`gitaly.internal:9999` from your GitLab server, and that your GitLab
+NFS shares are mounted at `/mnt/gitlab/default` and
+`/mnt/gitlab/storage1` respectively.
+
+Omnibus installations:
+
+```ruby
+# /etc/gitlab/gitlab.rb
+git_data_dirs({
+  { 'default' => { 'path' => '/mnt/gitlab/default', 'gitaly_address' => 'tcp://gitlab.internal:9999' } },
+  { 'storage1' => { 'path' => '/mnt/gitlab/storage1', 'gitaly_address' => 'tcp://gitlab.internal:9999' } },
+})
+```
+
+Source installations:
+
+```yaml
+# /home/git/gitlab/config/gitlab.yml
+gitlab:
+  repositories:
+    storages:
+      default:
+        path: /mnt/gitlab/default/repositories
+        gitaly_address: tcp://gitlab.internal:9999
+      storage1:
+        path: /mnt/gitlab/storage1/repositories
+        gitaly_address: tcp://gitlab.internal:9999
+```
+
+Now reconfigure (Omnibus) or restart (source). When you tail the
+Gitaly logs on your Gitaly server (`sudo gitlab-ctl tail gitaly` or
+`tail -f /home/git/gitlab/log/gitaly.log`) you should see requests
+coming in. One sure way to trigger a Gitaly request is to clone a
+repository from your GitLab server over HTTP.
+
 ## Configuring GitLab to not use Gitaly
 
 Gitaly is still an optional component in GitLab 9.3. This means you