BigW Consortium Gitlab

Skip to content

G
gitlab-ce
Commit 494c2785 by Gabriel Mazetto
Options

Fixed some documentation and moved Source install specific to other file.

parent 92e60372
Showing with 300 additions and 121 deletions
doc/administration/high_availability/redis.md
......@@ -19,7 +19,6 @@ that comes bundled with Omnibus GitLab packages.
- [Prerequisites](#prerequisites)
- [Redis setup](#redis-setup)
- [Existing single-machine installation](#existing-single-machine-installation)
- [Installation from source](#installation-from-source)
- [Omnibus packages](#omnibus-packages)
- [Configuring Sentinel](#configuring-sentinel)
- [How sentinel handles a failover](#how-sentinel-handles-a-failover)
......@@ -31,7 +30,6 @@ that comes bundled with Omnibus GitLab packages.
- [Redis replication](#redis-replication)
- [Sentinel](#sentinel)
- [Omnibus GitLab](#omnibus-gitlab)
- [Install from Source](#install-from-source)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
......@@ -161,58 +159,6 @@ To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`:
redis['enable'] = false
```
#### Installation from source
**Configuring Master Redis instance**
You need to make the following changes in `redis.conf`:
1. Define a `bind` address pointing to a local IP that your other machines
can reach you. If you really need to bind to an external acessible IP, make
sure you add extra firewall rules to prevent unauthorized access:
```conf
# By default, if no "bind" configuration directive is specified, Redis listens
# for connections from all the network interfaces available on the server.
# It is possible to listen to just one or multiple selected interfaces using
# the "bind" configuration directive, followed by one or more IP addresses.
#
# Examples:
#
# bind 192.168.1.100 10.0.0.1
# bind 127.0.0.1 ::1
bind 0.0.0.0 # This will bind to all interfaces
```
1. Define a `port` to force redis to listin on TCP so other machines can
connect to it:
```conf
# Accept connections on the specified port, default is 6379 (IANA #815344).
# If port 0 is specified Redis will not listen on a TCP socket.
port 6379
```
1. Set up password authentication (use the same password in all nodes)
```conf
requirepass "redis-password-goes-here"
masterauth "redis-password-goes-here"
```
1. Restart the Redis services for the changes to take effect.
**Configuring Slave Redis instance**
1. Follow same instructions from master, with the extra change in `redis.conf`:
```conf
# IP and port of the master Redis server
slaveof 10.10.10.10 6379
```
1. Restart the Redis services for the changes to take effect.
#### Omnibus packages
You need to install the Omnibus GitLab package in `3` independent machines.
......@@ -304,25 +250,16 @@ GitLab Enterprise Edition provides [automated way to setup and run](#sentinel-se
#### Sentinel setup
##### Community Edition
With GitLab Community Edition, you need to install, configure, execute and
monitor Sentinel from source. Omnibus GitLab Community Edition package does
not support Sentinel configuration.
A minimal configuration file (`sentinel.conf`) should contain the following:
```conf
bind 0.0.0.0 # bind to all interfaces or change to a specific IP
port 26379 # default sentinel port
sentinel auth-pass gitlab-redis redis-password-goes-here
sentinel monitor gitlab-redis 10.0.0.1 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel config-epoch gitlab-redis 0
sentinel leader-epoch gitlab-redis 0
```
See documentation for Source Install [here](redis_source.md).
##### Enterprise Edition
To setup sentinel, you edit `/etc/gitlab/gitlab.rb` file:
To setup sentinel, edit `/etc/gitlab/gitlab.rb` file:
```ruby
......@@ -336,7 +273,7 @@ redis_master_role['enable'] = true
## Enabled Sentinel and Redis Slave services
redis_sentinel_role['enable'] = true
redis_master_role['enable'] = true
redis_slave_role['enable'] = true
## Configure Redis
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
......@@ -345,7 +282,7 @@ redis['master_port'] = 6379 # port of the initial master redis instance
redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance
## Configure Sentinel
sentinel['bind'] = '0.0.0.0' # or specify an IP to bind to a single one
# sentinel['bind'] = '0.0.0.0' # bind to all interfaces, uncomment to specify an IP and bind to a single one
# sentinel['port'] = 26379 # uncomment to change default port
## Quorum must reflect the amount of voting sentinels it take to start a failover.
......@@ -435,24 +372,16 @@ master and the new sentinels servers.
### GitLab setup
You can enable or disable sentinel support at any time in new or existing
You can enable or disable Sentinel support at any time in new or existing
installations. From the GitLab application perspective, all it requires is
the correct credentials for the master Redis and for all Sentinel nodes.
the correct credentials for the Sentinel nodes.
It doesn't require a list of all Sentinel nodes, as in case of a failure,
the application will need to query only one of them.
While it doesn't require a list of all Sentinel nodes, in case of a failure,
it needs to access at one of listed ones.
>**Note:**
The following steps should be performed in the [GitLab application server](gitlab.md).
**For source based installations**
1. Edit `/home/git/gitlab/config/resque.yml` following the example in
`/home/git/gitlab/config/resque.yml.example`, and uncomment the sentinels
line, changing to the correct server credentials.
1. Restart GitLab for the changes to take effect.
**For Omnibus installations**
The following steps should be performed in the [GitLab application server](gitlab.md)
which ideally should not have Redis or Sentinels in the same machine for a HA setup.
1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines:
......@@ -466,7 +395,7 @@ The following steps should be performed in the [GitLab application server](gitla
]
```
1. [Reconfigure] the GitLab for the changes to take effect.
1. [Reconfigure] GitLab for the changes to take effect.
## Troubleshooting
......@@ -548,42 +477,6 @@ The way the redis connector `redis-rb` works with sentinel is a bit
non-intuitive. We try to hide the complexity in omnibus, but it still requires
a few extra configs.
#### Install from Source
If you get an error like: `Redis::CannotConnectError: No sentinels available.`,
there may be something wrong with your configuration files or it can be related
to [this issue][gh-531].
It's a bit non-intuitive the way you have to config `resque.yml` and
`sentinel.conf`, otherwise `redis-rb` will not work properly.
The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`)
**must** be used as the hostname in GitLab (`resque.yml` for source installations
or `gitlab-rails['redis_*']` in Omnibus):
```conf
# sentinel.conf:
sentinel monitor gitlab-redis 10.10.10.10 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel config-epoch gitlab-redis 0
sentinel leader-epoch gitlab-redis 0
```
```yaml
# resque.yaml
production:
url: redis://:myredispassword@gitlab-redis/
sentinels:
-
host: slave1.example.com # or use ip
port: 26380 # point to sentinel, not to redis port
-
host: slave2.exampl.com # or use ip
port: 26381 # point to sentinel, not to redis port
```
When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel)
---
To make sure your configuration is correct:
......@@ -611,8 +504,8 @@ To make sure your configuration is correct:
1. To simulate a failover on master Redis, SSH into the Redis server and run:
```bash
# port must match your master redis port
redis-cli -h localhost -p 6379 DEBUG sleep 60
# port must match your master redis port, and the sleep time must be a few seconds bigger than defined one
redis-cli -h localhost -p 6379 DEBUG sleep 20
```
1. Then back in the Rails console from the first step, run:
......
doc/administration/high_availability/redis_source.md 0 → 100644
# Configuring Redis for GitLab HA (Source Install)
We highly recommend that you use Omnibus GitLab packages, as we can optimize
required packages specifically for GitLab, and we will take care of upgrading
to the latest supported version.
If you are building packages for a specific distro, or trying to build some
internal automation, you can check this documentation to learn about the
minimal setup, required changes, etc.
If you want to see the documentation for Omnibus GitLab Install, please [read it here](redis.md).
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents**
- [Configure your own Redis server](#configure-your-own-redis-server)
- [Configuring Master Redis instance](#configuring-master-redis-instance)
- [Configuring Slave Redis instances](#configuring-slave-redis-instances)
- [Configuring Redis Sentinel instances](#configuring-redis-sentinel-instances)
- [GitLab setup](#gitlab-setup)
- [Example configurations](#example-configurations)
- [Configuring Redis Master](#configuring-redis-master)
- [Configuring Redis Slaves](#configuring-redis-slaves)
- [Configuring Redis Sentinel](#configuring-redis-sentinel)
- [Troubleshooting](#troubleshooting)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
## Configure your own Redis server
Redis server must be configured to use TCP connection instead of socket,
and since Redis `3.2`, you must define a password to receive external
connections (`requirepass`).
You will also need to define equal password for slave password definition
(`masterauth`), in the same instance, if you are using Redis with Sentinel.
To configure Redis to use TCP connection you need to define both
`bind` and `port`. You can bind to all interfaces (`0.0.0.0`) or specify the
ip of the desired interface (for ex. one from an internal network).
### Configuring Master Redis instance
You need to make the following changes in `redis.conf`:
1. Define a `bind` address pointing to a local IP that your other machines
can reach you. If you really need to bind to an external acessible IP, make
sure you add extra firewall rules to prevent unauthorized access:
1. Define a `port` to force redis to listen on TCP so other machines can
connect to it (default port is `6379`).
1. Set up password authentication (use the same password in all nodes).
The password should be defined equal for both `requirepass` and `masterauth`
when setting up Redis to use with Sentinel.
1. Restart the Redis services for the changes to take effect.
See [example configuration](#configuring-redis-master) below.
### Configuring Slave Redis instances
1. Follow same instructions for Redis Master
1. Define `slaveof` pointing to the Redis master instance with **IP** and **port**.
1. Restart the Redis services for the changes to take effect.
See [example configuration](#configuring-redis-slaves) below.
### Configuring Redis Sentinel instances
Sentinel is a special type of Redis server. It inherits most of the basic
configuration options you can define in `redis.conf`, with specific ones
starting with `sentinel` prefix.
You will need to define the initial configs to enable connectivity:
1. Define a `bind` address pointing to a local IP that your other machines
can reach you. If you really need to bind to an external acessible IP, make
sure you add extra firewall rules to prevent unauthorized access:
1. Define a `port` to force sentinel to listen on TCP so other machines can
connect to it (default port is `26379`).
And the sentinel specific ones:
1. Define with `sentinel auth-pass` the same shared password you have
defined for both Redis **Master** and **Slaves** instances.
1. Define with `sentinel monitor` the **IP** and **port** of the Redis
**Master** node, and the **quorum** required to start a failover.
If you need more information to understand about quorum, please
read the detailed explanation in the [HA documentation for Omnibus Installs](redis.md).
1. Define with `sentinel down-after-milliseconds` the ammount in `ms` of time
that an unresponsive server will be considered down.
1. Define a value for `sentinel failover_timeout` in `ms`. This has multiple
meanings:
* The time needed to re-start a failover after a previous failover was
already tried against the same master by a given Sentinel, is two
times the failover timeout.
* The time needed for a slave replicating to a wrong master according
to a Sentinel current configuration, to be forced to replicate
with the right master, is exactly the failover timeout (counting since
the moment a Sentinel detected the misconfiguration).
* The time needed to cancel a failover that is already in progress but
did not produced any configuration change (SLAVEOF NO ONE yet not
acknowledged by the promoted slave).
* The maximum time a failover in progress waits for all the slaves to be
reconfigured as slaves of the new master. However even after this time
the slaves will be reconfigured by the Sentinels anyway, but not with
the exact parallel-syncs progression as specified.
See [example configuration](#configuring-redis-sentinel) below.
## GitLab setup
You can enable or disable Sentinel support at any time in new or existing
installations. From the GitLab application perspective, all it requires is
the correct credentials for the Sentinel nodes.
While it doesn't require a list of all Sentinel nodes, in case of a failure,
it needs to access at one of listed ones.
>**Note:**
The following steps should be performed in the [GitLab application server](gitlab.md)
which ideally should not have Redis or Sentinels in the same machine for a HA setup.
1. Edit `/home/git/gitlab/config/resque.yml` following the example in
`/home/git/gitlab/config/resque.yml.example`, and uncomment the sentinels
lines, pointing to the correct server credentials.
1. Restart GitLab for the changes to take effect.
## Example configurations
In this example we consider that all servers have an internal network
interface with IPs in the `10.0.0.x` range, and that they can connect
to each other using these IPs.
In a real world usage, you would also setup firewall rules to prevent
unauthorized access from other machines, and block traffic from the
outside (Internet).
We will use the same `3` nodes with **Redis** + **Sentinel** topology
discussed in the [Configuring Redis for GitLab HA](redis.md) documentation.
Here is a list and description of each **machine** and the assined **ip**:
* `10.0.0.1`: Redis Master + Sentinel 1
* `10.0.0.2`: Redis Slave 1 + Sentinel 2
* `10.0.0.2`: Redis Slave 2 + Sentinel 3
Please note that after the initial configuration, if a failover is initiated
by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master**
will change permanently (including in `redis.conf`) from one node to the other,
until a new failover is initiated again.
The same thing will happen with `sentinel.conf` that will be overriten after the
initial execution, after any new sentinel node starts watching the **Master**,
or a failover promotes a different **Master** node.
### Configuring Redis Master
`redis.conf`:
```conf
bind 10.0.0.1
port 6379
requirepass redis-password-goes-here
masterauth redis-password-goes-here
```
### Configuring Redis Slaves
**Slave 1 - `redis.conf`:**
```conf
bind 10.0.0.2
port 6379
requirepass redis-password-goes-here
masterauth redis-password-goes-here
# IP and port of the master Redis server
slaveof 10.0.0.1 6379
```
**Slave 2 - `redis.conf`:**
```conf
bind 10.0.0.3
port 6379
requirepass redis-password-goes-here
masterauth redis-password-goes-here
# IP and port of the master Redis server
slaveof 10.0.0.1 6379
```
### Configuring Redis Sentinel
For this example, **Sentinel 1** will be configured in the same machine as the
**Redis Master**, **Sentinel 2** and **Sentinel 3** in the same machines as the
**Slave 1** and **Slave 2** respectively.
Sentinel 1 - `sentinel.conf`
```conf
bind 10.0.0.1
port 26379
sentinel auth-pass gitlab-redis redis-password-goes-here
sentinel monitor gitlab-redis 10.0.0.1 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel failover_timeout 30000
```
Sentinel 2 - `sentinel.conf`
```conf
bind 10.0.0.2
port 26379
sentinel auth-pass gitlab-redis redis-password-goes-here
sentinel monitor gitlab-redis 10.0.0.1 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel failover_timeout 30000
```
Sentinel 3 - `sentinel.conf`
```conf
bind 10.0.0.3
port 26379
sentinel auth-pass gitlab-redis redis-password-goes-here
sentinel monitor gitlab-redis 10.0.0.1 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel failover_timeout 30000
```
## Troubleshooting
We have a more detailed [Troubleshooting](redis.md#troubleshooting) explained in the documentation for Omnibus
Install. Here we will list only the things that are specific to a **Source** install.
If you get an error in GitLab like: `Redis::CannotConnectError: No sentinels available.`,
there may be something wrong with your configuration files or it can be related
to [this issue][gh-531].
It's a bit non-intuitive the way you have to config `resque.yml` and
`sentinel.conf`, otherwise `redis-rb` will not work properly.
The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`)
**must** be used as the hostname in GitLab (`resque.yml`):
```conf
# sentinel.conf:
sentinel monitor gitlab-redis 10.0.0.1 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel config-epoch gitlab-redis 0
sentinel leader-epoch gitlab-redis 0
```
```yaml
# resque.yaml
production:
url: redis://:myredispassword@gitlab-redis/
sentinels:
-
host: 10.0.0.1
port: 26379 # point to sentinel, not to redis port
-
host: 10.0.0.2
port: 26379 # point to sentinel, not to redis port
-
host: 10.0.0.3
port: 26379 # point to sentinel, not to redis port
```
When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel)
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment