BigW Consortium Gitlab

README.md 15.4 KB
Newer Older
1
## Migrate GitLab CI to GitLab CE or EE
2

3 4 5
Beginning with version 8.0 of GitLab Community Edition (CE) and Enterprise
Edition (EE), GitLab CI is no longer its own application, but is instead built
into the CE and EE applications.
6

7
This guide will detail the process of migrating your CI installation and data
8 9 10
into your GitLab CE or EE installation. **You can only migrate CI data from
GitLab CI 8.0 to GitLab 8.0; migrating between other versions (e.g.7.14 to 8.1)
is not possible.**
11

12 13
We recommend that you read through the entire migration process in this
document before beginning.
14

15
### Overview
16

17 18
In this document we assume you have a GitLab server and a GitLab CI server. It
does not matter if these are the same machine.
19

20 21
The migration consists of three parts: updating GitLab and GitLab CI, moving
data, and redirecting traffic.
22

23
Please note that CI builds triggered on your GitLab server in the time between
Jacob Vosmaer committed
24 25 26 27
updating to 8.0 and finishing the migration will be lost. Your GitLab server
can be online for most of the procedure; the only GitLab downtime (if any) is
during the upgrade to 8.0. Your CI service will be offline from the moment you
upgrade to 8.0 until you finish the migration procedure.
28

29
### Before upgrading
30

31
If you have GitLab CI installed using omnibus-gitlab packages but **you don't want to migrate your existing data**:
32 33 34 35 36

```bash
mv /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds.$(date +%s)
```

37 38 39
run `sudo gitlab-ctl reconfigure` and you can reach CI at `gitlab.example.com/ci`.

If you want to migrate your existing data, continue reading.
40

41 42 43 44 45
#### 0. Updating Omnibus from versions prior to 7.13

If you are updating from older versions you should first update to 7.14 and then to 8.0.
Otherwise it's pretty likely that you will encounter problems described in the [Troubleshooting](#troubleshooting).

46 47 48
#### 1. Verify that backups work

Make sure that the backup script on both servers can connect to the database.
49

50
```
51
# On your CI server:
52
# Omnibus
53
sudo chown gitlab-ci:gitlab-ci /var/opt/gitlab/gitlab-ci/builds
54
sudo gitlab-ci-rake backup:create
55

56 57 58 59
# Source
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production
```
60

61 62
Also check on your GitLab server.

63
```
64
# On your GitLab server:
65 66
# Omnibus
sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads
67

68 69 70 71
# Source
cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads
```
Robert Speicher committed
72

73 74
If this fails you need to fix it before upgrading to 8.0. Also see
https://about.gitlab.com/getting-help/
Robert Speicher committed
75

76 77 78
#### 2. Check source and target database types

Check what databases you use on your GitLab server and your CI server.
79 80 81 82
  Look for the 'adapter:' line. If your CI server and your GitLab server use
the same database adapter no special care is needed. If your CI server uses
MySQL and your GitLab server uses PostgreSQL you need to pass a special option
during the 'Moving data' part. **If your CI server uses PostgreSQL and your
83
GitLab server uses MySQL you cannot migrate your CI data to GitLab 8.0.**
Robert Speicher committed
84

85
```
86
# On your CI server:
87 88
# Omnibus
sudo gitlab-ci-rake env:info
89

90 91 92 93
# Source
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec rake env:info RAILS_ENV=production
```
94

95
```
96
# On your GitLab server:
97 98
# Omnibus
sudo gitlab-rake gitlab:env:info
99

100 101 102 103
# Source
cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production
```
104

105 106 107
#### 3. Storage planning

Decide where to store CI build traces on GitLab server. GitLab CI uses
108
  files on disk to store CI build traces. The default path for these build
Jacob Vosmaer committed
109
traces is `/var/opt/gitlab/gitlab-ci/builds` (Omnibus) or
110 111 112 113
`/home/git/gitlab/builds` (Source). If you are storing your repository data in
a special location, or if you are using NFS, you should make sure that you
store build traces on the same storage as your Git repositories.

Jacob Vosmaer committed
114
### I. Upgrading
115

116
From this point on, GitLab CI will be unavailable for your end users.
117

118 119 120
#### 1. Upgrade GitLab to 8.0

First upgrade your GitLab server to version 8.0:
121
https://about.gitlab.com/update/
122 123 124 125

#### 2. Disable CI on the GitLab server during the migration

After you update, go to the admin panel and temporarily disable CI.  As
126 127 128
  an administrator, go to **Admin Area** -> **Settings**, and under
**Continuous Integration** uncheck **Disable to prevent CI usage until rake
ci:migrate is run (8.0 only)**.
129 130 131 132

#### 3. CI settings are now in GitLab

If you want to use custom CI settings (e.g. change where builds are
133 134
  stored), please update `/etc/gitlab/gitlab.rb` (Omnibus) or
`/home/git/gitlab/config/gitlab.yml` (Source).
135 136 137 138

#### 4. Upgrade GitLab CI to 8.0

Now upgrade GitLab CI to version 8.0. If you are using Omnibus packages,
139
  this may have already happened when you upgraded GitLab to 8.0.
140

141 142 143
#### 5. Disable GitLab CI on the CI server

Disable GitLab CI after upgrading to 8.0.
144

145
```
146
# On your CI server:
147 148 149 150 151 152 153
# Omnibus
sudo gitlab-ctl stop ci-unicorn
sudo gitlab-ctl stop ci-sidekiq

# Source
sudo service gitlab_ci stop
cd /home/gitlab_ci/gitlab-ci
154
sudo -u gitlab_ci -H bundle exec whenever --clear-crontab RAILS_ENV=production
155
```
156

Jacob Vosmaer committed
157
### II. Moving data
158

159 160 161
#### 1. Database encryption key

Move the database encryption key from your CI server to your GitLab
162 163 164 165
  server. The command below will show you what you need to copy-paste to your
GitLab server. On Omnibus GitLab servers you will have to add a line to
`/etc/gitlab/gitlab.rb`. On GitLab servers installed from source you will have
to replace the contents of `/home/git/gitlab/config/secrets.yml`.
166

167
```
168
# On your CI server:
169 170
# Omnibus
sudo gitlab-ci-rake backup:show_secrets
171

172 173 174 175
# Source
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec rake backup:show_secrets RAILS_ENV=production
```
176

177 178 179
#### 2. SQL data and build traces

Create your final CI data export. If you are converting from MySQL to
180 181 182
  PostgreSQL, add ` MYSQL_TO_POSTGRESQL=1` to the end of the rake command. When
the command finishes it will print the path to your data export archive; you
will need this file later.
Robert Speicher committed
183

184
```
185
# On your CI server:
186
# Omnibus
187
sudo chown gitlab-ci:gitlab-ci /var/opt/gitlab/gitlab-ci/builds
188
sudo gitlab-ci-rake backup:create
189

190
# Source
191 192
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production
193
```
Kamil Trzcinski committed
194

195 196 197 198 199 200 201 202 203
#### 3. Copy data to the GitLab server

If you were running GitLab and GitLab CI on the same server you can skip this
step.

Copy your CI data archive to your GitLab server. There are many ways to do
this, below we use SSH agent forwarding and 'scp', which will be easy and fast
for most setups. You can also copy the data archive first from the CI server to
your laptop and then from your laptop to the GitLab server.
Robert Speicher committed
204

205 206 207 208 209
```
# Start from your laptop
ssh -A ci_admin@ci_server.example
# Now on the CI server
scp /path/to/12345_gitlab_ci_backup.tar gitlab_admin@gitlab_server.example:~
210
```
Kamil Trzcinski committed
211

212 213 214 215
#### 4. Move data to the GitLab backups folder

Make the CI data archive discoverable for GitLab. We assume below that you
store backups in the default path, adjust the command if necessary.
216

217
```
218
# On your GitLab server:
219 220
# Omnibus
sudo mv /path/to/12345_gitlab_ci_backup.tar /var/opt/gitlab/backups/
221

222 223 224
# Source
sudo mv /path/to/12345_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups/
```
Robert Speicher committed
225

226 227 228 229
#### 5. Import the CI data into GitLab.

This step will delete any existing CI data on your GitLab server. There should
be no CI data yet because you turned CI on the GitLab server off earlier.
230

231
```
232
# On your GitLab server:
233
# Omnibus
234
sudo chown git:git /var/opt/gitlab/gitlab-ci/builds
235
sudo gitlab-rake ci:migrate
236

237 238 239 240
# Source
cd /home/git/gitlab
sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production
```
241

242
#### 6. Restart GitLab
243

244
```
245
# On your GitLab server:
246 247 248
# Omnibus
sudo gitlab-ctl hup unicorn
sudo gitlab-ctl restart sidekiq
249

250 251 252
# Source
sudo service gitlab reload
```
Robert Speicher committed
253

Jacob Vosmaer committed
254
### III. Redirecting traffic
255

256 257
If you were running GitLab CI with Omnibus packages and you were using the
internal NGINX configuration your CI service should now be available both at
258
`ci.example.com` (the old address) and `gitlab.example.com/ci`. **You are done!**
259

260 261 262
If you installed GitLab CI from source we now need to configure a redirect in
NGINX so that existing CI runners can keep using the old CI server address, and
so that existing links to your CI server keep working.
263

264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311
#### 1. Update Nginx configuration

To ensure that your existing CI runners are able to communicate with the
migrated installation, and that existing build triggers still work, you'll need
to update your Nginx configuration to redirect requests for the old locations to
the new ones.

Edit `/etc/nginx/sites-available/gitlab_ci` and paste:

```nginx
# GITLAB CI
server {
  listen 80 default_server;         # e.g., listen 192.168.1.1:80;
  server_name YOUR_CI_SERVER_FQDN;  # e.g., server_name source.example.com;

  access_log  /var/log/nginx/gitlab_ci_access.log;
  error_log   /var/log/nginx/gitlab_ci_error.log;

  # expose API to fix runners
  location /api {
    proxy_read_timeout    300;
    proxy_connect_timeout 300;
    proxy_redirect        off;
    proxy_set_header      X-Real-IP $remote_addr;

    # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN
    resolver 8.8.8.8 8.8.4.4;
    proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
  }

  # redirect all other CI requests
  location / {
    return 301 $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
  }

  # adjust this to match the largest build log your runners might submit,
  # set to 0 to disable limit
  client_max_body_size 10m;
}
```

Make sure you substitute these placeholder values with your real ones:

1. `YOUR_CI_SERVER_FQDN`: The existing public-facing address of your GitLab CI
   install (e.g., `ci.gitlab.com`).
1. `YOUR_GITLAB_SERVER_FQDN`: The current public-facing address of your GitLab
   CE (or EE) install (e.g., `gitlab.com`).

312 313
**Make sure not to remove the `/ci$request_uri` part. This is required to
properly forward the requests.**
314 315 316

You should also make sure that you can:

317
1. `curl https://YOUR_GITLAB_SERVER_FQDN/` from your previous GitLab CI server.
318
1. `curl https://YOUR_CI_SERVER_FQDN/` from your GitLab CE (or EE) server.
319

320
#### 2. Check Nginx configuration
321 322 323

    sudo nginx -t

324
#### 3. Restart Nginx
325 326 327

    sudo /etc/init.d/nginx restart

328 329 330 331
#### Restore from backup

If something went wrong and you need to restore a backup, consult the [Backup
restoration](../raketasks/backup_restore.md) guide.
332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348

### Troubleshooting

#### show:secrets problem (Omnibus-only)
If you see errors like this:
```
Missing `secret_key_base` or `db_key_base` for 'production' environment. The secrets will be generated and stored in `config/secrets.yml`
rake aborted!
Errno::EACCES: Permission denied @ rb_sysopen - config/secrets.yml
```

This can happen if you are updating from versions prior to 7.13 straight to 8.0.
The fix for this is to update to Omnibus 7.14 first and then update it to 8.0.

#### Permission denied when accessing /var/opt/gitlab/gitlab-ci/builds
To fix that issue you have to change builds/ folder permission before doing final backup:
```
349 350 351 352 353 354
sudo chown -R gitlab-ci:gitlab-ci /var/opt/gitlab/gitlab-ci/builds
```

Then before executing `ci:migrate` you need to fix builds folder permission:
```
sudo chown git:git /var/opt/gitlab/gitlab-ci/builds
355 356 357
```

#### Problems when importing CI database to GitLab
Chris Spicer committed
358
If you were migrating CI database from MySQL to PostgreSQL manually you can see errors during import about missing sequences:
359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435
```
ALTER SEQUENCE
ERROR:  relation "ci_builds_id_seq" does not exist
ERROR:  relation "ci_commits_id_seq" does not exist
ERROR:  relation "ci_events_id_seq" does not exist
ERROR:  relation "ci_jobs_id_seq" does not exist
ERROR:  relation "ci_projects_id_seq" does not exist
ERROR:  relation "ci_runner_projects_id_seq" does not exist
ERROR:  relation "ci_runners_id_seq" does not exist
ERROR:  relation "ci_services_id_seq" does not exist
ERROR:  relation "ci_taggings_id_seq" does not exist
ERROR:  relation "ci_tags_id_seq" does not exist
CREATE TABLE
```

To fix that you need to apply this SQL statement before doing final backup:
```
# Omnibus
gitlab-ci-rails dbconsole <<EOF
-- ALTER TABLES - DROP DEFAULTS
ALTER TABLE ONLY ci_application_settings ALTER COLUMN id DROP DEFAULT;
ALTER TABLE ONLY ci_builds ALTER COLUMN id DROP DEFAULT;
ALTER TABLE ONLY ci_commits ALTER COLUMN id DROP DEFAULT;
ALTER TABLE ONLY ci_events ALTER COLUMN id DROP DEFAULT;
ALTER TABLE ONLY ci_jobs ALTER COLUMN id DROP DEFAULT;
ALTER TABLE ONLY ci_projects ALTER COLUMN id DROP DEFAULT;
ALTER TABLE ONLY ci_runner_projects ALTER COLUMN id DROP DEFAULT;
ALTER TABLE ONLY ci_runners ALTER COLUMN id DROP DEFAULT;
ALTER TABLE ONLY ci_services ALTER COLUMN id DROP DEFAULT;
ALTER TABLE ONLY ci_taggings ALTER COLUMN id DROP DEFAULT;
ALTER TABLE ONLY ci_tags ALTER COLUMN id DROP DEFAULT;
ALTER TABLE ONLY ci_trigger_requests ALTER COLUMN id DROP DEFAULT;
ALTER TABLE ONLY ci_triggers ALTER COLUMN id DROP DEFAULT;
ALTER TABLE ONLY ci_variables ALTER COLUMN id DROP DEFAULT;
ALTER TABLE ONLY ci_web_hooks ALTER COLUMN id DROP DEFAULT;

-- ALTER SEQUENCES
ALTER SEQUENCE ci_application_settings_id_seq OWNED BY ci_application_settings.id;
ALTER SEQUENCE ci_builds_id_seq OWNED BY ci_builds.id;
ALTER SEQUENCE ci_commits_id_seq OWNED BY ci_commits.id;
ALTER SEQUENCE ci_events_id_seq OWNED BY ci_events.id;
ALTER SEQUENCE ci_jobs_id_seq OWNED BY ci_jobs.id;
ALTER SEQUENCE ci_projects_id_seq OWNED BY ci_projects.id;
ALTER SEQUENCE ci_runner_projects_id_seq OWNED BY ci_runner_projects.id;
ALTER SEQUENCE ci_runners_id_seq OWNED BY ci_runners.id;
ALTER SEQUENCE ci_services_id_seq OWNED BY ci_services.id;
ALTER SEQUENCE ci_taggings_id_seq OWNED BY ci_taggings.id;
ALTER SEQUENCE ci_tags_id_seq OWNED BY ci_tags.id;
ALTER SEQUENCE ci_trigger_requests_id_seq OWNED BY ci_trigger_requests.id;
ALTER SEQUENCE ci_triggers_id_seq OWNED BY ci_triggers.id;
ALTER SEQUENCE ci_variables_id_seq OWNED BY ci_variables.id;
ALTER SEQUENCE ci_web_hooks_id_seq OWNED BY ci_web_hooks.id;

-- ALTER TABLES - RE-APPLY DEFAULTS
ALTER TABLE ONLY ci_application_settings ALTER COLUMN id SET DEFAULT nextval('ci_application_settings_id_seq'::regclass);
ALTER TABLE ONLY ci_builds ALTER COLUMN id SET DEFAULT nextval('ci_builds_id_seq'::regclass);
ALTER TABLE ONLY ci_commits ALTER COLUMN id SET DEFAULT nextval('ci_commits_id_seq'::regclass);
ALTER TABLE ONLY ci_events ALTER COLUMN id SET DEFAULT nextval('ci_events_id_seq'::regclass);
ALTER TABLE ONLY ci_jobs ALTER COLUMN id SET DEFAULT nextval('ci_jobs_id_seq'::regclass);
ALTER TABLE ONLY ci_projects ALTER COLUMN id SET DEFAULT nextval('ci_projects_id_seq'::regclass);
ALTER TABLE ONLY ci_runner_projects ALTER COLUMN id SET DEFAULT nextval('ci_runner_projects_id_seq'::regclass);
ALTER TABLE ONLY ci_runners ALTER COLUMN id SET DEFAULT nextval('ci_runners_id_seq'::regclass);
ALTER TABLE ONLY ci_services ALTER COLUMN id SET DEFAULT nextval('ci_services_id_seq'::regclass);
ALTER TABLE ONLY ci_taggings ALTER COLUMN id SET DEFAULT nextval('ci_taggings_id_seq'::regclass);
ALTER TABLE ONLY ci_tags ALTER COLUMN id SET DEFAULT nextval('ci_tags_id_seq'::regclass);
ALTER TABLE ONLY ci_trigger_requests ALTER COLUMN id SET DEFAULT nextval('ci_trigger_requests_id_seq'::regclass);
ALTER TABLE ONLY ci_triggers ALTER COLUMN id SET DEFAULT nextval('ci_triggers_id_seq'::regclass);
ALTER TABLE ONLY ci_variables ALTER COLUMN id SET DEFAULT nextval('ci_variables_id_seq'::regclass);
ALTER TABLE ONLY ci_web_hooks ALTER COLUMN id SET DEFAULT nextval('ci_web_hooks_id_seq'::regclass);
EOF

# Source
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec rails dbconsole production <<EOF
... COPY SQL STATEMENTS FROM ABOVE ...
EOF
```