BigW Consortium Gitlab

using_docker_images.md 18.9 KB
Newer Older
1
# Using Docker images
2

3
GitLab CI in conjunction with [GitLab Runner](../runners/README.md) can use
4
[Docker Engine](https://www.docker.com/) to test and build any application.
5

6 7
Docker is an open-source project that allows you to use predefined images to
run applications in independent "containers" that are run within a single Linux
8
instance. [Docker Hub][hub] has a rich database of pre-built images that can be
9
used to test and build your applications.
10

11
Docker, when used with GitLab CI, runs each job in a separate and isolated
12 13
container using the predefined image that is set up in
[`.gitlab-ci.yml`](../yaml/README.md).
14 15 16 17

This makes it easier to have a simple and reproducible build environment that
can also run on your workstation. The added benefit is that you can test all
the commands that we will explore later from your shell, rather than having to
18
test them on a dedicated CI server.
19

20
## Register Docker Runner
21

22 23 24 25
To use GitLab Runner with Docker you need to [register a new Runner][register]
to use the `docker` executor.

A one-line example can be seen below:
26 27

```bash
28 29
sudo gitlab-runner register \
  --url "https://gitlab.example.com/" \
30 31 32 33 34 35 36 37
  --registration-token "PROJECT_REGISTRATION_TOKEN" \
  --description "docker-ruby-2.1" \
  --executor "docker" \
  --docker-image ruby:2.1 \
  --docker-postgres latest \
  --docker-mysql latest
```

38
The registered runner will use the `ruby:2.1` Docker image and will run two
39 40 41
services, `postgres:latest` and `mysql:latest`, both of which will be
accessible during the build process.

Wilfred Hughes committed
42
## What is an image
43

44 45
The `image` keyword is the name of the Docker image the Docker executor
will run to perform the CI tasks.
46

47
By default, the executor will only pull images from [Docker Hub][hub],
48
but this can be configured in the `gitlab-runner/config.toml` by setting
49
the [Docker pull policy][] to allow using local images.
50 51 52

For more information about images and Docker Hub please read
the [Docker Fundamentals][] documentation.
53

Wilfred Hughes committed
54
## What is a service
55

56 57
The `services` keyword defines just another Docker image that is run during
your job and is linked to the Docker image that the `image` keyword defines.
58
This allows you to access the service image during build time.
59

60 61 62 63
The service image can run any application, but the most common use case is to
run a database container, eg. `mysql`. It's easier and faster to use an
existing image and run it as an additional container than install `mysql` every
time the project is built.
64

65 66 67 68 69
You are not limited to have only database services. You can add as many
services you need to `.gitlab-ci.yml` or manually modify `config.toml`.
Any image found at [Docker Hub][hub] or your private Container Registry can be
used as a service.

70 71
You can see some widely used services examples in the relevant documentation of
[CI services examples](../services/README.md).
72

73
### How services are linked to the job
74 75

To better understand how the container linking works, read
76
[Linking containers together][linking-containers].
77 78

To summarize, if you add `mysql` as service to your application, the image will
79
then be used to create a container that is linked to the job container.
80 81 82

The service container for MySQL will be accessible under the hostname `mysql`.
So, in order to access your database service you have to connect to the host
83 84 85 86 87 88 89
named `mysql` instead of a socket or `localhost`. Read more in [accessing the
services](#accessing-the-services).

### Accessing the services

Let's say that you need a Wordpress instance to test some API integration with
your application.
90

91 92
You can then use for example the [tutum/wordpress][] image in your
`.gitlab-ci.yml`:
93

94 95 96 97
```yaml
services:
- tutum/wordpress:latest
```
98

99
If you don't [specify a service alias](#available-settings-for-services),
100 101
when the job is run, `tutum/wordpress` will be started and you will have
access to it from your build container under two hostnames to choose from:
102

103 104
- `tutum-wordpress`
- `tutum__wordpress`
105

106 107 108 109 110 111 112 113 114 115 116 117 118 119
>**Note:**
Hostnames with underscores are not RFC valid and may cause problems in 3rd party
applications.

The default aliases for the service's hostname are created from its image name
following these rules:

- Everything after the colon (`:`) is stripped
- Slash (`/`) is replaced with double underscores (`__`) and the primary alias
  is created
- Slash (`/`) is replaced with a single dash (`-`) and the secondary alias is
  created (requires GitLab Runner v1.1.0 or higher)

To override the default behavior, you can
120
[specify a service alias](#available-settings-for-services).
121 122

## Define `image` and `services` from `.gitlab-ci.yml`
123 124

You can simply define an image that will be used for all jobs and a list of
125
services that you want to use during build time:
126 127

```yaml
128
image: ruby:2.2
129

130 131
services:
  - postgres:9.3
132

133
before_script:
134
  - bundle install
135

136 137 138 139 140
test:
  script:
  - bundle exec rake spec
```

141 142 143 144
It is also possible to define different images and services per job:

```yaml
before_script:
145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161
  - bundle install

test:2.1:
  image: ruby:2.1
  services:
  - postgres:9.3
  script:
  - bundle exec rake spec

test:2.2:
  image: ruby:2.2
  services:
  - postgres:9.4
  script:
  - bundle exec rake spec
```

162 163
Or you can pass some [extended configuration options](#extended-docker-configuration-options)
for `image` and `services`:
164 165 166 167

```yaml
image:
  name: ruby:2.2
168
  entrypoint: ["/bin/bash"]
169 170 171 172

services:
- name: my-postgres:9.4
  alias: db-postgres
173 174
  entrypoint: ["/usr/local/bin/db-postgres"]
  command: ["start"]
175 176 177 178 179 180 181 182 183

before_script:
- bundle install

test:
  script:
  - bundle exec rake spec
```

184
## Extended Docker configuration options
185

186
> Introduced in GitLab and GitLab Runner 9.4.
187

188 189
When configuring the `image` or `services` entries, you can use a string or a map as
options:
190

191 192 193 194 195
- when using a string as an option, it must be the full name of the image to use
  (including the Registry part if you want to download the image from a Registry
  other than Docker Hub)
- when using a map as an option, then it must contain at least the `name`
  option, which is the same name of the image as used for the string setting
196

197
For example, the following two definitions are equal:
198

199
1. Using a string as an option to `image` and `services`:
200

201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219
    ```yaml
    image: "registry.example.com/my/image:latest"

    services:
    - postgresql:9.4
    - redis:latest
    ```

1. Using a map as an option to `image` and `services`. The use of `image:name` is
   required:

    ```yaml
    image:
      name: "registry.example.com/my/image:latest"

    services:
    - name: postgresql:9.4
    - name: redis:latest
    ```
220 221 222

### Available settings for `image`

223
> Introduced in GitLab and GitLab Runner 9.4.
224

225 226 227 228
| Setting    | Required | GitLab version | Description |
|------------|----------|----------------| ----------- |
| `name`     | yes, when used with any other option      | 9.4 |Full name of the image that should be used. It should contain the Registry part if needed. |
| `entrypoint` | no     | 9.4 |Command or script that should be executed as the container's entrypoint. It will be translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`][entrypoint] directive, where each shell token is a separate string in the array. |
229 230

### Available settings for `services`
231

232
> Introduced in GitLab and GitLab Runner 9.4.
233

234 235 236 237 238 239
| Setting    | Required | GitLab version | Description |
|------------|----------|----------------| ----------- |
| `name`       | yes, when used with any other option  | 9.4 | Full name of the image that should be used. It should contain the Registry part if needed. |
| `entrypoint` | no     | 9.4 |Command or script that should be executed as the container's entrypoint. It will be translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`][entrypoint] directive, where each shell token is a separate string in the array. |
| `command`    | no       | 9.4 |Command or script that should be used as the container's command. It will be translated to arguments passed to Docker after the image's name. The syntax is similar to [`Dockerfile`'s `CMD`][cmd] directive, where each shell token is a separate string in the array. |
| `alias`      | no       | 9.4 |Additional alias that can be used to access the service from the job's container. Read [Accessing the services](#accessing-the-services) for more information. |
240

241
### Starting multiple services from the same image
242

243 244 245
> Introduced in GitLab and GitLab Runner 9.4. Read more about the [extended
configuration options](#extended-docker-configuration-options).

246 247
Before the new extended Docker configuration options, the following configuration
would not work properly:
248 249 250 251 252 253 254

```yaml
services:
- mysql:latest
- mysql:latest
```

255 256 257 258
The Runner would start two containers using the `mysql:latest` image, but both
of them would be added to the job's container with the `mysql` alias based on
the [default hostname naming](#accessing-the-services). This would end with one
of the services not being accessible.
259

260 261
After the new extended Docker configuration options, the above example would
look like:
262 263 264 265 266 267

```yaml
services:
- name: mysql:latest
  alias: mysql-1
- name: mysql:latest
268
  alias: mysql-2
269 270
```

271 272
The Runner will still start two containers using the `mysql:latest` image,
but now each of them will also be accessible with the alias configured
273 274
in `.gitlab-ci.yml` file.

275
### Setting a command for the service
276

277 278 279
> Introduced in GitLab and GitLab Runner 9.4. Read more about the [extended
configuration options](#extended-docker-configuration-options).

280 281 282 283 284
Let's assume you have a `super/sql:latest` image with some SQL database
inside it and you would like to use it as a service for your job. Let's also
assume that this image doesn't start the database process while starting
the container and the user needs to manually use `/usr/bin/super-sql run` as
a command to start the database.
285

286 287 288
Before the new extended Docker configuration options, you would need to create
your own image based on the `super/sql:latest` image, add the default command,
and then use it in job's configuration, like:
289 290 291

```Dockerfile
# my-super-sql:latest image's Dockerfile
292

293 294 295 296 297 298 299 300 301 302 303
FROM super/sql:latest
CMD ["/usr/bin/super-sql", "run"]
```

```yaml
# .gitlab-ci.yml

services:
- my-super-sql:latest
```

304 305
After the new extended Docker configuration options, you can now simply
set a `command` in `.gitlab-ci.yml`, like:
306 307 308 309 310 311

```yaml
# .gitlab-ci.yml

services:
- name: super/sql:latest
312
  command: ["/usr/bin/super-sql", "run"]
313 314
```

315
As you can see, the syntax of `command` is similar to [Dockerfile's `CMD`][cmd].
316

317
### Overriding the entrypoint of an image
318

319 320 321
> Introduced in GitLab and GitLab Runner 9.4. Read more about the [extended
configuration options](#extended-docker-configuration-options).

322 323 324 325 326 327 328
Let's assume you have a `super/sql:experimental` image with some SQL database
inside it and you would like to use it as a base image for your job because you
want to execute some tests with this database binary. Let's also assume that
this image is configured with `/usr/bin/super-sql run` as an entrypoint. That
means, that when starting the container without additional options, it will run
the database's process, while Runner expects that the image will have no
entrypoint or at least will start with a shell as its entrypoint.
329

330 331 332 333
Before the new extended Docker configuration options, you would need to create
your own image based on the `super/sql:experimental` image, set the entrypoint
to a shell and then use it in job's configuration, like:

334 335
```Dockerfile
# my-super-sql:experimental image's Dockerfile
336

337 338 339 340 341 342 343 344 345 346
FROM super/sql:experimental
ENTRYPOINT ["/bin/sh"]
```

```yaml
# .gitlab-ci.yml

image: my-super-sql:experimental
```

347 348
After the new extended Docker configuration options, you can now simply
set an `entrypoint` in `.gitlab-ci.yml`, like:
349 350 351 352 353 354

```yaml
# .gitlab-ci.yml

image:
  name: super/sql:experimental
355 356 357
  entrypoint: ["/bin/sh"]
```

358 359
As you can see the syntax of `entrypoint` is similar to
[Dockerfile's `ENTRYPOINT`][entrypoint].
360

361
## Define image and services in `config.toml`
362 363 364

Look for the `[runners.docker]` section:

365 366 367 368 369 370
```
[runners.docker]
  image = "ruby:2.1"
  services = ["mysql:latest", "postgres:latest"]
```

371
The image and services defined this way will be added to all job run by
372
that runner.
373

374
## Define an image from a private Container Registry
375

376 377 378 379 380 381 382 383 384 385 386
> **Notes:**
- This feature requires GitLab Runner **1.8** or higher
- For GitLab Runner versions **>= 0.6, <1.8** there was a partial
  support for using private registries, which required manual configuration
  of credentials on runner's host. We recommend to upgrade your Runner to
  at least version **1.8** if you want to use private registries.
- If the repository is private you need to authenticate your GitLab Runner in the
  registry. Learn more about how [GitLab Runner works in this case][runner-priv-reg].

As an example, let's assume that you want to use the `registry.example.com/private/image:latest`
image which is private and requires you to login into a private container registry.
387 388 389 390 391 392 393 394 395

Let's also assume that these are the login credentials:

| Key      | Value                |
|----------|----------------------|
| registry | registry.example.com |
| username | my_username          |
| password | my_password          |

396 397
To configure access for `registry.example.com`, follow these steps:

398 399 400
1. Find what the value of `DOCKER_AUTH_CONFIG` should be. There are two ways to
   accomplish this:
     - **First way -** Do a `docker login` on your local machine:
401

402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419
         ```bash
         docker login registry.example.com --username my_username --password my_password
         ```

          Then copy the content of `~/.docker/config.json`.
     - **Second way -** In some setups, it's possible that Docker client will use
       the available system keystore to store the result of `docker login`. In
       that case, it's impossible to read `~/.docker/config.json`, so you will
       need to prepare the required base64-encoded version of
       `${username}:${password}` manually. Open a terminal and execute the
       following command:

           ```bash
           echo -n "my_username:my_password" | base64

           # Example output to copy
           bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ=
           ```
420 421 422 423 424 425 426 427 428 429 430 431 432 433

1. Create a [secret variable] `DOCKER_AUTH_CONFIG` with the content of the
   Docker configuration file as the value:

     ```json
     {
         "auths": {
             "registry.example.com": {
                 "auth": "bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ="
             }
         }
     }
     ```

434 435
1. Optionally,if you followed the first way of finding the `DOCKER_AUTH_CONFIG`
   value, do a `docker logout` on your computer if you don't need access to the
436 437 438 439 440 441 442
   registry from it:

     ```bash
     docker logout registry.example.com
     ```

1. You can now use any private image from `registry.example.com` defined in
443
   `image` and/or `services` in your `.gitlab-ci.yml` file:
444 445 446 447 448 449 450 451 452 453

      ```yaml
      image: my.registry.tld:5000/namespace/image:tag
      ```

      In the example above, GitLab Runner will look at `my.registry.tld:5000` for the
      image `namespace/image:tag`.

You can add configuration for as many registries as you want, adding more
registries to the `"auths"` hash as described above.
454

455
## Configuring services
456

457 458
Many services accept environment variables which allow you to easily change
database names or set account names depending on the environment.
459

460 461
GitLab Runner 0.5.0 and up passes all YAML-defined variables to the created
service containers.
462

463 464
For all possible configuration variables check the documentation of each image
provided in their corresponding Docker hub page.
465

466 467
*Note: All variables will be passed to all services containers. It's not
designed to distinguish which variable should go where.*
468

469
### PostgreSQL service example
470

471 472
See the specific documentation for
[using PostgreSQL as a service](../services/postgres.md).
473

474
### MySQL service example
475

476 477
See the specific documentation for
[using MySQL as a service](../services/mysql.md).
478

479
## How Docker integration works
480

481
Below is a high level overview of the steps performed by Docker during job
482 483
time.

484
1. Create any service container: `mysql`, `postgresql`, `mongodb`, `redis`.
485 486
1. Create cache container to store all volumes as defined in `config.toml` and
   `Dockerfile` of build image (`ruby:2.1` as in above example).
487
1. Create build container and link any service container to build container.
488 489
1. Start build container and send job script to the container.
1. Run job script.
490 491 492 493 494
1. Checkout code in: `/builds/group-name/project-name/`.
1. Run any step defined in `.gitlab-ci.yml`.
1. Check exit status of build script.
1. Remove build container and all created service containers.

495
## How to debug a job locally
496 497

*Note: The following commands are run without root privileges. You should be
498
able to run Docker with your regular user account.*
499

500
First start with creating a file named `build_script`:
501

502
```bash
503
cat <<EOF > build_script
504 505
git clone https://gitlab.com/gitlab-org/gitlab-runner.git /builds/gitlab-org/gitlab-runner
cd /builds/gitlab-org/gitlab-runner
506
make
507 508 509
EOF
```

510 511 512 513 514 515 516
Here we use as an example the GitLab Runner repository which contains a
Makefile, so running `make` will execute the commands defined in the Makefile.
Your mileage may vary, so instead of `make` you could run the command which
is specific to your project.

Then create some service containers:

517
```
518 519
docker run -d --name service-mysql mysql:latest
docker run -d --name service-postgres postgres:latest
520 521
```

522 523 524 525 526 527 528
This will create two service containers, named `service-mysql` and
`service-postgres` which use the latest MySQL and PostgreSQL images
respectively. They will both run in the background (`-d`).

Finally, create a build container by executing the `build_script` file we
created earlier:

529
```
530
docker run --name build -i --link=service-mysql:mysql --link=service-postgres:postgres ruby:2.1 /bin/bash < build_script
531 532
```

533 534 535 536 537 538 539 540
The above command will create a container named `build` that is spawned from
the `ruby:2.1` image and has two services linked to it. The `build_script` is
piped using STDIN to the bash interpreter which in turn executes the
`build_script` in the `build` container.

When you finish testing and no longer need the containers, you can remove them
with:

541 542 543
```
docker rm -f -v build service-mysql service-postgres
```
544 545 546 547 548

This will forcefully (`-f`) remove the `build` container, the two service
containers as well as all volumes (`-v`) that were created with the container
creation.

549
[Docker Fundamentals]: https://docs.docker.com/engine/understanding-docker/
550
[docker pull policy]: https://docs.gitlab.com/runner/executors/docker.html#how-pull-policies-work
551 552
[hub]: https://hub.docker.com/
[linking-containers]: https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/
553 554 555
[tutum/wordpress]: https://hub.docker.com/r/tutum/wordpress/
[postgres-hub]: https://hub.docker.com/r/_/postgres/
[mysql-hub]: https://hub.docker.com/r/_/mysql/
556 557
[runner-priv-reg]: http://docs.gitlab.com/runner/configuration/advanced-configuration.html#using-a-private-container-registry
[secret variable]: ../variables/README.md#secret-variables
558 559 560
[entrypoint]: https://docs.docker.com/engine/reference/builder/#entrypoint
[cmd]: https://docs.docker.com/engine/reference/builder/#cmd
[register]: https://docs.gitlab.com/runner/register/