BigW Consortium Gitlab

index.md 6.59 KB
Newer Older
1 2 3
# GitLab Prometheus

>**Notes:**
4
- Prometheus and the various exporters listed in this page are bundled in the
5 6 7
  Omnibus GitLab package. Check each exporter's documentation for the timeline
  they got added. For installations from source you will have to install them
  yourself. Over subsequent releases additional GitLab metrics will be captured.
8
- Prometheus services are on by default with GitLab 9.0.
9 10
- Prometheus and its exporters do not authenticate users, and will be available
  to anyone who can access them.
11 12 13 14 15 16 17 18 19

[Prometheus] is a powerful time-series monitoring service, providing a flexible
platform for monitoring GitLab and other software products.
GitLab provides out of the box monitoring with Prometheus, providing easy
access to high quality time-series monitoring of GitLab services.

## Overview

Prometheus works by periodically connecting to data sources and collecting their
20 21 22 23
performance metrics via the [various exporters](#prometheus-exporters). To view
and work with the monitoring data, you can either
[connect directly to Prometheus](#viewing-performance-metrics) or utilize a
dashboard tool like [Grafana].
24 25 26 27

## Configuring Prometheus

>**Note:**
28
For installations from source you'll have to install and configure it yourself.
29

30 31 32 33
Prometheus and it's exporters are on by default, starting with GitLab 9.0.
Prometheus will run as the `gitlab-prometheus` user and listen on
`http://localhost:9090`. Each exporter will be automatically be set up as a
monitoring target for Prometheus, unless individually disabled.
34 35

To disable Prometheus and all of its exporters, as well as any added in the future:
36 37

1. Edit `/etc/gitlab/gitlab.rb`
38
1. Add or find and uncomment the following line, making sure it's set to `false`:
39 40

    ```ruby
41
    prometheus_monitoring['enable'] = false
42 43 44 45 46
    ```

1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
   take effect

47 48 49 50 51 52 53 54 55 56 57
## Changing the port Prometheus listens on

>**Note:**
The following change was added in [GitLab Omnibus 8.17][1261]. Although possible,
it's not recommended to change the default address and port Prometheus listens
on as this might affect or conflict with other services running on the GitLab
server. Proceed at your own risk.

To change the address/port that Prometheus listens on:

1. Edit `/etc/gitlab/gitlab.rb`
58
1. Add or find and uncomment the following line:
59 60 61 62 63 64 65 66 67 68 69

    ```ruby
    prometheus['listen_address'] = 'localhost:9090'
    ```

    Replace `localhost:9090` with the address/port you want Prometheus to
    listen on.

1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
   take effect

70
## Viewing performance metrics
71

72
You can visit `http://localhost:9090` for the dashboard that Prometheus offers by default.
73 74 75 76 77 78

>**Note:**
If SSL has been enabled on your GitLab instance, you may not be able to access
Prometheus on the same browser as GitLab due to [HSTS][hsts]. We plan to
[provide access via GitLab][multi-user-prometheus], but in the interim there are
some workarounds: using a separate browser for Prometheus, resetting HSTS, or
79
having [Nginx proxy it][nginx-custom-config].
80 81 82 83 84 85 86 87

The performance data collected by Prometheus can be viewed directly in the
Prometheus console or through a compatible dashboard tool.
The Prometheus interface provides a [flexible query language][prom-query] to work
with the collected data where you can visualize their output.
For a more fully featured dashboard, Grafana can be used and has
[official support for Prometheus][prom-grafana].

88 89 90 91 92 93 94
Sample Prometheus queries:

- **% Memory used:** `(1 - ((node_memory_MemFree + node_memory_Cached) / node_memory_MemTotal)) * 100`
- **% CPU load:** `1 - rate(node_cpu{mode="idle"}[5m])`
- **Data transmitted:** `irate(node_network_transmit_bytes[5m])`
- **Data received:** `irate(node_network_receive_bytes[5m])`

95 96
## Configuring Prometheus to monitor Kubernetes

97
> Introduced in GitLab 9.0.
98

Joshua Lambert committed
99
If your GitLab server is running within Kubernetes, Prometheus will collect metrics from the Nodes in the cluster including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration][] to monitor them.
100

101
To disable the monitoring of Kubernetes:
102 103

1. Edit `/etc/gitlab/gitlab.rb`
104
1. Add or find and uncomment the following line and set it to `false`:
105 106

    ```ruby
107
    prometheus['monitor_kubernetes'] = false
108 109 110 111 112
    ```

1. Save the file and [reconfigure GitLab][reconfigure] for the changes to
   take effect

113 114 115 116 117 118
## Prometheus exporters

There are a number of libraries and servers which help in exporting existing
metrics from third-party systems as Prometheus metrics. This is useful for cases
where it is not feasible to instrument a given system with Prometheus metrics
directly (for example, HAProxy or Linux system stats). You can read more in the
119
[Prometheus exporters and integrations upstream documentation][prom-exporters].
120 121 122 123 124 125 126

While you can use any exporter you like with your GitLab installation, the
following ones documented here are bundled in the Omnibus GitLab packages
making it easy to configure and use.

### Node exporter

127
The node exporter allows you to measure various machine resources such as
128 129
memory, disk and CPU utilization.

130
[➔ Read more about the node exporter.](node_exporter.md)
131

132 133 134 135 136 137
### Redis exporter

The Redis exporter allows you to measure various Redis metrics.

[➔ Read more about the Redis exporter.](redis_exporter.md)

138 139 140 141 142 143
### Postgres exporter

The Postgres exporter allows you to measure various PostgreSQL metrics.

[➔ Read more about the Postgres exporter.](postgres_exporter.md)

144 145 146 147 148 149
### GitLab monitor exporter

The GitLab monitor exporter allows you to measure various GitLab metrics.

[➔ Read more about the GitLab monitor exporter.](gitlab_monitor_exporter.md)

150
[grafana]: https://grafana.net
151 152
[hsts]: https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security
[multi-user-prometheus]: https://gitlab.com/gitlab-org/multi-user-prometheus
153
[nginx-custom-config]: https://docs.gitlab.com/omnibus/settings/nginx.html#inserting-custom-nginx-settings-into-the-gitlab-server-block
154
[prometheus]: https://prometheus.io
155
[prom-exporters]: https://prometheus.io/docs/instrumenting/exporters/
156 157 158 159
[prom-query]: https://prometheus.io/docs/querying/basics
[prom-grafana]: https://prometheus.io/docs/visualization/grafana/
[scrape-config]: https://prometheus.io/docs/operating/configuration/#%3Cscrape_config%3E
[reconfigure]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure
160
[1261]: https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1261
161
[prometheus integration]: ../../../user/project/integrations/prometheus.md
162
[prometheus-cadvisor-metrics]: https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md