Add guide for debugging issues with the Docker container registry

doc/container_registry/README.md

@@ -90,6 +90,10 @@ your `.gitlab-ci.yml`, you have to follow the
 [Using a private Docker Registry][private-docker]
 documentation. This workflow will be simplified in the future.
 
+## Troubleshooting
+
+See [the GitLab Docker registry troubleshooting guide](troubleshooting.md).
+
 [ce-4040]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4040
 [docker-docs]: https://docs.docker.com/engine/userguide/intro/
 [private-docker]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#using-a-private-docker-registry

doc/container_registry/troubleshooting.md

+# Troubleshooting the GitLab Container Registry
+
+## Basic Troubleshooting
+
+1. Check to make sure that the system clock on your Docker client and GitLab server have
+   been synchronized (e.g. via NTP).
+
+2. If you are using an S3-backed registry, double check that the IAM
+   permissions and the S3 credentials (including region) are correct. See [the
+   sample IAM policy](https://docs.docker.com/registry/storage-drivers/s3/)
+   for more details.
+
+3. Check the registry logs (e.g. `/var/log/gitlab/registry/current`) and the GitLab production logs
+   for errors (e.g. `/var/log/gitlab/gitlab-rails/production.log`). You may be able to find clues
+   there.
+
+# Advanced Troubleshooting
+
+NOTE: The following section is only recommended for experts.
+
+Sometimes it's not obvious what is wrong, and you may need to dive deeper into
+the communication between the Docker client and the registry to find out
+what's wrong. We will use a concrete example in the past to illustrate how to
+diagnose a problem with the S3 setup.
+
+## Example: Unexpected 403 error during push
+
+A user attempted to enable an S3-backed registry. The `docker login` step went
+fine. However, when pushing an image, the output showed:
+
+```
+The push refers to a repository [s3-testing.myregistry.com:4567/root/docker-test]
+dc5e59c14160: Pushing [==================================================>] 14.85 kB
+03c20c1a019a: Pushing [==================================================>] 2.048 kB
+a08f14ef632e: Pushing [==================================================>] 2.048 kB
+228950524c88: Pushing 2.048 kB
+6a8ecde4cc03: Pushing [==>                                                ] 9.901 MB/205.7 MB
+5f70bf18a086: Pushing 1.024 kB
+737f40e80b7f: Waiting
+82b57dbc5385: Waiting
+19429b698a22: Waiting
+9436069b92a3: Waiting
+error parsing HTTP 403 response body: unexpected end of JSON input: ""
+```
+
+This error is ambiguous, as it's not clear whether the 403 is coming from the GitLab Rails
+application, the Docker registry, or something else. In this case, we know that since
+the login succeeded, we probably need to look at the communication between the client
+and the registry.
+
+The REST API between the Docker client and registry is [described
+here](https://docs.docker.com/registry/spec/api/). Normally, one would just
+use Wireshark or tcpdump to capture the traffic and see where things went
+wrong.  However, since all communication between Docker clients and servers
+are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even
+if you know the private key. What can we do instead?
+
+## mitmproxy
+
+Enter [mitmproxy](https://mitmproxy.org/). This tool allows you to place a
+proxy between your client and server to inspect all traffic. One wrinkle is
+that your system needs to trust the mitmproxy SSL certificates for this
+to work.
+
+The following installation instructions assume you are running Ubuntu:
+
+1. Install mitmproxy (see http://docs.mitmproxy.org/en/stable/install.html)
+
+2. Run `mitmproxy --port 9000` to generate its certificates. Enter CTRL-C to quit.
+
+3. Install the certificate from ~/.mitmproxy to your system:
+
+    ```sh
+    sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt
+    sudo update-ca-certificates
+    ```
+
+If successful, the output should indicate that a certificate was added:
+
+```sh
+Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.
+Running hooks in /etc/ca-certificates/update.d....done.
+```
+
+## Verifying mitmproxy certifiactes
+
+To verify that the certificates are properly install, run:
+
+```sh
+mitmproxy --port 9000
+```
+
+This will run mitmproxy on port 9000. In another window, run:
+
+```sh
+curl --proxy http://localhost:9000 https://httpbin.org/status/200
+```
+
+If everything is setup correctly, then you will see information on the mitmproxy window and
+no errors from the curl commands.
+
+## Running the Docker daemon with a proxy
+
+For Docker to connect through a proxy, you must start the Docker daemon with the
+proper environment variables. The easiest way is to shutdown Docker (e.g. `sudo initctl stop docker`)
+and then run Docker by hand. As root, run:
+
+```sh
+export HTTP_PROXY="http://localhost:9000"
+export HTTPS_PROXY="https://localhost:9000"
+docker daemon --debug
+```
+
+This will launch the Docker daemon and proxy all connections through mitmproxy.
+
+## Running the Docker client
+
+Now that we have mitmproxy and Docker running, we can now attempt to login and push a container
+image. You may need to run as root to do this. For example:
+
+```sh
+docker login s3-testing.myregistry.com:4567
+docker push s3-testing.myregistry.com:4567/root/docker-test
+```
+
+In the example above, we see the following trace on the mitmproxy window:
+
+![mitmproxy output from Docker](img/mitmproxy-docker.png)
+
+The above image shows:
+
+* The initial PUT requests went through fine with a 201 status code.
+* The 201 redirected the client to the S3 bucket.
+* The HEAD request to the AWS bucket reported a 403 Unauthorized.
+
+What does this mean? This strongly suggests that the S3 user does not have the right
+[permissions to perform a HEAD request](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html).
+The solution: check the [IAM permissions again](https://docs.docker.com/registry/storage-drivers/s3/).
+Once the right permissions were set, the error went away.