BigW Consortium Gitlab

Skip to content

G
gitlab-ce
Commit 66a2e7bd by Achilleas Pipinellis
Options

Refactor the Git submodules with CI docs

[ci skip]
parent 09806605
Showing with 96 additions and 110 deletions
doc/ci/README.md
......@@ -21,6 +21,7 @@
- [CI services (linked docker containers)](services/README.md)
- [CI/CD pipelines settings](../user/project/pipelines/settings.md)
- [Review Apps](review_apps/index.md)
- [Git submodules](git_submodules.md) Using Git submodules in your CI jobs
## Breaking changes
......
doc/ci/git_submodules.md 0 → 100644
# Using Git submodules with GitLab CI
> **Notes:**
- GitLab 8.12 introduced a new [CI build permissions model][newperms] and you
are encouraged to upgrade your GitLab instance if you haven't done already.
If you are **not** using GitLab 8.12 or higher, you would need to work your way
around submodules in order to access the sources of e.g., `gitlab.com/group/project`
with the use of [SSH keys](ssh_keys/README.md).
- With GitLab 8.12 onward, your permissions are used to evaluate what a CI build
can access. More information about how this system works can be found in the
[Build permissions model](../user/permissions.md#builds-permissions).
- The HTTP(S) Git protocol [must be enabled][gitpro] in your GitLab instance.
## Configuring the `.gitmodules` file
If dealing with [Git submodules][gitscm], your project will probably have a file
named `.gitmodules`.
Let's consider the following example:
1. Your project is located at `https://gitlab.com/secret-group/my-project`.
1. To checkout your sources you usually use an SSH address like
`git@gitlab.com:secret-group/my-project.git`.
1. Your project depends on `https://gitlab.com/group/project`, which you want
to include as a submodule.
If you are using GitLab 8.12+ and your submodule is on the same GitLab server,
you must update your `.gitmodules` file to use **relative URLs**.
Since Git allows the usage of relative URLs for your `.gitmodules` configuration,
this easily allows you to use HTTP(S) for cloning all your CI builds and SSH
for all your local checkouts. The `.gitmodules` would look like:
```ini
[submodule "project"]
path = project
url = ../../group/project.git
```
The above configuration will instruct Git to automatically deduce the URL that
should be used when cloning sources. Whether you use HTTP(S) or SSH, Git will use
that same channel and it will allow to make all your CI builds use HTTP(S)
(because GitLab CI only uses HTTP(S) for cloning your sources), and all your local
clones will continue using SSH.
For all other submodules not located on the same GitLab server, use the full
HTTP(S) protocol URL:
```ini
[submodule "project-x"]
path = project-x
url = https://gitserver.com/group/project-x.git
```
Once `.gitmodules` is correctly configured, you can move on to
[configuring your `.gitlab-ci.yml`](#using-git-submodules-in-your-ci-jobs).
## Using Git submodules in your CI jobs
There are a few steps you need to take in order to make submodules work
correctly with your CI builds:
1. First, make sure you have used [relative URLs](#configuring-the-gitmodules-file)
for the submodules located in the same GitLab server.
1. Then, use `git submodule sync/update` in `before_script`:
```yaml
before_script:
- git submodule sync --recursive
- git submodule update --init --recursive
```
`--recursive` should be used in either both or none (`sync/update`) depending on
whether you have recursive submodules.
The rationale to set the `sync` and `update` in `before_script` is because of
the way Git submodules work. On a fresh Runner workspace, Git will set the
submodule URL including the token in `.git/config`
(or `.git/modules/<submodule>/config`) based on `.gitmodules` and the current
remote URL. On subsequent builds on the same Runner, `.git/config` is cached
and already contains a full URL for the submodule, corresponding to the previous
build, and to **a token from a previous build**. `sync` allows to force updating
the full URL.
[gitpro]: ../user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols
[gitscm]: https://git-scm.com/book/en/v2/Git-Tools-Submodules "Git submodules documentation"
[newperms]: ../user/project/new_ci_build_permissions_model.md
doc/user/permissions.md
......@@ -141,10 +141,9 @@ instance and project. In addition, all admins can use the admin interface under
| See events in the system | | | | ✓ |
| Admin interface | | | | ✓ |
### Build permissions
> Changed in GitLab 8.12.
### Builds permissions
>**Note:**
GitLab 8.12 has a completely redesigned build permissions system.
Read all about the [new model and its implications][new-mod].
......
doc/user/project/new_ci_build_permissions_model.md
......@@ -34,9 +34,9 @@ as created be the pusher (local push or via the UI) and any build created in thi
pipeline will have the permissions of the pusher.
This allows us to make it really easy to evaluate the access for all projects
that have Git submodules or are using container images that the pusher would
have access too. **The permission is granted only for time that build is running.
The access is revoked after the build is finished.**
that have [Git submodules][gitsub] or are using container images that the pusher
would have access too. **The permission is granted only for time that build is
running. The access is revoked after the build is finished.**
## Types of users
......@@ -141,7 +141,7 @@ with GitLab 8.12.
With the new build permissions model, there is now an easy way to access all
dependent source code in a project. That way, we can:
1. Access a project's Git submodules
1. Access a project's [Git submodules][gitsub]
1. Access private container images
1. Access project's and submodule LFS objects
......@@ -179,108 +179,8 @@ As a user:
### Git submodules
>
It often happens that while working on one project, you need to use another
project from within it; perhaps it’s a library that a third party developed or
you’re developing a project separately and are using it in multiple parent
projects.
A common issue arises in these scenarios: you want to be able to treat the two
projects as separate yet still be able to use one from within the other.
>
_Excerpt from the [Git website][git-scm] about submodules._
If dealing with submodules, your project will probably have a file named
`.gitmodules`. And this is how it usually looks like:
```
[submodule "tools"]
path = tools
url = git@gitlab.com/group/tools.git
```
> **Note:**
If you are **not** using GitLab 8.12 or higher, you would need to work your way
around this issue in order to access the sources of `gitlab.com/group/tools`
(e.g., use [SSH keys](../ssh_keys/README.md)).
>
With GitLab 8.12 onward, your permissions are used to evaluate what a CI build
can access. More information about how this system works can be found in the
[Build permissions model](../../user/permissions.md#builds-permissions).
To make use of the new changes, you have to update your `.gitmodules` file to
use a relative URL.
Let's consider the following example:
1. Your project is located at `https://gitlab.com/secret-group/my-project`.
1. To checkout your sources you usually use an SSH address like
`git@gitlab.com:secret-group/my-project.git`.
1. Your project depends on `https://gitlab.com/group/tools`.
1. You have the `.gitmodules` file with above content.
Since Git allows the usage of relative URLs for your `.gitmodules` configuration,
this easily allows you to use HTTP for cloning all your CI builds and SSH
for all your local checkouts.
For example, if you change the `url` of your `tools` dependency, from
`git@gitlab.com/group/tools.git` to `../../group/tools.git`, this will instruct
Git to automatically deduce the URL that should be used when cloning sources.
Whether you use HTTP or SSH, Git will use that same channel and it will allow
to make all your CI builds use HTTPS (because GitLab CI uses HTTPS for cloning
your sources), and all your local clones will continue using SSH.
Given the above explanation, your `.gitmodules` file should eventually look
like this:
```
[submodule "tools"]
path = tools
url = ../../group/tools.git
```
However, you have to explicitly tell GitLab CI to clone your submodules as this
is not done automatically. You can achieve that by adding a `before_script`
section to your `.gitlab-ci.yml`:
```
before_script:
- git submodule update --init --recursive
test:
script:
- run-my-tests
```
This will make GitLab CI initialize (fetch) and update (checkout) all your
submodules recursively.
If Git does not use the newly added relative URLs but still uses your old URLs,
you might need to add `git submodule sync --recursive` to your `.gitlab-ci.yml`,
prior to running `git submodule update --init --recursive`. This transfers the
changes from your `.gitmodules` file into the `.git` folder, which is kept by
runners between runs.
In case your environment or your Docker image doesn't have Git installed,
you have to either ask your Administrator or install the missing dependency
yourself:
```
# Debian / Ubuntu
before_script:
- apt-get update -y
- apt-get install -y git-core
- git submodule update --init --recursive
# CentOS / RedHat
before_script:
- yum install git
- git submodule update --init --recursive
# Alpine
before_script:
- apk add -U git
- git submodule update --init --recursive
```
To properly configure submodules with GitLab CI, read the
[Git submodules documentation][gitsub].
### Container Registry
......@@ -310,7 +210,7 @@ test:
[build permissions]: ../permissions.md#builds-permissions
[comment]: https://gitlab.com/gitlab-org/gitlab-ce/issues/22484#note_16648302
[ext]: ../permissions.md#external-users
[git-scm]: https://git-scm.com/book/en/v2/Git-Tools-Submodules
[gitsub]: ../../ci/git_submodules.md
[https]: ../admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols
[triggers]: ../../ci/triggers/README.md
[update-docs]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment