BigW Consortium Gitlab

manage_large_binaries_with_git_lfs.md 6.37 KB
Newer Older
Marin Jankovski committed
1 2
# Git LFS

3 4 5
Managing large files such as audio, video and graphics files has always been one
of the shortcomings of Git. The general recommendation is to not have Git repositories
larger than 1GB to preserve performance.
Marin Jankovski committed
6 7 8

## How it works

9 10 11
Git LFS client talks with the GitLab server over HTTPS. It uses HTTP Basic Authentication
to authorize client requests. Once the request is authorized, Git LFS client receives
instructions from where to fetch or where to push the large file.
Marin Jankovski committed
12

13
## GitLab server configuration
Marin Jankovski committed
14

15
Documentation for GitLab instance administrators is under [LFS administration doc](lfs_administration.md).
Marin Jankovski committed
16

17
## Requirements
Marin Jankovski committed
18

19
* Git LFS is supported in GitLab starting with version 8.2
20
* Git LFS must be enabled under project settings
21
* [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up
Marin Jankovski committed
22

23
## Known limitations
Marin Jankovski committed
24

25 26
* Git LFS v1 original API is not supported since it was deprecated early in LFS
  development
Marin Jankovski committed
27
* When SSH is set as a remote, Git LFS objects still go through HTTPS
28
* Any Git LFS request will ask for HTTPS credentials to be provided so a good Git
29 30
  credentials store is recommended
* Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have
31
  to add the URL to Git config manually (see [troubleshooting](#troubleshooting))
32 33 34 35
  
>**Note**: With 8.12 GitLab added LFS support to SSH. The Git LFS communication
 still goes over HTTP, but now the SSH client passes the correct credentials
 to the Git LFS client, so no action is required by the user.
Marin Jankovski committed
36 37 38

## Using Git LFS

39 40 41
Lets take a look at the workflow when you need to check large files into your Git
repository with Git LFS. For example, if you want to upload a very large file and
check it into your Git repository:
Marin Jankovski committed
42 43 44

```bash
git clone git@gitlab.example.com:group/project.git
45
git lfs install                       # initialize the Git LFS project
Marin Jankovski committed
46
git lfs track "*.iso"                 # select the file extensions that you want to treat as large files
47 48
```

49 50
Once a certain file extension is marked for tracking as a LFS object you can use
Git as usual without having to redo the command to track a file with the same extension:
51 52

```bash
Marin Jankovski committed
53
cp ~/tmp/debian.iso ./                # copy a large file into the current directory
54
git add .                             # add the large file to the project
Marin Jankovski committed
55 56 57 58
git commit -am "Added Debian iso"     # commit the file meta data
git push origin master                # sync the git repo and large file to the GitLab server
```

59
>**Note**: Make sure that `.gitattributes` is tracked by git. Otherwise Git
60 61 62 63 64
 LFS will not be working properly for people cloning the project.
 ```bash
 git add .gitattributes
 ```

65 66 67 68
Cloning the repository works the same as before. Git automatically detects the
LFS-tracked files and clones them via HTTP. If you performed the git clone
command with a SSH URL, you have to enter your GitLab credentials for HTTP
authentication.
Marin Jankovski committed
69 70 71 72 73

```bash
git clone git@gitlab.example.com:group/project.git
```

74 75
If you already cloned the repository and you want to get the latest LFS object
that are on the remote repository, eg. from branch `master`:
76 77 78 79

```bash
git lfs fetch master
```
Marin Jankovski committed
80

81
## Troubleshooting
Marin Jankovski committed
82 83 84

### error: Repository or object not found

85
There are a couple of reasons why this error can occur:
Marin Jankovski committed
86

87
* You don't have permissions to access certain LFS object
88

89
Check if you have permissions to push to the project or fetch from the project.
90

91 92
* Project is not allowed to access the LFS object

93 94
LFS object you are trying to push to the project or fetch from the project is not
available to the project anymore. Probably the object was removed from the server.
95

96
* Local git repository is using deprecated LFS API
Marin Jankovski committed
97

98
### Invalid status for `<url>` : 501
Marin Jankovski committed
99

100 101 102 103 104 105 106 107 108
Git LFS will log the failures into a log file.
To view this log file, while in project directory:

```bash
git lfs logs last
```

If the status `error 501` is shown, it is because:

109 110 111
* Git LFS is not enabled in project settings. Check your project settings and
  enable Git LFS.

112 113 114 115
* Git LFS support is not enabled on the GitLab server. Check with your GitLab
  administrator why Git LFS is not enabled on the server. See
  [LFS administration documentation](lfs_administration.md) for instructions
  on how to enable LFS support.
116

117 118 119 120 121
* Git LFS client version is not supported by GitLab server. Check your Git LFS
  version with `git lfs version`. Check the Git config of the project for traces
  of deprecated API with `git lfs -l`. If `batch = false` is set in the config,
  remove the line and try to update your Git LFS client. Only version 1.0.1 and
  newer are supported.
Marin Jankovski committed
122 123 124

### getsockopt: connection refused

125 126 127 128
If you push a LFS object to a project and you receive an error similar to:
`Post <URL>/info/lfs/objects/batch: dial tcp IP: getsockopt: connection refused`,
the LFS client is trying to reach GitLab through HTTPS. However, your GitLab
instance is being served on HTTP.
129

130 131
This behaviour is caused by Git LFS using HTTPS connections by default when a
`lfsurl` is not set in the Git config.
Marin Jankovski committed
132

133
To prevent this from happening, set the lfs url in project Git config:
Marin Jankovski committed
134 135

```bash
136
git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs"
Marin Jankovski committed
137 138 139 140
```

### Credentials are always required when pushing an object

141 142 143 144
>**Note**: With 8.12 GitLab added LFS support to SSH. The Git LFS communication
 still goes over HTTP, but now the SSH client passes the correct credentials
 to the Git LFS client, so no action is required by the user.

145 146
Given that Git LFS uses HTTP Basic Authentication to authenticate the user pushing
the LFS object on every push for every object, user HTTPS credentials are required.
Marin Jankovski committed
147

148 149
By default, Git has support for remembering the credentials for each repository
you use. This is described in [Git credentials man pages](https://git-scm.com/docs/gitcredentials).
Marin Jankovski committed
150

151 152
For example, you can tell Git to remember the password for a period of time in
which you expect to push the objects:
Marin Jankovski committed
153 154 155 156 157

```bash
git config --global credential.helper 'cache --timeout=3600'
```

158 159
This will remember the credentials for an hour after which Git operations will
require re-authentication.
Marin Jankovski committed
160

161 162
If you are using OS X you can use `osxkeychain` to store and encrypt your credentials.
For Windows, you can use `wincred` or Microsoft's [Git Credential Manager for Windows](https://github.com/Microsoft/Git-Credential-Manager-for-Windows/releases).
Marin Jankovski committed
163

164
More details about various methods of storing the user credentials can be found
165
on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage).