## Running Composer and NPM scripts with deployment via SCP
# Running Composer and NPM scripts with deployment via SCP in GitLab CI/CD
This guide covers the building dependencies of a PHP project while compiling assets via an NPM script.
...
...
@@ -39,13 +39,13 @@ In this particular case, the `npm deploy` script is a Gulp script that does the
All these operations will put all files into a `build` folder, which is ready to be deployed to a live server.
### How to transfer files to a live server?
## How to transfer files to a live server
You have multiple options: rsync, scp, sftp and so on. For now, we will use scp.
To make this work, you need to add a GitLab Secret Variable (accessible on _gitlab.example/your-project-name/variables_). That variable will be called `STAGING_PRIVATE_KEY` and it's the **private** ssh key of your server.
#### Security tip
### Security tip
Create a user that has access **only** to the folder that needs to be updated!
...
...
@@ -69,7 +69,7 @@ In order, this means that:
And this is basically all you need in the `before_script` section.
## How to deploy things?
## How to deploy things
As we stated above, we need to deploy the `build` folder from the docker image to our server. To do so, we create a new job:
1.`only:dev` means that this build will run only when something is pushed to the `dev` branch. You can remove this block completely and have everything be ran on every push (but probably this is something you don't want)
2.`ssh-add ...` we will add that private key you added on the web UI to the docker container
...
...
@@ -99,7 +99,7 @@ stage_deploy:
What's the deal with the artifacts? We just tell GitLab CI to keep the `build` directory (later on, you can download that as needed).
#### Why we do it this way?
### Why we do it this way
If you're using this only for stage server, you could do this in two steps:
...
...
@@ -112,7 +112,7 @@ The problem is that there will be a small period of time when you won't have the
So we use so many steps because we want to make sure that at any given time we have a functional app in place.
## Where to go next?
## Where to go next
Since this was a WordPress project, I gave real life code snippets. Some ideas you can pursuit:
# Test and Deploy a python application with GitLab CI/CD
This example will guide you how to run tests in your Python application and deploy it automatically as Heroku application.
You can checkout the example [source](https://gitlab.com/ayufan/python-getting-started) and check [CI status](https://gitlab.com/ayufan/python-getting-started/builds?scope=all).
You can checkout the [example source](https://gitlab.com/ayufan/python-getting-started).
## Configure project
### Configure project
This is what the `.gitlab-ci.yml` file looks like for this project:
```yaml
test:
script:
...
...
@@ -41,21 +44,25 @@ This project has three jobs:
2.`staging` - used to automatically deploy staging environment every push to `master` branch
3.`production` - used to automatically deploy production environmnet for every created tag
### Store API keys
## Store API keys
You'll need to create two variables in `Project > Variables`:
1.`HEROKU_STAGING_API_KEY` - Heroku API key used to deploy staging app,
2.`HEROKU_PRODUCTION_API_KEY` - Heroku API key used to deploy production app.
Find your Heroku API key in [Manage Account](https://dashboard.heroku.com/account).
### Create Heroku application
## Create Heroku application
For each of your environments, you'll need to create a new Heroku application.
You can do this through the [Dashboard](https://dashboard.heroku.com/).
### Create runner
## Create Runner
First install [Docker Engine](https://docs.docker.com/installation/).
To build this project you also need to have [GitLab Runner](https://about.gitlab.com/gitlab-ci/#gitlab-runner).
To build this project you also need to have [GitLab Runner](https://docs.gitlab.com/runner).
You can use public runners available on `gitlab.com`, but you can register your own:
# Test and Deploy a ruby application with GitLab CI/CD
This example will guide you how to run tests in your Ruby on Rails application and deploy it automatically as Heroku application.
You can checkout the example [source](https://gitlab.com/ayufan/ruby-getting-started) and check [CI status](https://gitlab.com/ayufan/ruby-getting-started/builds?scope=all).
### Configure project
## Configure the project
This is what the `.gitlab-ci.yml` file looks like for this project:
```yaml
test:
script:
...
...
@@ -36,23 +39,28 @@ This project has three jobs:
2.`staging` - used to automatically deploy staging environment every push to `master` branch
3.`production` - used to automatically deploy production environment for every created tag
### Store API keys
You'll need to create two variables in `Project > Variables`:
## Store API keys
You'll need to create two variables in your project's **Settings > CI/CD > Variables**:
1.`HEROKU_STAGING_API_KEY` - Heroku API key used to deploy staging app,
2.`HEROKU_PRODUCTION_API_KEY` - Heroku API key used to deploy production app.
Find your Heroku API key in [Manage Account](https://dashboard.heroku.com/account).
### Create Heroku application
## Create Heroku application
For each of your environments, you'll need to create a new Heroku application.
You can do this through the [Dashboard](https://dashboard.heroku.com/).
### Create runner
## Create Runner
First install [Docker Engine](https://docs.docker.com/installation/).
To build this project you also need to have [GitLab Runner](https://about.gitlab.com/gitlab-ci/#gitlab-runner).
You can use public runners available on `gitlab.com`, but you can register your own:
With the command above, you create a runner that uses [ruby:2.2](https://hub.docker.com/r/_/ruby/) image and uses [postgres](https://hub.docker.com/r/_/postgres/) database.
With the command above, you create a Runner that uses [ruby:2.2](https://hub.docker.com/r/_/ruby/) image and uses [postgres](https://hub.docker.com/r/_/postgres/) database.
To access PostgreSQL database you need to connect to `host: postgres` as user `postgres` without password.
This example will guide you how to run tests in your Clojure application.
You can checkout the example [source](https://gitlab.com/dzaporozhets/clojure-web-application) and check [CI status](https://gitlab.com/dzaporozhets/clojure-web-application/builds?scope=all).
### Configure project
## Configure the project
This is what the `.gitlab-ci.yml` file looks like for this project:
...
...
@@ -23,13 +23,13 @@ before_script:
-lein deps
-lein migratus migrate
test:
script:
test:
script:
-lein test
```
In before script we install JRE and [Leiningen](http://leiningen.org/).
Sample project uses [migratus](https://github.com/yogthos/migratus) library to manage database migrations.
In before script we install JRE and [Leiningen](http://leiningen.org/).
Sample project uses [migratus](https://github.com/yogthos/migratus) library to manage database migrations.
So we added database migration as last step of `before_script` section
You can use public runners available on `gitlab.com` for testing your application with such configuration.
- Small to medium size organisations using GitLab CE
*[James Mackey](#james-mackey)
...
...
@@ -7,16 +8,16 @@
*[Karolina Plaskaty](#karolina-plaskaty)
- Using GitLab.com for personal/hobby projects
- Would like to use GitLab at work
- Working for a medium to large size organisation
- Working for a medium to large size organisation
<hr>
---
### Nazim Ramesh
## Nazim Ramesh
- Small to medium size organisations using GitLab CE
<imgsrc="img/nazim-ramesh.png"width="300px">
#### Demographics
### Demographics
-**Age**<br>32 years old
-**Location**<br>Germany
...
...
@@ -26,7 +27,7 @@
-**Frequently used programming languages**<br>JavaScript, SQL, PHP
-**Hobbies / interests**<br>Functional programming, open source, gaming, web development and web security.
#### Motivations
### Motivations
Nazim works for a software development company which currently hires around 80 people. When Nazim first joined the company, the engineering team were using Subversion (SVN) as their primary form of source control. However, Nazim felt SVN was not flexible enough to work with many feature branches and noticed that developers with less experience of source control struggled with the central-repository nature of SVN. Armed with a wishlist of features, Nazim began comparing source control tools. A search for “self-hosted Git server repository management” returned GitLab. In his own words, Nazim explains why he wanted the engineering team to start using GitLab:
>
...
...
@@ -39,48 +40,48 @@ In his role as a full-stack web developer, Nazim could recommend products that h
“The biggest challenge...why should we change anything at all from the status quo? We needed to switch from SVN to Git. They knew they needed to learn Git and a Git workflow...using Git was scary to my colleagues...they thought it was more complex than SVN to use.”
>
Undeterred, Nazim decided to migrate a couple of projects across to GitLab.
Undeterred, Nazim decided to migrate a couple of projects across to GitLab.
>
“Old SVN users couldn’t see the benefits of Git at first. It took a month or two to convince them.”
>
Slowly, by showing his colleagues how easy it was to use Git, the majority of the team’s projects were migrated to GitLab.
Slowly, by showing his colleagues how easy it was to use Git, the majority of the team’s projects were migrated to GitLab.
The engineering team have been using GitLab CE for around 2 years now. Nazim credits himself as being entirely responsible for his company’s decision to move to GitLab.
The engineering team have been using GitLab CE for around 2 years now. Nazim credits himself as being entirely responsible for his company’s decision to move to GitLab.
#### Frustrations
##### Adoption to GitLab has been slow
### Frustrations
#### Adoption to GitLab has been slow
Not only has the engineering team had to get to grips with Git, they’ve also had to adapt to using GitLab. Due to lack of training and existing skills in other tools, the full feature set of GitLab CE is not being utilised. Nazim sold GitLab to his manager as an ‘all in one’ tool which would replace multiple tools used within the company, thus saving costs. Nazim hasn’t had the time to integrate the legacy tools to GitLab and he’s struggling to convince his peers to change their habits.
##### Missing Features
#### Missing Features
Nazim’s company want GitLab to be able to do everything. There isn’t a large budget for software, so they’re selective about what tools are implemented. It needs to add real value to the company. In order for GitLab to be widely adopted and to meet the requirements of different roles within the company, it needs a host of features. When an individual within Nazim’s company wants to know if GitLab has a specific feature or does a particular thing, Nazim is the person to ask. He becomes the point of contact to investigate, build or sometimes just raise the feature request. Nazim gets frustrated when GitLab isn’t able to do what he or his colleagues need it to do.
##### Regressions and bugs
#### Regressions and bugs
Nazim often has to calm down his colleagues, when a release contains regressions or new bugs. As he puts it “every new version adds something awesome, but breaks something”. He feels that “old issues for "minor" annoyances get quickly buried in the mass of open issues and linger for a very long time. More generally, I have the feeling that GitLab focus on adding new functionalities, but overlook a bunch of annoying minor regressions or introduced bugs.” Due to limited resource and expertise within the team, not only is it difficult to remain up-to-date with the frequent release cycle, it’s also counterproductive to fix workflows every month.
##### Uses too much RAM and CPU
#### Uses too much RAM and CPU
>
“Memory usages mean that if we host it from a cloud based host like AWS, we spend almost as much on the instance as what we would pay GitHub”
>
##### UI/UX
#### UI/UX
GitLab’s interface initially attracted Nazim when he was comparing version control software. He thought it would help his less technical colleagues to adapt to using Git and perhaps, GitLab could be rolled out to other areas of the business, beyond engineering. However, using GitLab’s interface daily has left him frustrated at the lack of personalisation / control over his user experience. He’s also regularly lost in a maze of navigation. Whilst he acknowledges that GitLab listens to its users and that the interface is improving, he becomes annoyed when the changes are too progressive. “Too frequent UI changes. Most of them tend to turn out great after a few cycles of fixes, but the frequency is still far too high for me to feel comfortable to always stay on the current release.”
#### Goals
### Goals
* To convince his colleagues to fully adopt GitLab CE, thus improving workflow and collaboration.
* To use a feature rich version control platform that covers all stages of the development lifecycle, in order to reduce dependencies on other tools.
* To use an intuitive and stable product, so he can spend more time on his core job responsibilities and less time bug-fixing, guiding colleagues, etc.
<hr>
---
### James Mackey
## James Mackey
- Medium to large size organisations using CE or EE
- Small organisations using EE
<imgsrc="img/james-mackey.png"width="300px">
#### Demographics
### Demographics
-**Age**<br>36 years old
-**Location**<br>US
...
...
@@ -90,7 +91,7 @@ GitLab’s interface initially attracted Nazim when he was comparing version con
-**Frequently used programming languages**<br>JavaScript, SQL, Node.js, Java, PHP, Python
-**Hobbies / interests**<br>DevOps, open source, web development, science, automation and electronics.
#### Motivations
### Motivations
James works for a research company which currently hires around 800 staff. He began using GitLab.com back in 2013 for his own open source, hobby projects and loved “the simplicity of installation, administration and use”. After using GitLab for over a year, he began to wonder about using it at work. James explains:
>
...
...
@@ -99,7 +100,7 @@ James works for a research company which currently hires around 800 staff. He be
James and his colleagues also reviewed competitor products including GitHub Enterprise, but they found it “less innovative and with considerable costs...GitLab had the features we wanted at a much lower cost per head than GitHub”.
The company James works for provides employees with a discretionary budget to spend how they want on software, so James and his team decided to upgrade to EE.
The company James works for provides employees with a discretionary budget to spend how they want on software, so James and his team decided to upgrade to EE.
James feels partially responsible for his organisation’s decision to start using GitLab.
...
...
@@ -107,33 +108,33 @@ James feels partially responsible for his organisation’s decision to start usi
“It's still up to the teams themselves [to decide] which tools to use. We just had a great experience moving our daily development to GitLab, so other teams have followed the path or are thinking about switching.”
>
#### Frustrations
##### Third Party Integration
### Frustrations
#### Third Party Integration
Some of GitLab EE’s features are too basic, in particular, issues boards which do not have the level of reporting that James and his team need. Subsequently, they still need to use GitLab EE in conjunction with other tools, such as JIRA. Whilst James feels it isn’t essential for GitLab to meet all his needs (his company are happy for him to use, and pay for, multiple tools), he sometimes isn’t sure what is/isn’t possible with plugins and what level of custom development he and his team will need to do.
##### UX/UI
#### UX/UI
James and his team use CI quite heavily for several projects. Whilst they’ve welcomed improvements to the builds and pipelines interface, they still have some difficulty following build process on the different tabs under Pipelines. Some confusion has arisen from not knowing where to find different pieces of information or how to get to the next stages logs from the current stage’s log output screen. They feel more intuitive linking and flow may alleviate the problem. Generally, they feel GitLab’s navigation needs to reviewed and optimised.
##### Permissions
#### Permissions
>
“There is no granular control over user or group permissions. The permissions for a project are too tightly coupled to the permissions for Gitlab CI/build pipelines.”
>
#### Goals
### Goals
* To be able to integrate third party tools easily with GitLab EE and to create custom integrations and patches where needed.
* To use GitLab EE primarily for code hosting, merge requests, continuous integration and issue management. James and his team want to be able to understand and use these particular features easily.
* To able to share one instance of GitLab EE with multiple teams across the business. Advanced user management, the ability to separate permissions on different parts of the source code, etc are important to James.
<hr>
---
### Karolina Plaskaty
## Karolina Plaskaty
- Using GitLab.com for personal/hobby projects
- Would like to use GitLab at work
- Working for a medium to large size organisation
- Working for a medium to large size organisation
<imgsrc="img/karolina-plaskaty.png"width="300px">
#### Demographics
### Demographics
-**Age**<br>26 years old
-**Location**<br>UK
...
...
@@ -143,22 +144,22 @@ James and his team use CI quite heavily for several projects. Whilst they’ve w
-**Frequently used programming languages**<br>JavaScript and SQL
-**Hobbies / interests**<br>Web development, mobile development, UX, open source, gaming and travel.
#### Motivations
### Motivations
Karolina has been using GitLab.com for around a year. She roughly spends 8 hours every week programming, of that, 2 hours is spent contributing to open source projects. Karolina contributes to open source projects to gain programming experience and to give back to the community. She likes GitLab.com for its free private repositories and range of features which provide her with everything she needs for her personal projects. Karolina is also a massive fan of GitLab’s values and the fact that it isn’t a “behemoth of a company”. She explains that “displaying every single thing (doc, culture, assumptions, development...) in the open gives me greater confidence to choose Gitlab personally and to recommend it at work.” She’s also an avid reader of GitLab’s blog.
Karolina works for a software development company which currently hires around 500 people. Karolina would love to use GitLab at work but the company has used GitHub Enterprise for a number of years. She describes management at her company as “old fashioned” and explains that it’s “less of a technical issue and more of a cultural issue” to convince upper management to move to GitLab. Karolina is also relatively new to the company so she’s apprehensive about pushing too hard to change version control platforms.
#### Frustrations
##### Unable to use GitLab at work
### Frustrations
#### Unable to use GitLab at work
Karolina wants to use GitLab at work but isn’t sure how to approach the subject with management. In her current role, she doesn’t feel that she has the authority to request GitLab.
##### Performance
#### Performance
GitLab.com is frequently slow and unavailable. Karolina has also heard that GitLab is a “memory hog” which has deterred her from running GitLab on her own machine for just hobby / personal projects.
##### UX/UI
#### UX/UI
Karolina has an interest in UX and therefore has strong opinions about how GitLab should look and feel. She feels the interface is cluttered, “it has too many links/buttons” and the navigation “feels a bit weird sometimes. I get lost if I don’t pay attention.” As Karolina also enjoys contributing to open-source projects, it’s important to her that GitLab is well designed for public repositories, she doesn’t feel that GitLab currently achieves this.
#### Goals
### Goals
* To develop her programming experience and to learn from other developers.
* To contribute to both her own and other open source projects.
* To use a fast and intuitive version control platform.
\ No newline at end of file
* To use a fast and intuitive version control platform.
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/2.6-to-3.0.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/2.9-to-3.0.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/3.0-to-3.1.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/3.1-to-4.0.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/4.0-to-4.1.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/4.1-to-4.2.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/4.2-to-5.0.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/5.0-to-5.1.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/5.1-to-5.2.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/5.1-to-5.4.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/5.1-to-6.0.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/5.2-to-5.3.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/5.3-to-5.4.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/5.4-to-6.0.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/6.0-to-6.1.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/6.1-to-6.2.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/6.2-to-6.3.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/6.3-to-6.4.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/6.4-to-6.5.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/6.5-to-6.6.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/6.6-to-6.7.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/6.7-to-6.8.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/6.8-to-6.9.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/6.9-to-7.0.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/6.x-or-7.x-to-7.14.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/7.0-to-7.1.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/7.1-to-7.2.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/7.2-to-7.3.md) for the most up to date instructions.*
*Make sure you view this [upgrade guide from the `master` branch](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update/7.3-to-7.4.md) for the most up to date instructions.*
