Introduction to GitLab Pipeline

What is GitlLab ?

 

GitLab is an open-source code repository and collaborative software development platform for large DevOps and DevSecOps projects. GitLab is free for individuals.

GitLab offers a location for online code storage and capabilities for issue tracking and CI/CD. The repository enables hosting different development chains and versions and allows users to inspect previous code and roll back to it in the event of unforeseen problems.

One of the most rudimentary work for a DevOps Engineer is to create a pipeline. Today we will explain what is GitLab pipeline? why it is used and as a bonus, we will explain it with an example.

 

What is Pipeline?

 

Pipelines are the top-level component of continuous integration, delivery, and deployment.

Pipelines comprise:

• Jobs, which define what to do. For example, jobs that compile or test code.

• Stages, which define when to run the jobs. For example, stages that run tests after stages that compile the code.

Jobs are executed by runners. Multiple jobs in the same stage are executed in parallel, if there are enough concurrent runners.

If all jobs in a stage succeed, the pipeline moves on to the next stage.

If any job in a stage fails, the next stage is not (usually) executed and the pipeline ends early.

 

There are typically 4 stages to GitLab Pipeline.

 

A build stage with a job called compile

A test stage with two jobs called test1 and test2

A staging stage with a job called deploy-to-stage

A production stage with a job called deploy-to-prod

 

Why use GitLab Pipeline?

 

In a world full of automation, our code deployment should also be automated. In order to achieve that we use GitLab Pipeline. It helps developers to build application faster and better.

Now on to the bonus part.

We will create a GitLab pipeline to deploy in AWS ECS (Elastic Container Service).

FrontEnd: React

BackEnd: NodeJS

DataBase: DocumentDB (I..e MongoDB)

Infra Hosted: AWS ECS

Note: You need to have maintainer access for creating pipeline

 

1. The first step of creating a GitLab pipeline for a specific repository is to create a gitlab-ci.yml file.
2. Click on the + option to create a new file in your desired branch. You will need to name the file as “gitlab-ci.yml”
   
   
3. After that, you need to add your configurations and script to deploy it to ECS.
4. First, you need to mention an image that you’re going to use and then you need to add the variables that you will be using.
5. All the crucial parameters of ECS should be mentioned in the variables. You will also need to mention the AWS Access key and AWS Secret key of your AWS account. The AWS access key and secret key of the user used should have suitable IAM policies for deploying to ECS.
6. The next step is to add the before script.
7. In before script, you will need to mention the command to install awscli and also .env file variables.Note: .env file is used to store database URLs. We have created a blank .env file in the same branch.
8. We are not saving the database URLs in the .env file on our repo. We will export them in “before script” method and then add them to our .env file. It will help us maintain security.
   

   before_script: 
     - apk add --no-cache python3 py3-pip 
     - pip install awscli 
     - IMAGE_TAG="$CI_COMMIT_SHORT_SHA" 
     - export DATABASE_URL=$DATABASE_URL 
     - export MONGO_URL=$MONGO_URL 
     - echo "Preparing .env file" 
     - echo DATABASE_URL=$DATABASE_URL >> .env - PORT=$PORT >> .env

9. Further, we need to mention the stages in our gitlab-ci.yml file. We are using 2 stages: build and deploy.

   stages: 
     - build 
     - deploy

10. In the build section, we need to create a docker image and save the image in Elastic Container Registry in AWS. In order to create a docker image, we will need a Dockerfile. We can get a basic Dockerfile structure on the internet. We just need to mention the port number which is used in ECS.
11. We will run all the commands needed to create a new image in ECR. After the ECR image is created our build stage will be completed.
12. We will then proceed to our deploy stage. In this stage, we will mention the script to add a new container definition, registering task definition, mention our task arn and the final step is to update the current service.
13. You might ask after all this process “How should I mention my desired branch” I want the pipeline to run on a specific branch.
14. In order to achieve that we have to mention the branch name at the end of every stage name.
15. We will use “only: -branch_name” to specify the branch.
16. These are the steps to create a pipeline that will deploy to ECS.

We are providing a code snippet of gitlab-ci.yml

 

image: docker:stable


# We need to mention all the variables we are going to use at the start of the file.
# These variables are mentioned in the CI/CD setting.

variables:
  REPOSITORY_URL: <REPOSITORY_URL>
  REGION: <AWS_DEFAULT_REGION>
  TASK_DEFINITION_NAME: <TASK_DEFINITION_NAME>
  CLUSTER_NAME: <CLUSTER_NAME>
  SERVICE_NAME: <SERVICE_NAME>
  CPU: <CPU>
  MEMORY: <MEMORY>
  TASKS_COUNT: <TASKS_COUNT>
  EXECUTION_ROLE_ARN: <EXECUTION_ROLE_ARN>
  AWS_ACCESS_KEY_ID: <AWS_ACCESS_KEY_ID>
  AWS_SECRET_ACCESS_KEY: <AWS_SECRET_ACCESS_KEY>


services:
  – name: docker:dind
    alias: docker

before_script:
  – apk add –no-cache python3 py3-pip
  – pip install awscli
  – IMAGE_TAG=”$CI_COMMIT_SHORT_SHA”
  – export DATABASE_URL=$DATABASE_URL
  – export MONGO_URL=$MONGO_URL
  – echo “Preparing .env file”
  – echo DATABASE_URL=$DATABASE_URL >> .env
  – echo PORT=$PORT >> .env

stages:
  – build
  – deploy


build:
  stage: build
  script:
    # All these commands are given in the ECR view push commands. You can just copy those commands are paste it here
    – aws ecr get-login-password –region us-east-1 | docker login –username AWS –password-stdin “AWS Account Number”.dkr.ecr.us-east-1.amazonaws.com
    – echo “Building image…”
    – docker build -t “ECR service name” . –no-cache
    – echo “Tagging image…”
    – IMAGE_TAG=”$CI_COMMIT_SHORT_SHA”
    – docker tag “ECR service name”:latest “AWS Account Number”.dkr.ecr.us-east-1.amazonaws.com/orderservice:latest
    – echo “Pushing image…”
    – docker push “AWS Account Number”.dkr.ecr.us-east-1.amazonaws.com/servicename:latest


only:
  – dev
  # In only, we are mentioning the specific branch name.

tags:
  – docker


deploy:
  stage: deploy
  script:
    – apk add –no-cache jq
    – TASK_DEFINITION=$(aws ecs describe-task-definition –task-definition “$TASK_DEFINITION_NAME” –region “${AWS_DEFAULT_REGION}”)
    – NEW_CONTAINER_DEFINTIION=$(echo $TASK_DEFINITION | jq –arg IMAGE “$REPOSITORY_URL:latest” ‘.taskDefinition.containerDefinitions[0].image = $IMAGE | .taskDefinition.containerDefinitions’)
    – echo “Registering new container definition…”
    – echo “aws ecs register-task-definition –region “${AWS_DEFAULT_REGION}” –family “${TASK_DEFINITION_NAME}” –container-definitions “${NEW_CONTAINER_DEFINTIION}” –cpu “${CPU}” –memory “${MEMORY}” –execution-role-arn “${EXECUTION_ROLE_ARN}” –network-mode “awsvpc” –requires-compatibilities “FARGATE””
    – aws ecs register-task-definition –region “${AWS_DEFAULT_REGION}” –family “${TASK_DEFINITION_NAME}” –container-definitions “${NEW_CONTAINER_DEFINTIION}” –cpu “${CPU}” –memory “${MEMORY}” –execution-role-arn “${EXECUTION_ROLE_ARN}” –network-mode “awsvpc” –requires-compatibilities “FARGATE” –task-role-arn “${EXECUTION_ROLE_ARN}”
    – TASK_ARN=$(aws ecs list-tasks –region “${AWS_DEFAULT_REGION}” –cluster “${CLUSTER_NAME}” –service-name “${SERVICE_NAME}”)
    – TASK_ID=$(echo ${TASK_ARN:72:36})
    – echo “Updating the service..”
    – echo “aws ecs update-service –region “${AWS_DEFAULT_REGION}” –cluster “${CLUSTER_NAME}” –service “${SERVICE_NAME}” –task-definition “${TASK_DEFINITION_NAME}” –desired-count “${TASKS_COUNT}””
    – aws ecs update-service –region “${AWS_DEFAULT_REGION}” –cluster “${CLUSTER_NAME}” –service “${SERVICE_NAME}” –task-definition “${TASK_DEFINITION_NAME}” –desired-count “${TASKS_COUNT}”
    – echo “Waiting until the previous task ${TASK_ID} is stopped…”

only:
  – dev

tags:
  – docker

 

Follow these steps and you can also create your own GitLab pipeline for deploying in the AWS ECS service.
We are hoping that this blog was useful for your project or to impress your boss ?

That’s all folks!!