Why Self-Host GitLab?

• Privacy: Keep your code and data on your own infrastructure.

• Customization: Tailor GitLab to your team’s needs.

• CI/CD: Automate builds, tests, and deployments with GitLab Runners.

Prerequisites

• A server with 8 vCPU, 7.2GB RAM, and 50GB+ disk space (e.g., Ubuntu 22.04).

• Root or sudo access.

• A domain name (e.g., git.example.com) or static IP.

Creating an EC2 server

1. Go to AWS console → EC2 → Instances

2. 

   Click on Launch Instance

3. 

   Fill the requirements

   • Name: GitLab Server

   • AMI: Ubuntu Server 24.04

   • 

     Instance type: t2.2xlarge

   • 

     Create a new key pair and save it

   • 

     Configure the Network settings

   • 

     Configure storage (set 50+ GiB of storage)

   • 

     Launch the Instance

     

     

Connect to the Gitlab Server using SSH

1. Copy the public IP of the Server

2. Open Your local terminal

3. Connect to the Instance via ssh command:

    ssh -i <path_of_key_pair_file> ubuntu@<public_ip>
   

   

4. Congrats! 🎉 Gitlab server is now connected to the local server via SSH

Step 1: Prepare the Server

Start by updating your server and installing dependencies:

sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install -y curl openssh-server ca-certificates tzdata perl

Optionally, install Postfix for email notifications:

sudo apt-get install -y postfix

Select Internet Site and enter your server’s domain or IP.

Step 2: Install Docker

Install Docker to run GitLab in a container:

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER

Log out and back in, then verify Docker:

docker --version

Log out and back in for the group change to take effect.

Step 3: Deploy GitLab

Create directories for persistent storage:

mkdir -p /srv/gitlab/config /srv/gitlab/logs /srv/gitlab/data

Run the GitLab container:

sudo docker run --detach \
  --hostname git.example.com \
  --publish 443:443 --publish 80:80 --publish 22:22 \
  --name gitlab \
  --restart always \
  --volume /srv/gitlab/config:/etc/gitlab \
  --volume /srv/gitlab/logs:/var/log/gitlab \
  --volume /srv/gitlab/data:/var/opt/gitlab \
  gitlab/gitlab-ce:latest

• Replace git.example.com with your domain or server IP.

• Wait 5-10 minutes for GitLab to initialize.

Check container status:

docker ps

Step 4: Secure the Instance

• Access GitLab: Navigate to http://<server-ip> or https://git.example.com.

• 

  Set a password for the root user when prompted.

• Log in: Use username root and the password you set.

Enable HTTPS by editing /srv/gitlab/config/gitlab.rb:

external_url 'https://git.example.com'
letsencrypt['enable'] = true
letsencrypt['contact_emails'] = ['your-email@example.com']

Reconfigure GitLab:

sudo docker exec -it gitlab gitlab-ctl reconfigure

Restrict access (optional): In the GitLab UI, go to Admin Area > Settings > General > Sign-up restrictions and disable sign-ups for a private setup.

Step 5: Set Up CI/CD with GitLab Runner

Install GitLab Runner:

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
sudo apt-get install -y gitlab-runner

Register the Runner:

• In GitLab UI, go to Settings > CI/CD > Runners to get the registration token and URL.

• 

  Register the runner:

sudo gitlab-runner register \
  --url https://git.example.com \
  --token <your-registration-token> \
  --description my-runner \
  --tag-list docker-runner \
  --executor docker \
  --docker-image alpine:latest

Verify Runner:

sudo gitlab-runner status

Test CI/CD with a Pipeline

Create a .gitlab-ci.yml file in your project to test the pipeline:

stages:
  - build
  - test

build:
  stage: build
  tags:
    - docker-runner
  script:
    - echo "Building the project..."
    - mkdir -p builds
    - touch builds/app.txt
  artifacts:
    paths:
      - builds/

test:
  stage: test
  tags:
    - docker-runner
  script:
    - echo "Running tests..."
    - test -f builds/app.txt

Push the file and check the pipeline in CI/CD > Pipelines

Conclusion

You now have a self-hosted GitLab instance with CI/CD, ready for private development and automation. Secure it with a firewall (ufw) and regular backups:

sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable

sudo docker exec -it gitlab gitlab-backup create

In today’s fast-paced software development world, automating the process of building, testing, and deploying applications is essential for delivering features quickly and reliably. That’s where CI/CD pipelines come into play.

In this project, I’ve built a complete, production-grade CI/CD pipeline using Jenkins, Kubernetes (EKS), and various open-source DevOps tools — all deployed on AWS. This setup not only automates the deployment of applications but also integrates monitoring (with Prometheus & Grafana) and secure ingress access with HTTPS using Nginx Ingress Controller and Cert-Manager.

Whether you're a DevOps beginner looking to understand real-world pipelines or a professional aiming to implement enterprise-grade CI/CD systems, this blog will walk you through every step — from infrastructure setup to continuous delivery and monitoring.

By the end of this guide, you'll have a fully functional, cloud-native CI/CD pipeline with all the essential components configured, deployed, and running on Kubernetes.

Architecture Diagram of the Project

📌 Source Code & Project Repositories

To keep things simple and organized, I’ve divided the entire project into three separate repositories, each focusing on a different part of the DevSecOps workflow:

🔧 Project Repository (CI Repo)

🚀 CD Repository

☁️ Infrastructure as Code (IaC) — Terraform for EKS

🔒 Configure AWS Security Group

A Security Group in AWS acts like a virtual firewall. It controls what kind of traffic can come into or go out of your EC2 instances or services—keeping your infrastructure secure.

For this project, we'll either create a new security group or update an existing one with the required rules.

📌 Essential Security Group Rules for Kubernetes Cluster

✅ Best Practices for Security Group Setup

• 🔐 Follow Least Privilege
  Only open the ports that your application actually needs. Avoid exposing everything “just in case.”

• 🛑 Restrict SSH Access (Port 22)
  Limit SSH access to your IP or admin IPs only. Never leave it open to the entire internet (0.0.0.0/0)—this is a big security risk. (I have done this for demo purpose only).

Create EC2 Instances for Required Tools

To run essential DevOps tools like Nexus, SonarQube, Jenkins, and manage infrastructure, you'll need to create four separate EC2 instances on AWS.

📋 What You’ll Be Creating:

🔧 Step 1: Launch EC2 Instances

1. Go to the AWS EC2 Dashboard

   

2. Click “Launch Instance”

   

⚙️ Step 2: Configure Instance

1. Set the number of instances to 4

   

2. AMI (Amazon Machine Image): Select the latest Ubuntu (e.g., Ubuntu 22.04 LTS)

   

3. Instance Type: Choose t2.large (2 vCPU, 8 GB RAM)

   

4. Key Pair: Select an existing key pair or create a new one to access your instances via SSH

   

5. Storage: Set root volume to at least 25 GB

   

6. Security Group: Use the security group you configured earlier (with necessary ports open)

   

7. Click Launch Instance

8. 

   Tags: Add a Name tag to each instance to identify them easily

🔗 Connecting to EC2 Instances via SSH

Once your EC2 instances are up and running, you can connect to them securely using SSH (Secure Shell) from your local terminal.

🧩 What You Need:

• The .pem file (private key) you downloaded or created while launching the EC2 instances

• The public IP address of each EC2 instance (you’ll find it in the EC2 dashboard)

💻 SSH Command

ssh -i <path-to-pem-file> ubuntu@<public-ip>

• Replace <path-to-pem-file> with the path to your .pem file

• Replace <public-ip> with the public IP of the instance you want to access

Repeat this for each instance:

• Nexus

• SonarQube

• Jenkins

• InfraServer

ssh -i <path-to-pem-file> ubuntu@<public-ip-of-SonarQube>
ssh -i <path-to-pem-file> ubuntu@<public-ip-of-Nexus>
ssh -i <path-to-pem-file> ubuntu@<public-ip-of-Jenkins>
ssh -i <path-to-pem-file> ubuntu@<public-ip-of-InfraServer>

Configure each server

To ensure your server is up to date, run the following command:

sudo apt update

This will refresh the package list and update any outdated software.

Configure the Infrastructure Server

Now, we need to make sure that the server has the necessary permissions to create resources on AWS.

1. Create an IAM Role in AWS:

   • Go to the AWS Management Console.

   • In the navigation bar, search for IAM and select Roles.

   • Click on Create role.

2. Set the Trusted Entity:

   • Trusted Entity Type: Select AWS service.

   • Use Case: Select EC2 (this allows EC2 instances to assume this role).

3. Attach Policies:

   • Click Next: Permissions.

   • In the search bar, search for AdministratorAccess.

   • Check the box next to AdministratorAccess to give the EC2 instance full permissions.

4. Assign a Role Name:

   • Choose a role name

   • Click Create role to finish creating the IAM role.

Attach the IAM Role to the EC2 Instance

Now that your IAM role is created, it’s time to attach it to your EC2 instance.

1. Go to the EC2 Dashboard in AWS.

2. Find the InfraServer instance and click on it to open the instance details.

3. Click Actions → Security → Modify IAM Role.

4. Under the IAM role section, select the role you just created.

   

5. Click Update IAM role to apply the changes.

InfraServer now has the necessary permissions to create AWS resources. 🚀

Install AWS CLI on the Infra Server

To manage AWS resources from your server, you need to install the AWS Command Line Interface (CLI).

1. Download AWS CLI: Run the following command to download the AWS CLI installation package:

    curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
   

2. Install unzip (if not already installed):

    sudo apt install unzip -y
   

3. Unzip the AWS CLI package:

    unzip awscliv2.zip
   

4. Install AWS CLI:

    sudo ./aws/install
   

Verify AWS CLI Installation

To ensure AWS CLI is installed correctly, run:

aws --version

This should display the installed version of AWS CLI.

Configure AWS CLI

Set the AWS region globally so that the AWS CLI knows where to create resources. Use the following command:

aws configure set region us-east-1

Replace us-east-1 with your preferred AWS region if needed.

Install Terraform

To use Terraform, follow these steps to install it on your InfraServer:

1. Update the server and install required dependencies:

    sudo apt-get update && sudo apt-get install -y gnupg software-properties-common
   

2. Download the HashiCorp GPG key:

    wget -O- https://apt.releases.hashicorp.com/gpg | \
    gpg --dearmor | \
    sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg > /dev/null
   

3. Verify the GPG key:

    gpg --no-default-keyring \
    --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg \
    --fingerprint
   

4. Add the HashiCorp repository:

    echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
    https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
    sudo tee /etc/apt/sources.list.d/hashicorp.list
   

5. Update the package list:

    sudo apt update
   

6. Install Terraform:

    sudo apt-get install terraform
   

Verify Terraform Installation

To confirm that Terraform is installed, run:

terraform -version

This will display the installed version of Terraform.

Clone the Infrastructure as Code (IaC) Repository

Clone GitHub repository to the InfraServer where the Terraform configuration files are stored.

1. Clone the repository:

    git clone https://github.com/praduman8435/EKS-Terraform.git
   

   

2. Navigate into the repository directory:

    cd EKS-Terraform
   

   

Create Resources on AWS Using Terraform

1. Initialize Terraform:

   Before applying the configuration, you need to initialize the Terraform working directory:

    terraform init
   

   

2. Check the Resources Terraform Will Create:

   Run the following command to see a preview of the resources Terraform will create:

    terraform plan
   

   

   

3. Apply the Terraform Configuration:

   Once you’re ready to create the resources, apply the configuration:

    terraform apply --auto-approve
   

   This command will automatically approve the changes without prompting for confirmation.

Now, sit back and relax for approximately 10 minutes as Terraform creates the resources in AWS. You can monitor the progress in the terminal.

Set Up the Jenkins Server

Now infrastructure is ready, let's set up Jenkins server. Jenkins will be the core tool for automating our CI/CD pipeline.

Step 1: Install Java

Jenkins requires Java to run. We’ll install OpenJDK 17 (a stable, widely-used version):

sudo apt install openjdk-17-jre-headless -y

Step 2: Install Jenkins

1. Add the Jenkins repository key:

    sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
    https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
   

2. Add the Jenkins repository to your system:

    echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
    https://pkg.jenkins.io/debian-stable binary/" | sudo tee \
    /etc/apt/sources.list.d/jenkins.list > /dev/null
   

3. Update your package list:

    sudo apt-get update
   

4. Install Jenkins:

    sudo apt-get install jenkins -y
   

Step 3: Access Jenkins Web UI

1. Get the Jenkins Server’s Public IP from the AWS EC2 dashboard.

2. In your browser, go to:

    http://<public-ip-of-your-Jenkins-server>:8080
   

   

3. Important: Make sure port 8080 is open in the security group attached to your Jenkins EC2 instance.

   • Go to EC2 → Security Groups → Edit inbound rules.

   • Add a rule to allow TCP traffic on port 8080 from your IP or anywhere (0.0.0.0/0) for testing.

Unlock Jenkins

1. On the Jenkins setup page, it will ask for the initial admin password.

2. Run this command on your server to get it:

    sudo cat /var/lib/jenkins/secrets/initialAdminPassword
   

   

3. Copy the password and paste it into the browser.

Step 5: Install Plugins & Create Admin User

1. Click "Install Suggested Plugins" when prompted.

   

   

2. Once the plugins are installed, create your admin user (username, password, email).

   

3. Click "Save and Continue".

   

4. Then click "Save and Finish".

🎉 Jenkins is Ready!

You’ve successfully installed and configured Jenkins! You can now start creating jobs and automating your CI/CD pipeline.

Set Up the SonarQube Server

SonarQube is a powerful tool for continuously inspecting code quality and security. In this step, we’ll install Docker and run SonarQube as a container on our server.

Step 1: Update the Server

Let’s start by updating the system packages:

sudo apt update

Step 2: Install Docker on the Server

To run SonarQube as a container, we first need Docker installed.

1. Install required dependencies:

    sudo apt-get install ca-certificates curl -y
   

2. Add Docker’s official GPG key:

    sudo install -m 0755 -d /etc/apt/keyrings
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    sudo chmod a+r /etc/apt/keyrings/docker.asc
   

3. Add the Docker repository:

    echo \
    "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
    https://download.docker.com/linux/ubuntu \
    $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
    sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
   

4. Update your package index again:

    sudo apt-get update
   

   

5. Install Docker Engine and tools:

    sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
   

Step 3: Enable Docker for Your User (Optional, but recommended)

To run Docker without sudo every time:

sudo usermod -aG docker $USER

⚠️ Important: You need to log out and log back in after running this command for the changes to take effect.

Step 4: Run SonarQube in a Docker Container

Now that Docker is ready, let’s launch SonarQube:

docker run -d --name sonarqube -p 9000:9000 sonarqube:lts

This command will:

• Download the latest LTS (Long-Term Support) version of SonarQube.

• Start it in a detached container.

• Expose it on port 9000.

Step 5: Access SonarQube Web UI

Open your browser and go to:

http://<public-ip-of-sonarqube>:9000

Make sure port 9000 is allowed in the EC2 security group.

Step 6: Login and Change Default Password

Use the default credentials to log in:

• Username: admin

• Password: admin

You’ll be prompted to change the password on your first login.

🎉 SonarQube is now up and running on your server!

Set Up the Nexus Server

Nexus is a repository manager where we can store and manage build artifacts like Docker images, Maven packages, and more. We’ll install and run Nexus in a Docker container.

Step 1: Update the System

Start by updating all the packages:

sudo apt update

Step 2: Install Docker

If Docker isn’t already installed on this server, follow these steps:

1. Install required packages:

    sudo apt-get install ca-certificates curl -y
   

2. Add Docker’s GPG key:

    sudo install -m 0755 -d /etc/apt/keyrings
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    sudo chmod a+r /etc/apt/keyrings/docker.asc
   

3. Add the Docker repository:

    echo \
    "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
    https://download.docker.com/linux/ubuntu \
    $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
    sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
   

4. Update package index:

    sudo apt-get update
   

   

5. Install Docker:

    sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
   

Step 3: Run Docker Without sudo (Optional)

To avoid typing sudo before every Docker command:

sudo usermod -aG docker $USER

🔁 Log out and log back in to apply this change.

Step 4: Run Nexus in a Docker Container

Now that Docker is ready, let’s launch the Nexus container:

docker run -d --name nexus -p 8081:8081 sonatype/nexus3:latest

• This runs Nexus in the background.

• The web interface will be available on port 8081.

Step 5: Access Nexus Web Interface

1. In your browser, go to:

    http://<public-ip-of-nexus>:8081
   

   

2. Make sure port 8081 is allowed in your EC2 instance’s Security Group.

Step 6: Retrieve the Admin Password

To sign in, you need the initial admin password which is stored inside the container.

Here’s how to get it:

1. Find the container ID:

    docker ps
   

2. Access the container shell:

    docker exec -it <container-id> /bin/bash
   

3. Print the password:

    cat /nexus-data/admin.password
   

4. Copy the password and go back to the Nexus UI.

Step 7: Login & Set a New Password

• Username: admin

• Password: (paste the password you retrieved)

After login, it will ask you to set a new admin password.

🎉 That’s it! Your Nexus repository manager is now ready to use.

Configure Jenkins Plugins and Docker

Now that Jenkins is up and running, let’s install the required plugins and set up Docker on the Jenkins server to enable full CI/CD functionality.

Step 1: Install Required Jenkins Plugins

1. Go to Jenkins Dashboard
   → Click Manage Jenkins
   → Click Manage Plugins
   → Go to the Available tab

   

2. Search and install the following plugins (you can select multiple at once):

   • Pipeline Stage View

   • Docker Pipeline

   • SonarQube Scanner

   • Config File Provider

   • Maven Integration

   • Pipeline Maven Integration

   • Kubernetes

   • Kubernetes CLI

   • Kubernetes Client API

   • Kubernetes Credentials

   • Kubernetes Credentials Provider

3. Click Install without restart and wait for all plugins to be installed.

   

4. Once done, restart Jenkins to apply all changes

Step 2: Install Docker on Jenkins Server

We’ll now install Docker so Jenkins jobs can build Docker images directly.

1. Update and install required packages:

    sudo apt-get update
    sudo apt-get install ca-certificates curl -y
   

2. Add Docker's GPG key:

    sudo install -m 0755 -d /etc/apt/keyrings
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    sudo chmod a+r /etc/apt/keyrings/docker.asc
   

3. Add the Docker repository:

    echo \
    "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
    https://download.docker.com/linux/ubuntu \
    $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
    sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
   

   

4. Install Docker:

    sudo apt-get update
    sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y
   

Step 3: Allow Jenkins User to Run Docker

To allow Jenkins to run Docker without needing sudo every time:

sudo usermod -aG docker $USER

🔁 Log out and log back in (or reboot) for the group change to take effect.

Step 4: Configure Maven and Sonar Scanner in Jenkins

1. Go to Jenkins Dashboard → Manage Jenkins → Global Tool Configuration

2. Scroll down to the Maven section:

   • Click Add Maven

   • Name it: maven3

     

   • Choose “Install automatically” (Jenkins will download it)

3. Scroll to the SonarQube Scanner section:

   • Click Add SonarQube Scanner

   • Name it: sonar-scanner

     

   • Enable “Install automatically”

4. Click Save or Apply to finish.

🎉 Done! Jenkins is now fully equipped with all the tools you need to build, analyze, and deploy your applications in a modern DevOps workflow.

Create and Configure Jenkins Pipeline

Step 1: Create a New Pipeline

1. Go to Jenkins Dashboard

2. Click New Item

3. Enter a name

4. Choose Pipeline as the item type

   

5. Click OK

6. Under Build Discarder:

   • Check Discard Old Builds

   • Set Max # of builds to keep = 3
     (Keeps Jenkins light and fast)

Step 2: Install Trivy on Jenkins Server

Trivy is used for container vulnerability scanning.

Run the following commands on your Jenkins server:

sudo apt-get install wget apt-transport-https gnupg lsb-release -y

wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | \
gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg > /dev/null

echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] \
https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | \
sudo tee -a /etc/apt/sources.list.d/trivy.list

sudo apt-get update
sudo apt-get install trivy -y

Check if Trivy is working:

trivy --version

Step 3: Add SonarQube Credentials in Jenkins

1. Go to SonarQube UI → Click on Administration → Security → Users

   

2. Generate a new token

   

Now, add the token in Jenkins:

1. Go to Jenkins Dashboard → Manage Jenkins → Credentials

   

2. Click on (global) → Add Credentials

   

3. Fill the form:

   • Kind: Secret Text

   • Secret: (Paste the token copied from SonarQube)

   • ID: sonar-token (We'll refer to this in the pipeline)

4. Click Create

   

Step 4: Configure SonarQube Server in Jenkins

1. Go to Jenkins Dashboard → Manage Jenkins → Configure System

2. Scroll to SonarQube servers section

   

3. Click Add SonarQube

4. Fill the details:

   • Name: sonar

   • Server Authentication Token: Choose sonar-token

   • Server URL: http://<public-ip-of-sonarqube>:9000

5. Click Save

Write Your Pipeline Script

You’re now ready to write your pipeline script under Pipeline → Pipeline Script section of the job.

pipeline {
    agent any

    tools{
        maven 'maven3'
    }
    environment {
        SCANNER_HOME= tool 'sonar-scanner'
    }

    stages {
        stage('Git Checkout') {
            steps {
                git branch: 'main', url: 'https://github.com/praduman8435/Capstone-Mega-DevOps-Project.git'
            }
        }
        stage('Compilation') {
            steps {
                sh 'mvn compile'
            }
        }
        stage('Testing') {
            steps {
                sh 'mvn test -DskipTests=true'
            }
        }
        stage('Trivy FS Scan') {
            steps {
                sh 'trivy fs --format table -o fs-report.html .'
            }
        }
        stage('Code Quality Analysis') {
            steps {
                withSonarQubeEnv('sonar') {
                    sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar-projectName=GCBank -Dsonar.projectKey=GCBank \
                            -Dsonar.java.binaries=target '''
                }
            }
        }
    }
}

Now, try to build pipeline and check till now everything works fine

Here, everything works as expected now move forward and add some more scripts inside pipeline

Implement Quality Gate Check

In SonarQube:

1. Go to: Administration → Configuration → Webhooks

   

   

2. Create New Webhook

   • Name: sonarqube-webhook

   • URL: http://<jenkins-public-ip>:8080/sonarqube-webhook/

• Click Create

This webhook will notify Jenkins once the SonarQube analysis is complete and the Quality Gate status is available.

Your Jenkins server should be publicly accessible (or at least reachable by the SonarQube server) on that webhook URL.

Update pom.xml with Nexus Repositories

In Nexus UI:

• Browse → Select maven-releases and maven-snapshots

  

• Copy both URLs (you’ll use them in pom.xml)

In your pom.xml: Search for <distributionManagement> block and update it like this:

<distributionManagement>
    <repository>
        <id>maven-releases</id>
        <url>http://<nexus-ip>:8081/repository/maven-releases/</url>
    </repository>
    <snapshotRepository>
        <id>maven-snapshots</id>
        <url>http://<nexus-ip>:8081/repository/maven-snapshots/</url>
    </snapshotRepository>
</distributionManagement>

💡 Don’t forget to:

• Replace <nexus-ip> with your actual Nexus IP

• Commit and push the change to GitHub

Configure Nexus Credentials in Jenkins via settings.xml

In Jenkins:

• Go to: Manage Jenkins → Managed Files → Add a new Config File

  

• Type: Global Maven settings.xml

• ID: Capstone

  

• click on next

• Now, Inside Content generated find the <servers> section

  

📄 In the <servers> section, add:

<servers>
  <server>
    <id>maven-releases</id>
    <username>admin</username>
    <password>heyitsme</password>
  </server>

  <server>
    <id>maven-snapshots</id>
    <username>admin</username>
    <password>heyitsme</password>
  </server>
</servers>

• Submit the changes

🔐 This ensures your Maven builds can authenticate with Nexus to deploy artifacts.

Add DockerHub Credentials in Jenkins

• Go to: Manage Jenkins → Credentials → Global → Add Credentials

  

  

• Kind: Username and Password

• ID: docker-cred

• Username: Your DockerHub username

• Password: Your DockerHub password

  

💡 You’ll reference this ID in your pipeline when logging in to DockerHub.

Add Jenkins User to Docker Group

Run this on Jenkins server:

sudo usermod -aG docker jenkins

Then restart the server or run:

sudo systemctl restart jenkins

So Jenkins can access Docker without sudo

Update the pipeline Script

pipeline {
    agent any

    tools{
        maven 'maven3'
    }
    environment {
        SCANNER_HOME= tool 'sonar-scanner'
        IMAGE_TAG= "v${BUILD_NUMBER}"
    }

    stages {
        stage('Git Checkout') {
            steps {
                git branch: 'main', url: 'https://github.com/praduman8435/Capstone-Mega-DevOps-Project.git'
            }
        }
        stage('Compilation') {
            steps {
                sh 'mvn compile'
            }
        }
        stage('Testing') {
            steps {
                sh 'mvn test -DskipTests=true'
            }
        }
        stage('Trivy FS Scan') {
            steps {
                sh 'trivy fs --format table -o fs-report.html .'
            }
        }
        stage('Code Quality Analysis') {
            steps {
                withSonarQubeEnv('sonar') {
                    sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar-projectName=GCBank -Dsonar.projectKey=GCBank \
                            -Dsonar.java.binaries=target '''
                }
            }
        }
        stage('Quality Gate Check'){
            steps{
                waitForQualityGate abortPipeline: false, credentialsId: 'sonar-token'
            }
        }
        stage('Build the Application'){
            steps{
                sh 'mvn package -DskipTests'
            }
        }
        stage('Push Artifacts to Nexus'){
            steps{
                withMaven(globalMavenSettingsConfig: 'Capstone', jdk: '', maven: 'maven3', mavenSettingsConfig: '', traceability: true) {
                    sh 'mvn clean deploy -DskipTests'
                }
            }
        }
        stage('Build & Tag Docker Image'){
            steps{
                script{
                    withDockerRegistry(credentialsId: 'docker-cred') {
                        sh 'docker build -t thepraduman/bankapp:$IMAGE_TAG .'
                    }
                }
            }
        }
        stage('Docker Image Scan') {
            steps {
                sh 'trivy image --format table -o image-report.html thepraduman/bankapp:$IMAGE_TAG'
            }
        }
        stage('Push Docker Image') {
            steps {
                script{
                    withDockerRegistry(credentialsId: 'docker-cred') {
                        sh 'docker push thepraduman/bankapp:$IMAGE_TAG'
                    }
                }
            }
        }
    }
}

Click on Build Now To check the pipeline successfully triggered or not

Automating Jenkins Pipeline Trigger with GitHub Webhook

Now that the CI pipeline is ready, let’s automate it so that it runs automatically every time new code is pushed to the GitHub repository. We’ll use the Generic Webhook Trigger plugin in Jenkins for this.

Install Generic Webhook Trigger Plugin

1. Go to Jenkins Dashboard → Manage Jenkins → Plugins

2. Under the Available plugins tab, search for Generic Webhook Trigger

   

3. Select it and click on Install

4. Restart Jenkins once the installation is complete

Configure Webhook Trigger in Your Pipeline

1. Go back to the Jenkins dashboard and open your pipeline job ( capstone_CI)

   

2. Click on Configure

3. Scroll down to the Build Triggers section and check the box for Generic Webhook Trigger

   

4. Under Post content parameters, add:

   • Variable: ref

   • Expression: $.ref

   • Content-Type: JSONPath

5. Add a token:

   • Token Name: capstone

6. (Optional) Add a filter to trigger the pipeline only for changes on the main branch:

   • Expression: refs/heads/main

   • Text: $ref

7. Click Save

Once saved, you’ll see a webhook URL under the token section, something like:

http://<your-jenkins-ip>:8080/generic-webhook-trigger/invoke?token=capstone

Configure GitHub Webhook

1. Go to your GitHub repository (the one used in the pipeline)

   

2. Click on Settings → Webhooks → Add Webhook

   

   

3. Fill out the form as follows:

   • Payload URL: Paste the webhook URL from Jenkins

   • Content Type: application/json

   • Leave the secret field blank (or add one and configure Jenkins accordingly)

   • Choose Just the push event

4. Click on Add Webhook

   

That’s it! 🎉 Now, every time a new commit is pushed to the main branch, Jenkins will automatically trigger the pipeline.

Setting Up CD Pipeline

With our CI pipeline automated, it’s time to set up the CD pipeline. The first step is ensuring we can update the Docker image tag in the Kubernetes deployment every time a new image is built by the CI pipeline.

We’ll start by granting Jenkins access to CD GitHub repository and setting up email notifications so that you receive updates when your pipeline fails or succeeds.

Add GitHub Credentials to Jenkins

This will allow Jenkins to clone or push changes to your GitHub CD repository.

1. Go to Jenkins Dashboard → Manage Jenkins → Credentials

2. Click on (global)

   

3. Click on Add Credentials

4. Fill in the following:

   • Kind: Username with password

   • Scope: Global

   • Username: Your GitHub username

   • Password: Your GitHub password or personal access token (recommended)

   • ID: github-cred

5. Click on Create

   

Configure Email Notifications in Jenkins

Generate Gmail App Password

To securely send emails from Jenkins, we’ll use a Gmail App Password instead of your actual Gmail password.

1. Log in to your Google account

2. Navigate to Security

3. Enable 2-Step Verification if it’s not already enabled

   

4. Scroll down to App Passwords

5. Generate a new app password:

   • App name: capstone

• Copy the generated token (you’ll need it in the next step)

Add Gmail Credentials to Jenkins

1. Go to Jenkins Dashboard → Manage Jenkins → Credentials

2. Click on (global) and then Add Credentials

3. Fill in the following:

   • Kind: Username with password

   • Scope: Global

   • Username: Your Gmail address

   • Password: The generated Gmail app password

   • ID: mail-cred

4. Click Create

Configure Jenkins Mail Server

1. Go to Manage Jenkins → Configure System

2. Scroll to Extended E-mail Notification and fill out:

   • SMTP Server: smtp.gmail.com

   • SMTP Port: 465

   • Credentials: Select mail-cred

   • Check Use SSL

3. Scroll to E-mail Notification section:

   • SMTP Server: smtp.gmail.com

   • Use SMTP Authentication: ✅

   • Username: Your Gmail address

   • Password: Your Gmail App Password (recently generated)

   • SMTP Port: 465

   • Use SSL: ✅

4. Click Save

🛡️ Make sure ports 465 and 587 are open in the Jenkins server’s security group to allow email traffic.

✅ Test the Email Setup

1. In the Extended E-mail Notification section, click on Test configuration by sending a test e-mail

2. Enter a recipient email address

3. Click Test Configuration

If everything is set up correctly, you should receive an email confirming that the Jenkins email notification system is working! 🎉

Add the Email notification script in the CI pipeline

Add this inside the pipeline but outside the stages block

post {
    always {
        script {
            def jobName = env.JOB_NAME
            def buildNumber = env.BUILD_NUMBER
            def pipelineStatus = currentBuild.result ?: 'UNKNOWN'
            def bannerColor = pipelineStatus.toUpperCase() == 'SUCCESS' ? 'green' : 'red'

            def body = """
                <html>
                    <body>
                        <div style="border: 4px solid ${bannerColor}; padding: 10px;">
                            <h2>${jobName} - Build #${buildNumber}</h2>
                            <div style="background-color: ${bannerColor}; padding: 10px;">
                                <h3 style="color: white;">Pipeline Status: ${pipelineStatus.toUpperCase()}</h3>
                            </div>
                            <p>Check the <a href="${env.BUILD_URL}">Console Output</a> for more details.</p>
                        </div>
                    </body>
                </html>
            """

            emailext(
                subject: "${jobName} - Build #${buildNumber} - ${pipelineStatus.toUpperCase()}",
                body: body,
                to: 'praduman.cnd@gmail.com',
                from: 'praduman.8435@gmail.com',
                replyTo: 'praduman.8435@gmail.com',
                mimeType: 'text/html',
                attachmentsPattern: 'fs-report.html'
            )
        }
    }
}

Now, The email successfully configured and sets up

🚀 Configure the InfraServer

Install Kubectl

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"

sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

kubectl version --client

Update the Kubeconfig File

Connect your Jenkins server to the EKS cluster:

aws eks update-kubeconfig \
  --region us-east-1 \
  --name capstone-cluster

Install eksctl

eksctl is a CLI tool that simplifies EKS cluster operations.

curl -sLO "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz"
tar -xzf eksctl_$(uname -s)_amd64.tar.gz
sudo mv eksctl /usr/local/bin

# Verify the installation
eksctl version

Install Helm

Helm is used for managing Kubernetes applications using Helm charts.

sudo apt update && sudo apt upgrade -y
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Associate IAM OIDC Provider with the Cluster

This step is needed to create service accounts with IAM roles.

eksctl utils associate-iam-oidc-provider \
  --cluster capstone-cluster \
  --region us-east-1 \
  --approve

Create IAM Service Account for EBS CSI Driver

This enables your cluster to dynamically provision EBS volumes.

eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster capstone-cluster \
  --region us-east-1 \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve \
  --override-existing-serviceaccounts

Deploy EBS CSI Driver

kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/ecr/?ref=release-1.30"

Install NGINX Ingress Controller

This is required for routing external traffic to your services inside Kubernetes:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/cloud/deploy.yaml

Install Cert-Manager (for TLS Certificates)

Cert-manager helps you manage SSL certificates inside Kubernetes:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.4/cert-manager.yaml

🔐 Configure RBAC (Role-Based Access Control)

To manage access control and permissions properly in your Kubernetes cluster, we’ll start by creating a dedicated namespace and then apply RBAC policies.

• Create a namespace with name webapps

    kubectl create ns webapps
  

  

• Create a Service Account

    vim service-account.yaml
  

  add the yaml file & save it

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: jenkins
      namespace: webapps
  

    kubectl apply -f service-account.yaml
  

• Create a Role

    vim role.yaml
  

  add the yaml file & save it

    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: jenkins-role
      namespace: webapps
    rules:
      - apiGroups:
            - ""
            - apps
            - networking.k8s.io
            - autoscaling
        resources:
          - secrets
          - configmaps
          - persistentvolumeclaims
          - services
          - pods
          - deployments
          - replicasets
          - ingresses
          - horizontalpodautoscalers
        verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  

    kubectl apply -f role.yaml
  

• Bind the role to service account

    vim role-binding.yaml
  

  add the yaml file & save it

    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: jenkins-rolebinding
      namespace: webapps 
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: jenkins-role 
    subjects:
    - namespace: webapps 
      kind: ServiceAccount
      name: jenkins
  

    kubectl apply -f role-binding.yaml
  

• Create Cluster role

    vim cluster-role.yaml
  

  add the yaml file & save it

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: jenkins-cluster-role
    rules:
    - apiGroups: [""]
      resources: 
         - persistentvolumes
      verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
    - apiGroups: ["storage.k8s.io"]
      resources: 
         - storageclasses
      verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
    - apiGroups: ["cert-manager.io"]
      resources: 
         - clusterissuers
      verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  

    kubectl apply -f cluster-role.yaml
  

• Bind cluster role to Service Account

    vim cluster-role-binding.yaml
  

  add the yaml file & save it

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: jenkins-cluster-rolebinding
    subjects:
    - kind: ServiceAccount
      name: jenkins
      namespace: webapps
    roleRef:
      kind: ClusterRole
      name: jenkins-cluster-role
      apiGroup: rbac.authorization.k8s.io
  

    kubectl apply -f cluster-role-binding.yaml
  

Grant Jenkins Access to Kubernetes for Deployments

To allow Jenkins to deploy applications to your EKS cluster, we need to create a service account token and configure it in Jenkins.

Create a Kubernetes Token for Jenkins

Create a secret token that Jenkins will use to authenticate with your Kubernetes cluster.

Run the following command to create and open a token manifest:

vim token.yaml

Paste the following YAML content into the file:

apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: jenkins-secret
  annotations:
    kubernetes.io/service-account.name: jenkins

Apply the secret to the webapps namespace:

kubectl apply -f token.yaml -n webapps

Now, retrieve the token using:

kubectl describe secret jenkins-secret -n webapps | grep token

Copy the generated token.

Add Kubernetes Token to Jenkins

1. Go to your Jenkins Dashboard → Manage Jenkins → Credentials.

   

2. Select the (global) domain and click Add Credentials.

   

3. Fill in the fields as follows:

   • Kind: Secret text

   • Secret: Paste the copied Kubernetes token

   • ID: k8s-cred

4. Click Create.

   

Install kubectl on Jenkins Server

To let Jenkins execute Kubernetes commands, install kubectl on the Jenkins machine:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"

sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Verify installation:

kubectl version --client

Setting Up the CD Pipeline in Jenkins

Now that CI is complete, let’s configure the Capstone Continuous Deployment (CD) Pipeline to automatically deploy your application to the Kubernetes cluster.

Step 1: Create a New Pipeline Job

1. Go to Jenkins Dashboard → New Item

   

2. Provide the following details:

   • Name: capstone_CD

   • Item Type: Select Pipeline

3. Click OK

Step 2: Configure Build Retention

1. Check the box "Discard old builds"

2. Set Max # of builds to keep to 3

   

   This helps conserve resources by keeping only the latest builds.

Step 3: Add the Deployment Pipeline Script

Scroll down to the Pipeline section and paste the following script:

groovyCopyEditpipeline {
    agent any

    stages {
        stage('Git Checkout') {
            steps {
                git branch: 'main', url: 'https://github.com/praduman8435/Capstone-Mega-CD-Pipeline.git'
            }  
        }

        stage('Deploy to Kubernetes') {
            steps {
                withKubeConfig(
                    credentialsId: 'k8s-cred',
                    clusterName: 'capstone-cluster',
                    namespace: 'webapps',
                    restrictKubeConfigAccess: false,
                    serverUrl: 'https://D133D06C5103AE18A950F2047A8EB7DE.gr7.us-east-1.eks.amazonaws.com'
                ) {
                    sh 'kubectl apply -f kubernetes/Manifest.yaml -n webapps'
                    sh 'kubectl apply -f kubernetes/HPA.yaml'
                    sleep 30
                    sh 'kubectl get pods -n webapps'
                    sh 'kubectl get svc -n webapps'
                }
            }  
        }
    }

    post {
        always {
            echo "Pipeline execution completed."
        }
    }
}

Step 4: Save & Run

1. Click Save

   

2. Click Build Now to trigger the pipeline

   

If everything is set up correctly, Jenkins will pull the deployment manifests from the CD GitHub repository and deploy your app to the webapps namespace in the EKS cluster.

Verifying Kubernetes Resources & Enabling HTTPS with Custom Domain

After setting up the CI/CD pipeline, it’s time to make sure everything is working perfectly and your application is accessible securely over HTTPS with a custom domain.

Step 1: Verify All Resources in the Cluster

On your Infra Server, run:

kubectl get all -n webapps

You should see all your application pods, services, and other resources running successfully. If everything looks good, proceed to the next step.

Step 2: Create a ClusterIssuer Resource for Let’s Encrypt

We'll use Cert-Manager to automatically provision SSL certificates from Let’s Encrypt.

1. Create a file called cluster-issuer.yaml:

vim cluster-issuer.yaml

2. Paste the following configuration:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: praduman.cnd@gmail.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

3. Apply the ClusterIssuer:

kubectl apply -f cluster-issuer.yaml

Step 3: Create an Ingress Resource for Your Application

1. Create an Ingress configuration file:

vim ingress.yaml

2. Paste the following content:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: bankapp-ingress
  namespace: webapps
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - www.capstonebankapp.in
    secretName: bankapp-tls-secret
  rules:
  - host: www.capstonebankapp.in
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: bankapp-service
            port:
              number: 80

3. Apply the ingress resource:

kubectl apply -f ingress.yaml

4. Check the status:

kubectl get ing -n webapps

Wait for a few moments, then run the command again. You’ll notice an external load balancer address (usually an AWS ELB) under the ADDRESS column.

Copy the bankapp ingress loadbalancer address

Step 4: Configure Your Custom Domain on GoDaddy

1. Log in to your GoDaddy account.

   

2. Navigate to My Products → DNS Settings for your domain (capstonebankapp.in).

3. Look for a CNAME record with name www and edit it.

   

4. In the Value field, paste the Load Balancer URL you got from the Ingress.

   

5. Save the changes.

⏳ Wait a few minutes for the DNS changes to propagate.

Step 5: Access Your Application

Open your browser and visit:

https://www.capstonebankapp.in/login

If everything was configured correctly, your application will now load securely over HTTPS with a valid Let’s Encrypt certificate.

Setup Monitoring with Prometheus & Grafana on EKS

After deploying your application, it's essential to monitor its health, performance, and resource usage. Let’s integrate Prometheus and Grafana into your Kubernetes cluster using Helm.

Add Prometheus Helm Repo

On your Infra Server, add the official Prometheus Community Helm chart repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Create values.yaml for Custom Configuration

We'll define how Prometheus and Grafana should be deployed, what metrics to scrape, and how to expose the services.

1. Create a file called values.yaml:

vi values.yaml

2. Paste the following configuration:

# values.yaml for kube-prometheus-stack

alertmanager:
  enabled: false

prometheus:
  prometheusSpec:
    service:
      type: LoadBalancer
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 5Gi
  additionalScrapeConfigs:
    - job_name: node-exporter
      static_configs:
        - targets:
            - node-exporter:9100
    - job_name: kube-state-metrics
      static_configs:
        - targets:
            - kube-state-metrics:8080

grafana:
  enabled: true
  service:
    type: LoadBalancer
  adminUser: admin
  adminPassword: admin123

prometheus-node-exporter:
  service:
    type: LoadBalancer

kube-state-metrics:
  enabled: true
  service:
    type: LoadBalancer

Save and exit the file.

Install Monitoring Stack with Helm

helm upgrade --install monitoring prometheus-community/kube-prometheus-stack -f values.yaml -n monitoring --create-namespace

Patch Services to Use LoadBalancer

(Optional if already configured in values.yaml, but ensures services are exposed)

kubectl patch svc monitoring-kube-prometheus-prometheus -n monitoring -p '{"spec": {"type": "LoadBalancer"}}'
kubectl patch svc monitoring-kube-state-metrics -n monitoring -p '{"spec": {"type": "LoadBalancer"}}'
kubectl patch svc monitoring-prometheus-node-exporter -n monitoring -p '{"spec": {"type": "LoadBalancer"}}'

Check Services & Access Grafana

Get all resources in the monitoring namespace:

kubectl get all -n monitoring
kubectl get svc -n monitoring

You’ll find External IPs assigned to services like Grafana and Prometheus.

➤ Access Grafana

• URL: http://<grafana-external-ip>

  

• Username: admin

• Password: admin123

➤ Access Prometheus

• URL: http://<prometheus-external-ip>:9090

  

• Go to Status → Targets to see what’s being monitored.

Configure Grafana Dashboard

1. Open the Grafana dashboard.

   

2. Go to Connections → Data Sources → Add new.

   

3. Search for Prometheus and select it.

4. In the URL, enter your Prometheus service URL (e.g., http://<prometheus-external-ip>:9090).

   

5. Click Save & Test.

   

🎯 View Dashboards

• Go to Dashboards → Browse.

  

• Explore default dashboards for Node Exporter, Kubernetes metrics, and more.

Now you have real-time observability into your EKS cluster!

✅ Conclusion

In this project, I've successfully built an enterprise-grade CI/CD pipeline from scratch using Jenkins, Kubernetes (EKS), Docker, GitHub, and other DevOps tools, all running on AWS.

I automated the entire workflow:

• From building and pushing Docker images in the CI pipeline

• To continuously deploying them into a secure, production-ready Kubernetes cluster via CD

• With proper ingress routing, TLS certificates, and monitoring in place using Prometheus and Grafana

This setup demonstrates how modern DevOps practices can streamline software delivery and infrastructure management. It not only improves deployment speed but also ensures reliability, scalability, and observability of applications.

💡 Let’s connect and discuss DevOps, cloud automation, and cutting-edge technology

🔗 LinkedIn | 💼 Upwork | 🐦 Twitter | 👨‍💻 GitHub

In today’s fast-paced software development world, automation is the key to delivering high-quality applications efficiently. That’s why enterprises need a robust, scalable, and secure CI/CD pipeline to streamline development, testing, and deployment.

This project is all about building the Ultimate Corporate-Grade CI/CD Pipeline—a fully automated, production-ready pipeline that follows DevSecOps best practices and is designed to work with Kubernetes on AWS.

What This Pipeline Covers?

✅ Kubernetes Cluster Setup – Using kubeadm to configure a highly available Kubernetes cluster on AWS EC2.
✅ Code Integration & Testing – Automating builds, tests, and security scans.
✅ Containerization & Orchestration – Using Docker and Kubernetes (EKS) for scalability.
✅ Automated Deployments – Implementing GitOps with ArgoCD for zero-downtime releases.
✅ Security & Compliance – Integrating Trivy, SonarQube, and Prometheus for DevSecOps.
✅ Observability & Monitoring – Setting up Grafana & Prometheus for real-time insights.

Why This Matters?

🔹 Enterprise-Grade Reliability – Built for scalability, high availability, and security.
🔹 Faster Releases – Automates everything from code commit to production deployment.
🔹 Cost-Effective – Optimized for AWS, reducing operational overhead.

By the end of this project, we’ll have a bulletproof CI/CD pipeline, complete with a Kubernetes cluster set up using kubeadm, ensuring a fully automated and secure software delivery process. Let’s dive in! 🚀

Source Code and Project Repository 📌

🚀 Setting Up Your Kubernetes Cluster with Kubeadm

🔒 Configure AWS Security Group

Before deploying Kubernetes on AWS, you need to configure security groups to control inbound and outbound traffic. Security groups act as a firewall, ensuring that only necessary connections are allowed while keeping your cluster protected.

You can either create a new security group or modify an existing one with the essential rules listed below:

📌 Essential Security Group Rules for Kubernetes

✅ Best Practices

• Follow Least Privilege: Only open the ports your cluster needs.

• Restrict SSH Access: Limit port 22 to specific IPs to enhance security.

• Secure API Server: Only allow trusted IPs for port 6443 to prevent unauthorized access.

Once your security groups are configured, you're now ready to initialize your Kubernetes cluster using Kubeadm! 🎯

🖥️ Create Virtual Machines (EC2 Instances) for the Cluster

To set up a Kubernetes cluster on AWS, you need to create three EC2 instances:

• One Master Node (Control Plane): Manages the cluster, schedules workloads, and maintains the desired state.

• Two Worker Nodes: Run application workloads and handle deployments.

🔧 Step 1: Launch EC2 Instances

1. Open the AWS Console and navigate to the EC2 Dashboard.

   

   

2. Click Launch Instance to create virtual machines.

   

⚙️ Step 2: Configure the Instances

1. Set the number of instances → 3 (1 Master + 2 Workers).

2. Choose Amazon Machine Image (AMI) → Latest Ubuntu AMI (recommended for Kubernetes).

3. Select an instance type → t2.xlarge

4. Assign Security Group → Use the previously configured security group to allow necessary traffic.

5. Set Storage → Minimum 30GB (for Kubernetes components, logs, and workloads).

6. Click Launch Instance to provision the virtual machines.

   

🏷️ Step 3: Naming the Instances

For easy identification, name your instances as follows:

• 🖥️ ControlPlane → Master Node

• 📦 WorkerNode1 → First Worker Node

• 📦 WorkerNode2 → Second Worker Node

Connecting to EC2 Instances via SSH

Once your EC2 instances are running, connect to them using SSH

Connect to the Control Plane Node

ssh -i <path-to-pem-file> ubuntu@<public-ip-of-controlplane>

Connect to Worker Nodes

ssh -i <path-to-pem-file> ubuntu@<public-ip-of-workernode1>
ssh -i <path-to-pem-file> ubuntu@<public-ip-of-workernode2>

🔹 Replace:

• <path-to-pem-file> → Your private key file path

• <public-ip> → The public IP address of each instance

Once your instances are up and running, you're ready to initialize the Kubernetes cluster with Kubeadm! 🚀

🔧 Updating and Installing Kubernetes on All Nodes

Before initializing your Kubernetes cluster, it's crucial to update your virtual machines, configure system settings, and install essential dependencies. This ensures a stable, secure, and well-optimized Kubernetes environment across all nodes.

Step 1: Disable Swap

Kubernetes requires swap to be disabled for optimal performance. Run the following commands on each node:

swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

This disables swap immediately and ensures it remains disabled after a reboot.

Step 2: Enable IP Forwarding and Allow Bridged Traffic

To enable proper Kubernetes networking, configure IP forwarding and iptables:

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

Now, set up required system parameters and apply them:

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system

✅ Verification

Ensure the necessary modules are loaded

lsmod | grep br_netfilter
lsmod | grep overlay

Check if system variables are correctly set

sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

Step 3: Update System Packages

Keeping your system updated ensures compatibility with the latest Kubernetes components. Run the following command on all nodes

sudo apt update && sudo apt upgrade -y

Step 4: Install Container Runtime (containerd)

Kubernetes requires a container runtime to manage containers. We'll use containerd

1. Install containerd

curl -LO https://github.com/containerd/containerd/releases/download/v1.7.14/containerd-1.7.14-linux-amd64.tar.gz
sudo tar Cxzvf /usr/local containerd-1.7.14-linux-amd64.tar.gz

2. Set up containerd as a service

curl -LO https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
sudo mkdir -p /usr/local/lib/systemd/system/
sudo mv containerd.service /usr/local/lib/systemd/system/
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

3. Enable systemd as the cgroup driver

sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml

4. Restart containerd and verify its status

sudo systemctl daemon-reload
sudo systemctl enable --now containerd
systemctl status containerd

Step 5: Install runc (Container Runtime Interface - CRI)

runc is required to run containers inside containerd. Install it using

curl -LO https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64
sudo install -m 755 runc.amd64 /usr/local/sbin/runc

Step 6: Install CNI (Container Network Interface) Plugins

CNI plugins allow pod-to-pod communication within the cluster. Install them with

curl -LO https://github.com/containernetworking/plugins/releases/download/v1.5.0/cni-plugins-linux-amd64-v1.5.0.tgz
sudo mkdir -p /opt/cni/bin
sudo tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.5.0.tgz

Step 7: Install Required Dependencies

Install essential tools for secure communication and package management

sudo apt install -y apt-transport-https ca-certificates curl gpg
sudo mkdir -p -m 755 /etc/apt/keyrings

Step 8: Install Kubernetes Components

1. Add the Kubernetes repository key and repository

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

2. Update the package list

sudo apt update

3. Install Kubernetes Tools (kubeadm, kubelet, kubectl)

sudo apt-get install -y kubelet=1.29.6-1.1 kubeadm=1.29.6-1.1 kubectl=1.29.6-1.1 --allow-downgrades --allow-change-held-packages

4. Prevent accidental updates that could break the cluster

sudo apt-mark hold kubeadm kubelet kubectl

Step 9: Configure crictl to Work with containerd

Set up crictl (a tool for interacting with container runtimes) to use containerd

sudo crictl config runtime-endpoint unix:///var/run/containerd/containerd.sock

🎯 Your Kubernetes Cluster is Ready for Initialization! 🚀

With all nodes updated and configured, your Kubernetes cluster is now ready for initialization using kubeadm in the next step!

🚀 Initializing the Kubernetes Cluster (Master Node Only)

Now that Kubernetes is installed on all nodes, it's time to initialize the cluster on the master node (Control Plane). This process sets up the control plane and defines the pod network, which is essential for communication between pods.

Step 1: Initialize the Kubernetes Control Plane

On the master node, run the following command to start the Kubernetes cluster:

sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=<private-ip-of-controlplane> --node-name controlplane

🔹 Replace <private-ip-of-controlplane> with the actual private IP of your Control Plane instance.

🔹 The --pod-network-cidr=192.168.0.0/16 flag specifies the pod network range.

Network CNI Options:

• Calico: 192.168.0.0/16 (default in this example)

• Flannel: 10.244.0.0/16

• Cilium: 10.217.0.0/16

✅ This command does the following:

• Sets up the Kubernetes control plane

• Defines the pod network for inter-pod communication

• Generates a join command to add worker nodes to the cluster

Step 2: Save the Worker Node Join Command

Once initialization is complete, Kubernetes will generate a command similar to this

sudo kubeadm join <master-node-ip>:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>

Copy and save this command—you will need it to connect worker nodes later.

Step 3: Configure kubectl on the Master Node

To start using Kubernetes, configure kubectl with the admin credentials

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

This allows you to run kubectl commands without root privileges.

Verify that the Control Plane is Running

kubectl get nodes

The master node should appear as NotReady—this is expected until a network plugin is installed.

Step 4: Deploy a Network Plugin (CNI)

Kubernetes requires a Container Network Interface (CNI) to enable pod-to-pod communication. Choose a networking solution and apply the corresponding configuration.

Install Calico (Recommended for Production)

Calico provides robust networking and security policies for Kubernetes clusters.

Run the following commands on the master node

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/tigera-operator.yaml
curl -O https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/custom-resources.yaml
kubectl apply -f custom-resources.yaml

This enables networking between pods across all nodes.

Verify Calico Installation

kubectl get pods -n calico-system

Step 5: Deploy the NGINX Ingress Controller

To manage external access to services inside the cluster, deploy the NGINX Ingress Controller:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml

🎯 Your Kubernetes Control Plane is Ready! 🚀

At this point:

• The control plane is up and running.

• Networking is configured, allowing pods to communicate.

• Ingress controller is deployed for external access.

🚀 Joining Worker Nodes to the Kubernetes Cluster

After setting up the control plane, the next step is to add worker nodes to the cluster. This allows them to host workloads and be managed by the master node.

Step 1: Join the Worker Nodes

Run the following command on each worker node to connect them to the Kubernetes cluster

sudo kubeadm join 172.31.38.39:6443 --token <your-token> \
    --discovery-token-ca-cert-hash sha256:<your-ca-hash>

Replace <your-token> and <your-ca-hash> with the values generated during kubeadm init.

✅ This command does the following:

• Connects worker nodes to the Kubernetes control plane.

• Allows the master node to schedule and manage workloads on worker nodes.

Once executed on all worker nodes, your Kubernetes cluster will be fully formed! 🎉

Step 2: Verify the Cluster Setup

To ensure that all nodes are properly connected, run the following command on the master node

kubectl get nodes

If everything is working correctly, you should see all three nodes (one master, two workers) in a Ready state.

Step 3: Verify All Pods Are Running

Check if all cluster components and networking pods are operational

kubectl get pods -A

✔ If all pods are running correctly, your cluster is now stable.

If some Calico pods are not running, follow the troubleshooting steps below.

Step 4: Disable Source/Destination Checks (AWS Only)

AWS enforces Source/Destination Checks, which can interfere with Kubernetes networking, especially when using Calico, Flannel, or Cilium CNIs.

To disable this check on all nodes (master + workers):

1. Go to AWS EC2 Dashboard → Instances

2. Select all Kubernetes nodes (master & workers)

3. Click Actions → Networking → Change Source/Destination Check

4. Select Disable and confirm ✅

🔍 Why Disable Source/Destination Checks?

• Kubernetes networking often involves asymmetric routing (packets leave from one interface but return through another).

• AWS blocks asymmetric routing by default, which breaks pod communication.

• Disabling this allows proper routing across nodes.

Step 5: Allow Bidirectional Traffic on TCP Port 179 (Calico CNI Only)

If using Calico CNI, you need to allow BGP (Border Gateway Protocol) traffic between nodes on TCP port 179.

Configure Security Group Rules

1. Go to AWS EC2 Dashboard → Security Groups

2. Find the security group attached to your Kubernetes nodes

3. Click on Inbound Rules → Edit Inbound Rules → Add Rule

• Type: Custom TCP Rule

• Protocol: TCP

• Port Range: 179

• Source:

  • 0.0.0.0/0 (not recommended for production)

  • OR Your VPC CIDR (e.g., 10.0.0.0/16)

4. Click Save Rules ✅

5. Now, edit Outbound Rules:

• Click Outbound Rules → Edit Outbound Rules → Add Rule

• Type: Custom TCP Rule

• Protocol: TCP

• Port Range: 179

• Destination:

  • 0.0.0.0/0 (for all hosts)

  • OR Your VPC CIDR (for internal traffic)

6️⃣ Click Save Rules ✅

🎯 Final Step: Verify Everything is Running

Run the following command again to ensure that all pods are running successfully:

kubectl get pods -A

🎉 Congratulations! Your Kubernetes Cluster is Now Fully Functional and Ready for Workloads! 🚀

Your Kubernetes cluster is now complete with a fully operational control plane and worker nodes. You’re now ready to deploy applications and manage workloads effectively.

🔒 Perform a Security Audit Using Kubeaudit

To ensure the security of your Kubernetes cluster, use Kubeaudit—a security auditing tool developed by Shopify. It helps identify misconfigurations, security vulnerabilities, and best practice violations in your cluster.

Step 1: Install Kubeaudit on Ubuntu

Run the following commands on your control plane (master node)

wget https://github.com/Shopify/kubeaudit/releases/download/v0.22.2/kubeaudit_0.22.2_linux_amd64.tar.gz
tar -xzf kubeaudit_0.22.2_linux_amd64.tar.gz  # Extract the tar.gz file
sudo mv kubeaudit /usr/local/bin/             # Move the binary to a system path
rm kubeaudit_0.22.2_linux_amd64.tar.gz        # Cleanup the tar.gz file
kubeaudit version                             # Verify installation

This will install Kubeaudit and verify that it's correctly set up.

Step 2: Scan the Cluster for Security Issues

Run the following command to audit the cluster and check for security misconfigurations

kubeaudit all

📌 What does this command do?

• Scans all Kubernetes objects for security vulnerabilities

• Detects misconfigurations in RBAC, PodSecurity, and container settings

• Provides remediation steps to fix issues

🛠 Setup SonarQube and Nexus Server

To implement code quality analysis (SonarQube) and artifact management (Nexus Repository Manager) in your CI/CD pipeline, follow these steps.

Launch EC2 Instances for SonarQube and Nexus

1. Go to AWS Console → Click Launch an Instance

2. Configure two EC2 instances:

   • AMI: Ubuntu

   • Instance Type: t2.medium

     

   • Storage: 20 GiB or more

     

   • Number of Instances: 2

     

3. Click Launch Instance

4. Rename instances to:

   • SonarQube

   • Nexus

Connect to Instances via SSH

Connect to each instance from your terminal:

ssh -i <path-to-pem-file> ubuntu@<private-ip>

Once connected, update the system:

sudo apt update

Install Docker on Both Servers

Step 1: Install Dependencies

sudo apt-get update
sudo apt-get install -y ca-certificates curl

Step 2: Add Docker's Official GPG Key

sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

Step 3: Add the Docker Repository

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update

Step 4: Install Docker

sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Step 5: Allow Docker Without sudo

sudo chmod 666 /var/run/docker.sock

Deploy SonarQube on the SonarQube Server

Run the following command on the SonarQube EC2 instance

docker run -d --name sonar -p 9000:9000 sonarqube:lts-community

Access SonarQube Dashboard

📌 Open your browser and go to:
http://<SonarQube-server-public-IP>:9000

Default Credentials

• Username: admin

• Password: admin

After logging in, update the default password for security

Deploy Nexus on the Nexus Server

Run this command on the Nexus EC2 instance:

docker run -d --name nexus -p 8081:8081 sonatype/nexus3

Access Nexus Dashboard

📌 Open your browser and go to: http://<Nexus-server-public-IP>:8081

Retrieve Admin Password

To log in, first find the default admin password:

docker exec -it nexus /bin/bash
cat sonatype-work/nexus3/admin.password

Use this admin password to log in.

• Username: admin

• Password: (from the above command)

Set a new password and enable anonymous access if needed.

Configure Nexus Repository in pom.xml

1. Go to Nexus Web UI

2. Click Browse → Look for:

   • Maven Releases

     

   • Maven Snapshots

     

3. Copy the repository URL

4. Add it to your project's pom.xml file:

<distributionManagement>
    <repository>
        <id>nexus-releases</id>
        <url>http://<Nexus-server-public-IP>:8081/repository/maven-releases/</url>
    </repository>
    <snapshotRepository>
        <id>nexus-snapshots</id>
        <url>http://<Nexus-server-public-IP>:8081/repository/maven-snapshots/</url>
    </snapshotRepository>
</distributionManagement>

🎯 Final Verification

1. Check if SonarQube is Running:

    docker ps | grep sonar
   

   You should see the sonarqube container running.

2. Check if Nexus is Running:

    docker ps | grep nexus
   

   You should see the nexus container running.

3. Test SonarQube Web Access:
   Open: http://<SonarQube-server-public-IP>:9000

4. Test Nexus Web Access:
   Open: http://<Nexus-server-public-IP>:8081

🎉 SonarQube and Nexus Are Now Set Up! 🚀

• SonarQube is ready for code quality analysis

• Nexus is ready for artifact storage

🚀 Set Up Jenkins Server on AWS

Jenkins is a powerful automation server widely used for CI/CD pipelines. This guide will help you set up Jenkins on an AWS EC2 instance, install necessary dependencies, and configure it for seamless use.

Launch an EC2 Instance for Jenkins

1. Go to AWS Console → Click Launch an Instance

   

2. Configure the instance:

   • Name: Jenkins Server

   • AMI: Ubuntu (latest LTS recommended)

     

   • Instance Type: t2.large (At least 2 vCPUs and 8 GB RAM for smooth builds)

     

   • Storage: 30 GiB (Jenkins jobs & artifacts require storage)

   • Number of Instances: 1

3. Click Launch Instance

   

4. Once running, copy the public IP of the Jenkins instance from the AWS Console.

Connect to the Jenkins Server via SSH

Use SSH to connect from your local machine:

ssh -i <path-of-pem-file> ubuntu@<jenkins-server-IP>

Replace <path-of-pem-file> with your private key path and <jenkins-server-IP> with your instance's public IP.

Update the System

To ensure the latest security patches and package versions, run:

sudo apt update && sudo apt upgrade -y

Install Jenkins

Step 1: Install Java (Required for Jenkins)

sudo apt install -y openjdk-17-jdk-headless

Step 2: Install Jenkins Using Official Repository

# Add Jenkins repository key
sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
  https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key

# Add Jenkins repository to sources list
echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
  https://pkg.jenkins.io/debian-stable binary/" | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null

# Update package lists and install Jenkins
sudo apt-get update
sudo apt-get install -y jenkins

Step 3: Start & Verify Jenkins Service

sudo systemctl start jenkins
sudo systemctl enable jenkins
sudo systemctl status jenkins

Install Docker (for Running Jenkins Builds in Containers)

Step 1: Install Dependencies

sudo apt-get install -y ca-certificates curl

Step 2: Add Docker’s Official GPG Key

sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

Step 3: Add Docker Repository

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
  https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Step 4: Install Docker

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Step 5: Allow Jenkins to Use Docker Without sudo

sudo chmod 666 /var/run/docker.sock

Install kubectl (for Kubernetes Integration)

Run the following command on the Jenkins server:

curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.19.6/2021-01-05/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin
kubectl version --short --client

Access Jenkins Web Interface

Jenkins runs on port 8080 by default.

📌 Open your browser and visit: http://<Jenkins-server-public-IP>:8080

Retrieve Initial Admin Password

Run this command:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Copy the admin password and paste it into the Jenkins web interface.

Jenkins Setup Wizard

1. Install Suggested Plugins (Recommended for most setups).

   

   

   

2. Create an Admin User (Set username, password, and email).

   

3. Start Using Jenkins 🎉

   

Install Required Jenkins Plugins

To enhance CI/CD functionality, install these plugins:

Step 1: Navigate to Plugin Manager

• Manage Jenkins → Manage Plugins → Available Plugins

  

Step 2: Install the Following Plugins

1. Eclipse Temurin Installer (Installs required Java versions)

2. Config File Provider (Manages configuration files for builds)

3. Pipeline Maven Integration (For running Maven builds inside pipelines)

4. SonarQube Scanner (For static code analysis integration)

5. Docker (Integrates Docker with Jenkins)

6. Docker Pipeline (Allows defining Docker containers in pipelines)

7. Kubernetes (For running Jenkins on Kubernetes)

8. Kubernetes CLI (Provides kubectl inside Jenkins)

9. Kubernetes Credentials (Manages Kubernetes authentication)

10. Kubernetes Client API (Allows Jenkins to interact with Kubernetes clusters)

11. Maven Integration (For building Java projects)

Click Install and wait for the installation to complete.

🎯 Final Verification

1. Check Jenkins Service

sudo systemctl status jenkins

You should see "active (running)" status.

2. Check Docker Installation

docker --version

It should display the installed Docker version.

3. Check kubectl Installation

kubectl version --short --client

It should show the Kubernetes client version.

4. Test Jenkins Web Access
   Open: http://<Jenkins-server-public-IP>:8080

🎉 Jenkins is Now Fully Set Up on AWS! 🚀

• Jenkins is installed & running

• Docker is set up for containerized builds

• kubectl is installed for Kubernetes integration

• Essential plugins are installed

Start Creating CI/CD Pipelines! 🚀

Jenkins is now fully set up! You can start creating Jenkins Pipelines to automate builds, tests, and deployments for your applications.

Step 1: Configure Essential Tools in Jenkins

1. Install Required Plugins

Install JDK

• Navigate to Manage Jenkins → Tools

  

• Under JDK, click Add JDK

• Name: jdk17

• Check Install automatically

• Installer: adoptium.net

• Version: jdk-17

Install SonarQube Scanner

• Under SonarQube Scanner, click Add SonarQube Scanner

• Name: sonar-scanner

• Check Install automatically

• Choose the latest version

Install Maven

• Under Maven, click Add Maven

• Name: maven3

• Check Install automatically

• Choose the latest version

Install Docker

• Under Docker, click Add Docker

• Name: docker

• Check Install automatically

• Installer: Download from docker.com

Once these tools are set up, you are ready to create a pipeline.

Step 2: Create a Jenkins Pipeline

A Jenkins pipeline automates the software development lifecycle, including building, testing, and deploying applications.

1. Access Jenkins Dashboard

• Log in to Jenkins using your browser.

• On the Jenkins Dashboard, click New Item to create a new job.

2. Define Pipeline Details

• Enter an Item Name (Example: BoardGame).

• Select Item Type → Choose Pipeline.

  

• Click OK to proceed.

3. Add GitHub Token in Jenkins Credentials

• Go to Jenkins Dashboard → Manage Jenkins → Credentials.

  

• Click on global → Add Credentials.

  

  

• Fill the details:

  • Kind: Username & Password

  • Scope: Global

  • Username: GitHub username

  • Password: Personal Access Token from GitHub

  • ID: git-cred

• Click Create.

4. Install Trivy on the Jenkins Instance

sudo apt-get install wget apt-transport-https gnupg lsb-release
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update
sudo apt-get install trivy -y

5. Add SonarQube Credentials in Jenkins

• Go to SonarQube Server → Administration → Security → Users.

  

  

• Update the token.

• Set a name and expiry date of the token and click on generate.

  

• Go to Jenkins Dashboard → Manage Jenkins → Credentials.

• Click global → Add new Credential.

• Kind: Secret text

• Scope: Global

• Secret: <token-generated>

• ID: sonar-token

  

• Click Create.

6. Set Up SonarQube Server in Jenkins

• Go to Jenkins Dashboard → Manage Jenkins → System.

  

• Click Add SonarQube:

  • Name: sonar

  • Server URL: <SonarQube-public-IP>:9000

  • Authentication Token: sonar-token

• Apply and Save.

7. Configure SonarQube Webhook

• Go to SonarQube Server → Administration → Configuration → Webhook.

  

• Create a new webhook:

  • URL: <Jenkins-public-IP>:8080/sonarqube-webhook

    

• Click Create.

8. Add Nexus Repository Credentials

• Go to Jenkins Dashboard → Manage Jenkins → Managed files.

  

• Add a new config: Global Maven settings.xml.

  

• ID: global-settings.

  

• In the content section, update <servers></servers>:

    <server>
        <id>maven-releases</id>
        <username>nexus-username</username>
        <password>nexus-password</password>
    </server>
    <server>
        <id>maven-snapshots</id>
        <username>nexus-username</username>
        <password>nexus-password</password>
    </server>
  

  

  

• Click Submit.

  

9. Add DockerHub Credentials

• Go to Jenkins Dashboard → Manage Jenkins → Credentials.

• Click global → Add Credential.

  

  

• Kind: Username with Password

• Scope: Global

• Username: DockerHub-username

• Password: DockerHub-password

• ID: docker-cred

  

• Click Create.

10. Create a Kubernetes Service Account

Create Namespace

kubectl create ns webapps

Create Service Account

apiVersion: v1
kind: ServiceAccount
metadata:
   name: jenkins
   namespace: webapps

kubectl apply -f svcacc.yaml

11. Create a Role using in kubernetes cluster

• create a role.yaml configuration file

    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: app-role
      namespace: webapps
    rules:
      - apiGroups:
            - ""
            - apps
            - autoscaling
            - batch
            - extensions
            - policy
            - rbac.authorization.k8s.io
        resources:
          - pods
          - secrets
          - componentstatuses
          - configmaps
          - daemonsets
          - deployments
          - events
          - endpoints
          - horizontalpodautoscalers
          - ingress
          - jobs
          - limitranges
          - namespaces
          - nodes
          - pods
          - persistentvolumes
          - persistentvolumeclaims
          - resourcequotas
          - replicasets
          - replicationcontrollers
          - serviceaccounts
          - services
        verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  

• create the role using the command

    kubectl apply -f role.yaml
  

12. Bind Role to Service Account

• create a bind.yaml configuration file

    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: app-rolebinding
      namespace: webapps 
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: app-role 
    subjects:
    - namespace: webapps 
      kind: ServiceAccount
      name: jenkins
  

• create the rolebinding using the command

    kubectl apply -f bind.yaml
  

13. Create Kubernetes Secret for Jenkins

• Create a secret resource configuration file: secrets.yaml

    apiVersion: v1
    kind: Secret
    metadata:
       name: mysecretname
       annotations:
          kubernetes.io/service-account.name: jenkins
    type: kubernetes.io/service-account-token
  

• Create secret using the kubectl command

    kubectl apply -f secrets.yaml -n webapps
  

  

• Get the token to connect to jenkins and copy that

    kubectl describe secret mysecretname -n webapps
  

  

• Copy the token and add it to Jenkins Credentials.

• Add the token in the jenkins credentials

  • Go to Jenkins Dashboard → Manage Jenkins → Credentials

  • Click on Global and Add a new credential

  • Kind: secret text

  • scope: global

  • secret: token-that-you-copied

  • ID: k8s-cred

Step 3: Configure Mail Notification

1. Generate Gmail App Password

• Search for App Password on any browser.

  

• Create a new App password with the name: Jenkins.

  

• Copy the generated token.

2. Add Gmail Credentials to Jenkins

• Go to Jenkins Dashboard → Manage Jenkins → Credentials.

• Click on Global and add a new credential:

  • Kind: Username with password

  • Scope: Global

  • Username: your-gmail

  • Password: generated token

  • ID: mail-cred

3. Configure Mail Server in Jenkins

• Go to Jenkins Dashboard → Manage Jenkins → System.

• Search for Extended E-mail Notification.

  

• Fill in the details:

  • SMTP server: smtp.gmail.com

  • SMTP port: 465

  • Credentials: Choose mail-cred

  • Check Use SSL

• Find the E-mail Notification section and configure

  

  • SMTP server: smtp.gmail.com

  • SMTP port: 465

  • Check Use SSL

  • Check Use SMTP Authentication

  • Username: email ID

  • Password: copied app token

• Save the changes.

Step 4: Create a Jenkins Pipeline

A Jenkins pipeline automates the software development lifecycle, including building, testing, and deploying applications.

1. General Settings

• Check Discard Old Builds to limit stored build history.

• Configure:

  • Max builds to keep: 10

  • Max days to keep builds: 30

2. Define the Pipeline Script

pipeline {
    agent any

    tools{
        jdk 'jdk17'
        maven 'maven3'
    }
    environment {
        SCANNER_HOME= tool 'sonar-scanner'
    }

    stages {
        stage('Git Checkout') {
            steps {
                git credentialsId: 'git-cred', url: 'https://github.com/praduman8435/Boardgame.git'
            }
        }
        stage('Compile') {
            steps {
                sh "mvn compile"
            }
        }
        stage('Test') {
            steps {
                sh "mvn test"
            }
        }
        stage('File System Scan') {
            steps {
                sh "trivy fs --format table -o trivy-fs-report.html ."
            }
        }
        stage('Code Quality Analysis') {
            steps {
                withSonarQubeEnv('sonar') {
                    sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar-projectName=BoardGame -Dsonar.projectKey=BoardGame \
                            -Dsonar.java.binaries=. '''
                }
            }
        }
        stage('Quality Gate') {
            steps {
                script {
                    waitForQualityGate abortPipeline: false, credentialsId: 'sonar-token'
                }
            }
        }
        stage('Build') {
            steps {
                sh "mvn package"
            }
        }
        stage('Publish Artifacts to Nexus') {
            steps {
                withMaven(globalMavenSettingsConfig: 'global-settings', jdk: 'jdk17', maven: 'maven3', mavenSettingsConfig: '', traceability: true) {
                    sh "mvn deploy"
                }
            }
        }
        stage('Build & Tag Docker Image') {
            steps {
                script {
                    withDockerRegistry(credentialsId: 'docker-cred', toolName: 'docker') {
                        sh "docker build -t thepraduman/boardgame:latest ."
                    }
                }
            }
        }
        stage('Docker Image Scan') {
            steps {
                sh "trivy image --format table -o trivy-image-report.html  thepraduman/boardgame:latest"
            }
        }
        stage('Push Docker Image') {
            steps {
                script {
                    withDockerRegistry(credentialsId: 'docker-cred', toolName: 'docker') {
                        sh "docker push thepraduman/boardgame:latest"
                    }
                }
            }
        }
        stage('Deploy to k8s') {
            steps {
                withKubeConfig(caCertificate: '', clusterName: 'kubernetes', contextName: '', credentialsId: 'k8s-cred', namespace: 'webapps', restrictKubeConfigAccess: false, serverUrl: 'https://172.31.39.125:6443') {
                    sh "kubectl apply -f k8s-manifest"
                }
            }
        }
        stage('Verify the deployment') {
            steps {
                withKubeConfig(caCertificate: '', clusterName: 'kubernetes', contextName: '', credentialsId: 'k8s-cred', namespace: 'webapps', restrictKubeConfigAccess: false, serverUrl: 'https://172.31.39.125:6443') {
                    sh "kubectl get pods -n webapps"
                    sh "kubectl get svc -n webapps"
                }
            }
        }
    }
}

3. Build the Pipeline

Click on Build Now to build the pipeline

Check the SonarQube Dashboard

Setup Monitoring (Prometheus & Grafana)

Create an EC2 Instance for Monitoring

1. Go to AWS console and launch an EC2 instance.

   

2. Instance Name: Monitoring

3. AMI: Ubuntu

   

4. Instance Type: t2.large

   

5. Key-Pair: Create a new key-pair or use an existing one.

6. Security Group: Use the previously created one.

7. Storage: 20 GiB

   

8. Click Launch Instance.

SSH into the Monitoring Server

1. Copy the public IP of the monitoring server.

   

2. Connect via SSH from your local machine:

    ssh -i <path-to-pem-file> ubuntu@<public-IP>
   

3. Update the server:

    sudo apt update
   

Install Prometheus

1. Download Prometheus:

    wget https://github.com/prometheus/prometheus/releases/download/v3.2.1/prometheus-3.2.1.linux-amd64.tar.gz
   

   

2. Extract the tar file:

    tar -xvf prometheus-3.2.1.linux-amd64.tar.gz
   

   

3. Navigate to the Prometheus directory:

    cd prometheus-3.2.1.linux-amd64
   

   

4. Run Prometheus in the background:

    ./prometheus &
   

   

5. Access Prometheus at:

    http://<public-IP-of-monitoring>:9090
   

   

Install Grafana

1. Install dependencies:

    sudo apt-get install -y adduser libfontconfig1 musl
   

2. Download and install Grafana:

    wget https://dl.grafana.com/enterprise/release/grafana-enterprise_11.5.2_amd64.deb
    sudo dpkg -i grafana-enterprise_11.5.2_amd64.deb
   

   

3. Start Grafana:

    sudo systemctl start grafana-server
   

4. Access Grafana at:

    http://<public-IP-of-monitoring>:3000
   

   

5. Default credentials:

   • Username: admin

   • Password: admin (Change it after logging in)

Install Blackbox Exporter

1. Download Blackbox Exporter:

    wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.26.0/blackbox_exporter-0.26.0.linux-amd64.tar.gz
   

   

2. Extract the tar file:

    tar -xvf blackbox_exporter-0.26.0.linux-amd64.tar.gz
   

   

3. Navigate to the directory:

    cd blackbox_exporter-0.26.0.linux-amd64
   

4. Run Blackbox Exporter:

    ./blackbox_exporter &
   

   

5. Access at:

    http://<public-IP-of-monitoring>:9115
   

   

Configure Prometheus

1. Edit prometheus.yml:

    vim prometheus.yml
   

2. Add Blackbox Exporter job:

    - job_name: 'blackbox'
      metrics_path: /probe
      params:
        module: [http_2xx]  # Look for a HTTP 200 response.
      static_configs:
        - targets:
          - http://prometheus.io    # Target to probe with http.
          - https://prometheus.io   # Target to probe with https.
          - http://example.com:8080 # Target to probe with http on port 8080.
      relabel_configs:
        - source_labels: [__address__]
          target_label: __param_target
        - source_labels: [__param_target]
          target_label: instance
        - target_label: __address__
          replacement: 127.0.0.1:9115  # The blackbox exporter's real hostname:port.
   

3. Restart Prometheus:

    pgrep prometheus
   

    kill <PID>
   

    ./prometheus &
   

4. Check Prometheus Dashboard → Status → Target.

   

Add Prometheus as DataSource in Grafana

1. Go to Grafana Dashboard.

2. Click Data Sources under Connections.

   

3. Search and select Prometheus.

   

4. Add a new data source and enter the Prometheus server URL.

   

5. Click Save & Test.

Import Grafana Dashboard

1. Go to Grafana Home Page.

2. Click + (Add) → Import Dashboard.

   

3. Enter 7587 as the dashboard ID and click Load.

   

4. Select Prometheus as the data source and click Import.

   

Install Prometheus Plugin on Jenkins

1. Go to Jenkins Dashboard → Manage Jenkins → Plugins.

2. Search for Prometheus and install it.

   

3. Restart Jenkins.

Install Node Exporter on Monitoring Server

1. Download Node Exporter:

    wget https://github.com/prometheus/node_exporter/releases/download/v1.9.0/node_exporter-1.9.0.linux-amd64.tar.gz
   

   

2. Extract the tar file:

    tar -xvf node_exporter-1.9.0.linux-amd64.tar.gz
   

   

3. Navigate to the directory:

    cd node_exporter-1.9.0.linux-amd64/
   

   

4. Run Node Exporter:

    ./node_exporter &
   

   

5. Access at:

    http://<public-IP-of-monitoring-server>:9100
   

   

Configure Prometheus for Jenkins

1. Go to Jenkins Dashboard → Manage Jenkins → System

   

2. search for prometheus section

   

3. Check if other things required or leave it default and save it

Edit the prometheus.yml file in monitoring server

1. Edit prometheus.yml:

    vim prometheus.yml
   

2. Add the following jobs:

    - job_name: node
      static_configs:
        - targets: ['<jenkins-public-IP>:9100']
    - job_name: jenkins
      metrics_path: '/prometheus'
      static_configs:
        - targets: ['<jenkins-public-IP>:8080']
   

3. Restart Prometheus:

    pgrep prometheus
   

    kill <PID>
   

   

    ./prometheus &
   

   

4. Verify in Prometheus → Status → Target Health.

   

Import Node Exporter Dashboard in Grafana

1. Go to Grafana Dashboard → Import.

   

2. Enter ID: 1860 and click Load.

   

3. Select Prometheus as the data source and import.

   

Monitoring setup with Prometheus, Grafana, and Jenkins is now complete! 🚀

Enjoyed the post? to support my writing!

Conclusion

This project covered the complete setup of a DevSecOps pipeline, from infrastructure provisioning to CI/CD automation, security, and monitoring. We deployed a Kubernetes cluster on AWS, automated deployments using Jenkins, Terraform, and ArgoCD, and integrated security tools like Trivy and SonarQube. For monitoring, we set up Prometheus and Grafana along with Node Exporter and Blackbox Exporter. By combining these tools, we built a secure, automated, and scalable environment that ensures smooth deployments, security compliance, and real-time monitoring.

💡 Let’s connect and discuss DevOps, cloud automation, and cutting-edge technology

🔗 LinkedIn | 💼 Upwork | 🐦 Twitter | 👨‍💻 GitHub

In today’s fast-paced tech world, speed and security go hand in hand. You can’t just build and deploy apps quickly—you need to keep them secure from day one. That’s where DevSecOps comes in! It blends development, security, and operations into one seamless process, ensuring that security is baked into every stage of the pipeline instead of being an afterthought.

This Ultimate DevSecOps Project is all about deploying a three-tier application on AWS EKS with a fully automated CI/CD pipeline. The goal? To make sure every piece of code is secure, high-quality, and production-ready before it even goes live.

What’s Inside This Project?

We’ll be using some of the best DevSecOps tools out there to make this happen:

✅ Jenkins – Automates the entire CI/CD pipeline.
✅ SonarQube & OWASP Dependency-Check – Keep the code clean, secure, and compliant.
✅ Trivy – Scans container images for security vulnerabilities before deployment.
✅ Terraform – Automates infrastructure setup on AWS.
✅ ArgoCD – Ensures Kubernetes deployments stay in sync with Git (GitOps).
✅ Prometheus & Grafana – Provide real-time monitoring and insights.

By the end of this project, you’ll have a fully functional, security-first DevSecOps pipeline that not only deploys applications but also keeps them safe, scalable, and efficient.

🚀 Ready to dive in? Let’s build something amazing!

Source Code & Repository 👇

You can find the source code for this project here:

Step 1: Set Up a Jenkins Server on AWS EC2

1. Log in to AWS and Launch an EC2 Instance

1. Go to the AWS Console.

2. Navigate to EC2 (Elastic Compute Cloud).

3. Click Launch Instance to create a new virtual machine.

2. Configure the EC2 Instance

• AMI (Amazon Machine Image): Choose Ubuntu Server.

• 

  Instance Type: Select t2.2xlarge (8 vCPUs, 32GB RAM) for better performance.

• 

  Key Pair: No need to create a key pair (proceed without key pair).

  

3. Configure Security Group (Firewall Rules)

Set up inbound rules to allow required network traffic:

Note: For security, avoid opening all these ports to the public. Instead, restrict access to trusted IPs or internal networks.

• Choose the created security group in Network setting

4. Configure Storage and IAM Role

• Storage: Set at least 30 GiB.

• 

  IAM Role: Attach an IAM profile with administrative access to allow Jenkins to manage AWS resources.

  Create an IAM profile with Administrator Access and attach it to EC2 instance

  

  

5. Automate Installation with User Data

Instead of manually installing required tools, you can automate it using a User Data script. This script will automatically install:

• Jenkins

• Docker

• Terraform

• AWS CLI

• SonarQube (running in a container)

• Trivy (for security scanning)

#!/bin/bash
# For Ubuntu 22.04
# Intsalling Java
sudo apt update -y
sudo apt install openjdk-17-jre -y
sudo apt install openjdk-17-jdk -y
java --version

# Installing Jenkins
curl -fsSL https://pkg.jenkins.io/debian/jenkins.io-2023.key | sudo tee \
  /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
  https://pkg.jenkins.io/debian binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update -y
sudo apt-get install jenkins -y

# Installing Docker 
#!/bin/bash
sudo apt update
sudo apt install docker.io -y
sudo usermod -aG docker jenkins
sudo usermod -aG docker ubuntu
sudo systemctl restart docker
sudo chmod 777 /var/run/docker.sock

# If you don't want to install Jenkins, you can create a container of Jenkins
# docker run -d -p 8080:8080 -p 50000:50000 --name jenkins-container jenkins/jenkins:lts

# Run Docker Container of Sonarqube
#!/bin/bash
docker run -d  --name sonar -p 9000:9000 sonarqube:lts-community


# Installing AWS CLI
#!/bin/bash
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
sudo apt install unzip -y
unzip awscliv2.zip
sudo ./aws/install

# Installing Kubectl
#!/bin/bash
sudo apt update
sudo apt install curl -y
sudo curl -LO "https://dl.k8s.io/release/v1.28.4/bin/linux/amd64/kubectl"
sudo chmod +x kubectl
sudo mv kubectl /usr/local/bin/
kubectl version --client


# Installing eksctl
#! /bin/bash
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
eksctl version

# Installing Terraform
#!/bin/bash
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install terraform -y

# Installing Trivy
#!/bin/bash
sudo apt-get install wget apt-transport-https gnupg lsb-release -y
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
echo deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt update
sudo apt install trivy -y


# Intalling Helm
#! /bin/bash
sudo snap install helm --classic

Why use User Data?

• Automates setup.

• Ensures all required tools are installed before first login.

• Saves time compared to manual installation.

6. Launch the Instance

• Click Launch Instance to start your Jenkins server.

7. Connect to the Instance

Since SSH is disabled for security reasons, use the EC2 Instance Connect feature:

• Go to the AWS EC2 Console.

• Select your Jenkins instance.

• 

  Click the "Connect" button at the top.

• Choose the "EC2 Instance Connect" tab.

• 

  Click "Connect" to open a web-based terminal directly in your browser.

8. Monitor Running Processes

To check what commands are running inside the instance, use:

htop

This command provides a real-time view of system performance and running processes.

Step 2: Set Up a Jenkins Pipeline to Deploy EKS and Networking Services

1. Access Jenkins

1. Open your browser and go to:

    http://<public-ip-of-jenkins-server>:8080
   

   

2. Retrieve the initial Jenkins admin password by running the following command in the jenkins Instance:

    sudo cat /var/lib/jenkins/secrets/initialAdminPassword
   

3. Follow the setup wizard:

   • Install Suggested Plugins.

   • 

     Create an Admin Username & Password.

   • Complete the basic configuration.

     

Now, Jenkins is ready to use.

2. Install Required Plugins

To enable Jenkins to work with AWS and Terraform, install the following plugins manually:

1. Click on "Manage Jenkins" > "Plugins Manager".

   

2. Search for and install these plugins:

   • AWS Credentials → Securely stores AWS access keys.

   • Pipeline: AWS Steps → Adds built-in AWS-specific pipeline steps.

   • Terraform → Enables Terraform automation in Jenkins.

   • Pipeline: Stage View → Provides a visual representation of pipeline stages.

3. Configure AWS Credentials in Jenkins

1. Go to "Manage Jenkins" > "Credentials".

2. 

   Click "Global" > "Add Credentials".

   

3. Fill in the details:

   • Kind: AWS Credentials

   • Scope: Global

   • ID: aws-creds

   • Access Key ID: From your AWS IAM user

   • Secret Access Key: From your AWS IAM user

4. 

   Click "Create".

Note: Create an IAM User in AWS with Administrator Access and use its credentials here.

4. Configure Terraform in Jenkins

1. Go to "Manage Jenkins" > "Tools".

2. Scroll to Terraform Installation.

   

3. Provide:

   • Name: terraform

   • Installation Directory: Find Terraform’s installation path by running below command in jenkins instance:

       whereis terraform
     

4. 

   

   Click Save.

5. Create a New Jenkins Pipeline

In this pipeline i will use this repository here all the terraform source code present to create production grade infrastructure

1. In Jenkins, go to Dashboard > New Item.

2. 

   Enter an item name.

3. 

   Select Pipeline and click OK.

4. Scroll to the Pipeline section:

   • Under Definition, select Pipeline script.

   • Copy and paste the following pipeline script:

    properties([
        parameters([
            string(
                defaultValue: 'dev',
                name: 'Environment'
            ),
            choice(
                choices: ['plan', 'apply', 'destroy'], 
                name: 'Terraform_Action'
            )
        ])
    ])

    pipeline {
        agent any
        stages {
            stage('Preparing') {
                steps {
                    sh 'echo Preparing'
                }
            }
            stage('Git Pulling') {
                steps {
                    git branch: 'main', url: 'https://github.com/praduman8435/Production-ready-EKS-with-automation.git'
                }
            }
            stage('Init') {
                steps {
                    withAWS(credentials: 'aws-creds', region: 'ap-south-1') {
                        sh 'terraform -chdir=eks/ init'
                    }
                }
            }
            stage('Validate') {
                steps {
                    withAWS(credentials: 'aws-creds', region: 'ap-south-1') {
                        sh 'terraform -chdir=eks/ validate'
                    }
                }
            }
            stage('Action') {
                steps {
                    withAWS(credentials: 'aws-creds', region: 'ap-south-1') {
                        script {    
                            if (params.Terraform_Action == 'plan') {
                                sh "terraform -chdir=eks/ plan -var-file=${params.Environment}.tfvars"
                            } else if (params.Terraform_Action == 'apply') {
                                sh "terraform -chdir=eks/ apply -var-file=${params.Environment}.tfvars -auto-approve"
                            } else if (params.Terraform_Action == 'destroy') {
                                sh "terraform -chdir=eks/ destroy -var-file=${params.Environment}.tfvars -auto-approve"
                            } else {
                                error "Invalid value for Terraform_Action: ${params.Terraform_Action}"
                            }
                        }
                    }
                }
            }
        }
    }

6. Finalize Pipeline Setup

1. Enable Groovy Sandbox: Check the box for "Use Groovy Sandbox".

2. Click "Save".

7. Run the Pipeline

1. Wait for a minute, then click "Build with Parameters".

2. 

   Select a Terraform action (plan, apply, or destroy).

3. Click "Build".

4. 

   

   Navigate to the Console Output to track progress.

   

What This Pipeline Does?

• Connects Jenkins to AWS using stored credentials.

• Fetches Terraform code from GitHub.

• Initializes Terraform for EKS and networking setup.

• Validates Terraform code before deployment.

• Executes Terraform actions based on user selection (plan, apply, or destroy)

Step 3: Set Up the Jump Server

Why Do You Need a Jump Server?

Since your EKS cluster is inside a VPC, it cannot be accessed directly from the internet. A Jump Server (Bastion Host) acts as a secure gateway, allowing access to private resources within your VPC.

How It Works:

• Your EKS cluster and other private resources don’t have public IPs, so they can't be accessed directly.

• Instead of exposing these private resources, you connect to a Jump Server first.

• The Jump Server has a public IP and is placed in a public subnet, acting as an intermediary to access the private cluster securely.

1. Create a Jump Server in AWS

1. Go to AWS EC2 Console and click "Launch Instance".

2. Configure the Instance:

   • Instance Name: jump-server

   • 

     AMI: Ubuntu

     

   • Instance Type: t2.medium

   • Key Pair: No need to attach a key pair (SSH disabled for security).

     

   • Network Settings:

     • VPC: Select the VPC created by the Jenkins Terraform pipeline.

     • Subnet: Choose any public subnet.

   • Storage: At least 30 GiB.

   • 

     IAM Role: Attach an IAM profile with administrative access.

3. Install required tools on the Jump Server automatically, by adding the following script in the User Data field:

    sudo apt update -y
   
    # Installing AWS CLI
    #!/bin/bash
    curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
    sudo apt install unzip -y
    unzip awscliv2.zip
    sudo ./aws/install
   
    # Installing Kubectl
    #!/bin/bash
    sudo apt update
    sudo apt install curl -y
    sudo curl -LO "https://dl.k8s.io/release/v1.28.4/bin/linux/amd64/kubectl"
    sudo chmod +x kubectl
    sudo mv kubectl /usr/local/bin/
    kubectl version --client
   
    # Intalling Helm
    #! /bin/bash
    sudo snap install helm --classic
   
    # Installing eksctl
    #! /bin/bash
    curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
    sudo mv /tmp/eksctl /usr/local/bin
    eksctl version
   

4. Launch the Instance by clicking "Launch Instance".

   

2. Connect to the Jump Server and Verify Access

Once the instance is running, access it using EC2 Instance Connect:

1. Go to AWS EC2 Console.

2. Select your Jump Server instance.

3. Click "Connect" > "EC2 Instance Connect".

4. 

   Click "Connect" to open a web terminal.

3. Configure AWS Credentials on the Jump Server

To allow the Jump Server to interact with AWS services, configure AWS CLI:

aws configure

• Enter AWS Access Key ID (from IAM user).

• Enter AWS Secret Access Key (from IAM user).

• Default region: Set your AWS region (e.g., us-east-1).

• Output format: Press Enter (default is json).

4. Update kubeconfig to Access the EKS Cluster

Run the following command to configure kubectl for your EKS cluster:

aws eks update-kubeconfig --name <your-eks-cluster-name> --region <your-region>

5. Verify the Cluster Connection

Check if the Jump Server can access the EKS cluster by listing the worker nodes:

kubectl get nodes

If you see the nodes, your Jump Server setup is successful! 🎉

Step 4: Set Up an AWS Load Balancer for EKS

In order to configure an AWS Load Balancer in our EKS cluster, we need a Service Account that allows Kubernetes to create and manage the load balancer automatically.

1. Create an IAM-Backed Service Account

The AWS Load Balancer Controller requires an IAM role with the necessary permissions to create and manage Elastic Load Balancers (ELB) in AWS.

Run the following command to create a service account with the required IAM role inside the EKS cluster:

eksctl create iamserviceaccount \
  --cluster=<eks-cluster-name> \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name AmazonEKSLoadBalancerRole \
  --attach-policy-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve \
  --region=ap-south-1

📌 Explanation:

• --cluster=<eks-cluster-name> → Name of your EKS cluster.

• --namespace=kube-system → Deploys the service account in the kube-system namespace.

• --name=aws-load-balancer-controller → Creates a service account with this name.

• --role-name AmazonEKSLoadBalancerRole → Assigns an IAM role to the service account.

• --attach-policy-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy → Attaches the AWS Load Balancer Controller IAM policy.

• --approve → Automatically applies the changes.

2. Add the AWS EKS Helm Repository

To deploy the AWS Load Balancer Controller, we use Helm, a package manager for Kubernetes.

Add the official AWS EKS Helm repository:

helm repo add eks https://aws.github.io/eks-charts

This repository contains pre-packaged Helm charts for essential AWS EKS components such as:
✅ AWS Load Balancer Controller
✅ EBS CSI Driver (for dynamic volume provisioning)
✅ VPC CNI Plugin (for networking enhancements)
✅ Cluster Autoscaler (for automatic scaling)

Update the repository to get the latest charts:

helm repo update

3. Install the AWS Load Balancer Controller

Now, install the AWS Load Balancer Controller using Helm:

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  --namespace kube-system \
  --set clusterName=<eks-cluster-name> \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

📌 Explanation:

• --namespace kube-system → Deploys the controller in the kube-system namespace.

• --set clusterName=<eks-cluster-name> → Associates it with your EKS cluster.

• --set serviceAccount.create=false → Uses the existing IAM-backed service account.

• --set serviceAccount.name=aws-load-balancer-controller → Specifies the service account created earlier.

4. Verify the AWS Load Balancer Controller

Check if the Load Balancer Controller is running correctly:

kubectl get pods -n kube-system | grep aws-load-balancer-controller

4. Fixing Pods in Error or CrashLoopBackOff

If your AWS Load Balancer Controller pods are in Error or CrashLoopBackOff, it’s likely due to misconfiguration. To fix this, upgrade the Helm release with the correct settings:

helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller \
  --set clusterName=<cluster-name> \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region=<your-region> \
  --set vpcId=<your-vpc-id> \
  -n kube-system

• <your-vpc-id>: VPC ID where your EKS cluster runs (e.g., vpc-0123456789abcdef0).

• <cluster-name>: Your EKS cluster name (e.g., dev-medium-eks-cluster).

• <your-region>: AWS region of your cluster (e.g., us-west-1).

What This Does?

• Upgrades/Installs: Updates the Helm release or installs it if missing.

• Configures Correctly: Ensures the controller uses the right cluster, service account, region, and VPC.

Check if the pods are running:

kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller

🚀 Your AWS Load Balancer Controller is now ready to manage Kubernetes services! 🎉

Step 5: Set Up and Configure ArgoCD on EKS

ArgoCD is a GitOps continuous delivery tool that automates the deployment of applications to Kubernetes. We will install ArgoCD in our EKS cluster and expose its UI for external access.

1. Create a Separate Namespace for ArgoCD

To keep ArgoCD components organized, create a dedicated namespace:

kubectl create namespace argocd

2. Install ArgoCD Using Manifests

Apply the official ArgoCD installation YAML to deploy its components:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

This will install all necessary ArgoCD components inside the argocd namespace.

3. Verify ArgoCD Installation

Check if all ArgoCD pods are running:

kubectl get pods -n argocd

4. Expose ArgoCD Server

By default, ArgoCD runs as a ClusterIP service, meaning it is only accessible inside the cluster. To access the UI externally, change it to a LoadBalancer service:

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

This will assign a public IP to the ArgoCD server, making it accessible via an Elastic Load Balancer (ELB) in AWS.

5. Retrieve the External IP of ArgoCD

Run the following command to get the external URL:

kubectl get svc -n argocd argocd-server

Look for the EXTERNAL-IP in the output. This is the URL you’ll use to access the ArgoCD UI.

6. Get the ArgoCD Admin Password

By default, ArgoCD generates an admin password stored as a Kubernetes secret. Retrieve it using:

kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 --decode

Use this password to log in as the admin user.

7. Access the ArgoCD UI

Now, you can access the UI of argoCD using the elastic loadbalancer created on the aws

Login using:

• Username: admin

• Password: (retrieved from the secret above)

ArgoCD is now ready! You can start managing your Kubernetes deployments using GitOps. 🚀

Step 6: Configure SonarQube for DevSecOps Pipeline

SonarQube is a crucial tool for static code analysis, ensuring code quality and security in your DevSecOps pipeline. We will configure it within Jenkins for automated code scanning.

1. Verify if SonarQube is Running

Since SonarQube is running as a Docker container on the Jenkins server, check its status with:

docker ps

You should see a running SonarQube container exposed on port 9000.

2. Access SonarQube UI

Open any web browser and visit:

http://<public-ip-of-jenkins-server>:9000

Log in using the default credentials:

• Username: admin

• Password: admin

Once logged in, set a new password for security.

3. Generate an Authentication Token

Jenkins needs a token to authenticate with SonarQube for automated scans.

1. Go to Administration → Security → Users.

2. 

   

   Click on Update Token.

3. Provide a name and set an expiration date (or leave it as "No Expiration").

   

4. Click Generate Token.

5. Copy and save the token securely (you will need it for Jenkins).

4. Create a Webhook for Jenkins Notifications

A webhook will notify Jenkins once SonarQube completes an analysis.

1. Navigate to Administration → Configuration → Webhooks.

2. 

   Click Create Webhook.

   

3. Enter the details:

   • Name: Jenkins Webhook

   • 

     URL: http://<public-ip-of-jenkins-server>:8080/sonarqube-webhook

   • Secret: (Leave blank)

4. Click Create.

   

Now, the webhook will trigger Jenkins when a project analysis is complete.

5. Create a SonarQube Project for Code Analysis

SonarQube will analyze the frontend and backend code separately.

Frontend Analysis Configuration

1. Go to Projects → Manually → Create a New Project.

2. 

   Fill in the required details (Project Name, Key, etc.).

3. 

   Click Setup.

4. Choose Analyze Locally.

5. Select Use an Existing Token and paste the token generated earlier.

6. 

   Choose Other if your build type is not listed.

7. 

   Select OS: Linux.

8. 

   SonarQube will generate a command for analysis—copy and save it.

   

9. Add the command to your Jenkins pipeline

Backend Analysis Configuration

Repeat the same steps for the backend project:

1. Go to Projects → Create a New Project.

   

2. Fill in the required details.

3. 

   Click Setup → Analyze Locally.

4. Use the previously generated token.

5. 

   Choose Other as the build type if needed.

6. Select OS: Linux.

7. Copy the generated analysis command and save it.

8. 

   Add the command to your Jenkins pipeline

6. Final Verification

At this point:
✅ SonarQube is running and accessible.
✅ Jenkins has an authentication token to interact with SonarQube.
✅ A webhook is set up to notify Jenkins about completed scans.
✅ Projects are created, and analysis commands are ready for Jenkins execution.

Now, whenever Jenkins runs the pipeline, SonarQube will analyze the code and report quality & security issues. 🎯 ✅

Step 6: Create an ECR Repository for Docker Images

Amazon Elastic Container Registry (ECR) will store the frontend and backend Docker images used for deployment. Let's configure it and store necessary credentials in Jenkins.

1. Create Private ECR Repositories

1. Open AWS Console and navigate to the ECR service.

2. 

   Click Create Repository.

3. Choose Private Repository.

4. Repository Name: frontend.

5. 

   Repeat the same steps to create a backend repository.

   

2. Store Credentials in Jenkins

To integrate Jenkins with SonarQube, AWS ECR, and GitHub, we need to store various credentials securely.

a) Store SonarQube Token in Jenkins

1. Go to Jenkins Dashboard → Manage Jenkins → Credentials.

2. 

   Select the appropriate Global credentials domain.

   

   

3. Click Add Credentials and fill in the details:

   • Kind: Secret Text

   • Scope: Global

   • Secret: <sonar-qube-token>

   • ID: sonar-token

   • Click Create.

b) Store AWS Account ID in Jenkins

1. Go to Credentials → Add Credentials.

2. Enter the details:

   • Kind: Secret Text

   • Scope: Global

   • Secret: <AWS-Account-ID>

   • 

     ID: Account_ID

     

   • Click Create.

c) Store ECR Repository Names in Jenkins

For Frontend Repository:

1. Add New Credential:

   • Kind: Secret Text

   • Scope: Global

   • Secret: frontend

   • ID: ECR_REPO1

   • Click Create.

For Backend Repository:

1. Add New Credential:

   • Kind: Secret Text

   • Scope: Global

   • Secret: backend

   • ID: ECR_REPO2

     

   • Click Create.

d) Store GitHub Credentials in Jenkins

1. Add New Credential:

   • Kind: Username with Password

   • Scope: Global

   • Username: <GitHub-Username>

   • Password: <Personal-Access-Token>

   • ID: GITHUB-APP

e) Store GitHub Personal Access Token in Jenkins

1. Add New Credential:

   • Kind: Secret Text

   • Scope: Global

   • Secret: <Personal-Access-Token>

   • ID: github

here all required credentials are get added

Final Confirmation

✅ ECR Repositories Created
✅ SonarQube Token Stored in Jenkins
✅ AWS Account ID Saved
✅ ECR Repository Names Stored
✅ GitHub Credentials & Token Added

With these credentials configured, Jenkins can authenticate and push Docker images to AWS ECR seamlessly. 🎯

Install and Configure Essential Plugins & Tools in Jenkins

To ensure seamless containerized builds, security analysis, and CI/CD automation, install and configure the necessary Jenkins plugins and tools.

1. Install Required Plugins

Navigate to Jenkins Dashboard → Manage Jenkins → Plugins → Available Plugins, then search and install the following:

✅ Docker – Enables Docker integration.
✅ Docker Pipeline – Provides Docker support in Jenkins pipelines.
✅ Docker Commons – Manages shared Docker images.
✅ Docker API – Allows interaction with Docker daemon.
✅ NodeJS – Supports Node.js builds.
✅ OWASP Dependency-Check – Detects security vulnerabilities in dependencies.
✅ SonarQube Scanner – Enables code quality analysis with SonarQube.

Once installed, restart Jenkins to apply changes.

2. Configure Essential Tools in Jenkins

a) NodeJS Installation

1. Go to Manage Jenkins → Tools

2. Under NodeJS, click Add NodeJS.

3. Fill in the required details.

4. Check the box Install Automatically.

5. 

   Click Save.

b) SonarQube Scanner Installation

1. Under Tools Configuration, go to SonarQube Scanner.

2. Click Add SonarQube Scanner.

3. Fill in the required details.

4. Check Install Automatically.

5. 

   Click Save.

c) OWASP Dependency Check Installation

1. Under Tools Configuration, go to Dependency Check.

2. Click Add Dependency Check.

3. Check Install Automatically from GitHub.

4. 

   Click Save.

d) Docker Installation

1. Under Tools Configuration, go to Docker.

2. Click Add Docker.

3. Fill in the required details.

4. Check Install Automatically from Docker.com.

5. 

   Click Save & Apply.

3. Configure SonarQube Webhook in Jenkins

To enable SonarQube notifications in Jenkins, configure the webhook.

Add SonarQube Server in Jenkins

1. Navigate to Manage Jenkins → Configure System.

2. Scroll to the SonarQube installation section.

3. Click Add SonarQube and enter:

   • Name: sonar-server

   • Server URL: http://<public-ip-of-jenkins-server>:9000

   • Server Authentication Token: <sonar-qube-token-credential-name>

4. 

   Click Apply & Save.

Jenkins is now fully equipped to handle Docker builds, security analysis, and SonarQube scanning in the DevSecOps pipeline. 🚀

Step 7: Create a Jenkins Pipeline for Frontend

This pipeline automates the frontend build, security analysis, Docker image creation, and deployment updates for the DevSecOps pipeline.

1. Create a New Pipeline in Jenkins

1. Navigate to Jenkins Dashboard → New Item.

2. Enter a Pipeline Name (e.g., frontend-pipeline).

3. Select Pipeline as the item type.

4. 

   Click OK to proceed.

5. Scroll down to the Pipeline section and choose Pipeline script.

2. Add the Pipeline Script

Copy and paste the following Jenkinsfile:

pipeline {
    agent any 
    tools {
        nodejs 'nodejs'
    }
    environment {
        SCANNER_HOME = tool 'sonar-scanner'
        AWS_ACCOUNT_ID = credentials('Account_ID')
        AWS_ECR_REPO_NAME = credentials('ECR_REPO1')
        AWS_DEFAULT_REGION = 'ap-south-1'
        REPOSITORY_URI = "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/"
    }
    stages {
        stage('Cleaning Workspace') {
            steps {
                cleanWs()
            }
        }
        stage('Checkout from Git') {
            steps {
                git credentialsId: 'GITHUB-APP', url: 'https://github.com/praduman8435/DevSecOps-in-Action.git', branch: 'main'
            }
        }
        stage('SonarQube Analysis') {
            steps {
                dir('frontend') {
                    withSonarQubeEnv('sonar-server') { // Use withSonarQubeEnv wrapper
                        sh '''
                        $SCANNER_HOME/bin/sonar-scanner \
                        -Dsonar.projectName=frontend \
                        -Dsonar.projectKey=frontend \
                        -Dsonar.sources=.
                        '''
                    }
                }
            }
        }
        stage('Quality Check') {
            steps {
                script {
                    waitForQualityGate abortPipeline: false, credentialsId: 'sonar-token'
                }
            }
        }
        stage('OWASP Dependency-Check Scan') {
            steps {
                dir('Application-Code/backend') {
                    dependencyCheck additionalArguments: '--scan ./ --disableYarnAudit --disableNodeAudit', odcInstallation: 'DP-Check'
                    dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
                }
            }
        }
        stage('Trivy File Scan') {
            steps {
                dir('frontend') {
                    sh 'trivy fs . > trivyfs.txt'
                }
            }
        }
        stage('Docker Image Build') {
            steps {
                script {
                    dir('frontend') {
                        sh 'docker system prune -f'
                        sh 'docker container prune -f'
                        sh 'docker build -t ${AWS_ECR_REPO_NAME} .'
                    }
                }
            }
        }
        stage('ECR Image Pushing') {
            steps {
                script {
                    sh 'aws ecr get-login-password --region ${AWS_DEFAULT_REGION} | docker login --username AWS --password-stdin ${REPOSITORY_URI}'
                    sh 'docker tag ${AWS_ECR_REPO_NAME} ${REPOSITORY_URI}${AWS_ECR_REPO_NAME}:${BUILD_NUMBER}'
                    sh 'docker push ${REPOSITORY_URI}${AWS_ECR_REPO_NAME}:${BUILD_NUMBER}'
                }
            }
        }
        stage('Trivy Image Scan') {
            steps {
                sh 'trivy image ${REPOSITORY_URI}${AWS_ECR_REPO_NAME}:${BUILD_NUMBER} > trivyimage.txt'
            }
        }
        stage('Update Deployment File') {
            environment {
                GIT_REPO_NAME = "DevSecOps-in-Action"
                GIT_USER_NAME = "praduman8435"
            }
            steps {
                dir('k8s-manifests/frontend') {
                    withCredentials([string(credentialsId: 'github', variable: 'GITHUB_TOKEN')]) {
                        sh '''
                        git config user.email "praduman.cnd@gmail.com"
                        git config user.name "praduman"
                        BUILD_NUMBER=${BUILD_NUMBER}
                        imageTag=$(grep -oP '(?<=frontend:)[^ ]+' deployment.yaml)
                        sed -i "s/${AWS_ECR_REPO_NAME}:${imageTag}/${AWS_ECR_REPO_NAME}:${BUILD_NUMBER}/" deployment.yaml
                        git add deployment.yaml
                        git commit -m "Update deployment image to version ${BUILD_NUMBER}"
                        git push https://${GITHUB_TOKEN}@github.com/${GIT_USER_NAME}/${GIT_REPO_NAME} HEAD:main
                        '''
                    }
                }
            }
        }
    }
}

3. Build the Pipeline

1. Click Save & Apply.

2. Click Build Now to start the pipeline.

   

   

4. Verify SonarQube Analysis

1. Open SonarQube UI:

    http://<public-ip-of-jenkins-server>:9000
   

   

2. Check if the SonarQube scan results appear in the UI under the frontend project.

Pipeline Workflow Summary

✅ Code Checkout from GitHub.
✅ SonarQube Scan for code quality analysis.
✅ Security Scans using OWASP Dependency-Check and Trivy.
✅ Docker Build & Push to Amazon ECR.
✅ Deployment Update in Kubernetes manifests.

The frontend pipeline is now fully automated and integrated into the DevSecOps workflow! 🚀

Step 8: Create a Jenkins Pipeline for Backend

This pipeline automates the backend build, security scanning, Docker image creation, and Kubernetes deployment updates for the DevSecOps pipeline.

1. Create a New Pipeline in Jenkins

1. Go to Jenkins Dashboard → New Item.

2. Enter a Pipeline Name (e.g., backend-pipeline).

3. Select Pipeline as the item type.

4. 

   Click OK to proceed.

5. Scroll to the Pipeline section and choose Pipeline script.

2. Add the Pipeline Script

Copy and paste the following Jenkinsfile:

pipeline {
    agent any 
    tools {
        nodejs 'nodejs'
    }
    environment {
        SCANNER_HOME = tool 'sonar-scanner'
        AWS_ACCOUNT_ID = credentials('Account_ID')
        AWS_ECR_REPO_NAME = credentials('ECR_REPO2')
        AWS_DEFAULT_REGION = 'ap-south-1'
        REPOSITORY_URI = "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/"
    }
    stages {
        stage('Cleaning Workspace') {
            steps {
                cleanWs()
            }
        }
        stage('Checkout from Git') {
            steps {
                git credentialsId: 'GITHUB-APP', url: 'https://github.com/praduman8435/DevSecOps-in-Action.git', branch: 'main'
            }
        }
        stage('SonarQube Analysis') {
            steps {
                dir('backend') {
                    withSonarQubeEnv('sonar-server') { // Use withSonarQubeEnv wrapper
                        sh '''
                        $SCANNER_HOME/bin/sonar-scanner \
                        -Dsonar.projectName=backend \
                        -Dsonar.projectKey=backend \
                        -Dsonar.sources=.
                        '''
                    }
                }
            }
        }
        stage('Quality Check') {
            steps {
                script {
                    waitForQualityGate abortPipeline: false, credentialsId: 'sonar-token'
                }
            }
        }
        stage('OWASP Dependency-Check Scan') {
            steps {
                dir('backend') {
                    dependencyCheck additionalArguments: '--scan ./ --disableYarnAudit --disableNodeAudit', odcInstallation: 'DP-Check'
                    dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
                }
            }
        }
        stage('Trivy File Scan') {
            steps {
                dir('backend') {
                    sh 'trivy fs . > trivyfs.txt'
                }
            }
        }
        stage('Docker Image Build') {
            steps {
                script {
                    dir('backend') {
                        sh 'docker system prune -f'
                        sh 'docker container prune -f'
                        sh 'docker build -t ${AWS_ECR_REPO_NAME} .'
                    }
                }
            }
        }
        stage('ECR Image Pushing') {
            steps {
                script {
                    sh 'aws ecr get-login-password --region ${AWS_DEFAULT_REGION} | docker login --username AWS --password-stdin ${REPOSITORY_URI}'
                    sh 'docker tag ${AWS_ECR_REPO_NAME} ${REPOSITORY_URI}${AWS_ECR_REPO_NAME}:${BUILD_NUMBER}'
                    sh 'docker push ${REPOSITORY_URI}${AWS_ECR_REPO_NAME}:${BUILD_NUMBER}'
                }
            }
        }
        stage('Trivy Image Scan') {
            steps {
                sh 'trivy image ${REPOSITORY_URI}${AWS_ECR_REPO_NAME}:${BUILD_NUMBER} > trivyimage.txt'
            }
        }
        stage('Update Deployment File') {
            environment {
                GIT_REPO_NAME = "DevSecOps-in-Action"
                GIT_USER_NAME = "praduman8435"
            }
            steps {
                dir('k8s-manifests/backend') {
                    withCredentials([string(credentialsId: 'github', variable: 'GITHUB_TOKEN')]) {
                        sh '''
                        git config user.email "praduman.cnd@gmail.com"
                        git config user.name "praduman"
                        BUILD_NUMBER=${BUILD_NUMBER}
                        imageTag=$(grep -oP '(?<=backend:)[^ ]+' deployment.yaml)
                        sed -i "s/${AWS_ECR_REPO_NAME}:${imageTag}/${AWS_ECR_REPO_NAME}:${BUILD_NUMBER}/" deployment.yaml
                        git add deployment.yaml
                        git commit -m "Update deployment image to version ${BUILD_NUMBER}"
                        git push https://${GITHUB_TOKEN}@github.com/${GIT_USER_NAME}/${GIT_REPO_NAME} HEAD:main
                        '''
                    }
                }
            }
        }
    }
}

3. Build the Pipeline

1. Click Save & Apply.

2. Click Build Now to trigger the pipeline.

   

4. Verify SonarQube Analysis

1. Open SonarQube UI:

    http://<public-ip-of-jenkins-server>:9000
   

2. 

   Check the SonarQube scan results under the backend project.

Pipeline Workflow Summary

✅ Code Checkout from GitHub.
✅ SonarQube Scan for code quality analysis.
✅ Security Scans using OWASP Dependency-Check and Trivy.
✅ Docker Build & Push to Amazon ECR.
✅ Deployment Update in Kubernetes manifests.

The backend pipeline is now fully automated and integrated into the DevSecOps workflow! 🚀

Step 9: Setup Application in ArgoCD

In this step, we will deploy the application (frontend, backend, database, and ingress) to the EKS cluster using ArgoCD.

1. Open ArgoCD UI

1. Get the ArgoCD server External-IP:

    kubectl get svc -n argocd argocd-server
   

   

2. Access the ArgoCD UI using EXTERNAL-IP in the output.

3. Login using username and password created by you previously.

2. Connect GitHub Repository to ArgoCD

1. Go to Settings → Repositories.

2. 

   Click "Connect Repository using HTTPS".

3. Enter:

   • Project: default

   • Repository URL: https://github.com/praduman8435/DevSecOps-in-Action.git

   • Authentication: None (if public repo)

4. 

   Click "Connect".

3. Create Kubernetes Namespace for Deployment

1. Open terminal and run:

    kubectl create namespace three-tier
   

2. Verify the namespace:

    kubectl get namespaces
   

   

4. Deploy Database in ArgoCD

1. In ArgoCD UI, go to Applications → Click New Application.

2. Fill in the following details:

   • Application Name: three-tier-database

   • Project Name: default

   • Sync Policy: Automatic

   • Repository URL: https://github.com/praduman8435/DevSecOps-in-Action.git

   • Path: k8s-manifests/database

   • Cluster URL: https://kubernetes.default.svc

   • Namespace: three-tier

3. 

   Click Create.

5. Deploy Backend in ArgoCD

1. Go to Applications → Click New Application.

2. Fill in:

   • Application Name: three-tier-backend

   • Project Name: default

   • Sync Policy: Automatic

   • Repository URL: https://github.com/praduman8435/DevSecOps-in-Action.git

   • Path: k8s-manifests/backend

   • Cluster URL: https://kubernetes.default.svc

   • Namespace: three-tier

3. 

   Click Create.

6. Deploy Frontend in ArgoCD

1. Go to Applications → Click New Application.

2. Fill in:

   • Application Name: three-tier-frontend

   • Project Name: default

   • Sync Policy: Automatic

   • Repository URL: https://github.com/praduman8435/DevSecOps-in-Action.git

   • Path: k8s-manifests/frontend

   • Cluster URL: https://kubernetes.default.svc

   • Namespace: three-tier

3. Click Create.

7. Deploy Ingress in ArgoCD

1. Go to Applications → Click New Application.

2. Fill in:

   • Application Name: three-tier-ingress

   • Project Name: default

   • Sync Policy: Automatic

   • Repository URL: https://github.com/praduman8435/DevSecOps-in-Action.git

   • Path: k8s-manifests

   • Cluster URL: https://kubernetes.default.svc

   • Namespace: three-tier

3. Click Create.

8. Verify Deployment in ArgoCD

1. Go to Applications in ArgoCD UI.

2. Check if all applications are Synced and Healthy.

3. If needed, Manually Sync any pending application.

🎉 Congratulations! Your application is now fully deployed using ArgoCD! 🚀 and can be accessed at http://3-111-158-0.nip.io/

Step 10: Configure Monitoring using Prometheus and Grafana

In this step, we will install and configure Prometheus and Grafana using Helm charts to monitor the Kubernetes cluster.

1. Add Helm Repositories for Prometheus & Grafana

Run the following commands to add and update the Helm repositories:

helm repo add stable https://charts.helm.sh/stable
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

2. Install Prometheus and Grafana using Helm

helm install prometheus prometheus-community/kube-prometheus-stack \
  --set prometheus.server.persistentVolume.storageClass=gp2 \
  --set alertmanager.alertmanagerSpec.persistentVolume.storageClass=gp2 \

3. Access Prometheus UI

• Get the Prometheus service details:

    kubectl get svc 
    #look for prometheus-kube-prometheus-prometheus svc
  

• Change the serveice type from ClusterIP to Loadbalancer

    kubectl edit svc  prometheus-kube-prometheus-prometheus
  

  • Find the line type: ClusterIP and change it to type: LoadBalancer.

• You can now access the prometheus server using external-IP of prometheus-kube-prometheus-prometheus svc

    kubectl get svc prometheus-kube-prometheus-prometheus
  

• 

  Open <External-IP>:9999 in your browser.

• Click on Status and select Target. You'll see a list of Targets displayed. In Grafana, we'll use this as a data source.

  

4. Access Grafana UI

• Get the Grafana service details

    kubectl get svc
    #look for the prometheus-grafana svc
  

• 

• By default, it uses ClusterIP. Change it to LoadBalancer:

    kubectl edit svc prometheus-grafana
  

  • Find the line type: ClusterIP and change it to type: LoadBalancer.

• Get the external IP of Grafana:

    kubectl get svc prometheus-grafana
  

  

• Open <EXTERNAL-IP> in your browser.

5. Get Grafana Admin Password

kubectl get secret grafana -n default -o jsonpath="{.data.admin-password}" | base64 --decode

• Username: admin

• Password: (output from the above command)

6. Configure Prometheus as a Data Source in Grafana

1. Login to Grafana UI.

2. Go to Connections → Data Sources.

3. 

   

   Click Data source → Select Prometheus

   

4. Provide the Prometheus URL (<prometheus-loadbalancer-dns>:9090) if not get provided.

5. Click Save & Test.

   

🎉 Congratulations! Your Kubernetes cluster is now being monitored using Prometheus & Grafana!

7. Setting Up Dashboards in Grafana for Kubernetes Monitoring

Grafana allows us to visualize Kubernetes cluster and resource metrics effectively. We’ll set up two essential dashboards to monitor our cluster using Prometheus as the data source.

Dashboard 1: Kubernetes Cluster Monitoring

This dashboard provides an overview of the Kubernetes cluster, including node health, resource usage, and workload performance.

Steps to Import the Dashboard:

• Open the Grafana UI and navigate to Dashboards.

• 

  Click on New → Import.

  

  

• In the Import via Grafana.com field, enter 6417 (Prometheus Kubernetes Cluster Monitoring Dashboard).

  

• Click Load.

• Select Prometheus as the data source.

• Click Import.

  

You should now see a comprehensive dashboard displaying Kubernetes cluster metrics.

Dashboard 2: Kubernetes Resource Monitoring

This dashboard provides insights into individual Kubernetes resources such as pods, deployments, and namespaces.

Steps to Import the Dashboard:

1. Open the Grafana UI and navigate to Dashboards.

2. Click on New → Import.

3. Enter 17375 (Kubernetes Resources Monitoring Dashboard).

4. Click Load.

5. Select Prometheus as the data source.

6. 

   select the data source i.e. prometheus and click on import

7. Click Import.

   

Now, you have two powerful dashboards to monitor both the overall cluster health and specific Kubernetes resources in real-time.

Enjoyed the post? to support my writing!

Conclusion

This Ultimate DevSecOps Project is all about bringing security into the DevOps pipeline while deploying a scalable, secure, and fully automated three-tier application on AWS EKS. By combining the power of Jenkins, SonarQube, Trivy, OWASP Dependency-Check, Terraform, ArgoCD, Prometheus, and Grafana, we've built a robust CI/CD pipeline that ensures code quality, security, and smooth deployments—without any manual headaches!

With SonarQube and OWASP Dependency-Check, we keep our code secure and compliant. Trivy scans our Docker images before they even reach AWS ECR, blocking vulnerabilities before they hit production. Jenkins takes care of automation, while ArgoCD ensures our Kubernetes deployments stay in perfect sync. And of course, Prometheus and Grafana give us full visibility into system health and performance, so we're always on top of things.

This project isn't just a DevSecOps tutorial—it's a real-world playbook for modern software delivery. Whether you're a DevOps pro, security enthusiast, or just diving into cloud automation, this guide sets you up with the tools and best practices to master DevSecOps in Kubernetes.

🚀 Ready to take your DevSecOps game to the next level? Let’s build, secure, and deploy—without limits! 🔐🎯

💡 Let’s connect and discuss DevOps, cloud automation, and cutting-edge technology

🔗 LinkedIn | 💼 Upwork | 🐦 Twitter | 👨‍💻 GitHub

volumes are used to provide storage that pods and containers can access. Unlike regular container storage, which is ephemeral (lost when the container stops), volumes allow data to persist or be shared across containers in a pod. Volumes ensure that data survives container restarts and can be used to share data between containers within the same pod. Any no. of volumes can be attached to pod.

Types of volumes in Kubernetes

1. emptyDir

   • An emptyDir volume is created when a pod is assigned to a node and exists as long as the pod is running. Once the pod is deleted, the data in the emptyDir is also removed.

   • Use case: Temporary storage for data that needs to be shared between containers in the same pod.

   • Example:

       apiVersion: v1
       kind: Pod
       metadata:
         name: emptydir-pod
       spec:
         containers:
           - name: busybox
             image: busybox
             command: ['sh', '-c', 'echo "Writing data to /data/emptydir-volume..."; echo "Hello from Kubesimplify" > /data/emptydir-volume/hello.txt; sleep 3600']
             volumeMounts:
               - name: temp-storage
                 mountPath: /data/emptydir-volume
         volumes:
           - name: temp-storage
             emptyDir: {}
     

     The YAML configuration defines a Kubernetes pod named emptydir-pod that uses an emptyDir volume for temporary storage. A container running the busybox image executes a command to write "Hello from Kubesimplify" to a file named hello.txt at /data/emptydir-volume. This volume is created when the pod starts and exists only as long as the pod is running; all data is deleted when the pod is terminated, making it ideal for temporary storage needs.

       apiVersion: v1
       kind: Pod
       metadata:
         name: emptydir-pod
       spec:
         containers:
           - name: busybox
             image: busybox
             command: ['sh', '-c', 'echo "Writing data to /data/emptydir-volume..."; sleep 3600']
             volumeMounts:
               - name: temp-storage
                 mountPath: /data/emptydir-volume
         volumes:
           - name: temp-storage
             emptyDir:
               medium: Memory
               sizeLimit: 512Mi
     

     This pod runs a busybox container that writes data to a temporary, memory-based volume mounted at /data/emptydir-volume inside the container. The volume is limited to 512 MiB and will be deleted once the pod stops.

2. hostPath

   • A HostPath volume mounts a file or directory from the node’s filesystem (the host) into the pod, allowing the container to access or write data directly on the host.

   • Use case: Accessing specific files or logs on the host node.

   • Example:

       apiVersion: v1
       kind: Pod
       metadata:
         name: hostpath-pod
       spec:
         containers:
           - name: busybox
             image: busybox
             command: ['sh', '-c', 'echo "Writing data to /data/hostpath-volume..."; echo "Hello from Kubesimplify" > /data/hostpath-volume/hello.txt; sleep 3600']
             volumeMounts:
               - name: host-storage
                 mountPath: /data/hostpath-volume
         volumes:
           - name: host-storage
             hostPath:
               path: /tmp/hostpath
               type: DirectoryOrCreate
     

     The HostPath Volume mounts the directory /tmp/hostpath from the node's filesystem into the container at /data/hostpath-volume. The pod writes the text "Hello from Kubesimplify" to a file named hello.txt inside the container at /data/hostpath-volume/hello.txt. Since this is a HostPath volume, the file is actually created in /tmp/hostpath/hello.txt on the host node. Additionally, the data stored in the HostPath volume will persist even if the pod is deleted, as it is stored directly on the host node’s filesystem.

3. Persistent Volume (PV)

   • It is a piece of storage in a Kubernetes cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes.

   • PVs are cluster resources that exist independently of any individual pod and are designed for long-term data storage.

   • They can be backed by various types of storage systems, such as NFS, iSCSI, cloud storage (like AWS EBS, Google Persistent Disk), or local storage.

   • PVs have a lifecycle that is separate from the pods that use them, allowing data to persist even if pods are deleted or moved.