Implementing a Security-Centric Cloud Native CI/CD Pipeline: A Real-World Demonstration (using Terraform and GCP)

In today’s software development landscape, the swift delivery of new features and updates is paramount; however, this must be balanced against the increasing importance of robust security practices. CI/CD (Continuous Integration/Continuous Delivery or Deployment) pipelines provide a framework for automating the building, testing, and deployment of applications, bolstering both speed and reliability. This project demonstrates a CI/CD pipeline where security is integrated as a first-class citizen throughout every stage.

Project Overview

#

The core objective of this project is to establish a CI/CD pipeline that prioritizes the following principles:

• Security by Design: Security considerations are embedded in all phases of the development and deployment workflow.

• Automation: Leveraging automation to maximize efficiency, reduce potential human error, and enforce security best practices.

• Continuous Monitoring: Implementing systems and application-level monitoring for proactive issue detection and rapid response.

• Infrastructure as Code with Terraform: Utilizing Terraform, a popular Infrastructure as Code (IaC) tool, to predictably create, change, and improve cloud infrastructure.

• Google Cloud as the Cloud Platform: Leveraging Google Cloud’s compute engine services and products to provision and manage the required infrastructure components.

Key Technologies

#

• Kubernetes: Container orchestration for application deployment and management.

• Jenkins: CI/CD automation server.

• SonarQube: Static code analysis to ensure code quality and identify potential security issues.

• Aqua Trivy: Vulnerability scanning for code dependencies and container images.

• Nexus Repository: Secure storage for build artifacts.

• Docker: Containerization for application packaging.

• Docker Hub: Docker image registry.

• Kubeaudit: Tool to audit Kubernetes clusters for various different security concerns.

• Grafana: For system and application-level monitoring and alerting.

• Prometheus: For collecting and querying metrics from services and endpoints.

• Gmail: For status notifications and alerts.

Workflow

#

A client requests a feature or change in the application and creates a Jira ticket. The Jira ticket is then assigned to a developer.

1. Development and Version Control:

   • Developers work on feature branches within a Git repository (e.g., GitHub) and successfully test the new feature in local environment.

   • Once changes are pushed along with source code to GitHub repository, it trigger the CI/CD pipeline.

2. Build and Unit Testing:

   • The project’s build system (e.g., Maven) compiles the code, and find out if there are any syntax base errors in the code.

   • Once code compilation is successful, Unit tests are executed to validate code functionality.

3. Code Quality and Security Analysis:

   • SonarQube analyzes code for maintainability, potential bugs, and security vulnerabilities.

   • Aqua Trivy scans for vulnerabilities within project dependencies.

4. Artifact Creation and Storage:

   • A build artifact (e.g., JAR, WAR) is generated.

   • The artifact is pushed to Nexus Repository for secure storage and proper release.

5. Docker Image Creation:

   • Docker builds a container image incorporating the build artifact and tag it properly.

   • Aqua Trivy scans the image for vulnerabilities.

   • Docker image is pushed to dockerhub.

6. Kubernetes Deployment:

   • Kube Audit secures the Kubernetes cluster.

   • The image is deployed to the Kubernetes cluster if all security scans pass.

7. Mail Notification

   • Client and devops engineer will receive the mail notification if pipeline is successful or failed.

   • Notifications are sent to mail for deployment status, errors, and critical alerts.

8. Monitoring:

   • Monitoring tools (Prometheus and Grafana) track the health of the system and application.

   • For system-level monitoring of hardware, Jenkins is used. Node Exporter monitors Jenkins.

Step-by-Step Project Execution

#

Phase 1: Creating Infrastructure

#

The project kicks off with the creation of an isolated network environment a critical step for maintaining security and control. This phase includes:

1. Network Environment Creation: Setting up a dedicated and secure network environment to isolate and protect the project’s infrastructure from external threats.

   • Create a Custom VPC on Google Cloud Platform with custom firewall rules allowing traffic to specific ports.

     Terraform Code:

     #

     # Network creation
     #-----------------#
     resource "google_compute_network" "dev-cicd-vpc" {
         auto_create_subnetworks = true
         description             = "VPC for secure CICD pipeline."
         mtu                     = 1460
         name                    = "dev-cicd-vpc"
         project                 = var.gcp_project_id
         routing_mode            = "REGIONAL"
     }
     
     # Firewall Rules
     #-----------------#
     
     ## Custom firewall rules
     resource "google_compute_firewall" "dev-cicd-vpc-allow-custom" {
         name                    = "dev-cicd-vpc-allow-custom"
         project                 = var.gcp_project_id
         network                 = google_compute_network.dev-cicd-vpc.name
         description             = "Allows connection from any source to any instance on the network using custom protocols."
         direction               = "INGRESS"
         priority                = 65534
         source_ranges           = ["10.128.0.0/9", "0.0.0.0/0"] 
         allow {
             protocol = "tcp" 
             ports    = ["80", "443", "465", "6443", "3000-10000", "30000-32767"]
         }
     }
     
     ## ICMP
     resource "google_compute_firewall" "dev-cicd-vpc-allow-icmp" {
         network     = google_compute_network.dev-cicd-vpc.name
         project     = var.gcp_project_id
         direction   = "INGRESS"
         priority    = 65534
         source_ranges = ["0.0.0.0/0"]
         name        = "dev-cicd-vpc-allow-icmp" 
         description = "Allows ICMP connections from any source to any instance on the network."
         allow {
             protocol = "icmp"
         }
     }
     
     ## SSH
     resource "google_compute_firewall" "dev-cicd-vpc-allow-ssh" {
         network     = google_compute_network.dev-cicd-vpc.name
         project     = var.gcp_project_id
         direction   = "INGRESS"
         priority    = 65534
         source_ranges = ["0.0.0.0/0"]
         name        = "dev-cicd-vpc-allow-ssh"
         description = "Allows TCP connections from any source to any instance on the network using port 22."
         allow {
             protocol = "tcp" 
             ports    = ["22"]
         }
     }

Project Repo

2. Kubernetes Cluster Deployment: Establishing a robust Kubernetes (K8s) cluster as the foundation for the project’s deployment needs. This cluster is thoroughly scanned and secured to ensure a resilient platform.

   • Create 3 VM’s on GCP for Kubernetes (1 Master and 2 Slave Nodes) and configure Kubernetes on them.

     ## Master Node VM
     resource "google_compute_instance" "cluster-instances-node-master" {
         boot_disk {
             auto_delete = true
             device_name = "k8-cluster-nodes"
     
             initialize_params {
             image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
             size  = 25
             type  = "pd-balanced"
             }
     
             mode = "READ_WRITE"
         }
     
         can_ip_forward      = false
         deletion_protection = false
         enable_display      = false
     
         labels = {
             goog-ec-src = "vm_add-tf"
             node        = "master"
         }
     
         machine_type = "e2-medium"
     
         metadata = {
             startup-script = file("./scripts/masterVM.sh")
         }
     
         name = "cluster-instances-node-master"
     
         network_interface {
             access_config {
             network_tier = "STANDARD"
             }
     
             queue_count = 0
             stack_type  = "IPV4_ONLY"
             network     = google_compute_network.dev-cicd-vpc.name
         }
     
         scheduling {
             automatic_restart   = true
             on_host_maintenance = "MIGRATE"
             preemptible         = false
             provisioning_model  = "STANDARD"
         }
     
         service_account {
             email  = var.gcp_service_account_email
             scopes = ["https://www.googleapis.com/auth/cloud-platform"]
         }
     
         shielded_instance_config {
             enable_integrity_monitoring = true
             enable_secure_boot          = false
             enable_vtpm                 = true
         }
     
         tags = ["http-server", "https-server", "lb-health-check"]
         zone = "asia-south1-c"
     }
     
     ## Slave Node VM (2 nodes)
     resource "google_compute_instance" "cluster-instances-node-slave" {
         count = 2
     
         boot_disk {
             auto_delete = true
             device_name = "k8-cluster-nodes"
     
             initialize_params {
             image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
             size  = 25
             type  = "pd-balanced"
             }
     
             mode = "READ_WRITE"
         }
     
         can_ip_forward      = false
         deletion_protection = false
         enable_display      = false
     
         labels = {
             goog-ec-src = "vm_add-tf"
             node        = "slave0${count.index + 1}"
         }
     
         machine_type = "e2-medium"
     
         metadata = {
             startup-script = file("./scripts/slaveVM.sh")
         }
     
         name = "cluster-instances-node-slave0${count.index + 1}"
     
         network_interface {
             access_config {
             network_tier = "STANDARD"
             }
     
             queue_count = 0
             stack_type  = "IPV4_ONLY"
             network     = google_compute_network.dev-cicd-vpc.name
         }
     
         scheduling {
             automatic_restart   = true
             on_host_maintenance = "MIGRATE"
             preemptible         = false
             provisioning_model  = "STANDARD"
         }
     
         service_account {
             email  = var.gcp_service_account_email
             scopes = ["https://www.googleapis.com/auth/cloud-platform"]
         }
     
         shielded_instance_config {
             enable_integrity_monitoring = true
             enable_secure_boot          = false
             enable_vtpm                 = true
         }
     
         tags = ["http-server", "https-server", "lb-health-check"]
         zone = "asia-south1-c"
     }

3. Essential Server Setup: Configuring and deploying essential servers to support the CI/CD processes. This includes:

   • Jenkins Server: For continuous integration and deployment automation.

     # Jenkins VM
     #-----------------#
     
     resource "google_compute_instance" "jenkins" {
         boot_disk {
             auto_delete = true
             device_name = "jenkins-vm"
     
             initialize_params {
             image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
             size  = 30
             type  = "pd-balanced"
             }
     
             mode = "READ_WRITE"
         }
     
         can_ip_forward      = false
         deletion_protection = false
         enable_display      = false
     
         labels = {
             goog-ec-src = "vm_add-tf"
             jenkins     = ""
         }
     
         machine_type = "e2-standard-2"
     
         metadata = {
             startup-script = file("./scripts/jenkins.sh")
         }
     
         name = "jenkins"
     
         network_interface {
             access_config {
             network_tier = "STANDARD"
             }
     
             queue_count = 0
             stack_type  = "IPV4_ONLY"
             network     = google_compute_network.dev-cicd-vpc.name
         }
     
         scheduling {
             automatic_restart   = true
             on_host_maintenance = "MIGRATE"
             preemptible         = false
             provisioning_model  = "STANDARD"
         }
     
         service_account {
             email  = var.gcp_service_account_email
             scopes = ["https://www.googleapis.com/auth/cloud-platform"]
         }
     
         shielded_instance_config {
             enable_integrity_monitoring = true
             enable_secure_boot          = false
             enable_vtpm                 = true
         }
     
         tags = ["http-server", "https-server", "lb-health-check"]
         zone = "asia-south1-c"
     }

   • SonarQube Server: For continuous code quality and security analysis.

     # Sonarqube VM
     #---------------#
     resource "google_compute_instance" "sonarqube" {
         boot_disk {
             auto_delete = true
             device_name = "k8-cluster-nodes"
     
             initialize_params {
             image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
             size  = 20
             type  = "pd-balanced"
             }
     
             mode = "READ_WRITE"
         }
     
         can_ip_forward      = false
         deletion_protection = false
         enable_display      = false
     
         labels = {
             goog-ec-src = "vm_add-tf"
             sonarqube   = ""
         }
     
         machine_type = "e2-medium"
     
         metadata = {
             startup-script = file("./scripts/sonarqube.sh")
         }
     
         name = "sonarqube"
     
         network_interface {
             access_config {
             network_tier = "STANDARD"
             }
     
             queue_count = 0
             stack_type  = "IPV4_ONLY"
             network     = google_compute_network.dev-cicd-vpc.name
         }
     
         scheduling {
             automatic_restart   = true
             on_host_maintenance = "MIGRATE"
             preemptible         = false
             provisioning_model  = "STANDARD"
         }
     
         service_account {
             email  = var.gcp_service_account_email
             scopes = ["https://www.googleapis.com/auth/cloud-platform"]
         }
     
         shielded_instance_config {
             enable_integrity_monitoring = true
             enable_secure_boot          = false
             enable_vtpm                 = true
         }
     
         tags = ["http-server", "https-server", "lb-health-check"]
         zone = "asia-south1-c"
     }

   • Monitoring Servers: To support system and application-level monitoring from the outset.

     # Monitoring VM
     #-----------------#
     
     resource "google_compute_instance" "monitor" {
         boot_disk {
             auto_delete = true
             device_name = "monitoring"
     
             initialize_params {
             image = "projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20240307b"
             size  = 20
             type  = "pd-balanced"
             }
     
             mode = "READ_WRITE"
         }
     
         can_ip_forward      = false
         deletion_protection = false
         enable_display      = false
     
         labels = {
             goog-ec-src = "vm_add-tf"
             monitor     = ""
         }
     
         machine_type = "e2-standard-2"
     
         metadata = {
             startup-script = file("./scripts/monitoring.sh")
         }
     
         name = "monitor"
     
         network_interface {
             access_config {
             network_tier = "PREMIUM"
             }
     
             queue_count = 0
             stack_type  = "IPV4_ONLY"
             network     = google_compute_network.dev-cicd-vpc.name
         }
     
         scheduling {
             automatic_restart   = true
             on_host_maintenance = "MIGRATE"
             preemptible         = false
             provisioning_model  = "STANDARD"
         }
     
         service_account {
             email  = var.gcp_service_account_email
             scopes = ["https://www.googleapis.com/auth/cloud-platform"]
         }
     
         shielded_instance_config {
             enable_integrity_monitoring = true
             enable_secure_boot          = false
             enable_vtpm                 = true
         }
     
         tags = ["http-server", "https-server", "lb-health-check"]
         zone = "asia-south1-c"
     }

Phase 2: Version Control and Notification System

#

In this phase, the focus is on version control and establishing a notification system to keep stakeholders informed:

1. Version Control with GitHub:

   • Private Repository Setup: Source code is pushed to a private GitHub repository to ensure security and privacy.

   • Branching Strategy: Implementing a branching strategy (e.g., feature branches, development, main) to manage code changes efficiently.

2. Mail Notification System:

   • Configuration: Setting up an email notification system to alert stakeholders about the project’s progress and any critical issues.

   • Integration: Integrating the notification system with the CI/CD pipeline to provide real-time updates on build statuses, deployments, and alerts.

Phase 3: Building the CI/CD Pipeline

#

The CI/CD pipeline is the heart of the project, automating the deployment process and ensuring robust security measures:

1. Pipeline Design and Implementation:

   • Jenkins Pipeline Configuration: Defining a declarative Jenkins pipeline (Jenkinsfile) that outlines the steps for building, testing, and deploying the application.

2. Security Measures:

   • Code Quality Analysis with SonarQube: Integrating SonarQube to perform static code analysis and ensure code quality.

   • Vulnerability Scanning with Aqua Trivy: Scanning code dependencies and Docker images for vulnerabilities to enhance security.

3. Automated Deployment:

   • Artifact Creation and Storage: Building the application artifacts (e.g., JAR, WAR) and securely storing them in Nexus Repository.

   • Container Image Build and Scan: Creating Docker container images and scanning them for vulnerabilities before deployment.

   • Kubernetes Deployment: Automating the deployment of container images to the Kubernetes cluster, ensuring a smooth transition from development to production.

Phase 4: Monitoring at Scale

#

In the final phase, system-level and application-level monitoring are implemented to maintain the health of the application and provide real-time insights:

1. System Monitoring with Prometheus and Grafana:

   • Metric Collection: Prometheus collects metrics from various services and endpoints, providing detailed insights into system performance.

   • Visualization and Alerting: Grafana visualizes these metrics and sets up alerting mechanisms to notify of any performance issues or anomalies.

     

2. Application Monitoring:

   • Endpoint Monitoring with Blackbox Exporter: Ensuring that web endpoints are monitored for availability and responsiveness.

   • Traffic and Performance Monitoring: Monitoring website traffic and application performance to ensure optimal user experience.

3. Continuous Improvement:

   • Feedback Loop: Using monitoring data to identify areas for improvement and optimize the CI/CD pipeline and deployment processes continually.

   • Proactive Maintenance: Implementing proactive measures based on monitoring insights to prevent issues before they impact users.

Security Considerations

#

• Pre-deployment Kubernetes Security: The Kubernetes cluster is hardened to minimize potential attack surfaces.

• Pipeline as Code: Pipeline configuration (e.g., Jenkinsfile) is treated as code, enabling versioning and peer review.

• Vulnerability Scanning: Scans at the code and image stages proactively identify issues.

• Monitoring: Anomalies or suspicious behavior are detected early.

Benefits of a Secure CI/CD Pipeline

#

• Enhanced Security Posture: Integration of security measures throughout the process mitigates risks.

• Increased Confidence: Rigorous testing and security checks lead to more reliable releases.

• Improved Agility: Automation streamlines development and reduces time to market.

• Reduced Operational Overhead: Fewer manual processes minimize potential errors.

Conclusion

#

This project exemplifies a comprehensive CI/CD pipeline that can serve as a blueprint for organizations looking to implement or refine their own CI/CD strategies. It demonstrates the importance of each phase and the inter-connectedness of various tools and practices that contribute to a successful software delivery lifecycle.