Skip to content

🔄 CI/CD Pipeline Overview

What is CI/CD?

CI/CD (Continuous Integration/Continuous Deployment) is a software development practice that enables teams to deliver code changes more frequently and reliably. It combines continuous integration, continuous delivery, and continuous deployment to automate the software delivery process.

CI/CD Pipeline Architecture

graph LR
    subgraph "Development"
        DEV[Developer]
        GIT[Git Repository]
    end

    subgraph "CI/CD Pipeline"
        BUILD[Build & Test]
        SEC[Security Scan]
        DEPLOY[Deploy]
        MON[Monitor]
    end

    subgraph "Environments"
        DEV_ENV[Development]
        STAGE[Staging]
        PROD[Production]
    end

    DEV --> GIT
    GIT --> BUILD
    BUILD --> SEC
    SEC --> DEPLOY
    DEPLOY --> DEV_ENV
    DEPLOY --> STAGE
    DEPLOY --> PROD
    PROD --> MON

Pipeline Stages

1. Continuous Integration (CI)

Code Commit

# Trigger pipeline on code changes
on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

Build Process

# Build and test application
build:
  runs-on: ubuntu-latest
  steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'

    - name: Install dependencies
      run: |
        pip install -r requirements.txt
        pip install -r requirements-dev.txt

    - name: Run tests
      run: |
        python -m pytest tests/
        python -m pytest integration_tests/

    - name: Build Docker image
      run: |
        docker build -t my-app:${{ github.sha }} .
        docker push my-app:${{ github.sha }}

Quality Gates

# Quality checks
quality:
  runs-on: ubuntu-latest
  needs: build
  steps:
    - name: Code quality check
      run: |
        flake8 src/
        black --check src/
        isort --check-only src/

    - name: Security scan
      run: |
        bandit -r src/
        safety check

    - name: Coverage report
      run: |
        coverage run -m pytest
        coverage report --fail-under=80

2. Continuous Delivery (CD)

Environment Promotion

# Deploy to different environments
deploy-dev:
  runs-on: ubuntu-latest
  needs: [build, quality]
  environment: development
  steps:
    - name: Deploy to development
      run: |
        kubectl set image deployment/my-app my-app=my-app:${{ github.sha }} -n dev
        kubectl rollout status deployment/my-app -n dev

deploy-staging:
  runs-on: ubuntu-latest
  needs: deploy-dev
  environment: staging
  steps:
    - name: Deploy to staging
      run: |
        kubectl set image deployment/my-app my-app=my-app:${{ github.sha }} -n staging
        kubectl rollout status deployment/my-app -n staging

deploy-production:
  runs-on: ubuntu-latest
  needs: deploy-staging
  environment: production
  steps:
    - name: Deploy to production
      run: |
        kubectl set image deployment/my-app my-app=my-app:${{ github.sha }} -n prod
        kubectl rollout status deployment/my-app -n prod

Infrastructure as Code

# Deploy infrastructure
deploy-infrastructure:
  runs-on: ubuntu-latest
  steps:
    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v3

    - name: Terraform init
      run: terraform init

    - name: Terraform plan
      run: terraform plan -out=tfplan

    - name: Terraform apply
      run: terraform apply tfplan

3. Continuous Deployment

Automated Deployment

# Automated deployment with approval gates
deploy:
  runs-on: ubuntu-latest
  needs: [build, test, security]
  environment:
    name: production
    url: https://my-app.example.com
  steps:
    - name: Deploy application
      run: |
        # Deploy to Kubernetes
        kubectl apply -f k8s/

        # Wait for deployment
        kubectl rollout status deployment/my-app

        # Run health checks
        ./scripts/health-check.sh

    - name: Run smoke tests
      run: |
        ./scripts/smoke-tests.sh

    - name: Notify team
      run: |
        curl -X POST $SLACK_WEBHOOK \
          -H 'Content-type: application/json' \
          -d '{"text":"Deployment completed successfully!"}'

GitHub Actions Workflows

1. Main CI/CD Pipeline

# .github/workflows/ci-cd.yml
name: CI/CD Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          pip install -r requirements-dev.txt

      - name: Run tests
        run: |
          pytest tests/ --cov=src --cov-report=xml

      - name: Upload coverage
        uses: codecov/codecov-action@v3
        with:
          file: ./coverage.xml

  security:
    runs-on: ubuntu-latest
    needs: test
    steps:
      - uses: actions/checkout@v4

      - name: Run security scan
        uses: snyk/actions/python@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          args: --severity-threshold=high

  build:
    runs-on: ubuntu-latest
    needs: [test, security]
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4

      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push Docker image
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  deploy-dev:
    runs-on: ubuntu-latest
    needs: build
    environment: development
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to development
        run: |
          kubectl config use-context dev-cluster
          kubectl set image deployment/my-app my-app=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} -n dev
          kubectl rollout status deployment/my-app -n dev

  deploy-staging:
    runs-on: ubuntu-latest
    needs: deploy-dev
    environment: staging
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to staging
        run: |
          kubectl config use-context staging-cluster
          kubectl set image deployment/my-app my-app=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} -n staging
          kubectl rollout status deployment/my-app -n staging

  deploy-production:
    runs-on: ubuntu-latest
    needs: deploy-staging
    environment: production
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to production
        run: |
          kubectl config use-context prod-cluster
          kubectl set image deployment/my-app my-app=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }} -n prod
          kubectl rollout status deployment/my-app -n prod

      - name: Run post-deployment tests
        run: |
          ./scripts/post-deployment-tests.sh

2. Infrastructure Pipeline

# .github/workflows/infrastructure.yml
name: Infrastructure Pipeline

on:
  push:
    branches: [main]
    paths: ['terraform/**', 'kubernetes/**']
  workflow_dispatch:

jobs:
  terraform-plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.5.0

      - name: Terraform Init
        run: terraform init
        working-directory: ./terraform

      - name: Terraform Plan
        run: terraform plan -out=tfplan
        working-directory: ./terraform

      - name: Upload Terraform Plan
        uses: actions/upload-artifact@v4
        with:
          name: terraform-plan
          path: ./terraform/tfplan

  terraform-apply:
    runs-on: ubuntu-latest
    needs: terraform-plan
    environment: production
    steps:
      - uses: actions/checkout@v4

      - name: Download Terraform Plan
        uses: actions/download-artifact@v4
        with:
          name: terraform-plan
          path: ./terraform

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.5.0

      - name: Terraform Apply
        run: terraform apply tfplan
        working-directory: ./terraform

3. Monitoring Pipeline

# .github/workflows/monitoring.yml
name: Monitoring Pipeline

on:
  schedule:
    - cron: '0 */6 * * *'  # Every 6 hours
  workflow_dispatch:

jobs:
  deploy-monitoring:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Deploy Prometheus
        run: |
          kubectl apply -f kubernetes/monitoring/prometheus/
          kubectl rollout status deployment/prometheus -n monitoring

      - name: Deploy Grafana
        run: |
          kubectl apply -f kubernetes/monitoring/grafana/
          kubectl rollout status deployment/grafana -n monitoring

      - name: Deploy AlertManager
        run: |
          kubectl apply -f kubernetes/monitoring/alertmanager/
          kubectl rollout status deployment/alertmanager -n monitoring

  chaos-testing:
    runs-on: ubuntu-latest
    needs: deploy-monitoring
    steps:
      - uses: actions/checkout@v4

      - name: Run chaos experiments
        run: |
          kubectl apply -f kubernetes/chaos-engineering/chaos-experiments.yaml
          sleep 300  # Wait for experiments to complete
          kubectl get chaos-experiments

Environment Management

1. Environment Configuration

Development Environment

# environments/dev.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: dev
data:
  DATABASE_URL: "postgresql://dev:password@dev-db:5432/dev"
  REDIS_URL: "redis://dev-redis:6379"
  LOG_LEVEL: "DEBUG"
  FEATURE_FLAGS: "experimental_features=true"

Staging Environment

# environments/staging.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: staging
data:
  DATABASE_URL: "postgresql://staging:password@staging-db:5432/staging"
  REDIS_URL: "redis://staging-redis:6379"
  LOG_LEVEL: "INFO"
  FEATURE_FLAGS: "experimental_features=false"

Production Environment

# environments/prod.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: prod
data:
  DATABASE_URL: "postgresql://prod:password@prod-db:5432/prod"
  REDIS_URL: "redis://prod-redis:6379"
  LOG_LEVEL: "WARNING"
  FEATURE_FLAGS: "experimental_features=false"

2. Environment Promotion

# Promote configuration between environments
promote-config:
  runs-on: ubuntu-latest
  steps:
    - name: Promote to staging
      run: |
        kubectl get configmap app-config -n dev -o yaml | \
        sed 's/namespace: dev/namespace: staging/' | \
        kubectl apply -f -

    - name: Promote to production
      run: |
        kubectl get configmap app-config -n staging -o yaml | \
        sed 's/namespace: staging/namespace: prod/' | \
        kubectl apply -f -

Security in CI/CD

1. Secret Management

# Use GitHub Secrets for sensitive data
env:
  DATABASE_PASSWORD: ${{ secrets.DATABASE_PASSWORD }}
  API_KEY: ${{ secrets.API_KEY }}
  SSL_CERT: ${{ secrets.SSL_CERT }}

2. Security Scanning

# Security scanning in pipeline
security-scan:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4

    - name: Run Trivy vulnerability scanner
      uses: aquasecurity/trivy-action@master
      with:
        image-ref: 'my-app:${{ github.sha }}'
        format: 'sarif'
        output: 'trivy-results.sarif'

    - name: Upload Trivy scan results
      uses: github/codeql-action/upload-sarif@v2
      with:
        sarif_file: 'trivy-results.sarif'

3. Compliance Checks

# Compliance checking
compliance:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4

    - name: Run compliance checks
      run: |
        # Check for sensitive data in code
        trufflehog --regex --entropy=False .

        # Check for license compliance
        license-checker --summary

        # Check for dependency vulnerabilities
        npm audit --audit-level=high

Monitoring and Observability

1. Pipeline Metrics

# Track pipeline metrics
metrics:
  runs-on: ubuntu-latest
  steps:
    - name: Collect pipeline metrics
      run: |
        # Build time
        echo "BUILD_TIME=$(date +%s)" >> $GITHUB_ENV

        # Test results
        echo "TEST_PASSED=${{ steps.test.outcome == 'success' }}" >> $GITHUB_ENV

        # Deployment time
        echo "DEPLOY_TIME=$(date +%s)" >> $GITHUB_ENV

2. Application Monitoring

# Deploy monitoring stack
monitoring:
  runs-on: ubuntu-latest
  steps:
    - name: Deploy Prometheus
      run: |
        helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
        helm install prometheus prometheus-community/kube-prometheus-stack

    - name: Deploy Grafana
      run: |
        helm repo add grafana https://grafana.github.io/helm-charts
        helm install grafana grafana/grafana

Best Practices

1. Pipeline Design

  • Keep pipelines fast: Minimize build and test time
  • Use caching: Cache dependencies and build artifacts
  • Parallel execution: Run independent jobs in parallel
  • Fail fast: Stop pipeline early if critical tests fail

2. Security

  • Scan for vulnerabilities: Regular security scanning
  • Use secrets management: Never hardcode secrets
  • Implement least privilege: Minimal required permissions
  • Audit pipeline changes: Track all pipeline modifications

3. Reliability

  • Implement rollback: Quick rollback procedures
  • Use blue-green deployment: Zero-downtime deployments
  • Monitor deployments: Real-time deployment monitoring
  • Test in production: Use feature flags and canary deployments

4. Observability

  • Comprehensive logging: Log all pipeline activities
  • Metrics collection: Track pipeline performance
  • Alerting: Proactive alerting for issues
  • Dashboards: Visual pipeline monitoring

Troubleshooting

1. Common Issues

Build Failures

# Debug build issues
docker build --progress=plain --no-cache .
docker logs <container-id>

Test Failures

# Debug test issues
pytest -v --tb=short
pytest --lf  # Run last failed tests

Deployment Failures

# Debug deployment issues
kubectl describe pod <pod-name>
kubectl logs <pod-name>
kubectl get events --sort-by=.metadata.creationTimestamp

2. Pipeline Debugging

# Enable debug mode
env:
  ACTIONS_STEP_DEBUG: true
  ACTIONS_RUNNER_DEBUG: true

3. Rollback Procedures

# Automated rollback
rollback:
  runs-on: ubuntu-latest
  if: failure()
  steps:
    - name: Rollback deployment
      run: |
        kubectl rollout undo deployment/my-app
        kubectl rollout status deployment/my-app

1. GitOps

  • Declarative deployments: Infrastructure as code
  • Automated synchronization: Git as source of truth
  • Pull-based deployments: Security and audit benefits

2. Progressive Delivery

  • Canary deployments: Gradual traffic shifting
  • Feature flags: Runtime feature management
  • A/B testing: Data-driven deployment decisions

3. AI/ML Integration

  • Intelligent testing: AI-powered test generation
  • Predictive deployment: ML-based deployment decisions
  • Automated optimization: Self-optimizing pipelines

4. Platform Engineering

  • Internal developer platforms: Self-service infrastructure
  • Developer experience: Improved developer productivity
  • Standardization: Consistent development practices