DevOps and CI/CD

CI/CD stands for Continuous Integration/Continuous Deployment (or Delivery).

Continuous Integration (CI):

• Developers frequently merge code changes into a central repository (multiple times a day)
• Automated builds and tests run on every commit
• Quickly detects integration bugs and code quality issues

Continuous Deployment/Delivery (CD):

• Continuous Delivery: Code is automatically prepared for release to production
• Continuous Deployment: Every change that passes tests is automatically deployed to production

Example CI/CD Pipeline for .NET Core:

# Azure DevOps Pipeline
trigger:
  - main

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: UseDotNet@2
  inputs:
    version: '8.x'

- task: DotNetCoreCLI@2
  displayName: 'Restore packages'
  inputs:
    command: 'restore'
    projects: '**/*.csproj'

- task: DotNetCoreCLI@2
  displayName: 'Build'
  inputs:
    command: 'build'
    arguments: '--configuration Release'

- task: DotNetCoreCLI@2
  displayName: 'Run Tests'
  inputs:
    command: 'test'
    projects: '**/*Tests.csproj'

- task: DotNetCoreCLI@2
  displayName: 'Publish'
  inputs:
    command: 'publish'
    publishWebProjects: true
    arguments: '--configuration Release --output $(Build.ArtifactStagingDirectory)'

- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: 'drop'

Why CI/CD is Important:

1. Faster Time to Market: Automated deployments reduce release cycles from weeks to hours
2. Higher Code Quality: Automated testing catches bugs early
3. Reduced Risk: Small, frequent deployments are easier to troubleshoot than large releases
4. Better Collaboration: Teams can work on features independently without integration hell
5. Faster Feedback: Developers get immediate feedback on code changes
6. Consistency: Eliminates "works on my machine" problems with standardized builds

Yes, Docker is essential for modern .NET Core application deployment.

Containerization is a lightweight virtualization method that packages an application and all its dependencies into a standardized unit called a container.

Key Concepts:

Container vs Virtual Machine:

• VM: Includes full OS, takes GBs of space, slower to start
• Container: Shares host OS kernel, MBs in size, starts in seconds

Docker Components:

• Image: Read-only template with application code and dependencies
• Container: Running instance of an image
• Dockerfile: Instructions to build an image
• Docker Hub/Registry: Repository for storing images

Example Dockerfile for .NET Core API:

# Multi-stage build for optimization
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /src

# Copy csproj and restore dependencies
COPY ["MyApi/MyApi.csproj", "MyApi/"]
RUN dotnet restore "MyApi/MyApi.csproj"

# Copy everything else and build
COPY . .
WORKDIR "/src/MyApi"
RUN dotnet build "MyApi.csproj" -c Release -o /app/build

# Publish the application
FROM build AS publish
RUN dotnet publish "MyApi.csproj" -c Release -o /app/publish /p:UseAppHost=false

# Build runtime image
FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS final
WORKDIR /app
EXPOSE 80
EXPOSE 443
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "MyApi.dll"]

Docker Compose for Multi-Container Setup:

version: '3.8'

services:
  api:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "5000:80"
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ConnectionStrings__DefaultConnection=Server=db;Database=MyDb;User=sa;Password=YourPassword123!
    depends_on:
      - db
    networks:
      - app-network

  db:
    image: mcr.microsoft.com/mssql/server:2022-latest
    environment:
      - ACCEPT_EULA=Y
      - SA_PASSWORD=YourPassword123!
    ports:
      - "1433:1433"
    volumes:
      - sqldata:/var/opt/mssql
    networks:
      - app-network

volumes:
  sqldata:

networks:
  app-network:
    driver: bridge

Benefits of Containerization:

1. Consistency: Same environment in dev, test, and production
2. Isolation: Applications run independently without conflicts
3. Portability: Run anywhere Docker is supported
4. Scalability: Easy to scale horizontally
5. Resource Efficiency: Lightweight compared to VMs
6. Version Control: Images are versioned and immutable

Common Docker Commands:

# Build image
docker build -t myapi:v1 .

# Run container
docker run -d -p 5000:80 --name myapi-container myapi:v1

# View running containers
docker ps

# View logs
docker logs myapi-container

# Stop container
docker stop myapi-container

# Remove container
docker rm myapi-container

Kubernetes (K8s) is an open-source container orchestration platform that automates deployment, scaling, and management of containerized applications.

Problems Kubernetes Solves:

1. Container Management at Scale: Managing hundreds or thousands of containers manually is impossible
2. High Availability: Ensures applications stay running even when containers or nodes fail
3. Load Balancing: Distributes traffic across multiple container instances
4. Auto-Scaling: Automatically scales applications based on demand
5. Rolling Updates: Deploy new versions without downtime
6. Service Discovery: Containers can find and communicate with each other
7. Storage Orchestration: Manages persistent storage for stateful applications
8. Self-Healing: Automatically restarts failed containers

Kubernetes Architecture:

Control Plane Components:

• API Server: Frontend for Kubernetes control plane
• etcd: Distributed key-value store for cluster data
• Scheduler: Assigns pods to nodes
• Controller Manager: Runs controller processes

Node Components:

• Kubelet: Agent that runs on each node
• Container Runtime: Docker, containerd, etc.
• Kube-proxy: Network proxy on each node

Key Kubernetes Objects:

1. Pod - Smallest deployable unit (one or more containers)

apiVersion: v1
kind: Pod
metadata:
  name: myapi-pod
spec:
  containers:
  - name: myapi
    image: myapi:v1
    ports:
    - containerPort: 80

2. Deployment - Manages ReplicaSets and rolling updates

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapi-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapi
  template:
    metadata:
      labels:
        app: myapi
    spec:
      containers:
      - name: myapi
        image: myapi:v1
        ports:
        - containerPort: 80
        env:
        - name: ASPNETCORE_ENVIRONMENT
          value: "Production"
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 80
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5

3. Service - Exposes pods to network traffic

apiVersion: v1
kind: Service
metadata:
  name: myapi-service
spec:
  type: LoadBalancer
  selector:
    app: myapi
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

4. ConfigMap - Stores configuration data

apiVersion: v1
kind: ConfigMap
metadata:
  name: myapi-config
data:
  appsettings.json: |
    {
      "Logging": {
        "LogLevel": {
          "Default": "Information"
        }
      }
    }

5. Secret - Stores sensitive data

apiVersion: v1
kind: Secret
metadata:
  name: myapi-secrets
type: Opaque
data:
  connectionstring: U2VydmVyPW15c3FsO0RhdGFiYXNlPW15ZGI7VXNlcj1yb290O1Bhc3N3b3JkPXBhc3M=

6. Ingress - HTTP/HTTPS routing

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapi-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
  - hosts:
    - api.mydomain.com
    secretName: myapi-tls
  rules:
  - host: api.mydomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapi-service
            port:
              number: 80

Common kubectl Commands:

# Deploy application
kubectl apply -f deployment.yaml

# Get resources
kubectl get pods
kubectl get deployments
kubectl get services

# Scale deployment
kubectl scale deployment myapi-deployment --replicas=5

# View logs
kubectl logs myapi-pod

# Execute command in pod
kubectl exec -it myapi-pod -- /bin/bash

# Update deployment (rolling update)
kubectl set image deployment/myapi-deployment myapi=myapi:v2

# Rollback deployment
kubectl rollout undo deployment/myapi-deployment

# Delete resources
kubectl delete -f deployment.yaml

When to Use Kubernetes:

✅ Use K8s when:

• Running microservices architecture
• Need high availability and auto-scaling
• Managing multiple environments (dev, staging, prod)
• Need container orchestration at scale

❌ Avoid K8s when:

• Simple monolithic application
• Small team with limited DevOps expertise
• Low traffic applications
• Development/testing only

Infrastructure as Code (IaC) is the practice of managing and provisioning infrastructure through machine-readable configuration files rather than manual processes or interactive configuration tools.

Key Principles:

1. Declarative vs Imperative:

   • Declarative: Define the desired end state (Terraform, ARM templates)
   • Imperative: Define steps to achieve the state (scripts, Ansible)

2. Version Control: Infrastructure definitions stored in Git

3. Idempotency: Running the same code multiple times produces the same result

4. Reproducibility: Create identical environments reliably

Benefits of IaC:

1. Consistency: Eliminates configuration drift between environments
2. Speed: Infrastructure can be provisioned in minutes
3. Documentation: Code serves as documentation
4. Version Control: Track changes and rollback if needed
5. Testing: Test infrastructure changes before production
6. Disaster Recovery: Quickly rebuild infrastructure from code
7. Cost Management: Easily create/destroy environments to save costs

Popular IaC Tools:

1. Terraform (Multi-Cloud)

# main.tf - Deploy .NET Core app to Azure
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "rg" {
  name     = "myapp-rg"
  location = "East US"
}

resource "azurerm_app_service_plan" "asp" {
  name                = "myapp-asp"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  kind                = "Linux"
  reserved            = true

  sku {
    tier = "Standard"
    size = "S1"
  }
}

resource "azurerm_linux_web_app" "webapp" {
  name                = "myapi-webapp"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  service_plan_id     = azurerm_app_service_plan.asp.id

  site_config {
    application_stack {
      dotnet_version = "8.0"
    }
  }

  app_settings = {
    "ASPNETCORE_ENVIRONMENT" = "Production"
  }
}

resource "azurerm_sql_server" "sql" {
  name                         = "myapp-sqlserver"
  resource_group_name          = azurerm_resource_group.rg.name
  location                     = azurerm_resource_group.rg.location
  version                      = "12.0"
  administrator_login          = "sqladmin"
  administrator_login_password = var.sql_password
}

resource "azurerm_sql_database" "db" {
  name                = "myapp-db"
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location
  server_name         = azurerm_sql_server.sql.name
  edition             = "Standard"
  requested_service_objective_name = "S0"
}

output "webapp_url" {
  value = azurerm_linux_web_app.webapp.default_hostname
}

2. ARM Templates (Azure-Specific)

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "webAppName": {
      "type": "string",
      "metadata": {
        "description": "Name of the web app"
      }
    },
    "location": {
      "type": "string",
      "defaultValue": "[resourceGroup().location]"
    }
  },
  "resources": [
    {
      "type": "Microsoft.Web/serverfarms",
      "apiVersion": "2022-03-01",
      "name": "[concat(parameters('webAppName'), '-plan')]",
      "location": "[parameters('location')]",
      "sku": {
        "name": "S1",
        "tier": "Standard"
      },
      "kind": "linux",
      "properties": {
        "reserved": true
      }
    },
    {
      "type": "Microsoft.Web/sites",
      "apiVersion": "2022-03-01",
      "name": "[parameters('webAppName')]",
      "location": "[parameters('location')]",
      "dependsOn": [
        "[resourceId('Microsoft.Web/serverfarms', concat(parameters('webAppName'), '-plan'))]"
      ],
      "properties": {
        "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', concat(parameters('webAppName'), '-plan'))]",
        "siteConfig": {
          "linuxFxVersion": "DOTNETCORE|8.0"
        }
      }
    }
  ]
}

3. Bicep (Azure - ARM Template Alternative)

// main.bicep
param webAppName string
param location string = resourceGroup().location
param sqlAdminPassword string

resource appServicePlan 'Microsoft.Web/serverfarms@2022-03-01' = {
  name: '${webAppName}-plan'
  location: location
  sku: {
    name: 'S1'
    tier: 'Standard'
  }
  kind: 'linux'
  properties: {
    reserved: true
  }
}

resource webApp 'Microsoft.Web/sites@2022-03-01' = {
  name: webAppName
  location: location
  properties: {
    serverFarmId: appServicePlan.id
    siteConfig: {
      linuxFxVersion: 'DOTNETCORE|8.0'
      appSettings: [
        {
          name: 'ASPNETCORE_ENVIRONMENT'
          value: 'Production'
        }
      ]
    }
  }
}

output webAppUrl string = webApp.properties.defaultHostName

4. Pulumi (Code-Based IaC with C#)

using Pulumi;
using Pulumi.Azure.Core;
using Pulumi.Azure.AppService;

class MyStack : Stack
{
    public MyStack()
    {
        var resourceGroup = new ResourceGroup("myapp-rg");

        var appServicePlan = new Plan("myapp-asp", new PlanArgs
        {
            ResourceGroupName = resourceGroup.Name,
            Kind = "Linux",
            Reserved = true,
            Sku = new PlanSkuArgs
            {
                Tier = "Standard",
                Size = "S1",
            },
        });

        var app = new AppService("myapp", new AppServiceArgs
        {
            ResourceGroupName = resourceGroup.Name,
            AppServicePlanId = appServicePlan.Id,
            SiteConfig = new AppServiceSiteConfigArgs
            {
                LinuxFxVersion = "DOTNETCORE|8.0",
            },
        });

        this.Endpoint = app.DefaultSiteHostname;
    }

    [Output]
    public Output Endpoint { get; set; }
}

Best Practices:

1. Use Version Control: Store IaC in Git repositories
2. Modularization: Break down into reusable modules
3. Environment Separation: Use workspaces or separate state files
4. Secret Management: Never hardcode secrets; use Azure Key Vault, AWS Secrets Manager
5. State Management: Use remote state storage (Azure Storage, S3)
6. Code Review: Treat infrastructure changes like application code
7. Automated Testing: Test infrastructure before deploying
8. Documentation: Comment complex configurations

Here's an overview of commonly used cloud services for .NET Core applications:

Azure Services

Compute:

1. Azure App Service
   • Managed platform for hosting web apps, APIs, and mobile backends
   • Built-in CI/CD, auto-scaling, custom domains, SSL
   • Best for: Web APIs, web applications, background jobs

// Program.cs - optimized for Azure App Service
var builder = WebApplication.CreateBuilder(args);

// Azure App Service provides HTTPS automatically
builder.Services.AddHttpsRedirection(options =>
{
    options.HttpsPort = 443;
});

// Add Application Insights
builder.Services.AddApplicationInsightsTelemetry();

var app = builder.Build();

// Configure for Azure App Service
app.UseForwardedHeaders();
app.UseHttpsRedirection();
app.MapHealthChecks("/health");

app.Run();

1. Azure Functions
   • Serverless compute for event-driven workloads
   • Pay per execution, auto-scaling
   • Best for: Background tasks, webhooks, scheduled jobs

[Function("ProcessOrder")]
public async Task Run(
    [HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequestData req,
    [QueueOutput("orders")] ICollector orderQueue)
{
    var order = await req.ReadFromJsonAsync();
    orderQueue.Add(order);
    
    var response = req.CreateResponse(HttpStatusCode.OK);
    return response;
}

1. Azure Kubernetes Service (AKS)

   • Managed Kubernetes cluster
   • Best for: Microservices, complex containerized applications

2. Azure Container Instances (ACI)

   • Run containers without managing servers
   • Best for: Simple containerized apps, batch jobs

Data Storage:

1. Azure SQL Database
   • Managed SQL Server database
   • Automatic backups, patching, high availability

// Connection string from Azure Key Vault
builder.Configuration.AddAzureKeyVault(
    new Uri($"https://{keyVaultName}.vault.azure.net/"),
    new DefaultAzureCredential());

builder.Services.AddDbContext(options =>
    options.UseSqlServer(
        builder.Configuration["ConnectionStrings:DefaultConnection"]));

1. Azure Cosmos DB
   • Globally distributed NoSQL database
   • Multiple APIs: SQL, MongoDB, Cassandra, Gremlin

builder.Services.AddSingleton(sp =>
{
    var connectionString = builder.Configuration["CosmosDb:ConnectionString"];
    return new CosmosClient(connectionString);
});

1. Azure Blob Storage
   • Object storage for unstructured data (files, images, videos)

builder.Services.AddSingleton(x =>
{
    var connectionString = builder.Configuration["AzureStorage:ConnectionString"];
    return new BlobServiceClient(connectionString);
});

// Usage
public class FileService
{
    private readonly BlobServiceClient _blobServiceClient;
    
    public async Task UploadFileAsync(Stream fileStream, string fileName)
    {
        var containerClient = _blobServiceClient.GetBlobContainerClient("uploads");
        await containerClient.CreateIfNotExistsAsync();
        
        var blobClient = containerClient.GetBlobClient(fileName);
        await blobClient.UploadAsync(fileStream, overwrite: true);
        
        return blobClient.Uri.ToString();
    }
}

1. Azure Table Storage / Azure Storage Queues
   • NoSQL key-value storage / Message queuing

Messaging & Integration:

1. Azure Service Bus
   • Enterprise message broker with topics, queues, subscriptions

builder.Services.AddSingleton(sp =>
{
    var connectionString = builder.Configuration["ServiceBus:ConnectionString"];
    return new ServiceBusClient(connectionString);
});

// Sending messages
public async Task SendMessageAsync(Order order)
{
    var sender = _serviceBusClient.CreateSender("orders");
    var message = new ServiceBusMessage(JsonSerializer.Serialize(order));
    await sender.SendMessageAsync(message);
}

1. Azure Event Grid

   • Event routing service for reactive programming

2. Azure Event Hubs

   • Big data streaming platform and event ingestion service

Security & Identity:

1. Azure Active Directory (Azure AD / Entra ID)
   • Identity and access management

builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddMicrosoftIdentityWebApi(builder.Configuration.GetSection("AzureAd"));

builder.Services.AddAuthorization();

1. Azure Key Vault
   • Secrets, keys, and certificate management

var keyVaultUri = new Uri(builder.Configuration["KeyVault:Uri"]);
builder.Configuration.AddAzureKeyVault(keyVaultUri, new DefaultAzureCredential());

Monitoring & Logging:

1. Application Insights
   • APM and monitoring solution

builder.Services.AddApplicationInsightsTelemetry(options =>
{
    options.ConnectionString = builder.Configuration["ApplicationInsights:ConnectionString"];
});

// Custom telemetry
public class OrderService
{
    private readonly TelemetryClient _telemetry;
    
    public async Task ProcessOrder(Order order)
    {
        var stopwatch = Stopwatch.StartNew();
        try
        {
            // Process order
            _telemetry.TrackEvent("OrderProcessed", 
                new Dictionary { ["OrderId"] = order.Id });
        }
        catch (Exception ex)
        {
            _telemetry.TrackException(ex);
            throw;
        }
        finally
        {
            _telemetry.TrackMetric("OrderProcessingTime", stopwatch.ElapsedMilliseconds);
        }
    }
}

1. Azure Monitor & Log Analytics
   • Infrastructure and application monitoring

DevOps:

1. Azure DevOps

   • CI/CD pipelines, repos, boards, artifacts

2. Azure Container Registry (ACR)

   • Private Docker registry

AWS Services

Compute:

1. AWS Elastic Beanstalk

   • Managed platform for .NET applications
   • Similar to Azure App Service

2. AWS Lambda

   • Serverless functions (supports .NET)

public class Function
{
    public async Task FunctionHandler(
        APIGatewayProxyRequest request, ILambdaContext context)
    {
        return new APIGatewayProxyResponse
        {
            StatusCode = 200,
            Body = JsonSerializer.Serialize(new { message = "Hello from Lambda" })
        };
    }
}

1. Amazon ECS / EKS

   • Container orchestration (ECS = proprietary, EKS = Kubernetes)

2. Amazon EC2

   • Virtual machines

Data Storage:

1. Amazon RDS (SQL Server)

   • Managed relational database

2. Amazon DynamoDB

   • NoSQL database

3. Amazon S3

   • Object storage (like Azure Blob Storage)

var s3Client = new AmazonS3Client(RegionEndpoint.USEast1);

// Upload file
var putRequest = new PutObjectRequest
{
    BucketName = "my-bucket",
    Key = "uploads/file.txt",
    ContentBody = "Hello S3"
};
await s3Client.PutObjectAsync(putRequest);

Messaging:

1. Amazon SQS

   • Message queuing service

2. Amazon SNS

   • Pub/sub messaging

3. Amazon EventBridge

   • Event bus for serverless applications

Security:

1. AWS IAM

   • Identity and access management

2. AWS Secrets Manager

   • Secrets management

3. AWS Cognito

   • User authentication and authorization

Typical .NET Core Architecture on Azure:

Internet
    ↓
Azure Front Door / Application Gateway (CDN, WAF, Load Balancer)
    ↓
Azure App Service / AKS (Multiple instances)
    ↓
    ├─→ Azure SQL Database (Relational data)
    ├─→ Azure Cosmos DB (NoSQL data)
    ├─→ Azure Cache for Redis (Caching)
    ├─→ Azure Blob Storage (File storage)
    ├─→ Azure Service Bus (Async messaging)
    └─→ Azure Key Vault (Secrets)
    
Monitoring: Application Insights + Azure Monitor
Identity: Azure AD
CI/CD: Azure DevOps / GitHub Actions

Cost Optimization Tips:

1. Use Azure App Service for simple apps (easier than AKS)
2. Use serverless (Functions, Logic Apps) for sporadic workloads
3. Enable auto-scaling to match demand
4. Use Azure Reserved Instances for predictable workloads (save 30-70%)
5. Implement caching (Redis) to reduce database load
6. Use Azure CDN for static content
7. Monitor with Azure Cost Management