•
kubernetes, devops, cloud-native

Understanding kubectl apply Workflow

Master kubectl apply workflow from YAML files to running pods in production, including deep analysis of core concepts, common issues, and best practices with detailed Mermaid sequence diagrams

kubernetes devops container-orchestration production microservices

Understanding kubectl apply Workflow

Kubernetes, as the de facto standard for container orchestration, has become the core infrastructure for modern cloud-native applications. This article provides an in-depth analysis of the kubectl apply workflow, core concepts, common production environment issues, and best practices to help DevOps engineers and SRE teams better understand and apply Kubernetes.

Table of Contents

Deep Analysis of Kubernetes Workflows

kubectl apply Command Execution Flow

When executing the kubectl apply command, a series of complex interactions occur within Kubernetes. Let’s understand this process through a detailed sequence diagram:

Mermaid Diagram
Rendering diagram...

Detailed Workflow Analysis

1. Client Processing Stage

# Detailed execution process of kubectl apply command
kubectl apply -f deployment.yaml

Step 1: YAML Parsing and Validation

  • kubectl parses the YAML file
  • Validates syntax and semantics of resource definitions
  • Checks completeness of required fields

Step 2: Resource Conversion

  • Convert YAML to Kubernetes API objects
  • Apply default values and auto-generated fields
  • Generate unique identifiers for resources

2. API Server Processing Stage

# Typical Deployment resource definition
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.21
        ports:
        - containerPort: 80

Step 3: Authentication and Authorization

  • Verify client certificates or tokens
  • Check RBAC permissions
  • Validate namespace access permissions

Step 4: Admission Controller Validation

  • Execute MutatingAdmissionWebhooks
  • Execute ValidatingAdmissionWebhooks
  • Apply resource quota limits

3. Storage and State Management

Step 5: etcd Storage

  • Store resource definitions in etcd
  • Maintain resource versions and state history
  • Ensure data consistency and persistence

4. Controller Loop and Scheduling

Step 6: Controller Detection

  • Deployment Controller detects new Deployment
  • Create ReplicaSet resources
  • ReplicaSet Controller creates Pod resources

Step 7: Scheduling Process

  • Scheduler evaluates cluster node resources
  • Apply scheduling policies and constraints
  • Bind Pods to appropriate nodes

5. Node Execution Stage

Step 8: Pod Creation

  • Kubelet monitors API Server changes
  • Download Pod definitions and container images
  • Create container runtime environment

Step 9: Network Configuration

  • CNI plugin configures Pod networking
  • Assign IP addresses and network policies
  • Establish service discovery mechanisms

Detailed Explanation of Kubernetes Core Concepts

1. Pod - Minimum Deployment Unit

A Pod is the smallest deployable unit in Kubernetes, typically containing one or more tightly coupled containers.

apiVersion: v1
kind: Pod
metadata:
  name: multi-container-pod
spec:
  containers:
  - name: web-server
    image: nginx:1.21
    ports:
    - containerPort: 80
  - name: sidecar
    image: busybox
    command: ['sh', '-c', 'while true; do echo "Sidecar running"; sleep 30; done']
  volumes:
  - name: shared-data
    emptyDir: {}

Pod Lifecycle States:

  • Pending: Pod has been scheduled but containers have not started
  • Running: Pod is bound to a node and all containers are running normally
  • Succeeded: All containers have terminated successfully and will not restart
  • Failed: All containers have terminated and at least one container failed
  • Unknown: Unable to obtain Pod status

2. Deployment - Declarative Updates

Deployment provides declarative updates and manages the lifecycle of ReplicaSets and Pods.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.21
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5

3. Service - Service Discovery and Load Balancing

Service provides a stable network access entry point for Pods.

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: ClusterIP

Service Types:

  • ClusterIP: Cluster-internal access
  • NodePort: Node port access
  • LoadBalancer: Cloud load balancer
  • ExternalName: External service mapping

4. ConfigMap and Secret - Configuration Management

# ConfigMap example
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  database_url: "mysql://db:3306/app"
  log_level: "info"
  feature_flags: |
    new_ui=true
    beta_features=false

---
# Secret example
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  username: YWRtaW4=  # base64 encoded
  password: cGFzc3dvcmQ=

5. Namespace - Resource Isolation

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    environment: production
    tier: backend

Common Production Environment Issues and Solutions

1. Resource Insufficiency Issues

Problem Description:

  • Pods cannot be scheduled to nodes
  • Node resource utilization is too high
  • Application performance degradation

Solutions:

# Resource requests and limits configuration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: resource-optimized-app
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: app
        image: myapp:latest
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        # Resource monitoring
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3

Resource Monitoring Script:

#!/bin/bash
# Monitor cluster resource usage

# Check node resources
kubectl top nodes

# Check Pod resource usage
kubectl top pods --all-namespaces

# Check resource quotas
kubectl describe quota --all-namespaces

# Check node capacity
kubectl describe nodes | grep -A 5 "Capacity:"

2. Network Connectivity Issues

Problem Diagnosis:

# Check Service and Endpoints
kubectl get svc,ep -n production

# Check Pod network connectivity
kubectl exec -it pod-name -- nslookup service-name

# Check network policies
kubectl get networkpolicies

# Check DNS resolution
kubectl exec -it pod-name -- cat /etc/resolv.conf

Network Policy Configuration:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: backend
    ports:
    - protocol: TCP
      port: 9090

3. Storage Issues

Persistent Storage Configuration:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: fast-ssd

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-with-storage
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: app
        image: myapp:latest
        volumeMounts:
        - name: data-volume
          mountPath: /app/data
      volumes:
      - name: data-volume
        persistentVolumeClaim:
          claimName: app-data-pvc

4. Application Startup and Health Check Issues

Health Check Configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: health-checked-app
spec:
  template:
    spec:
      containers:
      - name: app
        image: myapp:latest
        ports:
        - containerPort: 8080
        # 启动探针
        startupProbe:
          httpGet:
            path: /health/startup
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 30
        # 存活探针
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
        # 就绪探针
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 3

Kubernetes Best Practices

1. Resource Management Best Practices

Resource Requests and Limits:

# Production environment resource configuration example
apiVersion: apps/v1
kind: Deployment
metadata:
  name: production-app
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: app
        image: myapp:v1.2.3
        resources:
          # Request resources (scheduling basis)
          requests:
            memory: "512Mi"
            cpu: "500m"
            ephemeral-storage: "1Gi"
          # Limit resources (hard limits)
          limits:
            memory: "1Gi"
            cpu: "1000m"
            ephemeral-storage: "2Gi"
        # Resource monitoring
        env:
        - name: JVM_OPTS
          value: "-Xms512m -Xmx1g"

2. Security Best Practices

Pod Security Context:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-app
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 2000
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: app
        image: myapp:latest
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
          capabilities:
            drop:
            - ALL
        volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: cache
          mountPath: /app/cache
      volumes:
      - name: tmp
        emptyDir: {}
      - name: cache
        emptyDir: {}

RBAC Configuration:

# ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-service-account
  namespace: production

---
# Role
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-role
  namespace: production
rules:
- apiGroups: [""]
  resources: ["configmaps", "secrets"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "update"]

---
# RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-role-binding
  namespace: production
subjects:
- kind: ServiceAccount
  name: app-service-account
  namespace: production
roleRef:
  kind: Role
  name: app-role
  apiGroup: rbac.authorization.k8s.io

3. Monitoring and Observability

Prometheus Monitoring Configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)

Application Metrics Exposure:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: monitored-app
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: app
        image: myapp:latest
        ports:
        - containerPort: 8080
          name: http
        - containerPort: 9090
          name: metrics
        env:
        - name: METRICS_ENABLED
          value: "true"

4. Rolling Update Strategy

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rolling-update-app
spec:
  replicas: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 2
  template:
    spec:
      containers:
      - name: app
        image: myapp:v1.2.3
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3

5. Multi-Environment Management

Environment-Specific Configuration:

# Development environment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-dev
  namespace: development
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: app
        image: myapp:dev
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"

---
# Production environment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-prod
  namespace: production
spec:
  replicas: 5
  template:
    spec:
      containers:
      - name: app
        image: myapp:v1.2.3
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1000m"

Monitoring and Observability

1. Cluster Monitoring Architecture

graph TB
    subgraph "Kubernetes Cluster"
        subgraph "Monitoring Stack"
            Prometheus[Prometheus Server]
            Grafana[Grafana Dashboard]
            AlertManager[Alert Manager]
        end
        
        subgraph "Application Pods"
            App1[Application 1]
            App2[Application 2]
            App3[Application 3]
        end
        
        subgraph "Infrastructure"
            Node1[Node 1]
            Node2[Node 2]
            Node3[Node 3]
        end
    end
    
    subgraph "External Services"
        Slack[Slack Notifications]
        Email[Email Alerts]
        PagerDuty[PagerDuty]
    end
    
    App1 --> Prometheus
    App2 --> Prometheus
    App3 --> Prometheus
    Node1 --> Prometheus
    Node2 --> Prometheus
    Node3 --> Prometheus
    
    Prometheus --> Grafana
    Prometheus --> AlertManager
    AlertManager --> Slack
    AlertManager --> Email
    AlertManager --> PagerDuty

2. Key Metrics Monitoring

Cluster Resource Monitoring:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring
data:
  cluster-metrics.yml: |
    # CPU usage rate
    - alert: HighCPUUsage
      expr: (100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > 80
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High CPU usage detected"
        description: "CPU usage is above 80% for more than 5 minutes"
    
    # Memory usage rate
    - alert: HighMemoryUsage
      expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High memory usage detected"
        description: "Memory usage is above 85% for more than 5 minutes"
    
    # Pod restart count
    - alert: PodRestarting
      expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "Pod is restarting frequently"
        description: "Pod {{ $labels.pod }} is restarting {{ $value }} times per second"

Security Best Practices

1. Network Security

Network Policy Configuration:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

2. Image Security

Image Scanning and Policies:

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  containers:
  - name: app
    image: myapp:v1.2.3@sha256:abc123...
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 1000
      capabilities:
        drop:
        - ALL

Performance Optimization Strategies

1. Resource Optimization

HPA Configuration:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app-deployment
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

VPA Configuration:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app-deployment
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: app
      minAllowed:
        cpu: 100m
        memory: 128Mi
      maxAllowed:
        cpu: 1000m
        memory: 1Gi

2. Storage Optimization

Storage Class Configuration:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Summary and Future Outlook

Kubernetes, as the core platform for modern cloud-native applications, requires deep understanding of its working principles and best practices due to its complexity and powerful capabilities. Through the detailed analysis in this article, we have learned:

  1. Deep Workflow Analysis: Complete process from kubectl apply to Pod execution
  2. Core Concept Mastery: Key components like Pod, Deployment, Service, etc.
  3. Production Environment Problem Solving: Common issues with resources, networking, storage, etc.
  4. Best Practice Application: Strategies for security, monitoring, performance optimization, etc.

Key Points Summary

  • Resource Management: Properly set requests and limits to avoid resource contention
  • Health Checks: Configure comprehensive probes to ensure application stability
  • Security Practices: Implement principle of least privilege and use security contexts
  • Monitoring and Alerting: Establish complete observability systems
  • Rolling Updates: Adopt progressive deployment strategies to reduce risks
  1. Service Mesh Integration: Service mesh technologies like Istio, Linkerd
  2. GitOps Practices: Git-based declarative deployment management
  3. Edge Computing: Kubernetes deployment in edge environments
  4. AI/ML Workloads: Kubernetes-native AI/ML platforms
  5. Multi-Cloud Management: Unified management across cloud platforms

Through continuous learning and practice, we can better leverage Kubernetes’ powerful capabilities to build stable, secure, and efficient cloud-native application platforms.

Sep 15, 2025

Comprehensive DNS Guide

A complete guide to DNS concepts, troubleshooting, and best practices.

No Next Article

This is the latest article

YH

Youqing Han

DevOps Engineer

Share this article:

Stay Updated

Get the latest DevOps insights and best practices delivered to your inbox

No spam, unsubscribe at any time