---
title: "Deep Dive: Kubernetes Kubelet"
source: "https://support.tools/training/kubernetes-deep-dive/kubelet/"
published: 2025-01-01
categories: ["Kubernetes"]
tags: ["kubernetes","kubelet","node","containers"]
---


The Kubelet is the primary node agent in Kubernetes, responsible for managing containers and maintaining node state. This deep dive explores its architecture, container lifecycle management, and internal operations.

# [Architecture Overview](#architecture)

## Component Architecture
```plaintext
API Server -> Kubelet -> Container Runtime -> Containers
                     -> Volume Plugins   -> Volumes
                     -> Network Plugins  -> Pod Networking
```

## Key Responsibilities
1. **Pod Management**
   - Pod lifecycle
   - Container creation/deletion
   - Volume management
   - Network setup

2. **Node Management**
   - Resource monitoring
   - Health checking
   - Node status reporting

3. **Container Runtime Interface (CRI)**
   - Container operations
   - Image management
   - Runtime status

# [Pod Lifecycle Management](#pod-lifecycle)

## 1. Pod Admission
```go
// Pod admission workflow
func (kl *Kubelet) admitPod(pod *v1.Pod) error {
    // Check node resources
    if !kl.canAdmitPod(pod) {
        return fmt.Errorf("insufficient resources")
    }
    
    // Validate pod fields
    if err := kl.validatePod(pod); err != nil {
        return err
    }
    
    // Admit pod
    return kl.podManager.AddPod(pod)
}
```

## 2. Container Lifecycle
```yaml
apiVersion: v1
kind: Pod
metadata:
  name: lifecycle-demo
spec:
  containers:
  - name: lifecycle-demo
    image: nginx
    lifecycle:
      postStart:
        exec:
          command: ["/bin/sh", "-c", "echo Hello from postStart handler > /usr/share/message"]
      preStop:
        exec:
          command: ["/bin/sh","-c","nginx -s quit; while killall -0 nginx; do sleep 1; done"]
```

# [Container Runtime Interface](#cri)

## 1. Runtime Operations
```go
type RuntimeService interface {
    RunPodSandbox(config *PodSandboxConfig) (string, error)
    StopPodSandbox(podSandboxID string) error
    RemovePodSandbox(podSandboxID string) error
    CreateContainer(podSandboxID string, config *ContainerConfig, sandboxConfig *PodSandboxConfig) (string, error)
    StartContainer(containerID string) error
    StopContainer(containerID string, timeout int64) error
    RemoveContainer(containerID string) error
    ListContainers(filter *ContainerFilter) ([]*Container, error)
    ContainerStatus(containerID string) (*ContainerStatus, error)
    UpdateContainerResources(containerID string, resources *ResourceConfig) error
    ExecSync(containerID string, cmd []string, timeout time.Duration) (stdout []byte, stderr []byte, err error)
    Exec(request *ExecRequest) (*ExecResponse, error)
    Attach(req *AttachRequest) (*AttachResponse, error)
    PortForward(req *PortForwardRequest) (*PortForwardResponse, error)
}
```

## 2. Image Operations
```go
type ImageService interface {
    ListImages(filter *ImageFilter) ([]*Image, error)
    ImageStatus(image *ImageSpec) (*Image, error)
    PullImage(image *ImageSpec, auth *AuthConfig) (string, error)
    RemoveImage(image *ImageSpec) error
    ImageFsInfo() (*FsInfo, error)
}
```

# [Volume Management](#volumes)

## 1. Volume Plugin Integration
```go
type VolumePlugin interface {
    Init(host VolumeHost) error
    GetPluginName() string
    GetVolumeName(spec *Spec) (string, error)
    CanSupport(spec *Spec) bool
    RequiresRemount() bool
    NewMounter(spec *Spec, pod *v1.Pod, opts VolumeOptions) (Mounter, error)
    NewUnmounter(name string, podUID types.UID) (Unmounter, error)
}
```

## 2. Volume Mount Example
```yaml
apiVersion: v1
kind: Pod
metadata:
  name: volume-demo
spec:
  containers:
  - name: web
    image: nginx
    volumeMounts:
    - name: data
      mountPath: /usr/share/nginx/html
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data-pvc
```

# [Resource Management](#resources)

## 1. CPU Management
```yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cpuManagerPolicy: static
reservedSystemCPUs: "0-1"
```

## 2. Memory Management
```yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryManagerPolicy: Static
systemReserved:
  memory: 1Gi
kubeReserved:
  memory: 1Gi
```

## 3. Device Plugin Framework
```go
type DevicePlugin interface {
    GetDevicePluginOptions(context.Context, *Empty) (*DevicePluginOptions, error)
    ListAndWatch(*Empty, DevicePlugin_ListAndWatchServer) error
    Allocate(context.Context, *AllocateRequest) (*AllocateResponse, error)
    PreStartContainer(context.Context, *PreStartContainerRequest) (*PreStartContainerResponse, error)
}
```

# [Node Status Management](#node-status)

## 1. Node Registration
```yaml
apiVersion: v1
kind: Node
metadata:
  name: worker-1
  labels:
    kubernetes.io/hostname: worker-1
    node-role.kubernetes.io/worker: ""
spec:
  podCIDR: 10.244.1.0/24
status:
  capacity:
    cpu: "4"
    memory: 8Gi
    pods: "110"
```

## 2. Health Checking
```bash
# Kubelet health check endpoints
curl -k https://localhost:10250/healthz
curl -k https://localhost:10250/healthz/syncloop
```

# [Performance Tuning](#performance)

## 1. Kubelet Configuration
```yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 110
podsPerCore: 10
systemReserved:
  cpu: 500m
  memory: 1Gi
kubeReserved:
  cpu: 500m
  memory: 1Gi
evictionHard:
  memory.available: "500Mi"
  nodefs.available: "10%"
```

## 2. Container Runtime Settings
```toml
# containerd configuration
[plugins."io.containerd.grpc.v1.cri"]
  [plugins."io.containerd.grpc.v1.cri".containerd]
    snapshotter = "overlayfs"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v2"
```

# [Monitoring and Debugging](#monitoring)

## 1. Metrics Collection
```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kubelet
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: https-metrics
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
```

## 2. Log Analysis
```bash
# View kubelet logs
journalctl -u kubelet

# Check container logs
kubectl logs pod-name container-name

# Debug pod
kubectl debug node/worker-1 -it --image=ubuntu
```

# [Troubleshooting](#troubleshooting)

## Common Issues

1. **Pod Scheduling Failures**
```bash
# Check node capacity
kubectl describe node worker-1
# View kubelet logs
journalctl -u kubelet | grep "Failed to admit pod"
```

2. **Container Runtime Issues**
```bash
# Check CRI status
crictl info
# List containers
crictl ps -a
```

3. **Volume Mount Problems**
```bash
# Check volume mounts
findmnt | grep kubelet
# View volume manager logs
journalctl -u kubelet | grep "Volume Manager"
```

# [Best Practices](#best-practices)

1. **Resource Management**
   - Configure appropriate resource reservations
   - Enable CPU and memory management
   - Set proper eviction thresholds

2. **Security**
   - Enable node authorization
   - Configure TLS properly
   - Use secure container runtime settings

3. **Monitoring**
   - Implement proper metrics collection
   - Set up log aggregation
   - Configure alerts for critical issues

For more information, check out:
- [Container Runtime Deep Dive](/training/kubernetes-deep-dive/containerd/)
- Node Management
- Resource Management