Grafana Loki Log Aggregation at Scale: Production Architecture for Multi-Tenant Kubernetes
Centralized log aggregation is essential for troubleshooting and monitoring distributed systems, but traditional solutions like Elasticsearch can be resource-intensive and complex to operate. Grafana Loki provides a cost-effective, scalable alternative designed specifically for Kubernetes environments. This guide covers production-grade Loki deployment in microservices mode with multi-tenancy, object storage backends, and high-performance querying.
Grafana Loki Log Aggregation at Scale
Executive Summary
Grafana Loki is a horizontally scalable, highly available log aggregation system inspired by Prometheus. Unlike other logging solutions, Loki indexes only metadata (labels) rather than full-text content, making it significantly more cost-effective at scale. This guide demonstrates deploying Loki in microservices mode for enterprise Kubernetes environments, implementing multi-tenancy, configuring retention policies, and optimizing query performance.
Loki Architecture Overview
Microservices Mode Components
┌─────────────┐
│ Distributor│◄──── Promtail / Fluent Bit
└──────┬──────┘
│
▼
┌─────────────┐
│ Ingester │────► Object Storage (S3/GCS)
└──────┬──────┘
│
▼
┌─────────────┐
│ Querier │◄──── Grafana
└──────┬──────┘
│
▼
┌─────────────┐
│Query Frontend│
└─────────────┘
│
▼
┌─────────────┐
│ Compactor │
└─────────────┘
Components Explained
- Distributor: Validates and distributes incoming log streams
- Ingester: Writes log data to long-term storage
- Querier: Handles log queries
- Query Frontend: Query caching and splitting
- Compactor: Compacts and deduplicates index files
- Ruler: Evaluates LogQL recording and alerting rules
Core Infrastructure Setup
S3 Storage Backend Configuration
apiVersion: v1
kind: Secret
metadata:
name: loki-s3-config
namespace: logging
type: Opaque
stringData:
s3.yaml: |
s3:
endpoint: s3.amazonaws.com
bucketnames: loki-logs-prod
region: us-east-1
access_key_id: ${AWS_ACCESS_KEY_ID}
secret_access_key: ${AWS_SECRET_ACCESS_KEY}
s3forcepathstyle: false
insecure: false
http_config:
idle_conn_timeout: 90s
response_header_timeout: 0s
insecure_skip_verify: false
Loki Configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: loki-config
namespace: logging
data:
config.yaml: |
auth_enabled: true
server:
http_listen_port: 3100
grpc_listen_port: 9095
grpc_server_max_recv_msg_size: 104857600
grpc_server_max_send_msg_size: 104857600
log_level: info
distributor:
ring:
kvstore:
store: memberlist
heartbeat_timeout: 1m
memberlist:
node_name: ${HOSTNAME}
bind_port: 7946
join_members:
- loki-memberlist
ingester:
lifecycler:
ring:
kvstore:
store: memberlist
replication_factor: 3
heartbeat_timeout: 10m
num_tokens: 512
heartbeat_period: 5s
join_after: 30s
observe_period: 30s
final_sleep: 30s
chunk_idle_period: 1h
chunk_block_size: 262144
chunk_retain_period: 1m
max_chunk_age: 2h
wal:
enabled: true
dir: /loki/wal
replay_memory_ceiling: 4GB
checkpoint_duration: 10m
flush_on_shutdown: true
querier:
max_concurrent: 20
query_timeout: 5m
query_ingesters_within: 3h
engine:
timeout: 5m
max_look_back_period: 720h
query_frontend:
max_outstanding_per_tenant: 2048
compress_responses: true
log_queries_longer_than: 10s
downstream_url: http://loki-querier:3100
query_range:
align_queries_with_step: true
max_retries: 5
parallelise_shardable_queries: true
cache_results: true
results_cache:
cache:
memcached_client:
consistent_hash: true
host: memcached-frontend.logging.svc.cluster.local
service: memcached
timeout: 500ms
max_idle_conns: 100
query_scheduler:
max_outstanding_requests_per_tenant: 256
ruler:
enable_api: true
enable_alertmanager_v2: true
alertmanager_url: http://alertmanager.monitoring.svc.cluster.local:9093
ring:
kvstore:
store: memberlist
rule_path: /loki/rules
storage:
type: s3
s3:
bucketnames: loki-ruler-prod
endpoint: s3.amazonaws.com
region: us-east-1
compactor:
working_directory: /loki/compactor
shared_store: s3
compaction_interval: 10m
retention_enabled: true
retention_delete_delay: 2h
retention_delete_worker_count: 150
schema_config:
configs:
# Schema v11 (current)
- from: 2024-01-01
store: boltdb-shipper
object_store: s3
schema: v11
index:
prefix: loki_index_
period: 24h
# Schema v12 (future)
- from: 2025-01-01
store: tsdb
object_store: s3
schema: v12
index:
prefix: loki_tsdb_index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /loki/index
cache_location: /loki/index-cache
cache_ttl: 24h
shared_store: s3
index_gateway_client:
server_address: dns:///loki-index-gateway:9095
tsdb_shipper:
active_index_directory: /loki/tsdb-index
cache_location: /loki/tsdb-cache
cache_ttl: 24h
shared_store: s3
index_gateway_client:
server_address: dns:///loki-index-gateway:9095
aws:
bucketnames: loki-chunks-prod
region: us-east-1
s3forcepathstyle: false
chunk_store_config:
max_look_back_period: 720h
chunk_cache_config:
memcached_client:
consistent_hash: true
host: memcached-chunks.logging.svc.cluster.local
service: memcached
timeout: 500ms
max_idle_conns: 100
write_dedupe_cache_config:
memcached_client:
consistent_hash: true
host: memcached-chunks.logging.svc.cluster.local
service: memcached
timeout: 500ms
max_idle_conns: 100
limits_config:
# Ingestion limits
ingestion_rate_strategy: global
ingestion_rate_mb: 50
ingestion_burst_size_mb: 100
max_streams_per_user: 0
max_global_streams_per_user: 100000
# Query limits
max_query_length: 721h
max_query_parallelism: 32
max_query_series: 10000
max_entries_limit_per_query: 10000
max_cache_freshness_per_query: 10m
# Cardinality limits
max_label_name_length: 1024
max_label_value_length: 2048
max_label_names_per_series: 30
# Retention
retention_period: 720h
# Per-tenant overrides
per_tenant_override_config: /etc/loki/overrides.yaml
table_manager:
retention_deletes_enabled: true
retention_period: 720h
frontend:
compress_responses: true
max_outstanding_per_tenant: 256
log_queries_longer_than: 10s
frontend_worker:
frontend_address: loki-query-frontend:9095
grpc_client_config:
max_send_msg_size: 104857600
parallelism: 10
match_max_concurrent: true
---
apiVersion: v1
kind: ConfigMap
metadata:
name: loki-overrides
namespace: logging
data:
overrides.yaml: |
overrides:
# Production tenant with higher limits
prod:
ingestion_rate_mb: 100
ingestion_burst_size_mb: 200
max_global_streams_per_user: 200000
max_query_parallelism: 64
retention_period: 2160h # 90 days
# Development tenant with standard limits
dev:
ingestion_rate_mb: 10
ingestion_burst_size_mb: 20
max_global_streams_per_user: 50000
max_query_parallelism: 16
retention_period: 168h # 7 days
# Critical services with extended retention
critical:
ingestion_rate_mb: 200
ingestion_burst_size_mb: 400
max_global_streams_per_user: 300000
max_query_parallelism: 128
retention_period: 8760h # 1 year
Deploying Loki Components
Distributor Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: loki-distributor
namespace: logging
labels:
app: loki
component: distributor
spec:
replicas: 3
selector:
matchLabels:
app: loki
component: distributor
template:
metadata:
labels:
app: loki
component: distributor
loki-gossip-member: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "3100"
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: component
operator: In
values:
- distributor
topologyKey: kubernetes.io/hostname
containers:
- name: distributor
image: grafana/loki:2.9.3
args:
- -config.file=/etc/loki/config.yaml
- -target=distributor
- -config.expand-env=true
env:
- name: HOSTNAME
valueFrom:
fieldRef:
fieldPath: metadata.name
ports:
- name: http
containerPort: 3100
- name: grpc
containerPort: 9095
- name: gossip
containerPort: 7946
volumeMounts:
- name: config
mountPath: /etc/loki
- name: overrides
mountPath: /etc/loki/overrides.yaml
subPath: overrides.yaml
resources:
requests:
memory: "512Mi"
cpu: "500m"
limits:
memory: "1Gi"
cpu: "1000m"
livenessProbe:
httpGet:
path: /ready
port: 3100
initialDelaySeconds: 45
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 3100
initialDelaySeconds: 30
periodSeconds: 5
volumes:
- name: config
configMap:
name: loki-config
- name: overrides
configMap:
name: loki-overrides
---
apiVersion: v1
kind: Service
metadata:
name: loki-distributor
namespace: logging
labels:
app: loki
component: distributor
spec:
type: ClusterIP
selector:
app: loki
component: distributor
ports:
- name: http
port: 3100
targetPort: 3100
- name: grpc
port: 9095
targetPort: 9095
Ingester StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: loki-ingester
namespace: logging
labels:
app: loki
component: ingester
spec:
serviceName: loki-ingester
replicas: 3
podManagementPolicy: Parallel
selector:
matchLabels:
app: loki
component: ingester
template:
metadata:
labels:
app: loki
component: ingester
loki-gossip-member: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "3100"
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: component
operator: In
values:
- ingester
topologyKey: kubernetes.io/hostname
securityContext:
fsGroup: 10001
runAsNonRoot: true
runAsUser: 10001
containers:
- name: ingester
image: grafana/loki:2.9.3
args:
- -config.file=/etc/loki/config.yaml
- -target=ingester
- -config.expand-env=true
env:
- name: HOSTNAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: loki-s3-credentials
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: loki-s3-credentials
key: AWS_SECRET_ACCESS_KEY
ports:
- name: http
containerPort: 3100
- name: grpc
containerPort: 9095
- name: gossip
containerPort: 7946
volumeMounts:
- name: config
mountPath: /etc/loki
- name: overrides
mountPath: /etc/loki/overrides.yaml
subPath: overrides.yaml
- name: data
mountPath: /loki
resources:
requests:
memory: "4Gi"
cpu: "2000m"
limits:
memory: "8Gi"
cpu: "4000m"
livenessProbe:
httpGet:
path: /ready
port: 3100
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /ready
port: 3100
initialDelaySeconds: 45
periodSeconds: 5
timeoutSeconds: 5
volumes:
- name: config
configMap:
name: loki-config
- name: overrides
configMap:
name: loki-overrides
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: fast-ssd
resources:
requests:
storage: 100Gi
---
apiVersion: v1
kind: Service
metadata:
name: loki-ingester
namespace: logging
labels:
app: loki
component: ingester
spec:
type: ClusterIP
clusterIP: None
selector:
app: loki
component: ingester
ports:
- name: http
port: 3100
targetPort: 3100
- name: grpc
port: 9095
targetPort: 9095
Querier Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: loki-querier
namespace: logging
labels:
app: loki
component: querier
spec:
replicas: 3
selector:
matchLabels:
app: loki
component: querier
template:
metadata:
labels:
app: loki
component: querier
loki-gossip-member: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "3100"
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: component
operator: In
values:
- querier
topologyKey: kubernetes.io/hostname
containers:
- name: querier
image: grafana/loki:2.9.3
args:
- -config.file=/etc/loki/config.yaml
- -target=querier
- -config.expand-env=true
env:
- name: HOSTNAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: loki-s3-credentials
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: loki-s3-credentials
key: AWS_SECRET_ACCESS_KEY
ports:
- name: http
containerPort: 3100
- name: grpc
containerPort: 9095
- name: gossip
containerPort: 7946
volumeMounts:
- name: config
mountPath: /etc/loki
- name: overrides
mountPath: /etc/loki/overrides.yaml
subPath: overrides.yaml
- name: cache
mountPath: /loki
resources:
requests:
memory: "2Gi"
cpu: "1000m"
limits:
memory: "4Gi"
cpu: "2000m"
livenessProbe:
httpGet:
path: /ready
port: 3100
initialDelaySeconds: 45
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 3100
initialDelaySeconds: 30
periodSeconds: 5
volumes:
- name: config
configMap:
name: loki-config
- name: overrides
configMap:
name: loki-overrides
- name: cache
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: loki-querier
namespace: logging
labels:
app: loki
component: querier
spec:
type: ClusterIP
selector:
app: loki
component: querier
ports:
- name: http
port: 3100
targetPort: 3100
- name: grpc
port: 9095
targetPort: 9095
Query Frontend Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: loki-query-frontend
namespace: logging
labels:
app: loki
component: query-frontend
spec:
replicas: 2
selector:
matchLabels:
app: loki
component: query-frontend
template:
metadata:
labels:
app: loki
component: query-frontend
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "3100"
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: component
operator: In
values:
- query-frontend
topologyKey: kubernetes.io/hostname
containers:
- name: query-frontend
image: grafana/loki:2.9.3
args:
- -config.file=/etc/loki/config.yaml
- -target=query-frontend
- -config.expand-env=true
env:
- name: HOSTNAME
valueFrom:
fieldRef:
fieldPath: metadata.name
ports:
- name: http
containerPort: 3100
- name: grpc
containerPort: 9095
volumeMounts:
- name: config
mountPath: /etc/loki
- name: overrides
mountPath: /etc/loki/overrides.yaml
subPath: overrides.yaml
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"
livenessProbe:
httpGet:
path: /ready
port: 3100
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 3100
initialDelaySeconds: 15
periodSeconds: 5
volumes:
- name: config
configMap:
name: loki-config
- name: overrides
configMap:
name: loki-overrides
---
apiVersion: v1
kind: Service
metadata:
name: loki-query-frontend
namespace: logging
labels:
app: loki
component: query-frontend
spec:
type: LoadBalancer
selector:
app: loki
component: query-frontend
ports:
- name: http
port: 3100
targetPort: 3100
- name: grpc
port: 9095
targetPort: 9095
Compactor StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: loki-compactor
namespace: logging
labels:
app: loki
component: compactor
spec:
serviceName: loki-compactor
replicas: 1
selector:
matchLabels:
app: loki
component: compactor
template:
metadata:
labels:
app: loki
component: compactor
loki-gossip-member: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "3100"
spec:
securityContext:
fsGroup: 10001
runAsNonRoot: true
runAsUser: 10001
containers:
- name: compactor
image: grafana/loki:2.9.3
args:
- -config.file=/etc/loki/config.yaml
- -target=compactor
- -config.expand-env=true
env:
- name: HOSTNAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: loki-s3-credentials
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: loki-s3-credentials
key: AWS_SECRET_ACCESS_KEY
ports:
- name: http
containerPort: 3100
- name: grpc
containerPort: 9095
- name: gossip
containerPort: 7946
volumeMounts:
- name: config
mountPath: /etc/loki
- name: overrides
mountPath: /etc/loki/overrides.yaml
subPath: overrides.yaml
- name: data
mountPath: /loki
resources:
requests:
memory: "2Gi"
cpu: "1000m"
limits:
memory: "4Gi"
cpu: "2000m"
livenessProbe:
httpGet:
path: /ready
port: 3100
initialDelaySeconds: 60
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 3100
initialDelaySeconds: 45
periodSeconds: 5
volumes:
- name: config
configMap:
name: loki-config
- name: overrides
configMap:
name: loki-overrides
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: fast-ssd
resources:
requests:
storage: 50Gi
Memberlist Service
apiVersion: v1
kind: Service
metadata:
name: loki-memberlist
namespace: logging
labels:
app: loki
spec:
type: ClusterIP
clusterIP: None
publishNotReadyAddresses: true
selector:
loki-gossip-member: "true"
ports:
- name: gossip
port: 7946
targetPort: 7946
protocol: TCP
Log Shippers Configuration
Promtail DaemonSet
apiVersion: v1
kind: ConfigMap
metadata:
name: promtail-config
namespace: logging
data:
promtail.yaml: |
server:
http_listen_port: 9080
grpc_listen_port: 0
log_level: info
positions:
filename: /run/promtail/positions.yaml
clients:
- url: http://loki-distributor.logging.svc.cluster.local:3100/loki/api/v1/push
tenant_id: prod
batchwait: 1s
batchsize: 1048576
timeout: 10s
backoff_config:
min_period: 500ms
max_period: 5m
max_retries: 10
external_labels:
cluster: production
datacenter: us-east-1
scrape_configs:
# Kubernetes pods
- job_name: kubernetes-pods
pipeline_stages:
# Extract pod metadata
- cri: {}
# Parse JSON logs
- json:
expressions:
level: level
msg: message
timestamp: timestamp
# Extract level
- labels:
level:
# Drop debug logs
- drop:
source: level
expression: "debug"
kubernetes_sd_configs:
- role: pod
relabel_configs:
# Only scrape pods with correct annotation
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape_logs]
action: keep
regex: true
# Add namespace label
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
# Add pod name label
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
# Add container name label
- source_labels: [__meta_kubernetes_pod_container_name]
target_label: container
# Add app label
- source_labels: [__meta_kubernetes_pod_label_app]
target_label: app
# Add job label from app label
- source_labels: [__meta_kubernetes_pod_label_app]
target_label: job
# Add node name label
- source_labels: [__meta_kubernetes_pod_node_name]
target_label: node
# Set path to container logs
- source_labels: [__meta_kubernetes_pod_uid, __meta_kubernetes_pod_container_name]
target_label: __path__
separator: /
replacement: /var/log/pods/*$1/*.log
# System logs
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: system
__path__: /var/log/*.log
# Kubernetes audit logs
- job_name: kubernetes-audit
static_configs:
- targets:
- localhost
labels:
job: kubernetes-audit
__path__: /var/log/kubernetes/audit/*.log
pipeline_stages:
- json:
expressions:
verb: verb
user: user.username
resource: objectRef.resource
- labels:
verb:
user:
resource:
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: promtail
namespace: logging
labels:
app: promtail
spec:
selector:
matchLabels:
app: promtail
template:
metadata:
labels:
app: promtail
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9080"
spec:
serviceAccountName: promtail
containers:
- name: promtail
image: grafana/promtail:2.9.3
args:
- -config.file=/etc/promtail/promtail.yaml
env:
- name: HOSTNAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
ports:
- name: http
containerPort: 9080
volumeMounts:
- name: config
mountPath: /etc/promtail
- name: positions
mountPath: /run/promtail
- name: varlog
mountPath: /var/log
readOnly: true
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "256Mi"
cpu: "200m"
securityContext:
privileged: true
runAsUser: 0
volumes:
- name: config
configMap:
name: promtail-config
- name: positions
hostPath:
path: /var/lib/promtail
type: DirectoryOrCreate
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
tolerations:
- effect: NoSchedule
operator: Exists
Grafana Integration
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-datasources
namespace: logging
data:
loki.yaml: |
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
url: http://loki-query-frontend.logging.svc.cluster.local:3100
jsonData:
maxLines: 1000
derivedFields:
# Extract trace ID from logs
- datasourceUid: tempo
matcherRegex: "traceID=(\\w+)"
name: TraceID
url: "$${__value.raw}"
# Extract request ID
- matcherRegex: "requestID=(\\w+)"
name: RequestID
url: "/explore?left={\"queries\":[{\"expr\":\"{job=\\\"nginx\\\"} |= \\\"$${__value.raw}\\\"\"}]}"
LogQL Query Examples
# Show all logs for a specific pod
{namespace="production", pod="api-server-xyz"}
# Filter by log level
{namespace="production"} |= "level=error"
# JSON parsing and filtering
{app="api-server"} | json | level="error" | line_format "{{.timestamp}} {{.message}}"
# Rate queries
sum(rate({namespace="production"}[5m])) by (pod)
# Top 10 error-prone pods
topk(10, sum(rate({namespace="production"} |= "error" [5m])) by (pod))
# Latency analysis
quantile_over_time(0.99, {app="api"} | json | __error__="" | unwrap duration [5m]) by (endpoint)
# Log pattern detection
{namespace="production"} |= "error" | pattern "<_> error: <error_msg>"
# Multi-line log parsing
{app="java-app"} | multiline "^\\d{4}-\\d{2}-\\d{2}" firstline ^\\d{4}
# Label formatting
{app="nginx"} | json | line_format "{{.method}} {{.path}} {{.status}} {{.duration}}ms"
Performance Optimization
Memcached for Caching
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: memcached-frontend
namespace: logging
spec:
serviceName: memcached-frontend
replicas: 3
selector:
matchLabels:
app: memcached
component: frontend
template:
metadata:
labels:
app: memcached
component: frontend
spec:
containers:
- name: memcached
image: memcached:1.6-alpine
args:
- -m 2048
- -c 1024
- -I 32m
- -t 4
ports:
- name: memcached
containerPort: 11211
resources:
requests:
memory: "2Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
---
apiVersion: v1
kind: Service
metadata:
name: memcached-frontend
namespace: logging
spec:
type: ClusterIP
clusterIP: None
selector:
app: memcached
component: frontend
ports:
- port: 11211
targetPort: 11211
Monitoring and Alerting
apiVersion: v1
kind: ConfigMap
metadata:
name: loki-rules
namespace: logging
data:
alerts.yaml: |
groups:
- name: loki_alerts
interval: 30s
rules:
# High error rate
- alert: HighErrorRate
expr: |
sum(rate({namespace="production"} |= "error" [5m])) by (namespace, app)
> 10
for: 5m
labels:
severity: warning
annotations:
summary: "High error rate detected"
description: "{{ $labels.app }} in {{ $labels.namespace }} has error rate of {{ $value }} errors/sec"
# Ingester not ready
- alert: LokiIngesterNotReady
expr: |
kube_pod_container_status_ready{namespace="logging",container="ingester"} == 0
for: 5m
labels:
severity: critical
annotations:
summary: "Loki ingester not ready"
# Request failures
- alert: LokiRequestErrors
expr: |
sum(rate(loki_request_duration_seconds_count{status_code=~"5.."}[5m])) by (job, route)
/ sum(rate(loki_request_duration_seconds_count[5m])) by (job, route)
> 0.05
for: 15m
labels:
severity: warning
Best Practices
- Use appropriate cardinality - limit label values
- Implement tenant isolation with multi-tenancy
- Configure proper retention based on compliance needs
- Use structured logging for better parsing
- Enable caching for query performance
- Monitor Loki metrics for health
- Use LogQL patterns for efficient queries
- Implement rate limiting per tenant
- Configure index caching for faster queries
- Regular compaction for storage efficiency
Conclusion
Grafana Loki provides a cost-effective, scalable solution for log aggregation in Kubernetes environments. By deploying in microservices mode with proper multi-tenancy, caching, and retention policies, organizations can achieve enterprise-grade log management while minimizing infrastructure costs and operational complexity.