Falco Runtime Security: Enterprise Container Threat Detection and Response Guide
Falco has emerged as the de facto standard for runtime security and threat detection in cloud-native environments. As the first runtime security project to graduate from the CNCF, Falco provides real-time detection of anomalous behavior in applications, containers, and the underlying infrastructure using eBPF and kernel modules.
In this comprehensive guide, we’ll explore enterprise-grade Falco deployment patterns, custom rule development, integration with security orchestration platforms, and incident response workflows that have proven effective in production environments serving millions of requests.
Understanding Falco’s Architecture and Detection Capabilities
Core Components and Detection Engine
Falco operates at the kernel level, intercepting system calls to detect suspicious behavior. The architecture consists of several key components:
Driver Layer: Falco uses either an eBPF probe or kernel module to capture system calls with minimal performance overhead. The eBPF approach is preferred for modern kernels (4.14+) as it doesn’t require kernel module compilation.
Rules Engine: Falco’s rules engine processes system call events against a flexible rule set written in YAML. Rules can detect everything from unauthorized file access to cryptocurrency mining activities.
Output Framework: Detected events are formatted and sent to multiple destinations including stdout, files, syslog, and external systems via webhooks.
Let’s examine a production-ready Falco deployment architecture:
# falco-deployment.yaml - Production Falco DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: falco
namespace: falco-security
labels:
app: falco
app.kubernetes.io/name: falco
app.kubernetes.io/component: runtime-security
spec:
selector:
matchLabels:
app: falco
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
template:
metadata:
labels:
app: falco
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8765"
spec:
serviceAccountName: falco
hostNetwork: true
hostPID: true
priorityClassName: system-node-critical
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
containers:
- name: falco
image: falcosecurity/falco:0.36.2
securityContext:
privileged: true
args:
- /usr/bin/falco
- --cri
- /run/containerd/containerd.sock
- --cri
- /run/crio/crio.sock
- -K
- /var/run/secrets/kubernetes.io/serviceaccount/token
- -k
- https://kubernetes.default
- -pk
env:
- name: FALCO_BPF_PROBE
value: ""
- name: FALCO_GRPC_ENABLED
value: "true"
- name: FALCO_GRPC_BIND_ADDRESS
value: "0.0.0.0:5060"
- name: FALCO_METRICS_ENABLED
value: "true"
resources:
requests:
cpu: 100m
memory: 512Mi
limits:
cpu: 1000m
memory: 1Gi
volumeMounts:
- mountPath: /host/var/run/docker.sock
name: docker-socket
- mountPath: /host/run/containerd/containerd.sock
name: containerd-socket
- mountPath: /host/run/crio/crio.sock
name: crio-socket
- mountPath: /host/dev
name: dev-fs
readOnly: true
- mountPath: /host/proc
name: proc-fs
readOnly: true
- mountPath: /host/boot
name: boot-fs
readOnly: true
- mountPath: /host/lib/modules
name: lib-modules
- mountPath: /host/usr
name: usr-fs
readOnly: true
- mountPath: /host/etc
name: etc-fs
readOnly: true
- mountPath: /etc/falco
name: falco-config
- mountPath: /etc/falco/rules.d
name: falco-rules
volumes:
- name: docker-socket
hostPath:
path: /var/run/docker.sock
- name: containerd-socket
hostPath:
path: /run/containerd/containerd.sock
- name: crio-socket
hostPath:
path: /run/crio/crio.sock
- name: dev-fs
hostPath:
path: /dev
- name: proc-fs
hostPath:
path: /proc
- name: boot-fs
hostPath:
path: /boot
- name: lib-modules
hostPath:
path: /lib/modules
- name: usr-fs
hostPath:
path: /usr
- name: etc-fs
hostPath:
path: /etc
- name: falco-config
configMap:
name: falco-config
- name: falco-rules
configMap:
name: falco-rules
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: falco
namespace: falco-security
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: falco
rules:
- apiGroups:
- ""
resources:
- nodes
- namespaces
- pods
- replicationcontrollers
- services
verbs:
- get
- list
- watch
- apiGroups:
- apps
resources:
- daemonsets
- deployments
- replicasets
- statefulsets
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: falco
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: falco
subjects:
- kind: ServiceAccount
name: falco
namespace: falco-security
Production Falco Configuration
The Falco configuration file controls behavior, output formatting, and integration points:
# falco-config.yaml - Enterprise Falco Configuration
apiVersion: v1
kind: ConfigMap
metadata:
name: falco-config
namespace: falco-security
data:
falco.yaml: |
# Core engine settings
rules_file:
- /etc/falco/falco_rules.yaml
- /etc/falco/falco_rules.local.yaml
- /etc/falco/rules.d
# Performance tuning
syscall_event_drops:
actions:
- log
- alert
rate: 0.03333
max_burst: 10
# Enable modern eBPF probe
engine:
kind: ebpf
ebpf:
buf_size_preset: 4
drop_failed_exit: false
# Output formatting
json_output: true
json_include_output_property: true
json_include_tags_property: true
# Logging
log_stderr: true
log_syslog: false
log_level: info
# Priority threshold
priority: debug
# Buffering for high-volume environments
buffered_outputs: true
# Output channels
outputs:
rate: 0
max_burst: 1000
# Syslog output configuration
syslog_output:
enabled: false
# File output for audit trail
file_output:
enabled: true
keep_alive: false
filename: /var/log/falco/events.log
# Stdout output
stdout_output:
enabled: true
# HTTP output for webhook integration
http_output:
enabled: true
url: "http://falco-exporter.falco-security.svc.cluster.local:2801/events"
user_agent: "falco/0.36.2"
mtls: false
insecure: false
compress: true
# Program output for custom processing
program_output:
enabled: false
keep_alive: false
program: "jq '{text: .output}' | curl -d @- -X POST https://hooks.slack.com/services/XXX"
# gRPC API configuration
grpc:
enabled: true
bind_address: "0.0.0.0:5060"
threadiness: 8
# gRPC output for external consumers
grpc_output:
enabled: true
# Kubernetes metadata enrichment
kubernetes:
enabled: true
api_server: "https://kubernetes.default"
token_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
ssl_verify: true
# Metrics exposition
metrics:
enabled: true
interval: 1h
output_rule: true
resource_utilization_enabled: true
kernel_event_counters_enabled: true
libbpf_stats_enabled: true
convert_memory_to_mb: true
include_empty_values: false
# Web UI configuration
webserver:
enabled: true
listen_port: 8765
k8s_healthz_endpoint: /healthz
ssl_enabled: false
Custom Rule Development for Enterprise Environments
Understanding Falco Rule Syntax
Falco rules are written in a declarative YAML format that combines conditions with output templates. Let’s explore enterprise-grade custom rules:
# falco-custom-rules.yaml - Enterprise Custom Rules
apiVersion: v1
kind: ConfigMap
metadata:
name: falco-rules
namespace: falco-security
data:
custom_rules.yaml: |
# Macro definitions for reusability
- macro: sensitive_mount
condition: >
(container.mount.dest[/proc*] != "N/A" or
container.mount.dest[/var/run/docker.sock] != "N/A" or
container.mount.dest[/var/run/crio/crio.sock] != "N/A" or
container.mount.dest[/run/containerd/containerd.sock] != "N/A" or
container.mount.dest[/host] != "N/A" or
container.mount.dest[/etc] != "N/A")
- macro: production_namespace
condition: >
(k8s.ns.name in (production, prod, prd, default))
- macro: sensitive_file_access
condition: >
(fd.name startswith /etc/shadow or
fd.name startswith /etc/sudoers or
fd.name startswith /etc/pam.d or
fd.name startswith /etc/ssh/sshd_config or
fd.name startswith /root/.ssh or
fd.name startswith /home/*/.ssh/id_ or
fd.name startswith /etc/kubernetes/pki)
- macro: known_binary_locations
condition: >
(proc.pname in (docker, containerd, crio, kubelet, kube-proxy))
# List of approved container registries
- list: allowed_registries
items:
- docker.io/company
- gcr.io/company-project
- registry.company.com
- quay.io/company
# List of privileged service accounts
- list: privileged_service_accounts
items:
- falco
- cilium
- calico-node
- kube-proxy
- nvidia-device-plugin
# Rule: Detect unauthorized registry usage
- rule: Unauthorized Container Registry
desc: >
Container started from unauthorized registry.
Only approved internal registries should be used in production.
condition: >
container.start and
not container.image.repository in (allowed_registries) and
production_namespace
output: >
Unauthorized registry usage detected
(user=%user.name
user_uid=%user.uid
command=%proc.cmdline
container_id=%container.id
container_name=%container.name
image=%container.image.repository
namespace=%k8s.ns.name
pod=%k8s.pod.name)
priority: WARNING
tags: [container, registry, compliance]
source: syscall
# Rule: Detect cryptocurrency mining
- rule: Cryptocurrency Mining Activity
desc: >
Detection of cryptocurrency mining processes.
Common mining software or suspicious CPU-intensive processes detected.
condition: >
spawned_process and
(proc.name in (xmrig, minerd, cpuminer, ethminer, phoenixminer, t-rex, nbminer) or
(proc.name in (python, python3, node, java) and
proc.cmdline contains stratum))
output: >
Potential cryptocurrency mining detected
(user=%user.name
process=%proc.name
command=%proc.cmdline
parent=%proc.pname
container=%container.name
namespace=%k8s.ns.name
pod=%k8s.pod.name
cpu_usage=%proc.cpu.time)
priority: CRITICAL
tags: [malware, mining, threat]
source: syscall
# Rule: Sensitive file access in production
- rule: Sensitive File Access in Production
desc: >
Unauthorized access to sensitive configuration files in production containers.
This may indicate credential theft or privilege escalation attempts.
condition: >
open_read and
sensitive_file_access and
production_namespace and
not known_binary_locations and
not k8s.sa.name in (privileged_service_accounts)
output: >
Sensitive file accessed in production
(user=%user.name
process=%proc.name
command=%proc.cmdline
file=%fd.name
container=%container.name
namespace=%k8s.ns.name
pod=%k8s.pod.name
service_account=%k8s.sa.name)
priority: HIGH
tags: [filesystem, security, production]
source: syscall
# Rule: Detect reverse shell activity
- rule: Reverse Shell Connection
desc: >
Detection of reverse shell activity using common techniques.
Indicates potential container compromise or attacker access.
condition: >
spawned_process and
((proc.name = bash and
proc.cmdline contains "-i" and
(proc.cmdline contains "/dev/tcp" or
proc.cmdline contains "/dev/udp" or
proc.cmdline contains ">& /dev/tcp" or
proc.cmdline contains ">&/dev/tcp")) or
(proc.name in (nc, ncat, netcat, socat) and
(proc.cmdline contains "-e" or
proc.cmdline contains "-c" or
proc.cmdline contains "exec")) or
(proc.name = python and
proc.cmdline contains "socket" and
proc.cmdline contains "subprocess"))
output: >
Reverse shell activity detected
(user=%user.name
process=%proc.name
command=%proc.cmdline
parent=%proc.pname
container=%container.name
namespace=%k8s.ns.name
pod=%k8s.pod.name
connection=%fd.name)
priority: CRITICAL
tags: [network, shell, compromise]
source: syscall
# Rule: Package management in running container
- rule: Package Management in Running Container
desc: >
Package manager executed in running container.
Containers should be immutable and not have packages installed at runtime.
condition: >
spawned_process and
container and
proc.name in (apt, apt-get, yum, dnf, apk, pip, pip3, npm, gem) and
production_namespace
output: >
Package manager used in running container
(user=%user.name
process=%proc.name
command=%proc.cmdline
container=%container.name
image=%container.image.repository
namespace=%k8s.ns.name
pod=%k8s.pod.name)
priority: WARNING
tags: [container, immutability, compliance]
source: syscall
# Rule: Privileged container creation
- rule: Privileged Container Started
desc: >
Privileged container started outside of approved system namespaces.
Privileged containers can compromise node security.
condition: >
container.start and
container.privileged=true and
not k8s.ns.name in (kube-system, falco-security, cilium, calico-system, metallb-system)
output: >
Privileged container started
(user=%user.name
container=%container.name
image=%container.image.repository
namespace=%k8s.ns.name
pod=%k8s.pod.name
service_account=%k8s.sa.name)
priority: HIGH
tags: [container, privilege, security]
source: k8s_audit
# Rule: Kernel module loading
- rule: Kernel Module Load
desc: >
Kernel module loaded on host system.
Unexpected kernel module loading may indicate rootkit installation.
condition: >
spawned_process and
proc.name in (insmod, modprobe) and
not container
output: >
Kernel module loading detected
(user=%user.name
command=%proc.cmdline
module=%proc.args
parent=%proc.pname)
priority: CRITICAL
tags: [host, kernel, rootkit]
source: syscall
# Rule: Container drift detection
- rule: Container File System Modification
desc: >
File system modification in container that should be immutable.
Detects container drift from original image.
condition: >
open_write and
container and
not fd.name startswith /tmp and
not fd.name startswith /var/log and
not fd.name startswith /var/run and
not fd.name startswith /dev and
production_namespace
output: >
Container file system modified
(user=%user.name
process=%proc.name
command=%proc.cmdline
file=%fd.name
container=%container.name
image=%container.image.repository
namespace=%k8s.ns.name
pod=%k8s.pod.name)
priority: NOTICE
tags: [container, drift, immutability]
source: syscall
# Rule: Suspicious network activity
- rule: Outbound Connection to Known Malicious IP
desc: >
Outbound connection to IP address on threat intelligence blocklist.
Indicates potential data exfiltration or C2 communication.
condition: >
outbound and
fd.sip in (threat_intel_blocklist)
output: >
Connection to malicious IP detected
(user=%user.name
process=%proc.name
command=%proc.cmdline
destination=%fd.rip
port=%fd.rport
protocol=%fd.l4proto
container=%container.name
namespace=%k8s.ns.name
pod=%k8s.pod.name)
priority: CRITICAL
tags: [network, threat-intel, exfiltration]
source: syscall
# Rule: Service account token access
- rule: Service Account Token Access
desc: >
Suspicious access to Kubernetes service account token.
May indicate credential theft for cluster privilege escalation.
condition: >
open_read and
fd.name startswith /var/run/secrets/kubernetes.io/serviceaccount/token and
not proc.name in (kubelet, kube-proxy) and
not k8s.sa.name in (privileged_service_accounts)
output: >
Service account token accessed
(user=%user.name
process=%proc.name
command=%proc.cmdline
container=%container.name
namespace=%k8s.ns.name
pod=%k8s.pod.name
service_account=%k8s.sa.name)
priority: HIGH
tags: [kubernetes, credentials, privilege-escalation]
source: syscall
Integration with Security Orchestration Platforms
Falco Sidekick for Multi-Destination Routing
Falco Sidekick provides advanced output routing and transformation capabilities:
# falco-sidekick-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: falco-sidekick
namespace: falco-security
spec:
replicas: 2
selector:
matchLabels:
app: falco-sidekick
template:
metadata:
labels:
app: falco-sidekick
spec:
serviceAccountName: falco-sidekick
containers:
- name: falco-sidekick
image: falcosecurity/falco-sidekick:2.28.0
env:
# Slack integration
- name: SLACK_WEBHOOKURL
valueFrom:
secretKeyRef:
name: falco-sidekick-secrets
key: slack-webhook
- name: SLACK_MINIMUMPRIORITY
value: "warning"
- name: SLACK_MESSAGEFORMAT
value: "long"
# PagerDuty integration
- name: PAGERDUTY_ROUTINGKEY
valueFrom:
secretKeyRef:
name: falco-sidekick-secrets
key: pagerduty-key
- name: PAGERDUTY_MINIMUMPRIORITY
value: "critical"
# Elasticsearch integration
- name: ELASTICSEARCH_HOSTPORT
value: "https://elasticsearch.logging.svc.cluster.local:9200"
- name: ELASTICSEARCH_INDEX
value: "falco"
- name: ELASTICSEARCH_TYPE
value: "_doc"
- name: ELASTICSEARCH_MINIMUMPRIORITY
value: "debug"
- name: ELASTICSEARCH_USERNAME
valueFrom:
secretKeyRef:
name: falco-sidekick-secrets
key: elasticsearch-username
- name: ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
name: falco-sidekick-secrets
key: elasticsearch-password
# Loki integration
- name: LOKI_HOSTPORT
value: "http://loki.logging.svc.cluster.local:3100"
- name: LOKI_MINIMUMPRIORITY
value: "debug"
# Prometheus metrics
- name: PROMETHEUS_EXTRALABELS
value: "cluster:production,region:us-east-1"
# AWS Security Hub integration
- name: AWS_SECURITYHUB_REGION
value: "us-east-1"
- name: AWS_SECURITYHUB_ACCOUNTID
valueFrom:
secretKeyRef:
name: falco-sidekick-secrets
key: aws-account-id
- name: AWS_SECURITYHUB_MINIMUMPRIORITY
value: "high"
# Webhook for custom integration
- name: WEBHOOK_ADDRESS
value: "http://security-automation.security.svc.cluster.local:8080/falco"
- name: WEBHOOK_MINIMUMPRIORITY
value: "warning"
ports:
- name: http
containerPort: 2801
- name: metrics
containerPort: 2112
livenessProbe:
httpGet:
path: /ping
port: 2801
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /ping
port: 2801
initialDelaySeconds: 5
periodSeconds: 5
resources:
requests:
cpu: 50m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
---
apiVersion: v1
kind: Service
metadata:
name: falco-sidekick
namespace: falco-security
spec:
selector:
app: falco-sidekick
ports:
- name: http
port: 2801
targetPort: 2801
- name: metrics
port: 2112
targetPort: 2112
Custom Security Automation Response
Implement automated response to Falco events:
// security-automation-service.go
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"net/http"
"time"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
)
type FalcoEvent struct {
UUID string `json:"uuid"`
Output string `json:"output"`
Priority string `json:"priority"`
Rule string `json:"rule"`
Time time.Time `json:"time"`
OutputFields map[string]interface{} `json:"output_fields"`
Source string `json:"source"`
Tags []string `json:"tags"`
Hostname string `json:"hostname"`
}
type ResponseAction string
const (
ActionQuarantine ResponseAction = "quarantine"
ActionAlert ResponseAction = "alert"
ActionBlock ResponseAction = "block"
ActionLog ResponseAction = "log"
)
type SecurityAutomation struct {
k8sClient *kubernetes.Clientset
rules map[string]ResponseAction
}
func NewSecurityAutomation() (*SecurityAutomation, error) {
config, err := rest.InClusterConfig()
if err != nil {
return nil, fmt.Errorf("failed to get in-cluster config: %w", err)
}
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
return nil, fmt.Errorf("failed to create kubernetes client: %w", err)
}
return &SecurityAutomation{
k8sClient: clientset,
rules: map[string]ResponseAction{
"Reverse Shell Connection": ActionQuarantine,
"Cryptocurrency Mining Activity": ActionQuarantine,
"Kernel Module Load": ActionBlock,
"Privileged Container Started": ActionAlert,
"Sensitive File Access in Production": ActionAlert,
},
}, nil
}
func (sa *SecurityAutomation) HandleFalcoEvent(w http.ResponseWriter, r *http.Request) {
var event FalcoEvent
if err := json.NewDecoder(r.Body).Decode(&event); err != nil {
http.Error(w, "Invalid request body", http.StatusBadRequest)
return
}
log.Printf("Received Falco event: rule=%s priority=%s", event.Rule, event.Priority)
// Determine response action
action, exists := sa.rules[event.Rule]
if !exists {
action = ActionLog
}
// Execute response action
switch action {
case ActionQuarantine:
if err := sa.quarantinePod(event); err != nil {
log.Printf("Failed to quarantine pod: %v", err)
}
case ActionBlock:
if err := sa.blockAction(event); err != nil {
log.Printf("Failed to block action: %v", err)
}
case ActionAlert:
sa.sendAlert(event)
case ActionLog:
sa.logEvent(event)
}
w.WriteHeader(http.StatusOK)
}
func (sa *SecurityAutomation) quarantinePod(event FalcoEvent) error {
namespace := event.OutputFields["k8s.ns.name"].(string)
podName := event.OutputFields["k8s.pod.name"].(string)
log.Printf("Quarantining pod: %s/%s", namespace, podName)
// Add quarantine label
pod, err := sa.k8sClient.CoreV1().Pods(namespace).Get(
context.TODO(),
podName,
metav1.GetOptions{},
)
if err != nil {
return fmt.Errorf("failed to get pod: %w", err)
}
if pod.Labels == nil {
pod.Labels = make(map[string]string)
}
pod.Labels["security.falco.org/quarantined"] = "true"
pod.Labels["security.falco.org/quarantine-reason"] = event.Rule
pod.Labels["security.falco.org/quarantine-time"] = time.Now().Format(time.RFC3339)
// Apply network policy to isolate pod
_, err = sa.k8sClient.CoreV1().Pods(namespace).Update(
context.TODO(),
pod,
metav1.UpdateOptions{},
)
if err != nil {
return fmt.Errorf("failed to update pod labels: %w", err)
}
// Create incident ticket
sa.createIncident(event, "Pod quarantined due to security violation")
return nil
}
func (sa *SecurityAutomation) blockAction(event FalcoEvent) error {
// Implement blocking logic (e.g., update network policies)
log.Printf("Blocking action for event: %s", event.Rule)
return nil
}
func (sa *SecurityAutomation) sendAlert(event FalcoEvent) {
// Send to alerting system
log.Printf("ALERT: %s - %s", event.Rule, event.Output)
}
func (sa *SecurityAutomation) logEvent(event FalcoEvent) {
// Log to security event store
log.Printf("SECURITY_EVENT: %s", event.Output)
}
func (sa *SecurityAutomation) createIncident(event FalcoEvent, description string) {
// Integrate with incident management system
log.Printf("Creating incident: rule=%s description=%s", event.Rule, description)
}
func main() {
sa, err := NewSecurityAutomation()
if err != nil {
log.Fatalf("Failed to initialize security automation: %v", err)
}
http.HandleFunc("/falco", sa.HandleFalcoEvent)
http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
})
log.Println("Starting security automation service on :8080")
if err := http.ListenAndServe(":8080", nil); err != nil {
log.Fatalf("Server failed: %v", err)
}
}
Performance Tuning and Optimization
Reducing System Call Overhead
Configure Falco’s buffering and drop policies for high-traffic environments:
# Performance-optimized configuration
syscall_event_drops:
actions:
- log
- alert
rate: 0.03333 # Allow 3.3% drops before alerting
max_burst: 10 # Maximum consecutive drops
# Increase buffer sizes for high-volume
engine:
kind: ebpf
ebpf:
buf_size_preset: 7 # 8 = 16MB per CPU (maximum)
drop_failed_exit: true
# Rule optimization
base_syscalls:
custom_set:
- open
- openat
- openat2
- execve
- execveat
- connect
- accept
- socket
repair: false
Monitoring Falco Performance
Deploy Prometheus monitoring for Falco metrics:
# falco-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: falco
namespace: falco-security
spec:
selector:
matchLabels:
app: falco
endpoints:
- port: metrics
interval: 30s
path: /metrics
---
# Grafana dashboard ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: falco-dashboard
namespace: monitoring
data:
falco-dashboard.json: |
{
"dashboard": {
"title": "Falco Runtime Security",
"panels": [
{
"title": "Event Rate",
"targets": [{
"expr": "rate(falco_events_total[5m])"
}]
},
{
"title": "Drop Rate",
"targets": [{
"expr": "rate(falco_drops_total[5m])"
}]
},
{
"title": "Events by Priority",
"targets": [{
"expr": "sum(rate(falco_events_total[5m])) by (priority)"
}]
},
{
"title": "Events by Rule",
"targets": [{
"expr": "topk(10, sum(rate(falco_events_total[5m])) by (rule))"
}]
}
]
}
}
Incident Response Workflows
Automated Investigation Playbook
Create automated incident response playbooks:
# incident-response-playbook.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: incident-response-playbook
namespace: falco-security
data:
playbook.yaml: |
playbooks:
- name: cryptocurrency_mining
trigger:
rule: "Cryptocurrency Mining Activity"
priority: CRITICAL
steps:
- action: quarantine_pod
timeout: 30s
- action: capture_network_traffic
duration: 60s
- action: collect_process_dump
- action: save_container_logs
- action: snapshot_filesystem
- action: create_security_incident
severity: high
- action: notify_security_team
channel: slack
pagerduty: true
- name: reverse_shell
trigger:
rule: "Reverse Shell Connection"
priority: CRITICAL
steps:
- action: isolate_pod
- action: capture_memory_dump
- action: preserve_evidence
- action: terminate_pod
- action: create_security_incident
severity: critical
- action: escalate_to_security_team
- name: sensitive_file_access
trigger:
rule: "Sensitive File Access in Production"
priority: HIGH
steps:
- action: log_detailed_audit
- action: capture_process_tree
- action: review_rbac_permissions
- action: create_security_ticket
- action: notify_team_lead
Best Practices and Recommendations
Rule Development Guidelines
- Start Broad, Refine Gradually: Begin with default rules and tune based on false positive rates
- Use Macros and Lists: Improve maintainability by extracting common conditions
- Test in Non-Production: Validate new rules in development environments first
- Version Control: Store custom rules in Git with proper change management
- Document Exceptions: Clearly document why certain activities are whitelisted
Performance Considerations
- Buffer Sizing: Increase eBPF buffer sizes in high-traffic environments
- Rule Optimization: Minimize expensive operations like regex matching
- Selective Monitoring: Use namespace or label selectors to focus on critical workloads
- Drop Tolerance: Configure appropriate drop thresholds based on risk tolerance
- Resource Allocation: Ensure adequate CPU and memory for Falco DaemonSet
Integration Architecture
- Multi-Layer Defense: Combine Falco with admission controllers and network policies
- Centralized Logging: Forward all events to SIEM for correlation
- Automated Response: Implement graduated response based on severity
- Alert Fatigue Prevention: Tune rules to minimize false positives
- Compliance Mapping: Tag rules with relevant compliance requirements
Conclusion
Falco provides enterprise-grade runtime security monitoring with deep visibility into container and kernel behavior. By implementing custom rules tailored to your environment, integrating with security orchestration platforms, and establishing automated response workflows, you can detect and respond to threats in real-time.
The key to successful Falco deployment is continuous tuning based on your specific threat model and operational patterns. Start with conservative rules, monitor false positive rates, and gradually tighten security policies as you gain confidence in the system.
For production environments, the combination of Falco’s detection capabilities with automated response actions provides a powerful defense-in-depth strategy that significantly reduces the window of exposure for security incidents.