Monitoring on Support Tools

OpenTelemetry Distributed Tracing: Production Setup and Best Practices for Enterprise Microservices

Sat, 17 Oct 2026 00:00:00 -0500

Implementing distributed tracing in complex microservices environments requires a robust, vendor-neutral solution that can scale with your infrastructure. OpenTelemetry has emerged as the industry standard for observability, providing a unified approach to traces, metrics, and logs. This comprehensive guide covers production-grade OpenTelemetry deployment strategies for enterprise Kubernetes environments.

Prometheus TSDB Compaction Failed: The 64 GiB Index Ceiling and What It Says About Your Cardinality

Tue, 21 Jul 2026 09:00:00 -0500

Prometheus compaction had been running fine for months. Then this showed up in the logs, and kept showing up every couple of hours:

text
level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact [/var/lib/prometheus/metrics2/01KJQ22FD0C2VFGQC2T8QP70P2 /var/lib/prometheus/metrics2/01KK8B5EK0CPXWDGPRD3H89ZQP /var/lib/prometheus/metrics2/01KKSP6NZ0EYNDGJFJTD3D6N6F]: populate block: add series: write series data: \"/var/lib/prometheus/metrics2/01KKSSF1T0T0T3AN0PJ00WVQGJ.tmp-for-creation/index\" exceeding max size of 64GiB

add padding: \"/var/lib/prometheus/metrics2/01KKSSF1T0T0T3AN0PJ00WVQGJ.tmp-for-creation/index\" exceeding max size of 64GiB"

Searching that string gets you almost nothing: a handful of GitHub issues, no runbook, no explanation. So here is the explanation. Short version: this is a hard limit in the TSDB index file format, not a resource problem. You cannot tune it, add RAM to it, or upgrade past it. It is real and it needs work, but it is not a data-loss event, and the most likely root cause is cardinality. Specifically, the number of distinct series that existed anywhere inside the merged block’s time range, which is a very different number from your current head cardinality.

Custom Metrics with Prometheus Exporters: Building Production-Ready Exporters for Enterprise Systems

Wed, 03 Jun 2026 00:00:00 -0500

While Prometheus provides hundreds of community exporters, enterprises often need custom exporters for proprietary systems, legacy applications, or specialized monitoring requirements. This guide covers building production-grade Prometheus exporters from scratch, including best practices, performance optimization, and deployment strategies.

Cortex Multi-Tenant Monitoring: Horizontally Scalable Prometheus as a Service

Thu, 28 May 2026 00:00:00 -0500

Cortex provides horizontally scalable, multi-tenant Prometheus as a service, enabling organizations to offer monitoring capabilities to multiple teams or customers with isolation, custom limits, and high availability. This guide covers deploying Cortex in production with microservices architecture, per-tenant configuration, and performance optimization.

Falco Mastery: The Complete Guide to Runtime Security in Kubernetes Environments

Tue, 14 Apr 2026 09:00:00 -0500

Falco Mastery: The Complete Guide to Runtime Security in Kubernetes Environments

In today’s threat landscape, traditional perimeter security is insufficient for protecting cloud-native environments. Modern attacks target runtime environments directly, exploiting container vulnerabilities, privilege escalation, and lateral movement within Kubernetes clusters. This comprehensive guide explores Falco—the Cloud Native Computing Foundation’s runtime security project—providing advanced strategies for implementing enterprise-grade threat detection and response.

Whether you’re a security engineer implementing Zero Trust architecture or a platform team securing production workloads, this guide offers the deep expertise needed to master runtime security in Kubernetes environments and advance your career in cloud-native security.

Advanced Network Observability and Telemetry: Enterprise Infrastructure Guide

Sat, 11 Apr 2026 00:00:00 -0500

Advanced network observability and telemetry provide deep insights into network behavior, performance, and security posture. This comprehensive guide explores sophisticated monitoring strategies, telemetry collection frameworks, and production-ready observability architectures that enable proactive network management and rapid issue resolution in enterprise environments.

Advanced Network Monitoring with Flow Analysis: Enterprise Observability Guide

Fri, 10 Apr 2026 00:00:00 -0500

Advanced network monitoring and flow analysis provide critical visibility into network behavior, performance bottlenecks, and security threats. This comprehensive guide explores sophisticated monitoring architectures, flow analysis techniques, and enterprise-grade observability solutions for production network environments.

Advanced Monitoring and Observability with Prometheus and Grafana: Enterprise Production Framework 2026

Thu, 09 Apr 2026 00:00:00 -0500

Advanced monitoring and observability form the foundation of reliable enterprise systems, requiring sophisticated metrics collection, intelligent alerting, and comprehensive visualization strategies that provide actionable insights into system performance and behavior. This comprehensive guide explores enterprise-grade Prometheus and Grafana implementations, advanced observability patterns, and production-ready monitoring frameworks.

Managing Nagios Downtime via Command Line Using cURL: A Complete Guide

Sun, 15 Mar 2026 09:00:00 -0600

Master the art of managing Nagios downtime through the command line using cURL, enabling efficient automation and integration capabilities.

Running Nagios Checks as Root with NRPE: A Secure Implementation Guide

Thu, 15 Jan 2026 09:00:00 -0600

Master the art of securely running Nagios checks with root privileges using NRPE while maintaining system security and monitoring effectiveness.

Building a Comprehensive Container Monitoring Infrastructure in 2025

Tue, 09 Dec 2025 09:00:00 -0500

Effective monitoring is critical in containerized environments where systems are distributed, dynamic, and ephemeral. This guide walks through building a complete monitoring infrastructure using modern tools and techniques to provide full visibility into your container ecosystem.

Enterprise System Monitoring & Observability Guide 2025: Advanced Production Infrastructure Analytics

Mon, 28 Jul 2025 10:00:00 -0500

Enterprise system monitoring and observability require sophisticated frameworks that provide comprehensive visibility into infrastructure performance, application behavior, and business metrics across distributed systems. This guide covers advanced monitoring architectures, enterprise observability platforms, automated performance analytics, and production-grade monitoring solutions for critical business infrastructure.

Kubernetes Observability Best Practices for 2025

Thu, 17 Jul 2025 09:00:00 -0500

As Kubernetes environments grow in complexity, implementing effective observability becomes increasingly critical. In 2025, organizations face new challenges in monitoring and troubleshooting distributed systems, requiring a modernized approach to observability.

Monitoring PostgreSQL Using PostgreSQL Exporter on Kubernetes

Sun, 15 Jun 2025 18:15:00 -0500

Effective database monitoring is crucial for maintaining the performance, availability, and reliability of modern applications. PostgreSQL, a widely used open-source relational database, can be efficiently monitored using tools like PostgreSQL Exporter. In this guide, we’ll walk you through setting up PostgreSQL monitoring using PostgreSQL Exporter on a Kubernetes cluster.

How to Monitor External Services with Prometheus from Kubernetes

Sat, 07 Jun 2025 00:00:00 -0500

Monitoring external systems—like bare-metal servers, APIs, or other cloud services—from within Kubernetes can help unify your observability pipeline. With Prometheus Operator, you don’t need to deploy a second Prometheus instance. Instead, you can expose external targets using Kubernetes-native manifests: Service, Endpoints, and ServiceMonitor.

In this guide, we’ll walk through:

Monitoring a VM with Node Exporter
Monitoring a remote NGINX server
Scraping a custom REST API
Tips on TLS and firewalling external exporters

Managing Nagios Downtime via Command Line Using cURL

Thu, 20 Feb 2025 09:00:00 -0600

Discover how to programmatically manage Nagios downtimes using cURL, enabling automated maintenance windows and reducing alert noise during planned system work.

Running Nagios Checks as Root with NRPE

Mon, 17 Feb 2025 01:55:00 -0600

Learn how to configure NRPE to run Nagios checks with root privileges while maintaining security.

Monitoring is a Pain (and We're All Doing it Wrong)

Thu, 19 Dec 2024 12:00:00 -0500

And we’re all doing it wrong (including me).

Monitoring is supposed to make life easier for developers and operators, but it often does the opposite. Despite our best intentions, observability tools frequently fall short, leaving us with brittle systems, ballooning costs, and frustration.

Kubernetes Monitoring with Prometheus and Grafana

Thu, 24 Oct 2024 10:15:00 -0500

Monitoring is essential for operating Kubernetes clusters effectively. Prometheus and Grafana are two of the most popular open-source tools used to collect, query, and visualize metrics in Kubernetes environments. This post will walk you through how to set up Prometheus and Grafana for cluster monitoring and provide best practices for getting the most out of your metrics.

Why Monitoring is Critical in Kubernetes

Kubernetes environments are highly dynamic, and monitoring ensures:

Kubernetes Logging with Grafana Loki & Promtail

Tue, 08 Oct 2024 19:26:00 -0500

Logging in Kubernetes is crucial for monitoring, troubleshooting, and optimizing your cluster’s performance. However, managing logs from a distributed system like Kubernetes can be complex. Grafana Loki, in combination with Promtail, provides a scalable and efficient solution for log aggregation, allowing you to monitor logs from across your cluster in a centralized and easy-to-query way.

In this post, we’ll explore how to set up Kubernetes logging using Grafana Loki and Promtail to achieve a streamlined log management system.

Use External Labels with Prometheus Alerts

Sun, 29 Sep 2024 19:26:00 -0500

When managing a multi-cluster or multi-region Prometheus setup, external labels play a key role in differentiating alerts from different clusters. By using external labels in Prometheus alerts, you can prevent duplication of alerts and provide context such as the cluster, region, or environment from which the alert originated. In this post, we’ll explore how to configure and use external labels in Prometheus alerts.

Install Kubecost to Help Optimize Kubernetes Applications

Sat, 28 Sep 2024 19:26:00 -0500

As Kubernetes environments grow in size and complexity, managing and optimizing resource usage becomes increasingly important. Kubecost provides detailed insights into Kubernetes costs, helping teams optimize application performance, resource utilization, and reduce infrastructure expenses. In this post, we’ll walk through how to install Kubecost and start optimizing your Kubernetes applications.

Install and Configure Alertmanager with Slack Integration on Kubernetes

Wed, 25 Sep 2024 19:26:00 -0500

Setting up alert notifications is crucial for monitoring the health and performance of your Kubernetes cluster. Alertmanager, when integrated with Prometheus, helps handle alert routing, grouping, and silencing. By configuring Alertmanager with Slack integration, you can receive real-time alerts in your Slack channels. In this guide, we’ll walk through how to install and configure Alertmanager with Slack integration on Kubernetes.

Install Kube State Metrics on Kubernetes

Tue, 24 Sep 2024 19:26:00 -0500

Kube State Metrics is a powerful tool that exposes resource state metrics from Kubernetes objects, making it easier to monitor and gain insights into your cluster’s health and resource usage. It integrates seamlessly with Prometheus and helps track resource states such as pods, deployments, nodes, and more. In this guide, we’ll walk you through installing Kube State Metrics on Kubernetes.

Monitor Linux Servers with Grafana and Prometheus (node_exporter)

Mon, 23 Sep 2024 19:26:00 -0500

Monitoring Linux servers is critical for maintaining system health, optimizing performance, and identifying potential issues before they escalate. With Prometheus and Grafana, you can collect and visualize system metrics using node_exporter, a Prometheus exporter designed to gather hardware and OS metrics from Linux systems. In this post, we’ll guide you through setting up Prometheus, Grafana, and node_exporter to monitor your Linux servers.

Monitor HAProxy with Grafana and Prometheus (haproxy_exporter)

Sun, 22 Sep 2024 19:26:00 -0500

HAProxy is a widely used open-source load balancer and reverse proxy for TCP and HTTP-based applications. Monitoring HAProxy performance and availability is crucial for ensuring your services remain healthy and responsive. In this post, we’ll walk through how to monitor HAProxy using Grafana, Prometheus, and haproxy_exporter to provide real-time insights into your load balancer’s performance.

Monitor Bind DNS Server with Grafana and Prometheus (bind_exporter)

Thu, 19 Sep 2024 19:26:00 -0500

Monitoring your Bind DNS server is essential for maintaining optimal performance, availability, and security. With Prometheus and Grafana, you can collect and visualize real-time metrics from your Bind DNS server using bind_exporter, a tool designed to expose Bind metrics in a format Prometheus can scrape. In this guide, we’ll walk through how to set up monitoring for your Bind DNS server with Prometheus, Grafana, and bind_exporter.

Monitor Etcd Cluster with Grafana and Prometheus in RKE2

Tue, 17 Sep 2024 19:26:00 -0500

Monitoring your Etcd cluster is critical to ensure the health of your Kubernetes infrastructure, especially in an RKE2 environment. In this post, we’ll explore how to monitor an Etcd cluster using Grafana and Prometheus, giving you real-time insights and alerting capabilities to prevent issues before they impact your cluster’s availability.

Rancher Monitoring v2 Workshop: Comprehensive Guide

Wed, 10 Jan 2024 09:00:00 -0500

Rancher Monitoring v2 Workshop

Welcome to the Rancher Monitoring v2 Workshop! This workshop is designed to provide a comprehensive guide to deploying, managing, and troubleshooting Rancher Monitoring v2 in Kubernetes environments.

Overview

Rancher Monitoring v2 offers advanced monitoring capabilities for Kubernetes clusters managed by Rancher. It is a powerful tool that integrates Prometheus, Grafana, and other monitoring components to provide deep insights into the health and performance of clusters.

Goals of the Workshop

Understanding the architecture of Rancher Monitoring v2
Learning how to deploy and configure Rancher Monitoring v2
Best practices for monitoring Kubernetes clusters
Troubleshooting common issues in Rancher Monitoring v2
Hands-on experience with Rancher Monitoring v2
Setting up alerts and notifications
Setting up ingress for Grafana and Prometheus
Setting up Grafana dashboards

service rancher-monitoring-operator not found

Thu, 29 Sep 2022 00:51:00 -0500

Error:

Error: Internal error occurred: failed calling webhook "prometheusrulemutate.monitoring.coreos.com": failed to call webhook: Post "https://rancher-monitoring-operator.cattle-monitoring-system.svc:443/admission-prometheusrules/mutate?timeout=10s": service "rancher-monitoring-operator" not found

This error can be caused by a number of different issues. But first we need to understand what this error means.

The TLDR is that the webhook is not available. The webhook is a service that is used to validate and mutate resources. In this case, the webhook is used to validate and mutate PrometheusRule resources.

How to install kube-prometheus-stack

Tue, 12 Jul 2022 00:21:00 -0500

kube-prometheus-stack is a collection of Kubernetes manifests that includes the following:

The Prometheus operator

Prometheus
Alertmanager
Prometheus node-exporter
The Prometheus Adapter
kube-state-metrics
Grafana
Preconfigured to collect metrics from every Kubernetes component
Provides a set of default dashboards and alerts