By default, Kubernetes will pull from Docker Hub unless the deployment requests a fully qualified path. For example, if you deploy a pod with the image rancherlabs/swiss-army-knife, Kubernetes will default index.docker.io/rancherlabs/swiss-army-knife. This is works for environments that can pull images from the internet. But in environments that don’t have internet access or if your security team requires all images to be scanned before being deployed. To solve this issue, you’ll need to instruct your application teams to update to use a private registry IE private.example.com/rancherlabs/swiss-army-knife., But there is always a chance that an application team forgets to change their image path in their code. Now you have pods stuck in imagepullbackoff. To prevent this kind of issue, we will set up OPA Gatekeeper to block all deployment requests that are missing our private registry IE private.example.com.
When creating a new PV device using a previously partitioned disk, you may see the following error message.
GitHub[email protected]:~/ $ sudo pvcreate /dev/sdb
Device /dev/sda excluded by a filter.
[email protected]:~/
It’s often necessary to migrate from a self-signed or LetsEncrypt certificate to an externally created certificate like DigiCert or Comodo in Rancher v2.x.
systemd-resolved can cause issues with Kubernetes (not to mention the time spent troubleshooting various issues).
In Rancher v1.6, sometimes a service can be stuck in a removing state.
All containers of this service were already deleted in the user interface. I verified this on the Docker hosts using “docker ps -a,” and yes, all container instances were correctly removed. But the service in Rancher was still stuck in removing.
Furthermore, in Admin -> Processes the service.remove processes (which seem to because of being stuck in that service removing in progress) never disappeared and were re-started every 2 minutes:
Although I’m not sure what caused this, the reason might be several actions happening on that particular service almost at the same time:
This article will walk Rancher administrators through hardening the cluster communication between etcd nodes. We’ll go over configuring etcd to use specific ciphers which enable stronger encryption for securing intra-cluster etcd traffic.
The cipher suites defined in the example could trade off speed for stronger encryption. Consider the level of ciphers in use and how they could impact the performance of an etcd cluster. Testing should be done to factor the spec of your hosts (cpu, memory, disk, network, etc…) and the typical types of interacting with kubernetes as well as the amount of resources under management within the k8s cluster.
The Rancher cluster and project monitoring tools, allow you to monitor cluster components and nodes, as well as workloads and custom metrics from any HTTP or TCP/UDP metrics endpoint that these workloads expose.
This article will detail how to manually define additional scrape configs for either the cluster or project monitoring Prometheus instance, where you want to scrape other metrics.
Whether to define the additional scrape config at the cluster or project level would depend on the desired scope for the metrics and possible alerts. If you wish to scope the metrics scraped and likely alerts configured for these metrics, you could configure the additional scrape config at the project monitoring level to a project. If you wish to scope the metrics at the cluster level, only those with cluster-admin access could see the metrics or configure alerts. You could configure the additional scrape config at the cluster monitoring level.
When troubleshooting an issue with an RKE CLI or Rancher provisioned Kubernetes cluster, it may help to increase the verbosity of logging on one or more of the Kubernetes components above the default level. This article details the process of increasing logging on both those components that use the Kubernetes hyperkube image (kubelet, kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy) as well as the etcd component.
At times it’s necessary to block specific user agents from connecting to workloads within your cluster. Whether it’s bad actors or for compliance reasons, we’ll go through how to get it done with Rancher/RKE created clusters.
This article details how to enable Envoy’s access logging, for Rancher deployed Istio, in Rancher v2.3+
During a Rancher outage or other disaster event, you may lose access to a downstream cluster via Rancher and be unable to manage your applications. This process allows to bypass Rancher and connects directly to the downstream cluster.
This article details how to change the server URL for the Rancher v2.x cluster.
While migrating some VMs to a new storage array, I ran into an issue while moving the last VM, which, of course, was my vCenter appliance. When I right-clicked the VM in vCenter, then went to migrate. The migration option was greyed out, which blocked the migration.
TL;DR Follow VMware’s KB 1029926
It is a common practice to access servers remotely via SSH. Typically, you may have, which is commonly referred to as a “Jumpbox.” This server is accessible from the internet or other lesser trusted networks (sometimes this Jumpbox would be in a DMZ or have special firewall rules).
TL;DR
To SSH to a server through a jumpbox, you can use ssh -J [email protected] [email protected].
In Ubuntu, you may run into an issue when updating /etc/resolv.conf even tho you have root permissions.
Error:
rm: cannot remove '/etc/resolv.conf': Operation not permitted
After upgrading to Ubuntu 20.10, you may sometimes encounter an error when attempting to start PowerDNS:
Backend error: GSQLBackend unable to list keys: Could not prepare statement: select cryptokeys.id, flags, active, published, content from domains, cryptokeys where cryptokeys.domain_id=domains.id and name=?: Unknown column ‘published’ in ‘field list’
This article details how to enable debug level logging on the Alertmanager instance in a Rancher v2.x managed Kubernetes cluster, which may assist when troubleshooting cluster or project alerting.
This article details how to enable TLS 1.1 on the ingress-nginx controller in Rancher Kubernetes Engine (RKE) CLI or Rancher v2.x provisioned Kubernetes clusters.
This article details how to enable TLS 1.1 on the ingress-nginx controller in Rancher Kubernetes Engine (RKE) CLI or Rancher v2.x provisioned Kubernetes clusters.
This document covers setting up Rancher using an AWS SSL certificate and an ALB (Application Load Balancer).
In Ubuntu, you may sometimes encounter an error when attempting to run an apt command:
E: Could not get lock /var/lib/dpkg/lock-frontend - open (11: Resource temporarily unavailable)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
The docker0 bridge network has a default IP range of 172.17.0.0/16 (with an additional docker-sys bridge for system-docker using 172.18.0.0/16 by default on RancherOS). These ranges will be routed to these interfaces, per the below example of the route output. If the range(s) overlap with the internal IP space usage in your own network, the host will not be able to route packets to other hosts in your network that lie within these ranges. As a result you may wish to change the bridge range(s) to enable successful routing to hosts within these.
As part of my DevOps process, I needed to create some DNS records on my Windows DNS server. In order to script this out, I needed to install PowerShell on my Jenkins server which is running Ubuntu 18.04.
Recently while setting up Vault inside Rancher. I ran into an issue with the NGINX ingress terminating TLS and forwarding traffic unencrypted to Vault.
Recently I set about installing RancherOS. This was just to have a look and see what use cases it might help with. I chose to install RancherOS to a VM. In my case, VMware workstation. A roadblock I hit was providing an SSH key to the cloud-config.yml file.
The roadblock specifically, how can I send a file to a system I don’t have a password or SSH key for?
This article covers, generating an SSH key, SSH access to live CD and installation to hard disk.
Sometimes when you experiment with some apps and VMs (like hosting gitlab on a local server or running a Rancher lab cluster) you might want to setup SSL for the app to work, to mimic the live setup and to make the browser happy. In order to do that, you need a SSL certificate.
You can buy one for your domain from a trusted CA, but if you’re working on a local network, that might not be possible. The other solution is… becoming CA yourself and issuing and signing the certificate yourself!
It’s pretty easy, you need a linux box with openssl installed, then follow these instructions:
