Unlock root account on ESXi

If the root account gets locked out, you will not be able to access ESXi using SSH or vSphere Web client; please follow the below procedure to unlock the account.

Please note you will get an incorrect username/password error even though you are trying to log in with the correct username/password.

By default, the ESXi 6.x password requirements for lockout behavior are:

  • A maximum of ten failed attempts is allowed before the account is locked
  • Password lockout is active on SSH and the vSphere Web Service SDK
  • Password lockout is not active on the Direct Console Interface (DCUI) and the ESXi Shell

Raspberry Pi Boot Issue - Root account locked!

After a power outage, my Raspberry Pi 400 Keyboard Computer wouldn’t boot. The Pi would get stuck at boot with the following error message.

Cannot open access to console, the root account is locked.

vSphere 7 – Certificates with VMCA as Subordinate

For enterprises that need fully trusted SSL certificates for the vSphere 7.0 environment, you have two basic options:

Full Custom Mode: Manually replace all certificates for vCenter and the ESXi hosts with your trusted certificates. Subordinate CA Mode: Use the built-in VMCA service as an official subordinate CA of your existing PKI infrastructure. After the initial configuration, automates the issuing of SSL certs for your vSphere environment. This is the method covered in this blog post. VMware offers two other certificate options: Fully Managed and hybrid mode, for a total of four certificate options. You can find out more about all of them in this VMware blog post.

In a high-security environment, it is very likely the security team will NOT let you configure the vCenter VMCA as a subordinate CA. So, you will be left with the full custom mode if you want 100% of the certificates trusted. However, if you are in a situation where you can configure the VMCA as a subordinate CA, this post is for you!

Note: Before you do this replacement in production, I strongly urge you to set up a test vCenter instance and run this entire procedure. Botched certificate replacements can lead to bad days. And another tip for a lab test is to snapshot the vCenter VM before you try the process. That way, you can easily roll back should things go south.

How to grow a zvol in ZFS

In ZFS, there are two types of filesystems (datasets and zvol). The ZFS dataset can be grown by setting the quota and reservation properties. But zvols have to extend by setting the volsize property to a new size.

How to use OPA Gatekeeper to only allow images from a private registry.

By default, Kubernetes will pull from Docker Hub unless the deployment requests a fully qualified path. For example, if you deploy a pod with the image rancherlabs/swiss-army-knife, Kubernetes will default index.docker.io/rancherlabs/swiss-army-knife. This is works for environments that can pull images from the internet. But in environments that don’t have internet access or if your security team requires all images to be scanned before being deployed. To solve this issue, you’ll need to instruct your application teams to update to use a private registry IE private.example.com/rancherlabs/swiss-army-knife., But there is always a chance that an application team forgets to change their image path in their code. Now you have pods stuck in imagepullbackoff. To prevent this kind of issue, we will set up OPA Gatekeeper to block all deployment requests that are missing our private registry IE private.example.com.

How to use OPA Gatekeeper to require a label on all namespaces.

By default, Kubernetes will create namespaces without any labels. This can tracking the owner of the namespace difcult. Also, by having all namespaces labeled. You can use labels for running show-back and charge-back reports based on owner labels.

How to fix LVM Device /dev/sdb excluded by a filter.

When creating a new PV device using a previously partitioned disk, you may see the following error message.

[email protected]:~/ $ sudo pvcreate /dev/sdb
  Device /dev/sda excluded by a filter.
[email protected]:~/

How to change the Rancher v2.x Server certificate to an externally created certificate.

It’s often necessary to migrate from a self-signed or LetsEncrypt certificate to an externally created certificate like DigiCert or Comodo in Rancher v2.x.

How to disable systemd-resolved in Ubuntu

systemd-resolved can cause issues with Kubernetes (not to mention the time spent troubleshooting various issues).

How to resolve a service stuck in Removing in Rancher v1.6

In Rancher v1.6, sometimes a service can be stuck in a removing state.

All containers of this service were already deleted in the user interface. I verified this on the Docker hosts using “docker ps -a,” and yes, all container instances were correctly removed. But the service in Rancher was still stuck in removing.

Furthermore, in Admin -> Processes the service.remove processes (which seem to because of being stuck in that service removing in progress) never disappeared and were re-started every 2 minutes:

Although I’m not sure what caused this, the reason might be several actions happening on that particular service almost at the same time:

How to change etcd cipher suite in Rancher / RKE

This article will walk Rancher administrators through hardening the cluster communication between etcd nodes. We’ll go over configuring etcd to use specific ciphers which enable stronger encryption for securing intra-cluster etcd traffic.

The cipher suites defined in the example could trade off speed for stronger encryption. Consider the level of ciphers in use and how they could impact the performance of an etcd cluster. Testing should be done to factor the spec of your hosts (cpu, memory, disk, network, etc…) and the typical types of interacting with kubernetes as well as the amount of resources under management within the k8s cluster.

How to add additional scrape configs to a Rancher cluster or project monitoring Prometheus

The Rancher cluster and project monitoring tools, allow you to monitor cluster components and nodes, as well as workloads and custom metrics from any HTTP or TCP/UDP metrics endpoint that these workloads expose.

This article will detail how to manually define additional scrape configs for either the cluster or project monitoring Prometheus instance, where you want to scrape other metrics.

Whether to define the additional scrape config at the cluster or project level would depend on the desired scope for the metrics and possible alerts. If you wish to scope the metrics scraped and likely alerts configured for these metrics, you could configure the additional scrape config at the project monitoring level to a project. If you wish to scope the metrics at the cluster level, only those with cluster-admin access could see the metrics or configure alerts. You could configure the additional scrape config at the cluster monitoring level.

How to increase the log level of Kubernetes components in an RKE CLI or Rancher provisioned Kubernetes cluster

When troubleshooting an issue with an RKE CLI or Rancher provisioned Kubernetes cluster, it may help to increase the verbosity of logging on one or more of the Kubernetes components above the default level. This article details the process of increasing logging on both those components that use the Kubernetes hyperkube image (kubelet, kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy) as well as the etcd component.

How to block specific user agents from connecting through the nginx ingress controller

At times it’s necessary to block specific user agents from connecting to workloads within your cluster. Whether it’s bad actors or for compliance reasons, we’ll go through how to get it done with Rancher/RKE created clusters.

How to retrieve a kubeconfig from RKE v0.2.x+ or Rancher v2.2.x+

During a Rancher outage or other disaster event, you may lose access to a downstream cluster via Rancher and be unable to manage your applications. This process allows to bypass Rancher and connects directly to the downstream cluster.

How to change the Rancher 2.x Server URL

This article details how to change the server URL for the Rancher v2.x cluster.

How to Fix 'Migrate greyed out in vCenter for a single VM'

While migrating some VMs to a new storage array, I ran into an issue while moving the last VM, which, of course, was my vCenter appliance. When I right-clicked the VM in vCenter, then went to migrate. The migration option was greyed out, which blocked the migration.

TL;DR Follow VMware’s KB 1029926

How to access a protected server using a Jumpbox

It is a common practice to access servers remotely via SSH. Typically, you may have, which is commonly referred to as a “Jumpbox.” This server is accessible from the internet or other lesser trusted networks (sometimes this Jumpbox would be in a DMZ or have special firewall rules).

TL;DR To SSH to a server through a jumpbox, you can use ssh -J [email protected] [email protected].

How to Fix 'rm: cannot remove '/etc/resolv.conf': Operation not permitted'

In Ubuntu, you may run into an issue when updating /etc/resolv.conf even tho you have root permissions.

Error:

rm: cannot remove '/etc/resolv.conf': Operation not permitted

How to Fix 'Backend error: GSQLBackend unable to list keys' in PowerDNS

After upgrading to Ubuntu 20.10, you may sometimes encounter an error when attempting to start PowerDNS:

Backend error: GSQLBackend unable to list keys: Could not prepare statement: select cryptokeys.id, flags, active, published, content from domains, cryptokeys where cryptokeys.domain_id=domains.id and name=?: Unknown column ‘published’ in ‘field list’

How to enable debug level logging for the Rancher Cluster/Project Alerting Alertmanager instance, in a Rancher v2.x managed cluster?

This article details how to enable debug level logging on the Alertmanager instance in a Rancher v2.x managed Kubernetes cluster, which may assist when troubleshooting cluster or project alerting.

How to enable legacy TLS versions in Rancher's ingress-nginx

This article details how to enable TLS 1.1 on the ingress-nginx controller in Rancher Kubernetes Engine (RKE) CLI or Rancher v2.x provisioned Kubernetes clusters.

How to enable legacy TLS versions in Rancher's ingress-nginx

This article details how to enable TLS 1.1 on the ingress-nginx controller in Rancher Kubernetes Engine (RKE) CLI or Rancher v2.x provisioned Kubernetes clusters.

How to use External TLS Termination with AWS

This document covers setting up Rancher using an AWS SSL certificate and an ALB (Application Load Balancer).

How to Fix 'E: Could not get lock /var/lib/dpkg/lock-frontend - open (11: Resource temporarily unavailable)'

In Ubuntu, you may sometimes encounter an error when attempting to run an apt command:

E: Could not get lock /var/lib/dpkg/lock-frontend - open (11: Resource temporarily unavailable) E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?

Updating the docker bridge for Rancher managed clusters

The docker0 bridge network has a default IP range of 172.17.0.0/16 (with an additional docker-sys bridge for system-docker using 172.18.0.0/16 by default on RancherOS). These ranges will be routed to these interfaces, per the below example of the route output. If the range(s) overlap with the internal IP space usage in your own network, the host will not be able to route packets to other hosts in your network that lie within these ranges. As a result you may wish to change the bridge range(s) to enable successful routing to hosts within these.

Install PowerShell on Ubuntu

As part of my DevOps process, I needed to create some DNS records on my Windows DNS server. In order to script this out, I needed to install PowerShell on my Jenkins server which is running Ubuntu 18.04.

Rancher SSL Passthrough for NGINX ingress

Recently while setting up Vault inside Rancher. I ran into an issue with the NGINX ingress terminating TLS and forwarding traffic unencrypted to Vault.

Persisting RancherOS state

Recently I set about installing RancherOS. This was just to have a look and see what use cases it might help with. I chose to install RancherOS to a VM. In my case, VMware workstation. A roadblock I hit was providing an SSH key to the cloud-config.yml file.

The roadblock specifically, how can I send a file to a system I don’t have a password or SSH key for?

This article covers, generating an SSH key, SSH access to live CD and installation to hard disk.

How to become a local CA, and sign your own SSL certificates

Sometimes when you experiment with some apps and VMs (like hosting gitlab on a local server or running a Rancher lab cluster) you might want to setup SSL for the app to work, to mimic the live setup and to make the browser happy. In order to do that, you need a SSL certificate.

You can buy one for your domain from a trusted CA, but if you’re working on a local network, that might not be possible. The other solution is… becoming CA yourself and issuing and signing the certificate yourself!

It’s pretty easy, you need a linux box with openssl installed, then follow these instructions: