Documenting the challenges and solutions encountered while attempting to upgrade ArgoCD on Kubernetes to address a security fix. Learn from this experience to manage your ArgoCD upgrades more effectively.

(Not) Upgrading ArgoCD on Kubernetes

There’s a security fix that needs to be applied; there’s an arm64 release candidate. Time to upgrade ArgoCD.

Upgrade Attempt

The upgrade process should have been straightforward:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.3.0-rc5/manifests/install.yaml

However, it mostly was, except for this issue:

pod/argocd-applicationset-controller-9488fc486-jjt98   0/1     CrashLoopBackOff   4 (61s ago)   4m6s

Inspecting the Logs

Inspect the logs of the crashing pod:

kubectl --namespace argocd logs argocd-applicationset-controller-9488fc486-jjt98

Output:

standard_init_linux.go:228: exec user process caused: exec format error

Filed a bug report: argo-cd#8394.

Rolling Back the Upgrade

Attempted to roll back the deployments:

kubectl --namespace argocd rollout undo deployment/argocd-redis
kubectl --namespace argocd rollout undo deployment/argocd-repo-server
kubectl --namespace argocd rollout undo deployment/argocd-server
kubectl --namespace argocd rollout undo deployment/argocd-notifications-controller
kubectl --namespace argocd rollout undo deployment/argocd-dex-server
kubectl --namespace argocd rollout undo deployment/argocd-applicationset-controller
kubectl --namespace argocd rollout undo statefulset/argocd-application-controller

Deleting Deployments and Stateful Sets

Since rolling back didn’t work, I deleted all the deployments and stateful sets:

kubectl --namespace argocd delete statefulset argocd-application-controller
kubectl --namespace argocd delete deployment argocd-notifications-controller
kubectl --namespace argocd delete deployment argocd-redis
kubectl --namespace argocd delete deployment argocd-repo-server
kubectl --namespace argocd delete deployment argocd-dex-server
kubectl --namespace argocd delete deployment argocd-applicationset-controller
kubectl --namespace argocd delete deployment argocd-server

Reapplied the previous install manifest to restore a working state.

Conclusion

Upgrading ArgoCD on Kubernetes can be challenging, especially when dealing with architecture-specific issues. By documenting the problems encountered and the steps taken to resolve them, you can better prepare for similar situations in the future.

Keep an eye on the progress of the argo-cd#8394 issue to stay informed about potential fixes and improvements.

By following these steps and learning from this experience, you can manage your ArgoCD upgrades more effectively, ensuring minimal downtime and disruption to your services.