Deletion issue

After a few months of hard work, one of your colleague had successfully mastered the tic-tac-toe game, and decided it was time to remove this game from the OpenShift cluster. He deleted the whole namespace, but something went wrong. The namespace is still around, in a 'Terminating' state. He asked for your wise expertise.

What's going on?

You head straight to the exercise06-userXX namespace (from the developer perspective) and look at the project page. It says "Terminating", but nothing seems to progress.

Important: You will find you need to cleanup some resources. There is one resource in particular you should cleanup last, or you will lock yourself out of your namespace. I left some hints, it will become obvious which one I mean when you investigate.

See if you can investigate further, find what is blocking the termination of the namespace, and fix it. Good luck!

Hints

Hint 1

How to quickly check the common resource type inventory?

From the Developer perspective, you can get an overview of the common resource type in the project page (scroll down a bit)

How come there are still some resources hanging around?

Note: Not all resource type are listed, only the most common.

Hint 2

How to check the namespace deletion information?

The Developer perspective doesn't expose the YAML details of the namespace. You know that the YAML resource often expose useful information, and you want to check it. If you switch to the Administrator perspective, you will be able to check it's content, and as is sometime the case, find valuable information toward the end:

The three messages here are very useful, and should give you an idea on what is going on, and what to do next.

Hint 3

Why is the PVC stuck in terminating state?

From Hint 1 and 2, you probably saw there was a PVC that was not deleted. Unless you already got rid of the Pod to which this PVC was bound to, you will see that this PVC is stuck in terminating state. Why is that?

What is this kuberenetes.io/pvc-protection finalizer? You should check this paragraph of the Kubernetes documentation: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#storage-object-in-use-protection

Do you think you need to do something for this PVC at this stage?

Hint 4

Why is this pod stuck in terminating state?

From the first or second hint, you probably saw that one pod was not fully deleted. You won't see the pod in terminating state from the topology view. If you look at the Admin view however, you will get more info:

You will want to inspect the YAML section of this pod, and look carefully at the finalizers. If you need to know more about finalizers, head over this page: https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/

This blog post section is also quite useful (ignore the "Owner References" section): https://kubernetes.io/blog/2021/05/14/using-finalizers-to-control-deletion/#understanding-finalizers

Do you think you need to do something for this pod at this stage?

Hint 5

Why is this secret stuck in terminating state?

From the first or second hint, you probably saw that one secret was not fully deleted. Let's have a look at it's YAML content:

Do you think you need to do something for this pod at this stage?

Hint 5

Solution

Make sure to refresh your memory on finalizers, a few documentation link were provided in the previous hint sections. The PersistentVolumeClaim has a finalizer (pvc-protection). It's role is to prevent the deletion of the PVC while in used by pod. Since the pod is also in terminating state, and not fully deleted, the pvc finalizer remain, and block the deletion of the pvc. You don't really need to remove this legitimate finalizer, it will get automatically removed when the pod is deleted.

You can see that the pod has a finalizer that shouldn't be there. Just remove it. Note that as soon as you remove it:

1. the pod is fully deleted
2. the pvc is also fully deleted (check)

You can proceed the same way with the secret my-secret:

Finally , at the very last step, you should remove the rolebinding. Once removed, your namespace will finally be terminated.

Why do you think the rbac should be done at the last step?

If you remove it too early, you will lose access to your namespace, and only a cluster admin will be able to access the namespace and cleanup up for you.

Congratulation, the namespace is gone!

Did you know?

Common finalizers

Common finalizers include:

• kubernetes.io/pvc-protection which you saw today
• kubernetes.io/pv-protection which prevents a pv to be removed while it is bound to a pvc
• kubernetes for Namespaces only, in the .spec section your namespace has this finalizer, but you can't remove it manually.
• resources-finalizer.argocd.argoproj.io that controls cascading deletion of argocd applications (see https://argo-cd.readthedocs.io/en/stable/user-guide/app_deletion/#about-the-deletion-finalizer)

and there is more.

Owner references and deletion

An other mechanism that impact the way resource are deleted is owner references. You will find some good explanation here https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/ A typical example of ownership can be seen with Deployment -> ReplicaSet -> Pod. For example:

In the example above, what happens if someone delete the deployment, but the pod remains stuck in "terminating" state? The "blockOwnerDeletion" will prevent the replicaSet to be terminated, which in turn will prevent the Deployment to be terminated.

Take away

Finalizers (and owner references) play an important role in controlling kubernetes resource deletions. While in this exercise, we introduced a few "fake" finalizers, you will from time to time encounter finalizers related issue, especially when working with Persistent Volume and Persistent Volume Claims. It is therefore important to understand how they work.