[Starlingx-discuss] starlingx: unable to delete images from local registry
Hi,
We are using STX4.0
We have been trying to delete the images from the registry and it doesn't appear to be deleting. The "registry-image-delete" returns success but couldn't figure what is going wrong. Are there corresponding logs where the delete progress can be checked? Also, is there a workaround to delete them?
controller-0:/home/sysadmin# source /etc/platform/openrc [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# system registry-image-list --nowrap +----------------------------------------------------------------+ | Image Name | +----------------------------------------------------------------+ | docker.io/nfvpe/multus | | docker.io/starlingx/k8s-cni-sriov | | docker.io/starlingx/k8s-plugins-sriov-network-device | | docker.io/starlingx/n3000-opae | | gcr.io/kubernetes-helm/tiller | | k8s.gcr.io/coredns | | k8s.gcr.io/defaultbackend | | k8s.gcr.io/etcd | | k8s.gcr.io/kube-apiserver | | k8s.gcr.io/kube-controller-manager | | k8s.gcr.io/kube-proxy | | k8s.gcr.io/kube-scheduler | | k8s.gcr.io/pause | | quay.io/airshipit/armada | | quay.io/calico/cni | | quay.io/calico/kube-controllers | | quay.io/calico/node | | quay.io/calico/pod2daemon-flexvol | | quay.io/jetstack/cert-manager-acmesolver | | quay.io/jetstack/cert-manager-cainjector | | quay.io/jetstack/cert-manager-controller | | quay.io/jetstack/cert-manager-webhook | | quay.io/k8scsi/snapshot-controller | | quay.io/kubernetes-ingress-controller/nginx-ingress-controller | | quay.io/stackanetes/kubernetes-entrypoint | | vdu/du-l1 | | vdu/du-testmac | | vdu/dul1 | +----------------------------------------------------------------+ [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# system registry-image-delete vdu/du-testmac:v1 Image vdu/du-testmac:v1 deleted, please run garbage collect to free disk space. [root@controller-0 sysadmin(keystone_admin)]# system registry-image-delete vdu/du-l1:v1 Image vdu/du-l1:v1 deleted, please run garbage collect to free disk space. [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# system registry-image-list --nowrap +----------------------------------------------------------------+ | Image Name | +----------------------------------------------------------------+ | docker.io/nfvpe/multus | | docker.io/starlingx/k8s-cni-sriov | | docker.io/starlingx/k8s-plugins-sriov-network-device | | docker.io/starlingx/n3000-opae | | gcr.io/kubernetes-helm/tiller | | k8s.gcr.io/coredns | | k8s.gcr.io/defaultbackend | | k8s.gcr.io/etcd | | k8s.gcr.io/kube-apiserver | | k8s.gcr.io/kube-controller-manager | | k8s.gcr.io/kube-proxy | | k8s.gcr.io/kube-scheduler | | k8s.gcr.io/pause | | quay.io/airshipit/armada | | quay.io/calico/cni | | quay.io/calico/kube-controllers | | quay.io/calico/node | | quay.io/calico/pod2daemon-flexvol | | quay.io/jetstack/cert-manager-acmesolver | | quay.io/jetstack/cert-manager-cainjector | | quay.io/jetstack/cert-manager-controller | | quay.io/jetstack/cert-manager-webhook | | quay.io/k8scsi/snapshot-controller | | quay.io/kubernetes-ingress-controller/nginx-ingress-controller | | quay.io/stackanetes/kubernetes-entrypoint | | vdu/du-l1 | | vdu/du-testmac | | vdu/dul1 | +----------------------------------------------------------------+ [root@controller-0 sysadmin(keystone_admin)]# system registry-image-delete vdu/dul1:v1 Image vdu/dul1:v1 deleted, please run garbage collect to free disk space. [root@controller-0 sysadmin(keystone_admin)]# system registry-garbage-collect Running docker registry garbage collect [root@controller-0 sysadmin(keystone_admin)]# system registry-image-list --nowrap +----------------------------------------------------------------+ | Image Name | +----------------------------------------------------------------+ | docker.io/nfvpe/multus | | docker.io/starlingx/k8s-cni-sriov | | docker.io/starlingx/k8s-plugins-sriov-network-device | | docker.io/starlingx/n3000-opae | | gcr.io/kubernetes-helm/tiller | | k8s.gcr.io/coredns | | k8s.gcr.io/defaultbackend | | k8s.gcr.io/etcd | | k8s.gcr.io/kube-apiserver | | k8s.gcr.io/kube-controller-manager | | k8s.gcr.io/kube-proxy | | k8s.gcr.io/kube-scheduler | | k8s.gcr.io/pause | | quay.io/airshipit/armada | | quay.io/calico/cni | | quay.io/calico/kube-controllers | | quay.io/calico/node | | quay.io/calico/pod2daemon-flexvol | | quay.io/jetstack/cert-manager-acmesolver | | quay.io/jetstack/cert-manager-cainjector | | quay.io/jetstack/cert-manager-controller | | quay.io/jetstack/cert-manager-webhook | | quay.io/k8scsi/snapshot-controller | | quay.io/kubernetes-ingress-controller/nginx-ingress-controller | | quay.io/stackanetes/kubernetes-entrypoint | | vdu/du-l1 | | vdu/du-testmac | | vdu/dul1 | +----------------------------------------------------------------+ [root@controller-0 sysadmin(keystone_admin)]# system registry-image-tags vdu/dul1
[root@controller-0 sysadmin(keystone_admin)]# system registry-image-tags vdu/du-testmac
[root@controller-0 sysadmin(keystone_admin)]# system registry-image-tags vdu/du-l1
[root@controller-0 sysadmin(keystone_admin)]# system registry-image-list +----------------------------------------------------------------+ | Image Name | +----------------------------------------------------------------+ | docker.io/nfvpe/multus | | docker.io/starlingx/k8s-cni-sriov | | docker.io/starlingx/k8s-plugins-sriov-network-device | | docker.io/starlingx/n3000-opae | | gcr.io/kubernetes-helm/tiller | | k8s.gcr.io/coredns | | k8s.gcr.io/defaultbackend | | k8s.gcr.io/etcd | | k8s.gcr.io/kube-apiserver | | k8s.gcr.io/kube-controller-manager | | k8s.gcr.io/kube-proxy | | k8s.gcr.io/kube-scheduler | | k8s.gcr.io/pause | | quay.io/airshipit/armada | | quay.io/calico/cni | | quay.io/calico/kube-controllers | | quay.io/calico/node | | quay.io/calico/pod2daemon-flexvol | | quay.io/jetstack/cert-manager-acmesolver | | quay.io/jetstack/cert-manager-cainjector | | quay.io/jetstack/cert-manager-controller | | quay.io/jetstack/cert-manager-webhook | | quay.io/k8scsi/snapshot-controller | | quay.io/kubernetes-ingress-controller/nginx-ingress-controller | | quay.io/stackanetes/kubernetes-entrypoint | | vdu/du-l1 | | vdu/du-testmac | | vdu/dul1 | +----------------------------------------------------------------+ [root@controller-0 sysadmin(keystone_admin)]#
+ Hi Folks - any thoughts about the below issue?
From: Venkata Ramana Veldanda Sent: Tuesday, February 23, 2021 9:23 PM To: starlingx-discuss@lists.starlingx.io Subject: starlingx: unable to delete images from local registry
Hi,
We are using STX4.0
We have been trying to delete the images from the registry and it doesn't appear to be deleting. The "registry-image-delete" returns success but couldn't figure what is going wrong. Are there corresponding logs where the delete progress can be checked? Also, is there a workaround to delete them?
controller-0:/home/sysadmin# source /etc/platform/openrc [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# system registry-image-list --nowrap +----------------------------------------------------------------+ | Image Name | +----------------------------------------------------------------+ | docker.io/nfvpe/multus | | docker.io/starlingx/k8s-cni-sriov | | docker.io/starlingx/k8s-plugins-sriov-network-device | | docker.io/starlingx/n3000-opae | | gcr.io/kubernetes-helm/tiller | | k8s.gcr.io/coredns | | k8s.gcr.io/defaultbackend | | k8s.gcr.io/etcd | | k8s.gcr.io/kube-apiserver | | k8s.gcr.io/kube-controller-manager | | k8s.gcr.io/kube-proxy | | k8s.gcr.io/kube-scheduler | | k8s.gcr.io/pause | | quay.io/airshipit/armada | | quay.io/calico/cni | | quay.io/calico/kube-controllers | | quay.io/calico/node | | quay.io/calico/pod2daemon-flexvol | | quay.io/jetstack/cert-manager-acmesolver | | quay.io/jetstack/cert-manager-cainjector | | quay.io/jetstack/cert-manager-controller | | quay.io/jetstack/cert-manager-webhook | | quay.io/k8scsi/snapshot-controller | | quay.io/kubernetes-ingress-controller/nginx-ingress-controller | | quay.io/stackanetes/kubernetes-entrypoint | | vdu/du-l1 | | vdu/du-testmac | | vdu/dul1 | +----------------------------------------------------------------+ [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# system registry-image-delete vdu/du-testmac:v1 Image vdu/du-testmac:v1 deleted, please run garbage collect to free disk space. [root@controller-0 sysadmin(keystone_admin)]# system registry-image-delete vdu/du-l1:v1 Image vdu/du-l1:v1 deleted, please run garbage collect to free disk space. [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# system registry-image-list --nowrap +----------------------------------------------------------------+ | Image Name | +----------------------------------------------------------------+ | docker.io/nfvpe/multus | | docker.io/starlingx/k8s-cni-sriov | | docker.io/starlingx/k8s-plugins-sriov-network-device | | docker.io/starlingx/n3000-opae | | gcr.io/kubernetes-helm/tiller | | k8s.gcr.io/coredns | | k8s.gcr.io/defaultbackend | | k8s.gcr.io/etcd | | k8s.gcr.io/kube-apiserver | | k8s.gcr.io/kube-controller-manager | | k8s.gcr.io/kube-proxy | | k8s.gcr.io/kube-scheduler | | k8s.gcr.io/pause | | quay.io/airshipit/armada | | quay.io/calico/cni | | quay.io/calico/kube-controllers | | quay.io/calico/node | | quay.io/calico/pod2daemon-flexvol | | quay.io/jetstack/cert-manager-acmesolver | | quay.io/jetstack/cert-manager-cainjector | | quay.io/jetstack/cert-manager-controller | | quay.io/jetstack/cert-manager-webhook | | quay.io/k8scsi/snapshot-controller | | quay.io/kubernetes-ingress-controller/nginx-ingress-controller | | quay.io/stackanetes/kubernetes-entrypoint | | vdu/du-l1 | | vdu/du-testmac | | vdu/dul1 | +----------------------------------------------------------------+ [root@controller-0 sysadmin(keystone_admin)]# system registry-image-delete vdu/dul1:v1 Image vdu/dul1:v1 deleted, please run garbage collect to free disk space. [root@controller-0 sysadmin(keystone_admin)]# system registry-garbage-collect Running docker registry garbage collect [root@controller-0 sysadmin(keystone_admin)]# system registry-image-list --nowrap +----------------------------------------------------------------+ | Image Name | +----------------------------------------------------------------+ | docker.io/nfvpe/multus | | docker.io/starlingx/k8s-cni-sriov | | docker.io/starlingx/k8s-plugins-sriov-network-device | | docker.io/starlingx/n3000-opae | | gcr.io/kubernetes-helm/tiller | | k8s.gcr.io/coredns | | k8s.gcr.io/defaultbackend | | k8s.gcr.io/etcd | | k8s.gcr.io/kube-apiserver | | k8s.gcr.io/kube-controller-manager | | k8s.gcr.io/kube-proxy | | k8s.gcr.io/kube-scheduler | | k8s.gcr.io/pause | | quay.io/airshipit/armada | | quay.io/calico/cni | | quay.io/calico/kube-controllers | | quay.io/calico/node | | quay.io/calico/pod2daemon-flexvol | | quay.io/jetstack/cert-manager-acmesolver | | quay.io/jetstack/cert-manager-cainjector | | quay.io/jetstack/cert-manager-controller | | quay.io/jetstack/cert-manager-webhook | | quay.io/k8scsi/snapshot-controller | | quay.io/kubernetes-ingress-controller/nginx-ingress-controller | | quay.io/stackanetes/kubernetes-entrypoint | | vdu/du-l1 | | vdu/du-testmac | | vdu/dul1 | +----------------------------------------------------------------+ [root@controller-0 sysadmin(keystone_admin)]# system registry-image-tags vdu/dul1
[root@controller-0 sysadmin(keystone_admin)]# system registry-image-tags vdu/du-testmac
[root@controller-0 sysadmin(keystone_admin)]# system registry-image-tags vdu/du-l1
[root@controller-0 sysadmin(keystone_admin)]# system registry-image-list +----------------------------------------------------------------+ | Image Name | +----------------------------------------------------------------+ | docker.io/nfvpe/multus | | docker.io/starlingx/k8s-cni-sriov | | docker.io/starlingx/k8s-plugins-sriov-network-device | | docker.io/starlingx/n3000-opae | | gcr.io/kubernetes-helm/tiller | | k8s.gcr.io/coredns | | k8s.gcr.io/defaultbackend | | k8s.gcr.io/etcd | | k8s.gcr.io/kube-apiserver | | k8s.gcr.io/kube-controller-manager | | k8s.gcr.io/kube-proxy | | k8s.gcr.io/kube-scheduler | | k8s.gcr.io/pause | | quay.io/airshipit/armada | | quay.io/calico/cni | | quay.io/calico/kube-controllers | | quay.io/calico/node | | quay.io/calico/pod2daemon-flexvol | | quay.io/jetstack/cert-manager-acmesolver | | quay.io/jetstack/cert-manager-cainjector | | quay.io/jetstack/cert-manager-controller | | quay.io/jetstack/cert-manager-webhook | | quay.io/k8scsi/snapshot-controller | | quay.io/kubernetes-ingress-controller/nginx-ingress-controller | | quay.io/stackanetes/kubernetes-entrypoint | | vdu/du-l1 | | vdu/du-testmac | | vdu/dul1 | +----------------------------------------------------------------+ [root@controller-0 sysadmin(keystone_admin)]#
( cc’d jerry sun ... he may know )
Although the image did not get removed from the list, you notice that the image no longer has any tags (versions).
This has come up before ... I think this is just odd behaviour of the docker registry that it maintains the knowledge of the image even if it no longer has any tags/versions.
Jerry ?
Greg.
From: Venkata Ramana Veldanda Venkata.Veldanda@radisys.com Date: Tuesday, March 2, 2021 at 12:58 AM To: "starlingx-discuss@lists.starlingx.io" starlingx-discuss@lists.starlingx.io Cc: Srinivas Sadagopan Srinivas.Sadagopan@radisys.com Subject: Re: [Starlingx-discuss] starlingx: unable to delete images from local registry
[Please note: This e-mail is from an EXTERNAL e-mail address] + Hi Folks – any thoughts about the below issue?
From: Venkata Ramana Veldanda Sent: Tuesday, February 23, 2021 9:23 PM To: starlingx-discuss@lists.starlingx.io Subject: starlingx: unable to delete images from local registry
Hi,
We are using STX4.0
We have been trying to delete the images from the registry and it doesn’t appear to be deleting. The “registry-image-delete” returns success but couldn’t figure what is going wrong. Are there corresponding logs where the delete progress can be checked? Also, is there a workaround to delete them?
controller-0:/home/sysadmin# source /etc/platform/openrc [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# system registry-image-list --nowrap +----------------------------------------------------------------+ | Image Name | +----------------------------------------------------------------+ | docker.io/nfvpe/multus | | docker.io/starlingx/k8s-cni-sriov | | docker.io/starlingx/k8s-plugins-sriov-network-device | | docker.io/starlingx/n3000-opae | | gcr.io/kubernetes-helm/tiller | | k8s.gcr.io/coredns | | k8s.gcr.io/defaultbackend | | k8s.gcr.io/etcd | | k8s.gcr.io/kube-apiserver | | k8s.gcr.io/kube-controller-manager | | k8s.gcr.io/kube-proxy | | k8s.gcr.io/kube-scheduler | | k8s.gcr.io/pause | | quay.io/airshipit/armada | | quay.io/calico/cni | | quay.io/calico/kube-controllers | | quay.io/calico/node | | quay.io/calico/pod2daemon-flexvol | | quay.io/jetstack/cert-manager-acmesolver | | quay.io/jetstack/cert-manager-cainjector | | quay.io/jetstack/cert-manager-controller | | quay.io/jetstack/cert-manager-webhook | | quay.io/k8scsi/snapshot-controller | | quay.io/kubernetes-ingress-controller/nginx-ingress-controller | | quay.io/stackanetes/kubernetes-entrypoint | | vdu/du-l1 | | vdu/du-testmac | | vdu/dul1 | +----------------------------------------------------------------+ [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# system registry-image-delete vdu/du-testmac:v1 Image vdu/du-testmac:v1 deleted, please run garbage collect to free disk space. [root@controller-0 sysadmin(keystone_admin)]# system registry-image-delete vdu/du-l1:v1 Image vdu/du-l1:v1 deleted, please run garbage collect to free disk space. [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# [root@controller-0 sysadmin(keystone_admin)]# system registry-image-list --nowrap +----------------------------------------------------------------+ | Image Name | +----------------------------------------------------------------+ | docker.io/nfvpe/multus | | docker.io/starlingx/k8s-cni-sriov | | docker.io/starlingx/k8s-plugins-sriov-network-device | | docker.io/starlingx/n3000-opae | | gcr.io/kubernetes-helm/tiller | | k8s.gcr.io/coredns | | k8s.gcr.io/defaultbackend | | k8s.gcr.io/etcd | | k8s.gcr.io/kube-apiserver | | k8s.gcr.io/kube-controller-manager | | k8s.gcr.io/kube-proxy | | k8s.gcr.io/kube-scheduler | | k8s.gcr.io/pause | | quay.io/airshipit/armada | | quay.io/calico/cni | | quay.io/calico/kube-controllers | | quay.io/calico/node | | quay.io/calico/pod2daemon-flexvol | | quay.io/jetstack/cert-manager-acmesolver | | quay.io/jetstack/cert-manager-cainjector | | quay.io/jetstack/cert-manager-controller | | quay.io/jetstack/cert-manager-webhook | | quay.io/k8scsi/snapshot-controller | | quay.io/kubernetes-ingress-controller/nginx-ingress-controller | | quay.io/stackanetes/kubernetes-entrypoint | | vdu/du-l1 | | vdu/du-testmac | | vdu/dul1 | +----------------------------------------------------------------+ [root@controller-0 sysadmin(keystone_admin)]# system registry-image-delete vdu/dul1:v1 Image vdu/dul1:v1 deleted, please run garbage collect to free disk space. [root@controller-0 sysadmin(keystone_admin)]# system registry-garbage-collect Running docker registry garbage collect [root@controller-0 sysadmin(keystone_admin)]# system registry-image-list --nowrap +----------------------------------------------------------------+ | Image Name | +----------------------------------------------------------------+ | docker.io/nfvpe/multus | | docker.io/starlingx/k8s-cni-sriov | | docker.io/starlingx/k8s-plugins-sriov-network-device | | docker.io/starlingx/n3000-opae | | gcr.io/kubernetes-helm/tiller | | k8s.gcr.io/coredns | | k8s.gcr.io/defaultbackend | | k8s.gcr.io/etcd | | k8s.gcr.io/kube-apiserver | | k8s.gcr.io/kube-controller-manager | | k8s.gcr.io/kube-proxy | | k8s.gcr.io/kube-scheduler | | k8s.gcr.io/pause | | quay.io/airshipit/armada | | quay.io/calico/cni | | quay.io/calico/kube-controllers | | quay.io/calico/node | | quay.io/calico/pod2daemon-flexvol | | quay.io/jetstack/cert-manager-acmesolver | | quay.io/jetstack/cert-manager-cainjector | | quay.io/jetstack/cert-manager-controller | | quay.io/jetstack/cert-manager-webhook | | quay.io/k8scsi/snapshot-controller | | quay.io/kubernetes-ingress-controller/nginx-ingress-controller | | quay.io/stackanetes/kubernetes-entrypoint | | vdu/du-l1 | | vdu/du-testmac | | vdu/dul1 | +----------------------------------------------------------------+ [root@controller-0 sysadmin(keystone_admin)]# system registry-image-tags vdu/dul1
[root@controller-0 sysadmin(keystone_admin)]# system registry-image-tags vdu/du-testmac
[root@controller-0 sysadmin(keystone_admin)]# system registry-image-tags vdu/du-l1
[root@controller-0 sysadmin(keystone_admin)]# system registry-image-list +----------------------------------------------------------------+ | Image Name | +----------------------------------------------------------------+ | docker.io/nfvpe/multus | | docker.io/starlingx/k8s-cni-sriov | | docker.io/starlingx/k8s-plugins-sriov-network-device | | docker.io/starlingx/n3000-opae | | gcr.io/kubernetes-helm/tiller | | k8s.gcr.io/coredns | | k8s.gcr.io/defaultbackend | | k8s.gcr.io/etcd | | k8s.gcr.io/kube-apiserver | | k8s.gcr.io/kube-controller-manager | | k8s.gcr.io/kube-proxy | | k8s.gcr.io/kube-scheduler | | k8s.gcr.io/pause | | quay.io/airshipit/armada | | quay.io/calico/cni | | quay.io/calico/kube-controllers | | quay.io/calico/node | | quay.io/calico/pod2daemon-flexvol | | quay.io/jetstack/cert-manager-acmesolver | | quay.io/jetstack/cert-manager-cainjector | | quay.io/jetstack/cert-manager-controller | | quay.io/jetstack/cert-manager-webhook | | quay.io/k8scsi/snapshot-controller | | quay.io/kubernetes-ingress-controller/nginx-ingress-controller | | quay.io/stackanetes/kubernetes-entrypoint | | vdu/du-l1 | | vdu/du-testmac | | vdu/dul1 | +----------------------------------------------------------------+ [root@controller-0 sysadmin(keystone_admin)]#
participants (2)
-
Venkata Ramana Veldanda
-
Waines, Greg