Hi Bin, Forget my previous question, I think I understand the root cause: the oam network was not provisioned appropriately. I will figure out how to fix that. Thanks Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 -----Original Message----- From: Yang, Bin Sent: Thursday, July 18, 2019 9:13 AM To: 'Yang, Bin' Cc: starlingx-discuss@lists.starlingx.io Subject: RE: [Starlingx-discuss] controller-1 failed after unlock Hi Bin, Thanks a lot for the help. Indeed there is error there, it seems a timeout issue to pull docker images (again, I have met such issue during ansible-playbook on controller-0, the workaround is to increase the timeout in ansible playbook), any suggestion to workaround this issue? Thanks 2019-07-16T11:07:02.080 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.5: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m 2019-07-16T11:07:02.085 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m 2019-07-16T11:07:02.091 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.5: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m 2019-07-16T11:07:02.096 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m 2019-07-16T11:07:02.101 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.5: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m 2019-07-16T11:07:02.105 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m 2019-07-16T11:07:02.109 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.5: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m 2019-07-16T11:07:02.113 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m 2019-07-16T11:07:02.117 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m 2019-07-16T11:07:02.121 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m 2019-07-16T11:07:02.125 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m 2019-07-16T11:07:02.130 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m 2019-07-16T11:07:02.134 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`ESC[0m 2019-07-16T11:07:02.138 ESC[1;31mError: 2019-07-16 11:07:02 +0000 kubeadm init --config=/etc/kubernetes/kubeadm.yaml returned 1 instead of one of [0] BTW, It is very nice to talk to you, we are both 'Yang Bin' :) Best Regards, Bin Yang, Solution Engineering Team, Wind River ONAP Multi-VIM/Cloud PTL Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189 Skype: yangbincs993 -----Original Message----- From: Yang, Bin [mailto:bin.yang@intel.com] Sent: Wednesday, July 17, 2019 9:28 PM To: Yang, Bin Cc: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] controller-1 failed after unlock Hi Bin, This file is generated by kubeadm. And kubeadm is executed by puppet. Normally, you should find below log in /var/log/puppet/: Executing: 'kubeadm init --config=/etc/kubernetes/kubeadm.yaml' ... /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" ... If kubeadm does not run properly, you should find some error log. thanks, Bin On Wed, Jul 17, 2019 at 09:11:16AM +0000, Yang, Bin wrote:
Dear StarlingX experts,
I am exercising installing starlingx (with latest milestone3 image) with virtualbox following the wiki: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard
I am able to install/provision controller-0, compute-0, but failed to unlock controller-1, the issue seems to be kubelet.service cannot start due to missing of config file at /var/lib/kubelet/config.yaml :
controller-1:~# ls /var/lib/kubelet/ -al
total 8
drwxr-xr-x. 2 root root 4096 Jun 21 02:27 .
drwxr-xr-x. 60 root root 4096 Jul 16 10:43 ..
Could anyone who have insight of this issue shed some lights on this issue ? Thanks in advance
Below is the daemon log:
2019-07-17T02:06:13.623 controller-1 systemd[1]: info kubelet.service holdoff time over, scheduling restart.
2019-07-17T02:06:13.623 controller-1 systemd[1]: warning Cannot add dependency job for unit dev-hugepages.mount, ignoring: Unit is masked.
2019-07-17T02:06:13.641 controller-1 systemd[1]: info Stopped Kubernetes Kubelet Server.
2019-07-17T02:06:13.647 controller-1 systemd[1]: info Starting Kubernetes Kubelet Server...
2019-07-17T02:06:13.000 controller-1 root: info /usr/bin/kubelet-cgroup-setup.sh(481470): Nothing to do, already configured: /sys/fs/cgroup/cpuset/k8s-infra.
2019-07-17T02:06:13.664 controller-1 systemd[1]: info Started Kubernetes Kubelet Server.
2019-07-17T02:06:13.832 controller-1 kubelet[481475]: info Flag --feature-gates has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
2019-07-17T02:06:13.832 controller-1 kubelet[481475]: info Flag --cpu-manager-policy has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
2019-07-17T02:06:13.832 controller-1 kubelet[481475]: info F0717 02:06:13.831961 481475 server.go:189] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
2019-07-17T02:06:13.837 controller-1 systemd[1]: notice kubelet.service: main process exited, code=exited, status=255/n/a
2019-07-17T02:06:13.847 controller-1 systemd[1]: notice Unit kubelet.service entered failed state.
2019-07-17T02:06:13.847 controller-1 systemd[1]: warning kubelet.service failed.
Best Regards,
Bin Yang
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss