[Starlingx-discuss] controller-1 failed after unlock
Yang, Bin
Bin.Yang at windriver.com
Thu Jul 18 01:15:11 UTC 2019
Hi Bin,
Forget my previous question, I think I understand the root cause: the oam network was not provisioned appropriately. I will figure out how to fix that.
Thanks
Best Regards,
Bin Yang, Solution Engineering Team, Wind River
ONAP Multi-VIM/Cloud PTL
Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189
Skype: yangbincs993
-----Original Message-----
From: Yang, Bin
Sent: Thursday, July 18, 2019 9:13 AM
To: 'Yang, Bin'
Cc: starlingx-discuss at lists.starlingx.io
Subject: RE: [Starlingx-discuss] controller-1 failed after unlock
Hi Bin,
Thanks a lot for the help. Indeed there is error there, it seems a timeout issue to pull docker images (again, I have met such issue during ansible-playbook on controller-0, the workaround is to increase the timeout in ansible playbook), any suggestion to workaround this issue? Thanks
2019-07-16T11:07:02.080 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.5: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m
2019-07-16T11:07:02.085 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m
2019-07-16T11:07:02.091 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.5: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m
2019-07-16T11:07:02.096 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m
2019-07-16T11:07:02.101 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.5: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m
2019-07-16T11:07:02.105 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m
2019-07-16T11:07:02.109 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.5: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m
2019-07-16T11:07:02.113 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m
2019-07-16T11:07:02.117 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m
2019-07-16T11:07:02.121 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m
2019-07-16T11:07:02.125 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 128.224.230.5:9090: connect: no route to hostESC[0m
2019-07-16T11:07:02.130 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: , error: exit status 1ESC[0m
2019-07-16T11:07:02.134 ESC[mNotice: 2019-07-16 11:07:02 +0000 /Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`ESC[0m
2019-07-16T11:07:02.138 ESC[1;31mError: 2019-07-16 11:07:02 +0000 kubeadm init --config=/etc/kubernetes/kubeadm.yaml returned 1 instead of one of [0]
BTW, It is very nice to talk to you, we are both 'Yang Bin' :)
Best Regards,
Bin Yang, Solution Engineering Team, Wind River
ONAP Multi-VIM/Cloud PTL
Direct +86,10,84777126 Mobile +86,13811391682 Fax +86,10,64398189
Skype: yangbincs993
-----Original Message-----
From: Yang, Bin [mailto:bin.yang at intel.com]
Sent: Wednesday, July 17, 2019 9:28 PM
To: Yang, Bin
Cc: starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] controller-1 failed after unlock
Hi Bin,
This file is generated by kubeadm. And kubeadm is executed by puppet.
Normally, you should find below log in /var/log/puppet/:
Executing: 'kubeadm init --config=/etc/kubernetes/kubeadm.yaml'
...
/Stage[main]/Platform::Kubernetes::Master::Init/Exec[configure master node]/returns: [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
...
If kubeadm does not run properly, you should find some error log.
thanks,
Bin
On Wed, Jul 17, 2019 at 09:11:16AM +0000, Yang, Bin wrote:
> Dear StarlingX experts,
>
>
>
> I am exercising installing starlingx (with latest milestone3 image) with
> virtualbox following the wiki:
> https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard
>
>
>
> I am able to install/provision controller-0, compute-0, but failed to
> unlock controller-1, the issue seems to be kubelet.service cannot start
> due to missing of config file at /var/lib/kubelet/config.yaml :
>
>
>
> controller-1:~# ls /var/lib/kubelet/ -al
>
> total 8
>
> drwxr-xr-x. 2 root root 4096 Jun 21 02:27 .
>
> drwxr-xr-x. 60 root root 4096 Jul 16 10:43 ..
>
>
>
>
>
> Could anyone who have insight of this issue shed some lights on this issue
> ? Thanks in advance
>
>
>
> Below is the daemon log:
>
>
>
> 2019-07-17T02:06:13.623 controller-1 systemd[1]: info kubelet.service
> holdoff time over, scheduling restart.
>
> 2019-07-17T02:06:13.623 controller-1 systemd[1]: warning Cannot add
> dependency job for unit dev-hugepages.mount, ignoring: Unit is masked.
>
> 2019-07-17T02:06:13.641 controller-1 systemd[1]: info Stopped Kubernetes
> Kubelet Server.
>
> 2019-07-17T02:06:13.647 controller-1 systemd[1]: info Starting Kubernetes
> Kubelet Server...
>
> 2019-07-17T02:06:13.000 controller-1 root: info
> /usr/bin/kubelet-cgroup-setup.sh(481470): Nothing to do, already
> configured: /sys/fs/cgroup/cpuset/k8s-infra.
>
> 2019-07-17T02:06:13.664 controller-1 systemd[1]: info Started Kubernetes
> Kubelet Server.
>
> 2019-07-17T02:06:13.832 controller-1 kubelet[481475]: info Flag
> --feature-gates has been deprecated, This parameter should be set via the
> config file specified by the Kubelet's --config flag. See
> https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/
> for more information.
>
> 2019-07-17T02:06:13.832 controller-1 kubelet[481475]: info Flag
> --cpu-manager-policy has been deprecated, This parameter should be set via
> the config file specified by the Kubelet's --config flag. See
> https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/
> for more information.
>
> 2019-07-17T02:06:13.832 controller-1 kubelet[481475]: info F0717
> 02:06:13.831961 481475 server.go:189] failed to load Kubelet config file
> /var/lib/kubelet/config.yaml, error failed to read kubelet config file
> "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml:
> no such file or directory
>
> 2019-07-17T02:06:13.837 controller-1 systemd[1]: notice kubelet.service:
> main process exited, code=exited, status=255/n/a
>
> 2019-07-17T02:06:13.847 controller-1 systemd[1]: notice Unit
> kubelet.service entered failed state.
>
> 2019-07-17T02:06:13.847 controller-1 systemd[1]: warning kubelet.service
> failed.
>
>
>
>
>
>
>
>
>
> Best Regards,
>
> Bin Yang
>
>
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
More information about the Starlingx-discuss
mailing list