[Starlingx-discuss] About deployment issue: 06/08: Failed to execute controller manifest. Sporadic fail when config_controller

Tue Jul 24 13:41:00 UTC 2018

Cindy,

Sorry, I copied the wrong build error info in my first email today, which mislead you.
It should be below one 
=========================================
17:34:28 Debug: Packages still unresolved:
17:34:28
17:34:29 Warning: Infinite loop detected in dependency resolution.  See /localdisk/loadbuild/zhipengl/starlingx/export/deps.txt for details -- exiting
17:34:29 These RPMS had problems (likely version conflicts)
17:34:29        policycoreutils
17:34:29        policycoreutils-newrole
17:34:29        policycoreutils-python
17:34:29 Could not install dependencies
==============================================
From error detail log, it indicates selinux-policy depend on policycoreutils with version >= 2.5-18.

In current mirror, we only have two related RPM below.
policycoreutils-2.5-17.1.el7.x86_64.
policycoreutils-devel-2.5-17.1.el7.x86_64.rpm  
policycoreutils-python-2.5-17.1.el7.x86_64.rpm

So, I downloaded below 3 RPM, and build-iso pass  (with latest code today)
policycoreutils-2.5-22.el7.x86_64.rpm
policycoreutils-newrole-2.5-22.el7.x86_64.rpm
policycoreutils-python-2.5-22.el7.x86_64.rpm

I also deployed with the ISO I built without comment out kubernetes/docker/connectd.
Then it failed at step6/8 during Config_controller. With the error info in puppet.log as below.
=========================================================
Error: Systemd start for connectd failed!
Stage/Platform::Collectd/Service[collectd]/ensure:change from stopped to running failed: Systemd start for collectd failed.
=========================================================

@Erich and Abraham, any comment on my build and test result? Thanks!

Zhipeng

-----Original Message-----
From: Liu, ZhipengS [mailto:zhipengs.liu at intel.com] 
Sent: 2018年7月24日 21:10
To: Xie, Cindy <cindy.xie at intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran at intel.com>; Arce Moreno, Abraham <abraham.arce.moreno at intel.com>; Miller, Frank <Frank.Miller at windriver.com>; Hu, Yong <yong.hu at intel.com>; Little, Scott <Scott.Little at windriver.com>; Qi, Mingyuan <mingyuan.qi at intel.com>; 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] About deployment issue: 06/08: Failed to execute controller manifest. Sporadic fail when config_controller

Hi Cindy,

This is the key one, already in my code base.

BRs
zhipeng

-----Original Message-----
From: Xie, Cindy 
Sent: 2018年7月24日 19:20
To: Liu, ZhipengS <zhipengs.liu at intel.com>; Cordoba Malibran, Erich <erich.cordoba.malibran at intel.com>; Arce Moreno, Abraham <abraham.arce.moreno at intel.com>; Miller, Frank <Frank.Miller at windriver.com>; Hu, Yong <yong.hu at intel.com>; Little, Scott <Scott.Little at windriver.com>; Qi, Mingyuan <mingyuan.qi at intel.com>; 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>
Subject: RE: [Starlingx-discuss] About deployment issue: 06/08: Failed to execute controller manifest. Sporadic fail when config_controller

Zhipeng,
Please check if you have this patch included in your code base:
https://review.openstack.org/#/c/585043/

this one newly merged in the morning. GDC has built the ISO with the patch and Ada is doing the validation.

Thx. - cindy

-----Original Message-----
From: Liu, ZhipengS 
Sent: Tuesday, July 24, 2018 5:08 PM
To: Cordoba Malibran, Erich <erich.cordoba.malibran at intel.com>; Arce Moreno, Abraham <abraham.arce.moreno at intel.com>; Miller, Frank <Frank.Miller at windriver.com>; Xie, Cindy <cindy.xie at intel.com>; Hu, Yong <yong.hu at intel.com>; Little, Scott <Scott.Little at windriver.com>; Qi, Mingyuan <mingyuan.qi at intel.com>; 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>
Subject: RE: [Starlingx-discuss] About deployment issue: 06/08: Failed to execute controller manifest. Sporadic fail when config_controller

Hi Erich and Abraham,

According to the latest GUIDE, changes in imag.inc are reverted.
I synced the latest code and finished build. When build iso, there is a dependency issue I saw below.

16:25:59 These RPMS had problems (likely version conflicts)
16:25:59        containernetworking-cni
16:25:59        selinux-policy-base
16:25:59        selinux-policy-targeted
16:25:59 Could not install dependencies

After I downloaded below 3 rpms, build-iso pass.
policycoreutils-2.5-22.el7.x86_64.rpm
policycoreutils-newrole-2.5-22.el7.x86_64.rpm
policycoreutils-python-2.5-22.el7.x86_64.rpm

Do you have the same dependency issue? Need to update it to GUIDE if so.

Zhipeng

-----Original Message-----
From: Cordoba Malibran, Erich 
Sent: 2018年7月24日 11:38
To: Liu, ZhipengS <zhipengs.liu at intel.com>; Arce Moreno, Abraham <abraham.arce.moreno at intel.com>; Miller, Frank <Frank.Miller at windriver.com>; Xie, Cindy <cindy.xie at intel.com>; Hu, Yong <yong.hu at intel.com>; Little, Scott <Scott.Little at windriver.com>; Qi, Mingyuan <mingyuan.qi at intel.com>; 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>
Subject: Re: [Starlingx-discuss] About deployment issue: 06/08: Failed to execute controller manifest. Sporadic fail when config_controller

Hi, 

I think there are two issues here, 

1.- kubernetes-kubeadm requires a containernetworking-cni and that causes a failure in build-iso, this is reported here[0]
2.- With the dependencies resolved and with docker and kubernetes packages within the ISO, the config_controller command fails in step 6 trying to install docker-ce. We have seen this behavior in three different builds, also we have see it as sporadic, same ISO in some system finishes successful and in other fails.

Next time we see the problem we'll make a further debug.

[0] https://storyboard.openstack.org/#!/story/2003011

On 7/23/18, 9:12 PM, "Liu, ZhipengS" <zhipengs.liu at intel.com> wrote:

    Hi Abraham,

    Thanks for your quick feedback!
    I will also try it according to your updated GUIDE and update to you later today.
    Since dependency issue has been fixed, no need for my patch.

    zhipeng

    -----Original Message-----
    From: Arce Moreno, Abraham 
    Sent: 2018年7月24日 8:31
    To: Arce Moreno, Abraham <abraham.arce.moreno at intel.com>; Liu, ZhipengS <zhipengs.liu at intel.com>; Miller, Frank <Frank.Miller at windriver.com>; Xie, Cindy <cindy.xie at intel.com>; Hu, Yong <yong.hu at intel.com>; Little, Scott <Scott.Little at windriver.com>; Qi, Mingyuan <mingyuan.qi at intel.com>; 'starlingx-discuss at lists.starlingx.io' <starlingx-discuss at lists.starlingx.io>
    Subject: RE: [Starlingx-discuss] About deployment issue: 06/08: Failed to execute controller manifest. Sporadic fail when config_controller

    > > When built iso, it had some dependency issues, then commented out 
    > > below lines in build_iso/image.inc
    > >
    > > #selinux-policy
    > > ...
    > > #dockers-ce
    > > #kubernetes
    > > #kubernetes-client
    > > #kubernetes-kubeadm
    > > #kubernetes-master
    > > #kubernetes-node
    > 
    > Let us try today and will get back to this mailing list.

    Zhipeng,

    @Hayde has been able to build today from M/2018.07 [0] removing the need to comment above dependencies.

    Can you please help us confirm the steps are correct starting from scratch?

    Master compilation is in process using Hayde's fix. Developer Guide has been updated including notes for branches Master and M/2018.07
      https://wiki.openstack.org/wiki/StarlingX/Developer_Guide

    M/2018.07 generated ISO is being deployed today under Controller Storage and Simplex configurations following Installation Guide [1] and Validation Guide [2]:

    [0] http://lists.starlingx.io/pipermail/starlingx-discuss/2018-July/000321.html
    [1] https://wiki.openstack.org/wiki/StarlingX/Installation_Guide
    [2] https://wiki.openstack.org/wiki/StarlingX/Validation_Guide

    _______________________________________________
    Starlingx-discuss mailing list
    Starlingx-discuss at lists.starlingx.io
    http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss

_______________________________________________
Starlingx-discuss mailing list
Starlingx-discuss at lists.starlingx.io
http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss