From cindy.xie at intel.com Wed May 1 01:46:50 2019 From: cindy.xie at intel.com (Xie, Cindy) Date: Wed, 1 May 2019 01:46:50 +0000 Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190428 In-Reply-To: <0A5D9A624DF90343892F8F3FE7DE525A2A96A8DA@fmsmsx101.amr.corp.intel.com> References: <2FD5DDB5A04D264C80D42CA35194914F35F3DC92@SHSMSX104.ccr.corp.intel.com> <0A5D9A624DF90343892F8F3FE7DE525A2A96A8DA@fmsmsx101.amr.corp.intel.com> Message-ID: <2FD5DDB5A04D264C80D42CA35194914F35F3F3AC@SHSMSX104.ccr.corp.intel.com> Thanks Jose for the update regarding LP 1826445. I see you've added a notes there. Mingyuan will continue w/ the debug. From: Perez Carranza, Jose Sent: Wednesday, May 1, 2019 3:54 AM To: Miller, Frank ; Xie, Cindy ; Perez Ibarra, Maria G ; starlingx-discuss at lists.starlingx.io Subject: RE: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190428 Hi Frank LP 1826445 is updated with our latest findings, this seems to be a resources issue, Christopher was able complete successfully the "application apply" on a configuration were VMs hosting nodes have 8 cores and 32GB each controllers and computes. This is a considerable increase to the values mentioned on the wiki (16 GB for controllers and 10 GB for computes). Regards, José From: Miller, Frank [mailto:Frank.Miller at windriver.com] Sent: Tuesday, April 30, 2019 8:33 AM To: Xie, Cindy >; Perez Ibarra, Maria G >; starlingx-discuss at lists.starlingx.io Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190428 Cindy: I'll look for someone to triage 1824412 and update with any findings. But for 1826445 this is only reported in a virtual lab in an Intel site. We are not able to reproduce this issue in our labs or virtual environment. I suggest we wait for your team to return from Labor Day to reproduce and investigate. Frank From: Xie, Cindy [mailto:cindy.xie at intel.com] Sent: Tuesday, April 30, 2019 12:25 AM To: Miller, Frank >; Perez Ibarra, Maria G >; starlingx-discuss at lists.starlingx.io Subject: RE: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190428 Hi, Frank, I just assign LP#1824412 to Zhipeng. However, due to the fact that China is going to Labor Day holiday and will be black-out for 4 days, I am doubt how much progress we can make. It's appreciated if WR can have resource on both 1824412 and 1826445. Thanks. - cindy From: Miller, Frank [mailto:Frank.Miller at windriver.com] Sent: Tuesday, April 30, 2019 12:17 PM To: Perez Ibarra, Maria G >; starlingx-discuss at lists.starlingx.io Subject: Re: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190428 Maria - the sanities are much improved and I don't think they should be labelled RED. Looks more like YELLOW to me. There are 2 issues reported in this sanity. Can we get updates on the status of the 2 issues from the LP primes: · Cindy for: VM resize failed by "No valid host was found" https://bugs.launchpad.net/starlingx/+bug/1824412 · Mingyuan for: Failing application-apply due error on osh-openstack-openvswitch https://bugs.launchpad.net/starlingx/+bug/1826445 Let us know if you require assistance to triage/debug. Frank From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra at intel.com] Sent: Monday, April 29, 2019 10:25 PM To: starlingx-discuss at lists.starlingx.io Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190428 Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-APRIL-28 (link) Status: RED =========================================== Sanity Test is executed in a Containers - Bare Metal Environment AIO - Simplex Setup Manual [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [FAIL] | 03 TCs FAIL Sanity Platform 07 TCs [PASS] TOTAL: 57 TCS [Fail : 3 TCs] AIO - Duplex Setup Manual [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 05 TCs [PASS] TOTAL: 57 TCS PASS Standard - Local Storage (2+2) Setup Manual [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: 57 TCS PASS Standard - Dedicated Storage (2+2+2) Setup Manual [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 05 TCs [PASS] TOTAL: 57 TCS PASS Sanity Test is executed in a Containers - Virtual Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] | 3 TCs FAIL Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs ] [Fail : 3 TCs] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] Standard - Local Storage Setup 04 TCs [PASS] Provisioning 01 TCs [FAIL] Sanity OpenStack 49 TCs [FAIL] Sanity Platform 07 TCs [FAIL] TOTAL: [ 61 TCs PASS ] [Fail : 57 TCs] ---------------------------------------------------------- - VM resize failed by "No valid host was found" https://bugs.launchpad.net/starlingx/+bug/1824412 - Failing application-apply due error on osh-openstack-openvswitch https://bugs.launchpad.net/starlingx/+bug/1826445 For more detail of the tests: https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests#Sanity-OpenStack Regards Maria G. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cindy.xie at intel.com Wed May 1 02:08:26 2019 From: cindy.xie at intel.com (Xie, Cindy) Date: Wed, 1 May 2019 02:08:26 +0000 Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190430 In-Reply-To: References: Message-ID: <2FD5DDB5A04D264C80D42CA35194914F35F3F40B@SHSMSX104.ccr.corp.intel.com> Ada/Maria, I have a suggestion to the sanity test report, by looking at the cases you marked as "FAIL", actually there are quite some cases are actually "Blocked" due to the failure cases. For example: Standard - Local Storage Setup 04 TCs [PASS] Provisioning 01 TCs [FAIL] Sanity OpenStack 49 TCs [FAIL] Sanity Platform 07 TCs [FAIL] TOTAL: [ 61 TCs PASS ] [Fail : 57 TCs] Due to the fact that you're not able to provision Standard BM, they (49+7) cases were blocked. Normally I will mark those as "Blocked" instead of "Fail". Comments? Thx. - cindy From: Perez Ibarra, Maria G [mailto:maria.g.perez.ibarra at intel.com] Sent: Wednesday, May 1, 2019 7:26 AM To: starlingx-discuss at lists.starlingx.io Subject: [Starlingx-discuss] [Containers] Sanity Test - ISO 20190430 Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-APRIL-30 (link) Status: RED =========================================== Sanity Test is executed in a Containers - Bare Metal Environment AIO - Simplex Setup Manual [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [FAIL] | 03 TCs FAIL Sanity Platform 07 TCs [PASS] TOTAL: 57 TCS [Fail : 3 TCs] AIO - Duplex Setup Manual [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 05 TCs [PASS] TOTAL: 57 TCS PASS Standard - Local Storage (2+2) Setup Manual [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: 57 TCS PASS Standard - Dedicated Storage (2+2+2) Setup Manual [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 52 TCs [PASS] Sanity Platform 05 TCs [PASS] TOTAL: 57 TCS PASS Sanity Test is executed in a Containers - Virtual Environment AIO - Simplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] | 3 TCs FAIL Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs ] [Fail : 3 TCs] AIO - Duplex Setup 04 TCs [PASS] Provisioning 01 TCs [PASS] Sanity OpenStack 49 TCs [PASS] Sanity Platform 07 TCs [PASS] TOTAL: [ 61 TCs PASS ] Standard - Local Storage Setup 04 TCs [PASS] Provisioning 01 TCs [FAIL] Sanity OpenStack 49 TCs [FAIL] Sanity Platform 07 TCs [FAIL] TOTAL: [ 61 TCs PASS ] [Fail : 57 TCs] ---------------------------------------------------------- - VM resize failed by "No valid host was found" https://bugs.launchpad.net/starlingx/+bug/1824412 - Failing application-apply due error on osh-openstack-openvswitch https://bugs.launchpad.net/starlingx/+bug/1826445 For more detail of the tests: https://wiki.openstack.org/wiki/StarlingX/Test/SanityTests#Sanity-OpenStack Regards Maria G. -------------- next part -------------- An HTML attachment was scrubbed... URL: From austin.sun at intel.com Wed May 1 02:28:19 2019 From: austin.sun at intel.com (Sun, Austin) Date: Wed, 1 May 2019 02:28:19 +0000 Subject: [Starlingx-discuss] Build Docker images and how to use temporary image for debug in deploy env. In-Reply-To: <58CF5BABC9A76946A638A0E8AE48D1737183EE63@ALA-MBD.corp.ad.wrs.com> References: <6703202FD9FDFF4A8DA9ACF104AE129FBA492899@ALA-MBD.corp.ad.wrs.com> <58CF5BABC9A76946A638A0E8AE48D1737183EE63@ALA-MBD.corp.ad.wrs.com> Message-ID: Hi Gerry & Penny: Thanks a lot you. This is very useful for me . BR Austin Sun. From: Kopec, Gerald (Gerry) [mailto:Gerry.Kopec at windriver.com] Sent: Wednesday, May 1, 2019 6:35 AM To: Penney, Don ; Little, Scott ; starlingx-discuss at lists.starlingx.io Subject: Re: [Starlingx-discuss] Build Docker images and how to use temporary image for debug in deploy env. To load a private docker image onto your controller try these instructions: https://wiki.openstack.org/wiki/StarlingX/Containers/BuildingImages#Testing_Image_on_Running_System Then follow recommended link to: https://wiki.openstack.org/wiki/StarlingX/Containers/FAQ#How_do_I_make_changes_to_the_code_or_configuration_in_a_pod_for_debugging_purposes.3F Gerry From: Penney, Don Sent: Tuesday, April 30, 2019 5:41 PM To: Little, Scott; starlingx-discuss at lists.starlingx.io Cc: Kopec, Gerald (Gerry) Subject: RE: [Starlingx-discuss] Build Docker images and how to use temporary image for debug in deploy env. Sorry, Austin, I didn't see your email this morning. #1: The stx-mariadb image is not built in the dev build stream, much like other images. It now only builds in stable. The stx-fm-rest-api image, as well, only builds in the stable stream. The error you're getting is because there are a couple of centos_dev_docker_images.inc files with stale entries. This is just a warning to stderr and should not impact anything. #2: You can use the --push option of build-stx-images.sh, combined with --user and --registry options to tag and push the image(s) to a private registry or the docker hub. From there, I think there are instructions available on loading those images onto your controller. Gerry, are those instructions on a wiki? From: Scott Little [mailto:scott.little at windriver.com] Sent: Tuesday, April 30, 2019 5:02 PM To: starlingx-discuss at lists.starlingx.io Subject: Re: [Starlingx-discuss] Build Docker images and how to use temporary image for debug in deploy env. CENGN isn't building dev images these days. So it's not being tested in a while. cat stx/stx-integ/centos_dev_docker_images.inc database/mariadb However there is no file ... stx/stx-integ/database/mariadb/centos/stx-mariadb.dev_docker_image that's definitely inconsistent. Raise a launchpad. Scott On 2019-04-30 5:17 a.m., Sun, Austin wrote: Hi All: 1) When I tried to build dev images . I meet below error. "Unsupported BUILDER in /home/wrsroot/starlingx/workspace/localdisk/designer/wrsroot/starlingx/cgcs-root/stx/stx-integ/database/mariadb/centos/*.dev_docker_image:" Build command is something like (sudo ./build-stx-images.sh --os centos --stream dev --base starlingx/stx-centos:master-dev-latest --wheels ~/starlingx/wheel/stx-centos-stable-wheels.tar --only stx-fm-rest-api) Does mariadb not support for dev build ? 2) how to push image built by developer to deployed environment directly ? Do we have any wiki or guide about this ? Thanks. BR Austin Sun. _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss at lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From austin.sun at intel.com Wed May 1 02:59:19 2019 From: austin.sun at intel.com (Sun, Austin) Date: Wed, 1 May 2019 02:59:19 +0000 Subject: [Starlingx-discuss] 2nd node stuck at pxe installation In-Reply-To: <6703202FD9FDFF4A8DA9ACF104AE129FBA48BC76@ALA-MBD.corp.ad.wrs.com> References: <2125688793.1092495.1556635578983@communicator.strato.com> <6703202FD9FDFF4A8DA9ACF104AE129FBA48BC76@ALA-MBD.corp.ad.wrs.com> Message-ID: Hi Penny : Thanks a lot. some update from my side. From log Dracut [1]; squashfs.img was not successfully download on controller-1. And the ip on controller-1 during pxe boot was 192.168.204.4 , if use curl command "curl http://192.168.204.2:8080/feed/rel-19.01//LiveOS/squashfs.img -o mytest.img" can successfully download squashfs.img . But if running command "curl http://pxecontroller:8080/feed/rel-19.01//LiveOS/squashfs.img" , it will failed. From controller "nslookup pxecontroller" , the pxecontroller ip addr is 169.254.202.2 , not mgmt ip(192.168.204.2). so it will make sense pxecontroller can not be accessed by controller-1(pxe boot stage). If hack sysinv-conductor manager.py change pxecontroller to mgmt. ip (192.168.204.2) and restart sysinv-conductor, then the controller-1 can be installed successfully. BTW: This issue is 100% reproducible on certain servers, but not for all kinds of servers. [1] http://paste.openstack.org/show/750134/ [2] http://paste.openstack.org/show/750135/ Thanks. BR Austin Sun. -----Original Message----- From: Penney, Don [mailto:Don.Penney at windriver.com] Sent: Tuesday, April 30, 2019 11:04 PM To: Marcel Schaible ; starlingx-discuss at lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation You can also look in /var/log/daemon.log for TFTP logs showing the requests and transfers of files, and /www/var/log/lighttpd-access.log for transfer of the squashfs.img. Since the console showed "Loading rel-19.01/installer-initrd...........ready", it seems like the TFTP transfer was fine, which is why I suggested modifying the console setting to ensure you're seeing what's happening. This is what I'd expect to see if the console is set to serial, but there's no serial console attached (such as using virtual box without a serial console). For example, you can see the bzImage and initrd here being transferred via TFTP in daemon.log: 2019-04-26T16:27:34.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-bzImage to 192.168.202.24 2019-04-26T16:27:36.000 controller-0 dnsmasq-tftp[96915]: info sent /pxeboot/rel-19.01/installer-initrd to 192.168.202.24 followed by the squashfs.img in /www/var/log/lighttpd-access.log: 192.168.202.24 pxecontroller:8080 - [26/Apr/2019:16:28:12 +0000] "GET /feed/rel-19.01//LiveOS/squashfs.img HTTP/1.1" 200 368152576 "-" "curl/7.29.0" If only the bzImage and initrd is transferred, without seeing the logs on console, my guess would be an unsupported NIC preventing the initrd from accessing the management network in order to request the squashfs.img (which provides the installer rootfs) -----Original Message----- From: Marcel Schaible [mailto:marcel at schaible-consulting.de] Sent: Tuesday, April 30, 2019 10:46 AM To: starlingx-discuss at lists.starlingx.io Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation Hi, we have the same problem with the image from 20190411. It Looks like that controller-1 is not finding the pxe boot image. Kind regards Marcel > Message: 1 > Date: Tue, 30 Apr 2019 13:54:17 +0000 > From: "Penney, Don" > To: "Lin, Shuicheng" , > "starlingx-discuss at lists.starlingx.io" > > Subject: Re: [Starlingx-discuss] 2nd node stuck at pxe installation > Message-ID: > <6703202FD9FDFF4A8DA9ACF104AE129FBA48BB86 at ALA-MBD.corp.ad.wrs.com> > Content-Type: text/plain; charset="utf-8" > > You can set the console either via the web browser in the installation parameters, or at command-line, such as: > system host-update 2 personality=controller console= > > The default is the serial console, "ttyS0,115200". If you're using a graphical console, you can set it blank, or console=tty0, for example. This should allow you to see if there are some errors occurring trying to load the installer. > > > From: Lin, Shuicheng [mailto:shuicheng.lin at intel.com] > Sent: Tuesday, April 30, 2019 1:03 AM > To: starlingx-discuss at lists.starlingx.io > Subject: [Starlingx-discuss] 2nd node stuck at pxe installation > > Hi all, > I try to do duplex deploy, and find controller-1 stuck at pxe installation step. > The screen stuck at "Loading rel-19.01/installer-initrd...........ready". But package installation is not started as host-show command shown in controller-0. > I meet this issue with latest code. Do you know what cause the issue, or how to debug it? > Thanks. > > [wrsroot at controller-0 ~(keystone_admin)]$ system host-list > +----+--------------+-------------+----------------+-------------+--------------+ > | id | hostname | personality | administrative | operational | availability | > +----+--------------+-------------+----------------+-------------+--------------+ > | 1 | controller-0 | controller | unlocked | enabled | available | > | 2 | controller-1 | controller | locked | disabled | offline | > +----+--------------+-------------+----------------+-------------+--------------+ > [wrsroot at controller-0 ~(keystone_admin)]$ system host-show > controller-1 | grep install > | install_output | text | > | install_state | None | > | install_state_info | None | > | subfunction_avail | not-installed > > Best Regards > Shuicheng > _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss at lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss at lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss From Angie.Wang at windriver.com Wed May 1 04:11:54 2019 From: Angie.Wang at windriver.com (Wang, Jing (Angie)) Date: Wed, 1 May 2019 04:11:54 +0000 Subject: [Starlingx-discuss] Automation script needs to be updated because the coming changes for build-helm-charts.sh and system application-upload CLI Message-ID: Hi All, Please aware that there are two commits may be merged soon that will impact automated testing. The gerrit reviews: - The changes for build-helm-charts.sh: https://review.opendev.org/#/c/655494/ - The changes for system application-upload CLI: https://review.opendev.org/#/c/655497/ Once they are merged, - A metadata.yaml file will be built into stx-openstack application tarball which contains application name and version. The name of the application tarball will be changed to -(-