May 2020 - Starlingx-discuss - lists.starlingx.io

Re: [Starlingx-discuss] Sanity Master Test LAYERED build ISO 20200506T013246Z bug#1877035
by Sun, Austin 07 May '20

07 May '20

Hi All: After further analysis this issue with Yan. Suspect there is bug hiding in cengn layer build. Scott : would you like to check cengn build env ? The details analysis : Latest puppet-keystone rpm built from cengn should include change [1] which was merged by April 28th. >From distro layer build 0505 [3] and latest distro output , puppet-keystone includes change [1]. But if we extract puppet-keystone in 0506 iso built from layer build [2] or latest iso [5] , it is surprised this change [1] was not included. so bug#1877035 shows-up based on 0506 and 0507 iso. If we extract puppet-keystone from 0506 iso built from monolithic [4], the change [1] was included in puppet-keystone, and ansible provision pass it. BTW: layer build in my local setup (cross build flock or flock layer only build ) did not show this issue, so it should not be related to layer build script. [1] https://review.opendev.org/#/c/716842/ [2] http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/flock/20200… [3] http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/2020… [4] http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/monolithic/… [5] http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/flock/20200… [6] http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/distro/late… Thanks. BR Austin Sun. From: Jascanu, Nicolae <nicolae.jascanu(a)intel.com> Sent: Wednesday, May 6, 2020 10:06 PM To: starlingx-discuss(a)lists.starlingx.io Subject: [Starlingx-discuss] Sanity Master Test LAYERED build ISO 20200506T013246Z Sanity Test from 2020-May-06 (http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/flock/20200…) Status: RED Provisioning is failing for all configurations. I've added a comment to the bug: https://bugs.launchpad.net/starlingx/+bug/1877035 We used the helm chart from http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/latest_dock… Regards, STX Validation Team

3 3

[Starlingx-discuss] Minutes: StarlingX Release Meeting - May 7/2020
by Khalil, Ghada 07 May '20

07 May '20

Agenda/Minutes are posted at: https://etherpad.openstack.org/p/stx-releases stx.4.0 - Feature Tracking: https://docs.google.com/spreadsheets/d/1a93wt0XO0_JvajnPzQwnqFkXfdDysKVnHpb… - Specific Feature Follow-up --- features we haven't talked about in a while - Fault Containerization / Prime: Saul - Is this being worked? Is it going to make it for 5/15? Or should we mark it as out? - Someone is assigned from Intel Bangalore team, but not sure about progress. Action: Bruce will follow up. - python3 / Prime: Austin - A small # of reviews are posted & merging. - https://review.opendev.org/#/q/projects:+starlingx+topic:python3+(status:op… - What's the risk of the code being merged? Is the code benign? Do we keep this going until MS3 and resume after the stx.4.0 branch is created? - Agreed to evaluate risk on a commit-by-commit basis, especially as we get closer to MS3 and RC1. - Backup & Restore (k8s portion) / Prime: Frank - Is this done? - Main functionality is in, but some gaps have been identified and need to be re-worked. Forecast date revised. - Containerize Openstack clients / Prime: Bruce - Is this being worked? Is it going to make it for MS-3 (5/29)? Or should we mark it as out? - Still not resourced. Mark as Out of stx.4.0 - System Upgrade Support / Prime: Dariush - Reviews/code is being merged. - Where are we? What will be achieved by MS-3 (5/29)? - Expect the framework to be in place for MS-3 to allow a null upgrade (stx.4 to stx.4). Then need to determine what code changes are required on the stx3 side to enable an actual upgrade. This will likely do beyond the stx.4.0 release date. - IPv6 support for PXE boot network / Prime: Kunpeng - Is this being worked? - Ghada to ask Steve, the networking PL if he knows the status of this feature. - Layered Build / Prime: Frank - Is this considered done? Are there remaining work items? - Consider DONE. Frank to clean up the storyboard. Reported issues are tracked by launchpads. - Update on Feature Testing -- Yang / Nick - Feature Testing: https://docs.google.com/spreadsheets/d/1C9n4aRQT7xMyTDCT5sfuZGNI9ermAX5BYRy… - Things are going well overlap / making good progress in general - Action: Yang/Nick to update the feature testing dates in the feature tracking spreadsheet. - Kernel Upversion - Feature testing is underway with engineering build. Agreed to look at moving the remaining testing to stx master as the code is all merged now. - Openstack host services upversion to train - Feature testing complete with engineering build - Ceph Rook - Did a sanity with an engineering build. - Will start feature test-cases - Kata Containers - Need to wait - TSN - Not started. Will start to look into the setup requirements next week. - Cert Manager - Test was in progress, but blocked on a good load from stx master - Regression Testing: https://docs.google.com/spreadsheets/d/1gA3bnLS7aY2y8dKxm4MuqpWyELq3PVJMYti… - Working on test-case selection next week - Will start execution the week after - May 18 - Delay is due to lack of good loads from stx master - Sanity Issues - Sanity with monolithic build - Nick & team will try this today - Sanity with layered build - Saul working on a fix to address the issue causing the openstack failure on bootstrap - https://bugs.launchpad.net/starlingx/+bug/1877035

1 0

[Starlingx-discuss] Propose new Core for stx-root, stx-tools
by Miller, Frank 07 May '20

07 May '20

Folks: I'd like to propose adding an additional Core for the stx-root and stx-tools repos: Davlet Panech. Davlet is now a member of the Starlingx Build team and he would especially be able to help cover these repos when Scott or Don or Saul are on vacation. Frank

3 3

[Starlingx-discuss] Learnings from build failures for Train & kernel commits in past week
by Miller, Frank 07 May '20

07 May '20

StarlingX Developers: In the past 2 weeks we've had many days of build failures. A full timeline is below but the majority of the issues were associated with delivery of a large number of commits for 2 large features. And having this done just days after build layering was enabled for the project. Looking through the variety of failures, the Build Team identified the following recommendations to prevent similar issues in the future. These recommendations are for both the developers as well as the Cores. Recommendations for Developers: 1. Every developer must do a final build before removing WFL -1 from their commit. 2. For larger features or any commits that introduce or change an rpm, the developer build must be done in clean workspace and clean 'output' directories. This includes starting with a clean mirror download from CENGN. Address any failures seen in your build when using a clean workspace. Identify any mirror updates required and see [3] for how to handle. 3. Any changes to lst files need to be handled separate from rest of code/feature as follows: * Deliver a commit that adds all entries to lst files required for the feature * Wait for a successful CENGN build, confirm the new files appear in the CENGN mirror. * Rebuild the code/feature in a clean developer workspace and ensure a clean build with no additional lst changes needed. Then merge the code/feature commits. * Once the next CENGN build is successful, then deliver an additional commit to remove any old lst file entries that are no longer needed. 4. Community needs to move over to use the layered build approach for their developer builds. This is the production way of building and will save developer build times for most commits. Recommendations for Cores: 1. Ensure large features have followed the above guidelines before allowing the code to merge. If there are any other suggestions please provide your input. Thank-you. StarlingX Build Team ---------------------------------------- ----- Timeline of failures -------- April 28 * Train commits merged, build failure * Root cause: copy/paste error on the source repo config https://review.opendev.org/716883 * Don Penney found issue and provided fix: https://review.opendev.org/724127 * Preventative action: before merge do final test with build in developer workspace in clean workspace and clean 'output' directories. Address any failures seen in build from a clean workspace. April 29 * Train commits causing additional build failure * Root cause: missing dependency - need to add lst entry for python2-sphinxcontrib-apidoc. * Scott Little found issue and provided fix: https://review.opendev.org/#/c/724411/ * Preventative action: developers need to build before merge and start with clean build environment April 30 * Train commits still causing build failure * Root cause: missing dependency: need build dependency in python-neutronclient * Scott Little found issue and provided fix: https://review.opendev.org/#/c/724792/ May 1-3 * 4.18 kernel commits causing build failures * Root cause #1: CentOS 8 has released a newer kernel version than the one that was merged in our stx-kernel. CentOS 8 has an "obsoletes" on the kernel we had merged. The yumdownloader tool refused to download an obsoletes package. * Scott Little found issue, Austin Sun & Shuicheng Lin provided fixes: [1][2][3] * Preventative action: before merge do final test with build in clean workspace and clean 'output' directories. Address any failures seen in build from a clean workspace. * Additional action: How are we going to prevent build failures due to obsoletes when next CentOS 8 version is released and our current version is obsoleted? May 4 * 4.18 kernel commits causing additional build failures * Root cause #2: missing dependencies for kmods & mellanox - lst files needed to be updated * Scott Little provided fixes: [4][5] * Shuicheng Lin provided an additional fix to add perl-generators to lst: [6] May 5 * 4.18 kernel commits causing additional build failure during flock layer build * Root cause #3: Issue with build layering mirror update script colliding with a generic script running on CENGN * Scott Little provided fix in CENGN script [1] https://review.opendev.org/#/c/725214/ [2] https://review.opendev.org/#/c/724951/ [3] https://review.opendev.org/#/c/724955/ [4] https://review.opendev.org/#/c/725379/ [5] https://review.opendev.org/#/c/725371/ [6] https://review.opendev.org/#/c/725447/

1 0

[Starlingx-discuss] [ Test ] meeting notes - 05/05/2020
by Liu, Yang (YOW) 07 May '20

07 May '20

Agenda for 05/05/2020 Attendees: Yang, Cosmin, Nicolae, GeorgeP, Ruediger, Mihail, GeorgeC, Bruce · Sanity Status: * No build since apr28th. New build is on the way with possible kernal fix · stx4.0 testing: * Feature testing: § https://docs.google.com/spreadsheets/d/1C9n4aRQT7xMyTDCT5sfuZGNI9ermAX5BYRy… § Centos8: · Test in progress on same eng load. Will test TPM and Mellonox next. · TPM works on official load - will try on the new eng load · Mellonox is available - to be configured § Upversion Openstack services used by flock components on host: · Test completed. Feature merged - have not tested in offical load due to openstack apply issue and now the build issue. § Upgrade Containerized OpenStack to Ussuri (and OpenStack helm rebase) · Got eng load - going to test sanity on it § Windows Active directory completed § Red fish virual media support - testing completed § Kubernetes Upgrade Support in progress · Regression completed. k8s dashboard broken - LP to be reported. § Ceph - Rook · Got eng load - installing system by following instructions. Deployment is not successfully yet - investigating on whether its configuration/env issue, or procedural issue, etc. § Kata Containers - started, not a focus at the moment. § TSN - not started § Cert-Manager · Testing in progress - caused openstack apply to fail, now fixed. · ETA: May11th week § B&R with etcd database · Feature testing completed. Regression ran on regular basis. Often broken when new features introduced. * Regression testing: § Starting in mid-May. · Select regression test cases by May15th. § Nic: All types of systems are configured · Two Simplex servers are done · Two Duplex systems are done · Two Standard are done · Two Storage are done · Three virual setups - SX/DX, Standard, Storage · Open topic * Test automation § Need automated installation script · Robot framwork has installation script to setup and provision the system - can be starting point with focus on virtual installation o https://opendev.org/starlingx/test/src/branch/master/automated-robot-suite § Suggest to priorize the features without hardware depency § Right teams are loaded with new feature testing * Test in the open: § No update for past week. Need to check with Bill and Nic's team will try virtual box when they got some cycles. * zuul failed - Mihail sent email § GeorgeP has a fix in the code, so will go with this solution if works. § Al had a fix to clamping the pylint version https://review.opendev.org/#/c/724326/ - on hold unless first option doesn't work. * Zoom meeting schedule doesn't see to be right on the wiki § Need to follow up with Ildiko

1 0

[Starlingx-discuss] Sanity Master Test LAYERED build ISO 20200507T015208Z
by Jascanu, Nicolae 07 May '20

07 May '20

Sanity Test from 2020-May-07 (http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/flock/20200…) Status: RED Provisioning is failing with the existing bug 1877035. I've added a comment to: https://bugs.launchpad.net/starlingx/+bug/1877035 We used the helm chart from http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/… Regards, STX Validation Team

1 0

[Starlingx-discuss] Restarting Multi-OS project
by Saul Wold 06 May '20

06 May '20

Folks, As has been discussed in the TSC / Community meeting we are going to restart the Multi-OS project. This time around there is a strong effort being worked on by a different team at WindRiver to port the Flock and associated changes to an OpenEmbedded[0] base with the Yocto Project[1]. StarlingX actually has a history of using OpenEmbedded in the distant past! We will use the Thursday 7:30am PT slot initially and if/when the build team moves to bi-weekly, we might move the bi-weeky in the 7:00am slot opposite weeks from the build team. Initially I will again take the TL position and the upcoming election can figure out who the PL will be. Thanks for your support Sau! [0] https://www.openembedded.org/wiki/Main_Page [1] https://www.yoctoproject.org/

1 1

[Starlingx-discuss] [docs] [meeting] Docs team notes 2020-05-06
by Camp, MaryX 06 May '20

06 May '20

Hello all, Here are this week's docs team meeting minutes (short form). Details in [2]. Join us if you have interest in StarlingX docs! We meet on Wednesdays 12:30 PST. [1] Call logistics: https://wiki.openstack.org/wiki/Starlingx/Meetings [2] Our tracking Etherpad: https://etherpad.openstack.org/p/stx-documentation thanks, Mary Camp ========== 2020-05-06 . All -- reviews merged since last week: 3 . All -- bug status -- 7 total, 4 WIP o [ww18] admin endpoints in distributed cloud are changed to https [WIP] o [ww17] Debug guide [not started] o [ww16] Kata containers [WIP, location under Kubernetes operations] and Build Avoidance (not started; AR Mary compare bug text to existing guide o [ww15] Backup & restore procedure info [WIP] Pinged Mihnea again, cc Frank Miller. Add to Operations section. . Reviews in progress: o TSN in Kata containers - Yi Wang author - needs technical review. Mary will do clerical edits after merge. o Rook migration - Martin Chen author - needs author updates. Mary will do clerical edits after merge. o Modifying layered build commands (add pike / remove pike) . All -- Opens o From Bruce, feedback from the field team supporting STX customers: . Documentation: Documentation can be improved. The Getting started guide abruptly ends after setting up OpenStack but not networking (and the networking wiki while complete, isn't the best piece of literature. It's also not linked-to from within the getting started guide). Similarly, the lowlatency worker subfunction isn't documented at all (except for a mention in one of the mailing lists). There's much we can improve. . "lowlatency worker subfunction" = real time kernel AR Mary find this in the discuss archives and figure out where it belongs. . "networking wiki" is probably the TSN wiki page. [We think he's talking about networking in general] We could add a helpful example (even if very simple) and/or helpful link to OpenStack documentation about next steps. AR Mary ping Steve Webster for help on this. Could set up a LP issue for this, but ping him first. o Layered Build guide thread on discuss list: http://lists.starlingx.io/pipermail/starlingx-discuss/2020-May/008529.html [Shuicheng] and http://lists.starlingx.io/pipermail/starlingx-discuss/2020-May/008515.html [Saul] . Shuicheng is saying to update the existing build guide w/pointer to Layered Build guide; Saul says Layered Build guide is good and needs edits. AR Mary to email Poornima about this. . Discussion last week: o Bruce: attended marketing call with foundation, significant need for blog posts, esp new release, new content. Mary is available to help with edits/proofing as needed. . AR for Bruce and Greg to find likely authors for blog posts within the teams. o Search option for docs: Our docs page should have a search option. [Mary ask Kris about search, was it looked into?] o Website needs update. AR Mary to find how to change the website. Git repo, file an issue? who can help. . Discussion 2 weeks ago: o WIP: Docker proxy setup Priority 1 task . 06May: Discussion of draft diagrams. We agreed that the Rotated diagram is preferred, just need to switch public/company clouds (left/right)

1 0

[Starlingx-discuss] StarlingX and load balancers?
by Jones, Bruce E 06 May '20

06 May '20

I have a question from customer. Is it possible to put run load balancers for StarlingX worker nodes? Do we support the standard Kubernetes Ingress and Ingress Controller features? brucej

1 0

[Starlingx-discuss] Bandit
by Saul Wold 06 May '20

06 May '20

TSC Members: During the TSC call today, I brought up running bandit[0] to the flock repos. Bandit is a code scanner that looks for common security related issues in Python. It was initially an OpenStack project that has since migrated to Python Code Quality Authority. It is actively being maintained. Bandit is being used by other OpenStack projects via Tox and Zuul jobs such as Cinder and Armada[1]. There are other possible options[2] (not a comprehensive search), but they don't are either older (not maintained), python2 only or not truly opensource. If the community has other suggestions, I am welcome the additional data. The zuul job should not add too much time, since all the zuul jobs run in parallel. Currently the zuul job is no-voting, the initial scan has identified about 75 High issues. Most are dealing with subprocess being called with shell equals True, there is also a few deprecated libraries. These will need to be evaluated, it's on the schedule for the PTG. There are also options to set a baseline and use it for checking new failures. Based on the above are there any objections to continue enabling bandit for the StarlingX project's python code? Sau! [0] https://github.com/PyCQA/bandit [1] https://wiki.openstack.org/wiki/Security/Projects/Bandit [2] https://geekflare.com/find-python-security-vulnerabilities/

1 0