June 2019 - Starlingx-discuss - lists.starlingx.io

[Starlingx-discuss] [ Test ] Meeting notes - 06/24/2019
by Cabrales, Ada 28 Jun '19

28 Jun '19

Agenda for 06/24 Attendees: Elio, Richo, Al, Critopher, JC, Maria P, Numan, Yang, Bruce 1. Regression testing status - Elio, Numan Tracker - https://docs.google.com/spreadsheets/d/1MoBroFimeQjsJvCC_N_DSLYMQsNZVMzTwAV… MS-3 declared on 06/22 ISO for regression testing is from 06/21 Intel - 110 tests executed manually (began last week), 7 failures. Automated regression to begin today. Plan is to execute one domain per day, in 4 configs. Running on virtual, due to tests cases were created in this env and we are having problems running them on bare metal. Wind River - automated regression ran this weekend - results to be updated, launchpads in progress. Some failures in nova for live migration. Reports to be sent Tuesday and Thursday. Maria P to send the first report today. Goal is to finish by last week of July. Update the title (in the tracker) and status of the bugs found. Ask Ghada for creating a tag for bugs found during regression - stx.regression - Numan 2. Sanity status Two greens in a row. Today's looks good so far. The deployment times continue to increase Numan to send info about their setup and deployment times. Cristopher to collect the info from proxy config. Unlocking master controller is taking more and more time. Track this time. 3. Testing framework - Erich finished the review Elio and JC - please review the code. wrsroot patch is going to be submitted this week. For history - it doesn't make test to include it on this commit. The testing framework is a parallel project, it doesn't have impact on the final stx product. Saul - is there a deployment capability included into the testing framework, or covers the testing only? Yang - is for running the tests in an already configured system. How to address issues found in the framework (launchpads?) Launchpads will be the way of reporting, however, we have to make sure of not affecting stx statistics. Proposal is to a use prefix [stx-test] into the title and also, use a tag. For new features/tests, use storyboard, as the project does. Ada to begin a mail thread for notifying this. Yang - on vacations starting this Friday. 4. Feature testing status - Total/pass/fail/block Containers Ada - 75/40/1/2 Ask to be updated offline. Numan - 82/58/3/0 No update. OpenStack patch elimination Ada - 50/21/0/6 Testing has been resumed. Latests ISO includes features that were blocking execution. Numan - 8/4/4/0 Retesting launchpads. Centos 7.6 - QAT - 12/9/0/0 10 executed - 1 pending (requires image with QAT driver). REST API pending - help required to define the steps. Numan to check with Cris. Containerized OVS - 15/8/0/0 In progress - results to be updated soon. 50% already done. No problems identified so far. Ceph upgrade - 26/15/0/7 6 blocked, 3 for retesting, 1 deferred - storyboard to be created. i/o test to be run soon. 5. Opens Numan - some Intel people is going to WR for training. - This will be on late August. Request is in approval loop.

1 0

[Starlingx-discuss] High availability for instances
by Marcel Schaible 27 Jun '19

27 Jun '19

Hi, we have two compute nodes with 1 instance of a vm each running an application. Because thise application should be available 24/7 we want test a fail over from one compute node to the other. I found an user Story "High Availability for Virtual Machines" (http://specs.openstack.org/openstack/openstack-user-stories/user-stories/pr…) Is this sub-Project still activate and what is the current state? Any other idea how to accomplish a fail over with the upcoming relase of starlingx? Thanks! Marcel

1 0

[Starlingx-discuss] LP1827258 (OOM on compute node) analysis
by Yang, Bin 27 Jun '19

27 Jun '19

Hi, LP1827258 [0] is highlighted that "order zero" (4KB) allocation causes OOM. It looks unreasonable that 4KB allocation was failed. Here we have two problems: 1. Why 4KB memory allocation causes the OOM 2. Why system has no enough memory 1. Why 4KB memory allocation causes the OOM ======================================== >From compute-1_20190507.124154/var/log/kern.log [1] [ 1515.471830] calico-node invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=999 ... [ 1515.471950] Node 0 DMA free:15848kB min:60kB low:72kB high:88kB [ 1515.471954] lowmem_reserve[]: 0 2800 15838 15838 [ 1515.471956] Node 0 DMA32 free:63620kB min:11480kB low:14348kB high:17220kB [ 1515.471959] lowmem_reserve[]: 0 0 13038 13038 [ 1515.471961] Node 0 Normal free:53148kB min:53464kB low:66828kB high:80196kB [ 1515.471964] lowmem_reserve[]: 0 0 0 0 ... [ 1515.472142] Out of memory: Kill process 60356 (kubernetes-entr) score 1000 or sacrifice child gfp_mask=0x201da: it means the page allocation is from ZONE_NORMAL But Node 0 Normal, free:53148kB < min:53464kB: it has no enough space Conclusion: ******************************************************* * min is the oom watermark * * Since the free < min, kernel will start oom killer. * ******************************************************* We can find the kernel code for this logic: ------------------------------------------ mm/page_alloc.c: __zone_watermark_ok() /* * Check watermarks for an order-0 allocation request. If these * are not met, then a high-order request also cannot go ahead * even if a suitable page happened to be free. */ if (free_pages <= min + z->lowmem_reserve[classzone_idx]) return false; /* If this is an order-0 request then the watermark is fine */ if (!order) return true; And here is a related document: https://www.kernel.org/doc/Documentation/sysctl/vm.txt min_free_kbytes: --------------- This is used to force the Linux VM to keep a minimum number of kilobytes free. The VM uses this number to compute a watermark[WMARK_MIN] value for each lowmem zone in the system. Each lowmem zone gets a number of reserved free pages based proportionally on its size. Some minimal amount of memory is needed to satisfy PF_MEMALLOC allocations; if you set this to lower than 1024KB, your system will become subtly broken, and prone to deadlock under high loads. Setting this too high will OOM your machine instantly. And the watermarks are setup base on this kernel parameter: ---------------------------------------------------------- mm/page_alloc.c init_per_zone_wmark_min() Conclusion: *************************************************************************** * The default setting is calculated based on your total system memory size. * I think min watermark = 53464kB for 16GB is reasonable. *************************************************************************** 2. Why system has no enough memory ================================== from compute-1_20190507.124154/var/log/kern.log [1] 2019-05-06T16:24:12.266 localhost kernel: debug [ 0.000000] On node 0 totalpages: 4174118 ... 2019-05-06T17:28:56.749 compute-1 kernel: info [ 1515.471986] Node 0 hugepages_total=1 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB 2019-05-06T17:28:56.749 compute-1 kernel: info [ 1515.471987] Node 0 hugepages_total=6807 hugepages_free=6807 hugepages_surp=0 hugepages_size=2048kB ... from hieradata/192.168.204.77.yaml [1]: platform::compute::hugepage::params::vm_2M_pages: '"7024,7172"' ... platform::compute::params::worker_base_reserved: ("node0:8000MB:1" "node1:2000MB:1") from puppet.log [0] ... Exec[Allocate 7024 /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages] ... Exec[Allocate 7172 /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages] ... The total memory on node 0 is 16GB; >From kenrel log, 2M hugepage is 6807 which is smaller than 7024. It looks system has no 7024 2M pages But total expected hugepage size is: 7024*2M + 1G = 14.7GB It is not reasonable. We have several reserved memory resources as below: 1. 8GB reserved by worker_reserved.conf: WORKER_BASE_RESERVED=("node0:8000MB:1" "node1:2000MB:1") 2. 10% reserved by below code: sysinv host.py: vm_hugepages_nr_2M = int(m.vm_hugepages_possible_2M * 0.9) After code review, it looks the hugepage allocation check code has a problem. [2] It only check whether the total allocation memory is bigger than total node memory size. If the pending hugepage size is between max possible size and node total size, the check will be passed. By default, if no pending hugepage request, _update_huge_pages() will allocate (m.vm_hugepages_possible_2M*0.9) 2M hugepage. But vm_hugepages_nr_2M_pending will be used with priority. If user config 2M hugepage manually with a wrong size, this issue will be triggered. Then, the hugepage allocation size will be overflow. The normal 4K pages will not be enough. So the OOM min watermark will be hit. A patch had been submitted for review. [2] [0] https://bugs.launchpad.net/starlingx/+bug/1827258/ [1] https://bugs.launchpad.net/starlingx/+bug/1827258/+attachment/5262103/+file… [2] https://review.opendev.org/#/c/667811/1/sysinv/sysinv/sysinv/sysinv/api/con…

1 0

[Starlingx-discuss] STX 1.0 automatically shutdown and power off.
by Ezpeer Chen 27 Jun '19

27 Jun '19

Dear all, My STX 1.0 will be automatically shutdown and power off. Where could i check the logs about this issue? Thanks a lot.

3 12

[Starlingx-discuss] [Containers] Sanity Test - ISO 20190626
by Perez Ibarra, Maria G 27 Jun '19

27 Jun '19

Status of the Sanity Test for last CENGN ISO: bootimage.iso from 2019-Jun-26 (link<http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190626T01…>) Status: Green ====================== Bare Metal environment ====================== AIO - Simplex: Setup 03 TCs Provision-Containers 01 TCs Sanity-OpenStack 49 TCs Sanity-Platform 11 TCs ------------------------------ TOTAL: 64 TCs AIO - Duplex: Setup 03 TCs Provision-Containers 01 TCs Sanity-OpenStack 52 TCs Sanity-Platform 09 TCs ------------------------------ TOTAL: 65 TCs Standard - Local Storage (2+2): Setup 03 TCs Provision-Containers 01 TCs Sanity-OpenStack 52 TCs Sanity-Platform 09 TCs ------------------------------ TOTAL: 65 TCs Standard - External Storage (2+2+2): Setup 03 TCs Provision-Containers 01 TCs Sanity-OpenStack 52 TCs Sanity-Platform 05 TCs ------------------------------ TOTAL: 61 TCs =================== Virtual Environment =================== AIO - Simplex Setup 03 TCs Provisioning 01 TCs Sanity OpenStack 49 TCs Sanity Platform 07 TCs ------------------------------ TOTAL: 60 TCs AIO - Duplex Setup 03 TCs Provisioning 01 TCs Sanity OpenStack 51 TCs Sanity Platform 05 TCs ------------------------------ TOTAL: 61 TCs Standard - Local Storage (2+2): Setup 03 TCs Provisioning 01 TCs Sanity OpenStack 52 TCs Sanity Platform 05 TCs ------------------------------ TOTAL: 61 TCs Regards Maria G.

1 0

[Starlingx-discuss] [docs][meetings] docs team meeting minutes 6/26/2019
by Tullis, Michael L 27 Jun '19

27 Jun '19

Greg Waines (copied) was nominated as a core reviewer for docs, and the team is in full agreement, ratifying this nomination. Congratulations Greg! For other notes and new action items from our docs team meeting today, see our etherpad: https://etherpad.openstack.org/p/stx-documentation Join us if you have interest in StarlingX docs! We meet Wednesdays, and call logistics are here: https://wiki.openstack.org/wiki/Starlingx/Meetings. -- Mike

1 0

[Starlingx-discuss] Wiki update
by Jones, Bruce E 27 Jun '19

27 Jun '19

The new wiki home page is live! brucej

1 0

[Starlingx-discuss] CFP Deadline for Open Infrastructure Summit Shanghai
by Jimmy McArthur 27 Jun '19

27 Jun '19

Hi Everyone! The July 2 deadline to submit a presentation [1] for the Open Infrastructure Summit [2] in Shanghai is in less than one week! Submit your session today and join the global community in Shanghai, November 4-6, 2019. Sessions will be presented in both Mandarin and English, so you may submit your presentation in either language. Submit your presentations, panels, and hands-on workshops [3] before July 2 at 11:59 pm PT (July 3, 2019 at 15:00 China Standard Time). Tracks [4]: Container Infrastructure Hands-on Workshops AI, Machine Learning & HPC Private & Hybrid Cloud Public Cloud 5G, NFV & Edge Open Development Getting Started CI/CD Security Upcoming Shanghai Summit Deadlines * Register now [5] before the early bird registration deadline in early August (USD or RMB options available) * Apply for Travel Support [6] before August 8. For more information on the Travel Support Program, go here [7]. * Interested in sponsoring the Summit? [8]. * The content submission process for the Forum and Project Teams Gathering will be managed separately in the upcoming months. We look forward to your submissions! Cheers, Jimmy [1] https://cfp.openstack.org/ [2] https://www.openstack.org/summit/shanghai-2019/ [3] https://cfp.openstack.org/ [4] https://www.openstack.org/summit/shanghai-2019/summit-categories/ [5] https://www.openstack.org/summit/shanghai-2019/ [6] https://openstackfoundation.formstack.com/forms/travelsupportshanghai [7] https://www.openstack.org/summit/shanghai-2019/travel/ [8] https://www.openstack.org/summit/shanghai-2019/sponsors/

1 0

Re: [Starlingx-discuss] Community Call (June 26, 2019)
by Zvonar, Bill 27 Jun '19

27 Jun '19

Notes & actions from today's call... MS-3 is official! - Thank you everyone! - Reminder to clean up storyboard: https://storyboard.openstack.org/#!/story/list?status=active&project_group_… - reminder of code merge guidelines until RC1. This was discussed and sent out previously. - Priority goes to stx.2.0 bug fixes and stx.2.0 MS-3 exceptions (above) - For code unrelated to stx.2.0 (code for deferred items to stx.3.0, new stx.3.0 features, enhancements), only passive/disabled code should be merged in master until the stx.2.0 RC1 branch is created (Aug 5). We will leave it to the judgment of the technical leads and core reviewers to determine if code is safe to merge. If you need an opinion, the release planning team is happy to help (Ghada, Bill and Bruce). Feedback on the new wiki has been positive (Bruce). - Any objections to me making https://wiki.openstack.org/wiki/StarlingX/Draft_new_wiki_home_page the new wiki home page? - we agreed to go ahead with the new wiki, Bruce will update it later today, and include a link to the old wiki off the new page regression status - http://lists.starlingx.io/pipermail/starlingx-discuss/2019-June/005094.html - the 115 testcases that were run were all run manually - regression reports will be sent twice a week - Tuesdays & Thursdays - per Numan, we've started automated regression as well - that'll be an additional 1100+ testcases, over & above the 421 manual that Maria's reporting on - ACTION: Numan & Ada to sort out how they aggregate reporting will be done (manual & automated) automation framework (Numan) - thanks to Bruce, Saul & Erich et al for reviewing the automation framework that Yang is making available to the Community - once it's up, folks can start using it - it includes a good number of testcases - ACTION: Numan/Yang arrange an info session for the Community (in a few weeks after Yang's vacation) defect trend (Bill) - marginally better than last week, but still a major cause of concern Docker image list: http://lists.starlingx.io/pipermail/starlingx-discuss/2019-June/005063.html - per Brent / Frank, Al Bailey's going to work on this - he has an action to get a forecast for when it can be done - ACTION: Frank update on the forecast for this stx.3.0 next steps (Ghada) - Tentative Release Dates show MS-1 as July 17 -- based on a 6wk offset from the openstack train release - https://docs.google.com/spreadsheets/d/1ZFR-9-riwhIwiBYBmWmi1qOVyHtgJ8Xk_Fa… - MS-1 criteria - https://wiki.openstack.org/wiki/StarlingX/Release_Plan#Release_Milestones - Release priorities and major features defined. - a list of features - TSC is working through an overall list (etherpad) - the list for 3.0 will ultimately be tracked on the Release Plan google sheet - High level resourcing secured. - an intended resource plan for each feature that's on the list - we agreed to remove the word "tentative" from the 3.0 dates - those are the dates now - Release Candidates: https://etherpad.openstack.org/p/stx-r3-feature-candidates - So far, 4 features are marked as "Agreed for R3 / Do in R3": - Containerization Deferrals: - Backup and Restore - Containerized FM - Containerized openstack clients - CPU manager static policy for non-openstack worker nodes - Distributed Cloud - Upgrade from R2->R3 - Openstack Train Release Support - Python 2->3 cutover - Some specs are in progress: https://review.opendev.org/#/q/project:starlingx/specs+status:ope- - Containerized Ceph - Time Sensitive Networking - Infrastructure and Cluster Monitoring - OVS-DPDK Containerization - Redfish (coming soon) - Other stuff in progress - MultiOS w/ focus on OpenSuse - K8s device plugin integration - Intel GPU >> initial review underway. - Nvidia GPU - Intel FPGA >> investigation starting / spec to follow. - QAT - 17 stories in Storyboard: https://storyboard.openstack.org/#!/story/list?status=active&project_group_… - This covers some distro.other patch reduction work items (deferred from stx.2.0) - How do we move the planning forward? Can we achieve MS-1 on July 17? - we agreed to stay the course - look through the candidate list in the next TSC meeting first contact (Bill) - ACTION: Bill start checking if any 'new' people emails are going unresponded updates on actions - sanity - still not boring, but getting greener - we'll keep it in view - list of stx-sanity tagged lps: https://bugs.launchpad.net/starlingx/+bugs?field.tag=stx.sanity&orderby=-da… - big files - ACTION: Scott & Dean to talk about the mechanics of how this would work given a big honkin' storage somewhere - ACTION: Frank to talk to CENGN about getting sufficient space (pending any other parameters from Scott) - Mailing list post size (Brent) - Brent's had mailing list replies blocked due to size restrictions - per Dean, the limit is ~64k, he'll talk to Jeremy about making it higher - but it may be a global OpenStack limit - ACTION: Dean find out what our options for increasing this are - Mid-cycle Meeting? - (Saul) - possibly a mid-cycle meeting in September in Ottawa, align with some training sessions that are planned - ACTION: Bill check with Ian about the logistics/timing - stx.1.0 installation documentation (Cindy) - are we preserving the installation procedures for stx.1.0 as we work on the 2.0 - ACTION: Bruce (or Doc team) let us know if there is such a thing -----Original Message----- From: Zvonar, Bill Sent: Tuesday, June 25, 2019 8:45 AM To: 'starlingx-discuss(a)lists.starlingx.io' <starlingx-discuss(a)lists.starlingx.io> Subject: Community Call (June 26, 2019) Reminder of tomorrow's Community call, topics include... - MS-3 officially declared - policy for changes going forward - bug count / resolution forecast - updated wiki - first contact Please feel free to add topics to the agenda at [0]. Bill... [0] etherpad: https://etherpad.openstack.org/p/stx-status [1] call details: https://wiki.openstack.org/wiki/Starlingx/Meetings#7am_PDT_.2F_1400_UTC_-_C… [2] meeting start time in various time-zones: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20190626T1400

1 0

[Starlingx-discuss] Notes: Weekly StarlingX non-OpenStack distro meeting, 6/26
by Xie, Cindy 26 Jun '19

26 Jun '19

Agenda for 6/26 meeting: - stx.3.0 feature proposal (All) - Redfish support (https://storyboard.openstack.org/#!/story/2005861) - Python2to3 transition (https://wiki.openstack.org/wiki/StarlingX/Python2, story to be created) - non-Openstack patch cleanup (TBD) - Ceph containerization (https://storyboard.openstack.org/#!/story/2005527, spec under review: https://review.opendev.org/#/c/656371/) sizing the effort before we commit the feature into 3.0 timeline. Propose staging option. - Support Kata container (story to be created), not sure yet if this fits into 3.0. first step is to propose TSC and initiate the discussion. Best to put the discussion/proposal onto the mailing list. - QAT support in Cinder & Glance (story to be created). AR to Cindy to discussion w/ Vivian for details. - systemd standardization. Saul: found from multi-OS effort, systemd was used to launch services, there is some services used sysInv instead of systemd. Needs to standarization to use systemd and move away from hybrid mode.AR to Saul and Marcela to send the technical proposal to mailing list and create SB. Each team members can bring up your proposed work items so that we can bring up to TSC for discussion & approval. - Ceph test status report (Abraham/Fernando) test status tracking: https://docs.google.com/spreadsheets/d/15us6HWgcb0dmHHZe2SyOR_UI5BvVpoU8TCS… 16 pass, 6 blocked, 3 retest 2 patches to address task 30351 under SB#2003909 have been merged yesterday. 4 P1 tests shall be unblocked. AR: Abraham/Fernando to test the 4 P1 cases with the latest build. - QAT test status report (Ricardo) test status tracking: https://docs.google.com/spreadsheets/d/15us6HWgcb0dmHHZe2SyOR_UI5BvVpoU8TCS… Moving to embedded QAT devices but not yet replicate the success on PCIe card yet. Will continue to preopare the HW and continue the testing. Already have instructions from Shuicheng, expect to finish the setup by this week. RESTAPI disable/enable - pending Numan's advice for howto. - stx.2.0 bug triage and review (Tingjie/Ovidiu/Daniel/Martin; Bin) - 1829844 : patch available, under review - 1829855 , patch WIP will upload today. - 1830191 , coding done, working on dev testing will post for review tomorrow. - 1831300 , WIP and need retest. One problem is that we are not on latest mimic version. -1830938, maybe an upstream Ceph issue, cherry pick might be required. -1832854, under debug, root cause not clear yet. -1827258 /1832647, kernel log checked, when OOM found the system allocated >14G huge page memory but the whole system memory is only 16G. From Bin's analysis, there are 7000 huge pages (each huge page size is 2M).When OOM happens, the system was attempt to allocate 4K pages but failed. From Brent, the system reserve 14.5G for system usage, only if all those 14.5G are used up, then OOM will happen. - Opens (all) -----Original Message----- From: Xie, Cindy Sent: Tuesday, June 25, 2019 9:50 PM To: starlingx-discuss(a)lists.starlingx.io; Rowsell, Brent <Brent.Rowsell(a)windriver.com>; Wold, Saul <saul.wold(a)intel.com> Subject: RE: Weekly StarlingX non-OpenStack distro meeting Agenda for 6/26 meeting: - stx.3.0 feature proposal (All) - Ceph test status report (Abraham/Fernando) - QAT test status report (Ricardo) - stx.2.0 bug triage and review (Tingjie/Ovidiu/Daniel/Martin; Bin) - Opens (all) -----Original Appointment----- From: Xie, Cindy Sent: Thursday, April 25, 2019 5:42 PM To: Xie, Cindy; 'zhaos'; 'starlingx-discuss(a)lists.starlingx.io'; 'Rowsell, Brent'; Wold, Saul Cc: Hu, Wei W; Jones, Bruce E; 'Eslimi, Dariush'; Cobbley, David A; 'Zhi Zhi2 Chang'; Armstrong, Robert H; 'Waines, Greg'; 'Badea, Daniel'; 'Carlos Cebrian'; 'Seiler, Glenn'; Chen, Tingjie; 'Chen, Jacky'; Komiyama, Takeo; Gomez, Juan P; Peng Tan Subject: Weekly StarlingX non-OpenStack distro meeting When: Wednesday, June 26, 2019 9:00 PM-10:00 PM (UTC+08:00) Beijing, Chongqing, Hong Kong, Urumqi. Where: https://zoom.us/j/342730236 . Cadence and time slot: o Wednesday 9AM Winter EDT (10PM China time, US PDT Winter time 6AM) . Call Details: o Zoom link: https://zoom.us/j/342730236 o Dialing in from phone: o Dial(for higher quality, dial a number based on your current location): US: +1 669 900 6833 or +1 646 876 9923 o Meeting ID: 342 730 236 o International numbers available: https://zoom.us/u/ed95sU7aQ . Meeting Agenda and Minutes: o https://etherpad.openstack.org/p/stx-distro-other

1 0