[Starlingx-discuss] [ Test ] Meeting notes - 07/23/2019

[Cabrales, Ada]

Agenda for 07/23
Attendees: Cristopher, Yong, Elio, Dominig, Fer, JC, Jose, JP, Maria P, Numan, Yang, Ada, Richo
1. Sanity status - Cristopher
   Create instance from Image or from Volume fails -  https://bugs.launchpad.net/starlingx/+bug/1837241
   The fix has been merged, it will be available on tonight's build, an engineering build will be tested on virtual.

   Platform integ state stays in 'uploaded' status - should be fixed in next ISO. 

   On intel side, when a suite fails on a stage, we re-run it. If it runs with manual execution, we don't report the issue. 
   However, we are seeing intermittent failures that looks like they are gone with second execution. Let's report all these. 
   These could be a systemic failure. 

   There's a field in the launchpad template for stating if this is an intermittent issue or not. Use it.
   Numan: please send the list of bugs found in sanity by WR.

      Info of intel tests sent - last Wednesday

2. Regression testing status stx.2.0 - Elio, Numan
   Total / Pass / Fail / blocked / Obsolete - 493 / 287 / 11 / 37 / 21 - 
   Pass rate: 96.3%
   Regression due date is next week (Aug 2nd)
   ~80% of manual execution done, including the storage tests. 
   Let's mark the ones with low risk and already passed as 2nd priority.
   2nd run of automated regression is about 5% done.
   Failures - we don't have a way to verify encrypted devices. 
   Talk with Chris - Numan will also talk with him.  
   SRIOV - PCI pass through - work with Richo for getting help. Make sure the hardware is enabled for SRIOV - this is working for WR.
   Still investigating about NUMA tests
   IPv6 will be run for basic functionality. A lab will be also set in WR for testing.
    
   75 tests remaining for execution. By end of this week we will know if help is required or not. 
   Automated regression report sent by email - Ada to consolidate.
   Looks better than last run - ~81% of pass rate.

   Ada to build a full report for tests/failures/manual/automated 

3. Pytest framework committed - Ada
   Code is merged. Documentation in the wiki is missing. 
   Readme is already in rst. 
   Ada to talk with Abraham in order to get direction - 
   Jose states the wiki can be updated automatically - let's find out with Abraham. 
   Another way of doing it will be putting the link to the documentation into the readme file.
   Or add the info into the documentation and that's the one to be copied to the wiki. 
   For bugs / changes submit launchpads using the stx.testauto.

4. Robot framework - plan for having it outside? - Jose
   Code is being cleaned-up. 
   Documentation is also being completed.
   Plan is to have the code by first week of August. Process will be similar to what we did for the pytest FW. 

5. Pending actions:
   - Inclusion of report of automated tests run into the regression report
   - Dashboard for testing results
   -  Shared space for files (google drive might not the best option)
   - Project for consolidating the sanity run on both sides (WR and Intel)

6. Opens
   Yong - high launchpad - nova placement patch has been sent. Sent the info by email to schedule it.
   This requires a specific hardware (specific NICs) - Richo to review and schedule running it based on the hardware availability.

   Numan to JC - re-send the email with the sanity tests cases. 


Regards
Ada