[Starlingx-discuss] MultiOS: Integration of OpenSUSE build system (OBS) in StarlingX CI process

[Dominig ar Foll (Intel Open Source)]

Hello,

having received some requests for information, about the best way to
integrate the OpenSUSE Build System (OBS) in StartlingX CI system, I am
happy to share my finding.
My explanations are written using osc command lines, in order to ease
the checking of the described concepts, but obviously a CI integration
would be easier/safer made using the OBS API equivalences. For more
detail son the OBS API see:
   https://build.opensuse.org/apidocs/index

A) Testing the change submission
============================

1) Test via Local build
-----------------------------------
It is obviously possible for the CI, to create a local checkout  (osc co
ProjectName PackageName) of the project/package on which the change
submission is targetted, update the _service to point toward the change
submission to be tested, on git source repo.
Then follow by a local build for one or multiple distro target (repository).
  osc checkout --output-dir=CIlocalTestDir  ProjectName PackageName
  # copy a _service file pointing to the change submission to be tested. 
  osc build --trust-all-projects Repository
It would require to parse the osc build output to extract the final
status and potential a failure cause.
The local build present the advantage to be potentially quicker as we
would control the power allocated to the best build as well as the
scheduling but requires network and CPU resources allocation.

2) Test via build in public OBS
------------------------------------------------
In that model, we want to create a branch of the project to test our
change submission.
The osc branch command does not well fit the requirement, because it
immediately starts a build after it creation, what would slow the
process a lot. We want to change the source download origin, before
starting any build process.
So I propose to create a 'branch' manually to overcome that current OBS
limitation.

We need to create a project that will have an identical configuration to
the final target project.
To do so, we need to copy the meta prj and prjconf as well as creating a
test build project.
   # Extracting the target project configs
   osc meta prj TargetProjectName  >TargetProjectName.prj
   osc meta prjconf TargetProjectName >TargetProjectName.prjconf

   # Create the test project on OBS with an easy to track name
   osc meta prj -F TargetProjectName.prj TargetProjectName-submissionXYZ
   # Copy target prjconf in the newly created test project
   osc meta prjconf -F TargetProjectName.prjconf
TargetProjectName-submissionXYZpraget

   # Clean local tmp tiles
   rm -f TargetProjectName.prj TargetProjectName.prjconf

  # Installing the _service pointing to the commit submission
  # API based implementation is highly advise to better manage possible
errors
  osc co TargetProjectName-submissionXYZ
  # install the _service pointing to the commit submission
  osc ci -m "test build for submissionXYZ"
  # spec files and rpmlint files will soon direclty been download by the
_service. I assume that to be already the case in that process description.
 
  # getting build results (xml is easier to parse but other return
forTargetProjectName-submissionXYZmats are available if desired)
  # Note: my test shows the -w option is not that reliable and retries
handling would required.
  osc results  -w --xml
TargetProjectName-submissionXYZhttps://rabbit.opensuse.org/

  # Alternative method
  # OBS provides an event channel which provides many valuable
information for an integration activity.
     https://rabbit.opensuse.org/
  # list of all available events
     
https://openbuildservice.org/help/manuals/obs-admin-guide/obs.cha.administration.html#_message_bus

3) case of update via new patchset
----------------------------------------------------------
It's not uncommon, that a new patchset is created during the review
process before acceptance. This can be done before the finalisation of
any validation of the test build. In that case we would need to abort
the current build and restart a new one to speed up the process.
  # Install an updated _service pointing to the new patch set (depending
of git config that might not be required).
  # trigger an abort rebuild.
   osc restartbuild TargetProjectName-submissionXYZ
  # restart the build result monitoring process.

4) Using a local OBS
----------------------------------
Creating a local OBS linked to the public OBS is quite easy and likely a
better method that doing local build in the CI, would the control of the
speed of validation, via the public OBS, be an issue .

B) Accepting the submission
=======================
The process would be the same whatever is the test build method (local,
public/private OBS).
Once the submission is accepted you just need to retrigger a remote
service run on the public OBS.
This would download the source, spec and rpmlint from the git repo,
trigger a new build of the target package as well as all the depending
packages and publish results in public repo upon completion.

We can see here, one of the weakness of the proposed process. A change
in a package may be acceptable for that package but may break dependant
packages.
There is a model in OBS to tackle that issue via the use of staging
project, but it would slow down the acceptance of all change
submissions, what might be an overkill. Up the community to decide what
is the best compromise. I would favour to stick to single package check,
at least,  in a first step.

Once the submission request has been accepted, remember to delete all
the temporary projects where ever they are located. It is good idea to
keep them present until acceptance, as OBS as well as osc use caches
internally.  

Enjoy your day.

-- 
Dominig ar Foll
Senior Software Architect
Intel Open Source Technology Centre