[Starlingx-discuss] Proposal: Build and Multi-OS strategy

Fri Feb 15 16:46:10 UTC 2019

Hi all.

I've been gathering thoughs on how to achieve multi OS since the
beginning of the project and this is a good oppportunity to drop all of
them here :)

~~~~~~~~~

I think multi OS is a challenge with many faces that can be summarized
in two high level areas: building and running.

Building
========

Currently, developers plays a mixed role, they are developing and
packaging software. They have defined a workflow where in addition of
creating new features, fixing bugs and other tasks, they need to be
aware of the build system that consumes their work. This model is
similar to Yocto's (which I think was used in the past) where
developers needs to create/modify recipes to get their changes checked
in. This model ain't bad but I think it could be incomplatible with a
multi OS approach. It could enforce that developers needs to be aware
of additional build layers and add an extra effort everytime they want
to make changes.

If you take any distribution out there, all of them have the same entry
point: a software released by upstream. Let's say there is a `abc`
project and they release the `abc-1.2.3` version packaged in a `abc-
1.2.3.tar.gz` file. Any distribution that wants to include `abc` takes
that packaged file and create a distribution package. The `abc` team
doesn't care on how distros package their software, they just care on
creating a good piece of software that is flexible enough to run well
on any distribution. Anytime `abc` releases a new version, they
increment the version number accordingly and the distribution packagers
includes the new version.

In my opinion, having a separation between build system and our
software components will help the project in many ways. From one side,
building for multiple OS is easier because the multi OS team can only
care about threating our software as upstream, building, packaging and
eventually creating bugs for upstream (as all distros does). In the
other side, the software component maintainers can only care about
robustness of their project, that can be tested, builded and installed.

The downside in this separation is that we need things we currently
don't have, like proper versioning and release tools (autotools and
friends). But also the mindset to start threating our software
components as any software project out there.

I believe some of this intentions has been captured in the ongoing
multi OS specs, but I want to highlight the importance of the role of
software maintainers in this model. Some workflows might need to change
(if we take this path), but we can find equivalence in a new scenario.

Once we have this separation, the multi OS building effort relies more
on packaging and creating bugs for upstream. The next of the challenges
are in the running part.

Running
=======

In my opinion running in multiple OS is a challenge even higher than
creating the RPM o DEB packages. At this point we don't have enough
information on how the system will behave in Ubuntu, I've seen hard
coded paths in the source code, some for executables and some other for
configuration files. Extending the example, in distros like Clear
Linux, the configuration files needs to be in a different path, this is
also true for distros under the Atomic Project[0] (This is just an
example I know they are out of the scope right now).

The point here is that there's a high level of uncertainty on how the
entire system will behave. I think there are a lot of questions, for
example, the configuration will work in Ubuntu? (there's a plan to use
Ansible that could help), the installation process can work the same?
does anaconda works the same way in Ubuntu? What refactors are needed
in stx-update to support DEBs? and other questions. 

The multi OS doesn't have (and I think it shouldn't) enough vision in
these topics. The software component maintainers are the best people to
offer insights into how their projects will work in a multi OS
environment. I we take the build system and source code separation
again, then the software projects should only care that the software
are flexible and can run well in different operating systems (as
upstream source projects does).

I believe we might need more involvement of the community to solve this
kind of questions or, at least, raise red flags that could help to plan
and act accordingly.

[0] https://www.projectatomic.io/

On Fri, 2019-02-15 at 00:05 +0000, Jones, Bruce E wrote:
> I would like to start a thread that I hope will result in more focus
> and direction for the Build and Multi-OS teams. It’s right before I
> disappear for vacation, so I’ll ask Saul and Cesar to address any
> follow-ups.
>  
> What I would like to propose is that we as a community take on the
> task of delivering support for Ubuntu as a StarlingX host OS for the
> November 2019 release.  This would allow us to support the ~35% of
> the cloud ecosystem that doesn’t run on RHEL or CentOS.
>  
> It will require a lot of work and therefore we should start as soon
> as possible.  What I would propose we do is:
> 1)      The Build team to create a new and separate build system for
> an Ubuntu LTS hosted ISO [0]
> 2)      The MultiOS team to review the outstanding carried patches
> and apply those needed to the Ubuntu packages
> 3)      The MultiOS team to update the system as needed to use an
> Ubuntu installer to get controller-0 fully installed
> 4)      Which would then lead to work in the MultiOS team to bring up
> the Ubuntu hosted StarlingX in Simplex mode [1].
>  
> We would then have a (kind of) working StarlingX image that will
> enable the broader community to contribute to all of the other work
> needed to deliver a fully supported Ubuntu host for the November
> release.  That work would include bringing up the other
> configurations beyond Simplex, changes to the StarlingX software
> management and update services, the additional testing needed, and
> other tasks which can be parallelized.
>  
> Meanwhile, on the Intel side we have received new guidance from our
> new management on the requirement for Clear Linux support.  We will
> continue but slow down that work for now and focus on Ubuntu.  So for
> November, the goal is to support 2 Host OS’s, not 3.
>  
> We will need support and contributions from the community to achieve
> this goal in time for November.  The MultiOS team in particular will
> need help and additional contribtutors.
>  
>    Brucej
>  
> [0] Making the build system common between Ubuntu and CentOS is hard
> and probably should not be attempted.  We should leverage what we
> can, of course, from our own code and the broader ecosystem.
> [1] Or in which ever configuration is easier….
>  
>  
> _______________________________________________
> Starlingx-discuss mailing list
> Starlingx-discuss at lists.starlingx.io
> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss