Hi,
To have a better understanding process of the CI/CD setup on Arm platform, I summarize and come up with the following simple topology diagram. With action items list below. Welcome any comments and suggestions.
The proposed topology
Work steps
Current status
I am trying step 1, build ISO image with Jackie's porting patch series in our internal lab to understand the build process.
The previous build packages error is because I missed some patches from Jackie. I still need to figure out an error during generating ISO image with Jackie.
The auto test part in step 2 still needs some input.
What is test repo used? What is the minimal test set? Any quick get started guide?
Thanks,
Tianyu
+ Steve in the loop.
Hi Steve,
As you mentioned that Davlet or Scott may be on vacation, would you please help to see who could answer Tianyu’s question regarding the test cases?
Thanks
Chris
From: Tianyu Li <Tianyu.Li@arm.com>
Sent: Wednesday, August 30, 2023 3:32 PM
To: Huang, Jie (Jackie) <Jackie.Huang@windriver.com>; Little, Scott <Scott.Little@windriver.com>; Panech, Davlet <Davlet.Panech@windriver.com>
Cc: starlingx-discuss@lists.starlingx.io; Scott Kamp <scott.kamp@xunison.com>; Chris Wang <Chris.Wang@arm.com>; Lijian Zhang <Lijian.Zhang@arm.com>; Geary, Stephen (Steve) <Steve.Geary@windriver.com>; Wang, Linda <Linda.Wang@windriver.com>; nd <nd@arm.com>
Subject: Re: StarlingX CI/CD setup on Arm platform
Thanks Davlet and Scott for information,
I tried to setup the Jenkins pipeline jobs on local Arm server, after applied Jackie's Arm porting patches[0] with environment variable PATCH_LIST, the build job passed some stage and failed at 'download-prerequisites' with following log[1], looks like the downloader still trying to download some x86 packages. Any idea what could cause this?
I have another general question regarding testing stage in CI/CD process,
What is the minimum automation test case should be done after image is done?
If Davlet and Scott are not familiar with this area, who can help on this?
FYI, this is our plan to support StartingX CI/CD on Arm.
[1] downloaderERROR "The cache has no package named 'gcc-alpha-linux-gnu'"
Thanks,
Tianyu
From: Huang, Jie (Jackie) <Jackie.Huang@windriver.com>
Sent: Thursday, August 24, 2023 8:57 PM
To: Huang, Jie (Jackie) <Jackie.Huang@windriver.com>; Little, Scott <Scott.Little@windriver.com>; Panech, Davlet <Davlet.Panech@windriver.com>;
Tianyu Li <Tianyu.Li@arm.com>
Cc: starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io>; Scott Kamp <scott.kamp@xunison.com>;
Trevor Tao <Trevor.Tao@arm.com>; Geary, Stephen (Steve) <Steve.Geary@windriver.com>; Wang, Linda <Linda.Wang@windriver.com>
Subject: RE: StarlingX CI/CD setup on Arm platform
Hi Scott / Davlet,
As we discussed on the OS Distro call, a local CENGEN like server is needed for the local CI/CD setup,
Could you please share the info of how to setup a CENGEN like server, specifically how to prepare the
package mirror, we need to get it setup and prepare the mirror for ARM deb packages, and for the
LAT-SDK for ARM as well. Thanks!
Thanks,
Jackie
From: Huang, Jie (Jackie) <Jackie.Huang@windriver.com>
Sent: Friday, August 18, 2023 1:33 PM
To: Little, Scott <Scott.Little@windriver.com>; Panech, Davlet <Davlet.Panech@windriver.com>; Tianyu Li <Tianyu.Li@arm.com>
Cc: starlingx-discuss@lists.starlingx.io; Scott Kamp <scott.kamp@xunison.com>; Trevor Tao <Trevor.Tao@arm.com>;
Geary, Stephen (Steve) <Steve.Geary@windriver.com>; Wang, Linda <Linda.Wang@windriver.com>
Subject: Re: [Starlingx-discuss] StarlingX CI/CD setup on Arm platform
Hi Scott,
Thanks for sharing, it’s very helpful.
Thanks,
Jackie
From: Little, Scott <Scott.Little@windriver.com>
Sent: Thursday, August 17, 2023 11:48 PM
To: Huang, Jie (Jackie) <Jackie.Huang@windriver.com>; Panech, Davlet <Davlet.Panech@windriver.com>; Tianyu Li <Tianyu.Li@arm.com>
Cc: starlingx-discuss@lists.starlingx.io; Scott Kamp <scott.kamp@xunison.com>; Trevor Tao <Trevor.Tao@arm.com>;
Geary, Stephen (Steve) <Steve.Geary@windriver.com>; Wang, Linda <Linda.Wang@windriver.com>
Subject: Re: StarlingX CI/CD setup on Arm platform
There is also ...
Which contains the current state of the StarlingX jenkins setup, less a few credentials.
It includes 'Debian_build_server.txt' with crude documentation of how the server was setup.
Scott
On 2023-08-16 21:04, Huang, Jie (Jackie) wrote:
Hi Davlet,
Thanks for the detailed answers, which are very helpful, Tianyu and I will start by familiarizing ourselves and check if there are any more questions.
Thanks,
Jackie
From: Panech, Davlet <Davlet.Panech@windriver.com>
Sent: Wednesday, August 16, 2023 11:30 PM
To: Huang, Jie (Jackie) <Jackie.Huang@windriver.com>; Little, Scott <Scott.Little@windriver.com>; Tianyu Li <Tianyu.Li@arm.com>; Wang, Linda <Linda.Wang@windriver.com>
Cc: starlingx-discuss@lists.starlingx.io; Scott Kamp <scott.kamp@xunison.com>; Trevor Tao <Trevor.Tao@arm.com>; Geary, Stephen (Steve) <Steve.Geary@windriver.com>
Subject: Re: StarlingX CI/CD setup on Arm platform
Hi Jackie,
CI/CD can be thought of as 2 somewhat disconnected parts - build & automated testing. I know little about test automation, the current build set up is outlined below.
You would need a dedicated server:
1. Linux : we currently use Debian 11, but it shouldn't matter since most build steps run in docker containers
2. Docker
3. Kubernetes
4. Jenkins: integrated with k8s and able to run k8s pipelines
5. git
6. git-repo https://gerrit.googlesource.com/git-repo
Other prerequisites:
1. You would need some DockerHub account to avoid DockerHub's rate limits when pulling images
2. Once your ARM set up is ready we can replace that with a controlled account that we manage in order to push the images to Docker Hub.
Jenkins job definitions (Jenkinsfiles) are source-controlled here: https://opendev.org/starlingx/jenkins-pipelines . You would need to define Jenkins jobs manually in the GUI, and point them to the corresponding Jenkinsfiles in this git repo.
1. The repo itself acts as a pipeline library in Jenkins with shared functions under vars/ . The library must be defined at Jenkins' folder level.
2. pipelines/monolithic.Jenkinsfile : top-level job that calls other jobs under parts/
3. pipelines/parts/ : directory containing sub-job Jenkinsfiles. You would create a Jenkins folder in the GUI called "parts" and define individual jobs pointing to individual Jenkinsfiles from this subdirectory in git repo
4. scripts/ : shell scripts that implement individual build steps (called by Jenkinsfiles)
Required directories on the server, owned by jenkins user:
1. /localdisk/designer/jenkins/ root for per-project build areas
2. /localdisk/loadbuild/jenkins/ root for per-project build archives
At first execution the monolithic job will fail, but create an example configuration file in the project build area that you are expected to copy & edit:
1. /localdisk/designer/jenkins/$PROJECT/build.cond.example
There are many options with comments in it; the example file is also source controlled under scripts/templates/build.conf.example.in
Builds are slow and would greatly benefit from fast storage (SSDs etc).
Please start by familiarizing yourself with this repository and ping me if you have any more questions.
Hope this helps,
D.
From: Huang, Jie (Jackie) <Jackie.Huang@windriver.com>
Sent: August 9, 2023 10:28 AM
To: Little, Scott <Scott.Little@windriver.com>; Panech, Davlet <Davlet.Panech@windriver.com>; Tianyu Li <Tianyu.Li@arm.com>; Wang, Linda <Linda.Wang@windriver.com>
Cc: starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io>; Scott Kamp <scott.kamp@xunison.com>; Trevor Tao <Trevor.Tao@arm.com>; Geary, Stephen (Steve) <Steve.Geary@windriver.com>
Subject: RE: StarlingX CI/CD setup on Arm platform
Hi Scott and Davlet,
As mentioned in the Multi-OS call, I’m re-sending this with all the questions about StarlingX CI/CD, please help answer as many as you can, thanks!
There would be 6 Arm servers contributed to our community for CI/CD (2 for build, 4 for testing of different deployment configs), what we want to know is:
1. What’s the current status of StarlingX CICD for x86-64?
2. What needs to be done for the CICD workflow setup and how after the 6 ARM servers are ready and hosted in the lab?
a. E.g. how to setup build servers and testing env?
b. What CICD tools need to be setup? Zuul/Jenkins/etc.
c. Are there any docs or info of the steps for these setups?
3. How many efforts and how long will it take?
4. Can this only be done by specific person? Or anyone have permission to access to the servers? If the later, I think Tianyu from Arm team and I can also help
on that if we got enough information.
Tianyu, please add if you have any other questions.
Thanks,
Jackie
From: Huang, Jie (Jackie)
Sent: Monday, July 31, 2023 12:57 PM
To: Trevor Tao <Trevor.Tao@arm.com>; Geary, Stephen (Steve) <Steve.Geary@windriver.com>; Little, Scott <Scott.Little@windriver.com>; Scott Kamp <scott.kamp@xunison.com>
Cc: starlingx-discuss@lists.starlingx.io
Subject: RE: StarlingX CI/CD setup on Arm platform
Hi, Scott Kamp and Scott Little,
I found the following actions in the meeting minutes in stx-status | OpenDev Etherpad:
o ARM Support
§ Action: Scott Kamp to respond to Jackie/the mailing list to provide assistance/access to some arm machines
§ Action: Scott Little to respond regarding build questions
· Cannot mirror more files on CENGN; will need to wait until we transition away from CENGN
· Action: Scott Kamp to explore providing a hosting env temporarily
So I think you might be able to help on Trevor’s and my questions:
There would be 6 Arm servers contributed to our community for CI/CD (2 for build, 4 for testing of different deployment configs), what we want to know is:
1. What needs to be done for the CICD workflow setup and how?
a. E.g. how to setup build servers and testing env?
b. What CICD tools need to be setup? Zuul/Jenkins/etc.
c. Are there any docs for these setups?
2. How many efforts and how long will it take?
3. Is there a CICD team or someone that can help on the setup or just providing useful information?
Thanks,
Jackie
From: Trevor Tao <Trevor.Tao@arm.com>
Sent: Thursday, July 27, 2023 5:06 PM
To: Geary, Stephen (Steve) <Steve.Geary@windriver.com>; starlingx-discuss@lists.starlingx.io
Cc: Huang, Jie (Jackie) <Jackie.Huang@windriver.com>
Subject: StarlingX CI/CD setup on Arm platform
Hi Steve, StarlingX:
From the OS distro meeting yesterday, we knew that there would be 6 Arm servers contributed to our community to do the CI/CD related work.
So we would like to ask who can help on the setup of CI/CD environment on the arm platform, and any other useful information/details for the CI/CD issue here.
Thanks,
Best Regards,
Zijin Tao(Trevor Tao, 陶孜谨)
ARM Electronic Technology (Shanghai) Co., Ltd
安谋电子科技(上海)有限公司
Building 11, Shanghai Busininess ParkⅢ ,
No.1016 Tianlin Rd, Minhang District, Shanghai, 200233 China
上海市闵行区田林路1016号科技绿洲三期2号楼10楼,200233
Cell: +86-153 7109 6192
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.