[Starlingx-discuss] [Debian Build]: Layout of the debian folder
Hello Everyone: You may have known Wind River team is working on the transition to Debian OS, more details of the project can be found at https://docs.starlingx.io/specs/specs/stx-6.0/. My purpose is launching a discussion about the userspace packages transition to Debian. The spec can be found at https://docs.starlingx.io/specs/specs/stx-6.0/approved/starlingx_2008704_deb... In order to inherit the existing userspace construction as much as possible, our proposal is creating a 'debian' folder in same directory with 'centos' folder for each package. For example, the dhcp package in integ repo, it's centos folder is at: https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/centos We will create a debian folder in same location https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/debian. All materials in this folder control building dhcp under Debian host. The "stx_deb_folder_layout.rst" is the layout of debian folder. I also attach a couple of samples to demonstrate how to fill the debian folder for a debian package and a 3rd package. This layout is not the final version, I'm appreciated for any suggestion from you. Thanks, ytao
Hi, I had a look through the document and it looks okay. I just have a couple of comments/questions about the folder content: * **debver** - This is the package version of the debian package? This is generated from the debian/changelog when the package is built. I guess the problem is that you want to differentiate from a starlingx package and native package? My suggestion would be what the Ubuntu developers do. For example, if we have patches to apply for systemd, the debian version would be 247.3-3, while the StarlingX package would be 247.3-3stx0. Then a user or developer can quickly see that they are working with a native debian package or a starlingx modified package. * **deb_patches** - As a developer, if the package has its debian/patches I would rather use the debian/patches directory. If the directory doesnt exist I am more likely to create a debian/patches directory and then send a patch to debian to fix a problem. * **dl_path** - Debian already does this for you already, the debian/watch file keeps track of where to download a tarball, or a zip file, or a python wheel for you. You can use uscan(1), which is a part of devscripts package. Debian/Ubuntu developers have been doing this for years. (https://wiki.debian.org/debian/watch) If you have any questions please let me know. Regards chuck debian/watch - Debian Wiki<https://wiki.debian.org/debian/watch> debian/watch. The file named watch in the debian directory is used to check for newer versions of upstream software is available and to download it if necessary. The download itself will be performed with the uscan program from the devscripts package. It takes the path to the debian directory that uses the watch file as an argument or searches the directories underneath the current working ... wiki.debian.org ________________________________ From: ytao <yue.tao@windriver.com> Sent: July 26, 2021 3:33 AM To: starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io>; Asselstine, Mark <Mark.Asselstine@windriver.com> Subject: [Starlingx-discuss] [Debian Build]: Layout of the debian folder Hello Everyone: You may have known Wind River team is working on the transition to Debian OS, more details of the project can be found at https://docs.starlingx.io/specs/specs/stx-6.0/. My purpose is launching a discussion about the userspace packages transition to Debian. The spec can be found at https://docs.starlingx.io/specs/specs/stx-6.0/approved/starlingx_2008704_deb... In order to inherit the existing userspace construction as much as possible, our proposal is creating a 'debian' folder in same directory with 'centos' folder for each package. For example, the dhcp package in integ repo, it's centos folder is at: https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/centos We will create a debian folder in same location https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/debian. All materials in this folder control building dhcp under Debian host. The "stx_deb_folder_layout.rst" is the layout of debian folder. I also attach a couple of samples to demonstrate how to fill the debian folder for a debian package and a 3rd package. This layout is not the final version, I'm appreciated for any suggestion from you. Thanks, ytao
On 7/27/21 7:05 AM, Short, Charles wrote:
Hi,
First of all, I'm appreciated for your input.
I had a look through the document and it looks okay. I just have a couple of comments/questions about the folder content:
* **debver** - This is the package version of the debian package? This is generated from the debian/changelog when the package is built. I guess the problem is that you want to differentiate from a starlingx package and native package? My suggestion would be what the Ubuntu developers do. For example, if we have patches to apply for systemd, the debian version would be 247.3-3, while the StarlingX package would be 247.3-3stx0. Then a user or developer can quickly see that they are working with a native debian package or a starlingx modified package.
For a Debian native package, the *debver* is the debian version. We use it to download the src via "apt-source packagname=debver", for a starlingx package, it is up to the developer. I observed in CentOS version, they added a patch to change the package version, for example integ/base/lighttpd/centos/meta_patches/Update-package-versioning-for-TIS-format.patch, which appends a 'tis' to package version, that requests developer to update the version manually. After transiting to debian, OBS can achieve it automatically. It can append 'tis' or 'stx' to a package, so don't need developers to update it manually.
* **deb_patches** - As a developer, if the package has its debian/patches I would rather use the debian/patches directory. If the directory doesnt exist I am more likely to create a debian/patches directory and then send a patch to debian to fix a problem.
Exactly, we should send a fix to debian community and then uprev the package to get the fix, but we also need to consider local changes. The *patches" folder contains the local change for source codes, that will be copied to debian/patches (or create it if doesn't exist), and update the debian/patches/series. The *deb_patches* contains the patches to change debian folder, for example, we want to customize the installation of a package, we need to update override_dh_install stage in debian/rules. The patches will be applied to package/debian folder directly. The *deb_patches* is similar with centos/meta_patches The *patches* is similar with centos/patches
*
* **dl_path** - Debian already does this for you already, the debian/watch file keeps track of where to download a tarball, or a zip file, or a python wheel for you. You can use uscan(1), which is a part of devscripts package. Debian/Ubuntu developers have been doing this for years. (https://wiki.debian.org/debian/watch)
Thanks you for pointing me the debian/watch. I will take some time to learn about debian/watch and go back soon. It looks a debian file for monitoring the upstream version change. I am not sure if it is suitable for a no debian package. In our internal discussion, Davlet told me a special case, that the url of a tar ball likes this https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8... As Davlet's suggestion, the *dl_path* refers to example file <https://opendev.org/starlingx/integ/src/branch/master/centos_tarball-dl.lst> and the script that parses it <https://opendev.org/starlingx/tools/src/branch/master/centos-mirror-tools/dl_tarball.sh#L215>. I will do assessment for debian/watch. Thanks, ytao
If you have any questions please let me know.
Regards chuck
debian/watch - Debian Wiki <https://wiki.debian.org/debian/watch> debian/watch. The file named watch in the debian directory is used to check for newer versions of upstream software is available and to download it if necessary. The download itself will be performed with the uscan program from the devscripts package. It takes the path to the debian directory that uses the watch file as an argument or searches the directories underneath the current working ... wiki.debian.org
------------------------------------------------------------------------ *From:* ytao <yue.tao@windriver.com> *Sent:* July 26, 2021 3:33 AM *To:* starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io>; Asselstine, Mark <Mark.Asselstine@windriver.com> *Subject:* [Starlingx-discuss] [Debian Build]: Layout of the debian folder Hello Everyone:
You may have known Wind River team is working on the transition to Debian OS, more details of the project can be found at https://docs.starlingx.io/specs/specs/stx-6.0/. My purpose is launching a discussion about the userspace packages transition to Debian. The spec can be found at https://docs.starlingx.io/specs/specs/stx-6.0/approved/starlingx_2008704_deb...
In order to inherit the existing userspace construction as much as possible, our proposal is creating a 'debian' folder in same directory with 'centos' folder for each package. For example, the dhcp package in integ repo, it's centos folder is at:
https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/centos
We will create a debian folder in same location
https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/debian.
All materials in this folder control building dhcp under Debian host. The "stx_deb_folder_layout.rst" is the layout of debian folder. I also attach a couple of samples to demonstrate how to fill the debian folder for a debian package and a 3rd package.
This layout is not the final version, I'm appreciated for any suggestion from you.
Thanks,
ytao
Hi Charles: I did some investigation of debian/watch file, which is the input file for uscan command. It’s purpose is to detect if upstream has a newer release. I tried the uscan, which is limited to run with a full debian folder, not only a dedbian/watch. I also try to the option “—watchfile” to specify a watch file, but it still checks the debian/changelog file. $uscan –watchfile path/watch uscan die: Are you in the source code tree? Cannot find readable debian/changelog anywhere! at And the url in watch file must be a regex expression, for example, Allow http://ftp.isc.org/isc/dhcp/(\d.\d.\d*)/dhcp-(\d.*)\.(?:tgz|tbz2|txz|tar\.(?:gz|bz2|xz))<http://ftp.isc.org/isc/dhcp/(/d./d./d*)/dhcp-(/d.*)/.(?:tgz|tbz2|txz|tar\.(?:gz|bz2|xz))> Not allow http://ftp.isc.org/isc/dhcp/4.4.2/dhcp-4.4.2.tar.gz A developer only care about a special version, but he has to build a regex expression, of cause uscan has –download-version to specify a version, but watch file can’t meet a unregular url, like https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8..., which is a downloading url in Starlingx repositories. I think it’s better not to use the uscan+watch to download 3rd part packages. Thanks, ytao From: Tao, Yue <yue.tao@windriver.com> Sent: Tuesday, July 27, 2021 10:53 AM To: Short, Charles <Charles.Short@windriver.com>; starlingx-discuss@lists.starlingx.io; Asselstine, Mark <Mark.Asselstine@windriver.com>; Bai, Haiqing <Haiqing.Bai@windriver.com>; Panech, Davlet <Davlet.Panech@windriver.com> Subject: Re: [Starlingx-discuss] [Debian Build]: Layout of the debian folder On 7/27/21 7:05 AM, Short, Charles wrote: Hi, First of all, I'm appreciated for your input. I had a look through the document and it looks okay. I just have a couple of comments/questions about the folder content: * **debver** - This is the package version of the debian package? This is generated from the debian/changelog when the package is built. I guess the problem is that you want to differentiate from a starlingx package and native package? My suggestion would be what the Ubuntu developers do. For example, if we have patches to apply for systemd, the debian version would be 247.3-3, while the StarlingX package would be 247.3-3stx0. Then a user or developer can quickly see that they are working with a native debian package or a starlingx modified package. For a Debian native package, the *debver* is the debian version. We use it to download the src via "apt-source packagname=debver", for a starlingx package, it is up to the developer. I observed in CentOS version, they added a patch to change the package version, for example integ/base/lighttpd/centos/meta_patches/Update-package-versioning-for-TIS-format.patch, which appends a 'tis' to package version, that requests developer to update the version manually. After transiting to debian, OBS can achieve it automatically. It can append 'tis' or 'stx' to a package, so don't need developers to update it manually. * **deb_patches** - As a developer, if the package has its debian/patches I would rather use the debian/patches directory. If the directory doesnt exist I am more likely to create a debian/patches directory and then send a patch to debian to fix a problem. Exactly, we should send a fix to debian community and then uprev the package to get the fix, but we also need to consider local changes. The *patches" folder contains the local change for source codes, that will be copied to debian/patches (or create it if doesn't exist), and update the debian/patches/series. The *deb_patches* contains the patches to change debian folder, for example, we want to customize the installation of a package, we need to update override_dh_install stage in debian/rules. The patches will be applied to package/debian folder directly. The *deb_patches* is similar with centos/meta_patches The *patches* is similar with centos/patches * * **dl_path** - Debian already does this for you already, the debian/watch file keeps track of where to download a tarball, or a zip file, or a python wheel for you. You can use uscan(1), which is a part of devscripts package. Debian/Ubuntu developers have been doing this for years. (https://wiki.debian.org/debian/watch) Thanks you for pointing me the debian/watch. I will take some time to learn about debian/watch and go back soon. It looks a debian file for monitoring the upstream version change. I am not sure if it is suitable for a no debian package. In our internal discussion, Davlet told me a special case, that the url of a tar ball likes this https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8... As Davlet's suggestion, the *dl_path* refers to example file<https://opendev.org/starlingx/integ/src/branch/master/centos_tarball-dl.lst> and the script that parses it<https://opendev.org/starlingx/tools/src/branch/master/centos-mirror-tools/dl_tarball.sh#L215>. I will do assessment for debian/watch. Thanks, ytao If you have any questions please let me know. Regards chuck debian/watch - Debian Wiki<https://wiki.debian.org/debian/watch> debian/watch. The file named watch in the debian directory is used to check for newer versions of upstream software is available and to download it if necessary. The download itself will be performed with the uscan program from the devscripts package. It takes the path to the debian directory that uses the watch file as an argument or searches the directories underneath the current working ... wiki.debian.org ________________________________ From: ytao <yue.tao@windriver.com><mailto:yue.tao@windriver.com> Sent: July 26, 2021 3:33 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io>; Asselstine, Mark <Mark.Asselstine@windriver.com><mailto:Mark.Asselstine@windriver.com> Subject: [Starlingx-discuss] [Debian Build]: Layout of the debian folder Hello Everyone: You may have known Wind River team is working on the transition to Debian OS, more details of the project can be found at https://docs.starlingx.io/specs/specs/stx-6.0/. My purpose is launching a discussion about the userspace packages transition to Debian. The spec can be found at https://docs.starlingx.io/specs/specs/stx-6.0/approved/starlingx_2008704_deb... In order to inherit the existing userspace construction as much as possible, our proposal is creating a 'debian' folder in same directory with 'centos' folder for each package. For example, the dhcp package in integ repo, it's centos folder is at: https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/centos We will create a debian folder in same location https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/debian. All materials in this folder control building dhcp under Debian host. The "stx_deb_folder_layout.rst" is the layout of debian folder. I also attach a couple of samples to demonstrate how to fill the debian folder for a debian package and a 3rd package. This layout is not the final version, I'm appreciated for any suggestion from you. Thanks, ytao
Hi, You can specify the version you wish to download, I did the following: * apt-get source isc-dhcp * cd isc-dhcp-4.4.1 * uscan –download-version 4.3.5 * Regards chuck From: Tao, Yue <Yue.Tao@windriver.com> Sent: Friday, July 30, 2021 6:50 AM To: Tao, Yue <Yue.Tao@windriver.com>; Short, Charles <Charles.Short@windriver.com>; starlingx-discuss@lists.starlingx.io; Asselstine, Mark <Mark.Asselstine@windriver.com>; Bai, Haiqing <Haiqing.Bai@windriver.com>; Panech, Davlet <Davlet.Panech@windriver.com> Subject: RE: [Starlingx-discuss] [Debian Build]: Layout of the debian folder Hi Charles: I did some investigation of debian/watch file, which is the input file for uscan command. It’s purpose is to detect if upstream has a newer release. I tried the uscan, which is limited to run with a full debian folder, not only a dedbian/watch. I also try to the option “—watchfile” to specify a watch file, but it still checks the debian/changelog file. $uscan –watchfile path/watch uscan die: Are you in the source code tree? Cannot find readable debian/changelog anywhere! at And the url in watch file must be a regex expression, for example, Allow http://ftp.isc.org/isc/dhcp/(\d.\d.\d*)/dhcp-(\d.*)\.(?:tgz|tbz2|txz|tar\.(?:gz|bz2|xz))<http://ftp.isc.org/isc/dhcp/(/d./d./d*)/dhcp-(/d.*)/.(?:tgz|tbz2|txz|tar\.(?:gz|bz2|xz))> Not allow http://ftp.isc.org/isc/dhcp/4.4.2/dhcp-4.4.2.tar.gz A developer only care about a special version, but he has to build a regex expression, of cause uscan has –download-version to specify a version, but watch file can’t meet a unregular url, like https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8..., which is a downloading url in Starlingx repositories. I think it’s better not to use the uscan+watch to download 3rd part packages. Thanks, ytao From: Tao, Yue <yue.tao@windriver.com<mailto:yue.tao@windriver.com>> Sent: Tuesday, July 27, 2021 10:53 AM To: Short, Charles <Charles.Short@windriver.com<mailto:Charles.Short@windriver.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Asselstine, Mark <Mark.Asselstine@windriver.com<mailto:Mark.Asselstine@windriver.com>>; Bai, Haiqing <Haiqing.Bai@windriver.com<mailto:Haiqing.Bai@windriver.com>>; Panech, Davlet <Davlet.Panech@windriver.com<mailto:Davlet.Panech@windriver.com>> Subject: Re: [Starlingx-discuss] [Debian Build]: Layout of the debian folder On 7/27/21 7:05 AM, Short, Charles wrote: Hi, First of all, I'm appreciated for your input. I had a look through the document and it looks okay. I just have a couple of comments/questions about the folder content: * **debver** - This is the package version of the debian package? This is generated from the debian/changelog when the package is built. I guess the problem is that you want to differentiate from a starlingx package and native package? My suggestion would be what the Ubuntu developers do. For example, if we have patches to apply for systemd, the debian version would be 247.3-3, while the StarlingX package would be 247.3-3stx0. Then a user or developer can quickly see that they are working with a native debian package or a starlingx modified package. For a Debian native package, the *debver* is the debian version. We use it to download the src via "apt-source packagname=debver", for a starlingx package, it is up to the developer. I observed in CentOS version, they added a patch to change the package version, for example integ/base/lighttpd/centos/meta_patches/Update-package-versioning-for-TIS-format.patch, which appends a 'tis' to package version, that requests developer to update the version manually. After transiting to debian, OBS can achieve it automatically. It can append 'tis' or 'stx' to a package, so don't need developers to update it manually. * **deb_patches** - As a developer, if the package has its debian/patches I would rather use the debian/patches directory. If the directory doesnt exist I am more likely to create a debian/patches directory and then send a patch to debian to fix a problem. Exactly, we should send a fix to debian community and then uprev the package to get the fix, but we also need to consider local changes. The *patches" folder contains the local change for source codes, that will be copied to debian/patches (or create it if doesn't exist), and update the debian/patches/series. The *deb_patches* contains the patches to change debian folder, for example, we want to customize the installation of a package, we need to update override_dh_install stage in debian/rules. The patches will be applied to package/debian folder directly. The *deb_patches* is similar with centos/meta_patches The *patches* is similar with centos/patches * * **dl_path** - Debian already does this for you already, the debian/watch file keeps track of where to download a tarball, or a zip file, or a python wheel for you. You can use uscan(1), which is a part of devscripts package. Debian/Ubuntu developers have been doing this for years. (https://wiki.debian.org/debian/watch) Thanks you for pointing me the debian/watch. I will take some time to learn about debian/watch and go back soon. It looks a debian file for monitoring the upstream version change. I am not sure if it is suitable for a no debian package. In our internal discussion, Davlet told me a special case, that the url of a tar ball likes this https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8... As Davlet's suggestion, the *dl_path* refers to example file<https://opendev.org/starlingx/integ/src/branch/master/centos_tarball-dl.lst> and the script that parses it<https://opendev.org/starlingx/tools/src/branch/master/centos-mirror-tools/dl_tarball.sh#L215>. I will do assessment for debian/watch. Thanks, ytao If you have any questions please let me know. Regards chuck debian/watch - Debian Wiki<https://wiki.debian.org/debian/watch> debian/watch. The file named watch in the debian directory is used to check for newer versions of upstream software is available and to download it if necessary. The download itself will be performed with the uscan program from the devscripts package. It takes the path to the debian directory that uses the watch file as an argument or searches the directories underneath the current working ... wiki.debian.org ________________________________ From: ytao <yue.tao@windriver.com><mailto:yue.tao@windriver.com> Sent: July 26, 2021 3:33 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io>; Asselstine, Mark <Mark.Asselstine@windriver.com><mailto:Mark.Asselstine@windriver.com> Subject: [Starlingx-discuss] [Debian Build]: Layout of the debian folder Hello Everyone: You may have known Wind River team is working on the transition to Debian OS, more details of the project can be found at https://docs.starlingx.io/specs/specs/stx-6.0/. My purpose is launching a discussion about the userspace packages transition to Debian. The spec can be found at https://docs.starlingx.io/specs/specs/stx-6.0/approved/starlingx_2008704_deb... In order to inherit the existing userspace construction as much as possible, our proposal is creating a 'debian' folder in same directory with 'centos' folder for each package. For example, the dhcp package in integ repo, it's centos folder is at: https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/centos We will create a debian folder in same location https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/debian. All materials in this folder control building dhcp under Debian host. The "stx_deb_folder_layout.rst" is the layout of debian folder. I also attach a couple of samples to demonstrate how to fill the debian folder for a debian package and a 3rd package. This layout is not the final version, I'm appreciated for any suggestion from you. Thanks, ytao
Hi Exactly, --download-version specify a version to download, but the restriction of uscan is the url in debian/watch must be a regex expression $cat isc-dhcp-4.4.1/debian/watch version=3 opts="uversionmangle=s/(rc|a|b|c)/~$1/,pgpsigurlmangle=s/$/.sha512.asc/" \ http://ftp.isc.org/isc/dhcp/(\d.\d.\d*)/dhcp-(\d.*)\.(?:tgz|tbz2|txz|tar\.(?:gz|bz2|xz))<http://ftp.isc.org/isc/dhcp/(/d./d./d*)/dhcp-(/d.*)/.(?:tgz|tbz2|txz|tar\.(?:gz|bz2|xz))> As I mentioned before, some urls of 3rd part packages can’t be converted to a regex expression, so I think uscan may not suitable for downloading 3rd part packages. But uscan is a very powerful tool, probably we use it in other places. FYI the special urls. http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto/snapshot/linux-yocto-b... https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8... BTW, I attach a revision layout, which updates md5 checksum in dl_patch. Thanks, Ytao From: Short, Charles <Charles.Short@windriver.com> Sent: Friday, July 30, 2021 9:07 PM To: Tao, Yue <Yue.Tao@windriver.com>; starlingx-discuss@lists.starlingx.io; Asselstine, Mark <Mark.Asselstine@windriver.com>; Bai, Haiqing <Haiqing.Bai@windriver.com>; Panech, Davlet <Davlet.Panech@windriver.com> Subject: RE: [Starlingx-discuss] [Debian Build]: Layout of the debian folder Hi, You can specify the version you wish to download, I did the following: * apt-get source isc-dhcp * cd isc-dhcp-4.4.1 * uscan –download-version 4.3.5 * Regards chuck From: Tao, Yue <Yue.Tao@windriver.com<mailto:Yue.Tao@windriver.com>> Sent: Friday, July 30, 2021 6:50 AM To: Tao, Yue <Yue.Tao@windriver.com<mailto:Yue.Tao@windriver.com>>; Short, Charles <Charles.Short@windriver.com<mailto:Charles.Short@windriver.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Asselstine, Mark <Mark.Asselstine@windriver.com<mailto:Mark.Asselstine@windriver.com>>; Bai, Haiqing <Haiqing.Bai@windriver.com<mailto:Haiqing.Bai@windriver.com>>; Panech, Davlet <Davlet.Panech@windriver.com<mailto:Davlet.Panech@windriver.com>> Subject: RE: [Starlingx-discuss] [Debian Build]: Layout of the debian folder Hi Charles: I did some investigation of debian/watch file, which is the input file for uscan command. It’s purpose is to detect if upstream has a newer release. I tried the uscan, which is limited to run with a full debian folder, not only a dedbian/watch. I also try to the option “—watchfile” to specify a watch file, but it still checks the debian/changelog file. $uscan –watchfile path/watch uscan die: Are you in the source code tree? Cannot find readable debian/changelog anywhere! at And the url in watch file must be a regex expression, for example, Allow http://ftp.isc.org/isc/dhcp/(\d.\d.\d*)/dhcp-(\d.*)\.(?:tgz|tbz2|txz|tar\.(?:gz|bz2|xz))<http://ftp.isc.org/isc/dhcp/(/d./d./d*)/dhcp-(/d.*)/.(?:tgz|tbz2|txz|tar\.(?:gz|bz2|xz))> Not allow http://ftp.isc.org/isc/dhcp/4.4.2/dhcp-4.4.2.tar.gz A developer only care about a special version, but he has to build a regex expression, of cause uscan has –download-version to specify a version, but watch file can’t meet a unregular url, like https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8..., which is a downloading url in Starlingx repositories. I think it’s better not to use the uscan+watch to download 3rd part packages. Thanks, ytao From: Tao, Yue <yue.tao@windriver.com<mailto:yue.tao@windriver.com>> Sent: Tuesday, July 27, 2021 10:53 AM To: Short, Charles <Charles.Short@windriver.com<mailto:Charles.Short@windriver.com>>; starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io>; Asselstine, Mark <Mark.Asselstine@windriver.com<mailto:Mark.Asselstine@windriver.com>>; Bai, Haiqing <Haiqing.Bai@windriver.com<mailto:Haiqing.Bai@windriver.com>>; Panech, Davlet <Davlet.Panech@windriver.com<mailto:Davlet.Panech@windriver.com>> Subject: Re: [Starlingx-discuss] [Debian Build]: Layout of the debian folder On 7/27/21 7:05 AM, Short, Charles wrote: Hi, First of all, I'm appreciated for your input. I had a look through the document and it looks okay. I just have a couple of comments/questions about the folder content: * **debver** - This is the package version of the debian package? This is generated from the debian/changelog when the package is built. I guess the problem is that you want to differentiate from a starlingx package and native package? My suggestion would be what the Ubuntu developers do. For example, if we have patches to apply for systemd, the debian version would be 247.3-3, while the StarlingX package would be 247.3-3stx0. Then a user or developer can quickly see that they are working with a native debian package or a starlingx modified package. For a Debian native package, the *debver* is the debian version. We use it to download the src via "apt-source packagname=debver", for a starlingx package, it is up to the developer. I observed in CentOS version, they added a patch to change the package version, for example integ/base/lighttpd/centos/meta_patches/Update-package-versioning-for-TIS-format.patch, which appends a 'tis' to package version, that requests developer to update the version manually. After transiting to debian, OBS can achieve it automatically. It can append 'tis' or 'stx' to a package, so don't need developers to update it manually. * **deb_patches** - As a developer, if the package has its debian/patches I would rather use the debian/patches directory. If the directory doesnt exist I am more likely to create a debian/patches directory and then send a patch to debian to fix a problem. Exactly, we should send a fix to debian community and then uprev the package to get the fix, but we also need to consider local changes. The *patches" folder contains the local change for source codes, that will be copied to debian/patches (or create it if doesn't exist), and update the debian/patches/series. The *deb_patches* contains the patches to change debian folder, for example, we want to customize the installation of a package, we need to update override_dh_install stage in debian/rules. The patches will be applied to package/debian folder directly. The *deb_patches* is similar with centos/meta_patches The *patches* is similar with centos/patches * * **dl_path** - Debian already does this for you already, the debian/watch file keeps track of where to download a tarball, or a zip file, or a python wheel for you. You can use uscan(1), which is a part of devscripts package. Debian/Ubuntu developers have been doing this for years. (https://wiki.debian.org/debian/watch) Thanks you for pointing me the debian/watch. I will take some time to learn about debian/watch and go back soon. It looks a debian file for monitoring the upstream version change. I am not sure if it is suitable for a no debian package. In our internal discussion, Davlet told me a special case, that the url of a tar ball likes this https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8... As Davlet's suggestion, the *dl_path* refers to example file<https://opendev.org/starlingx/integ/src/branch/master/centos_tarball-dl.lst> and the script that parses it<https://opendev.org/starlingx/tools/src/branch/master/centos-mirror-tools/dl_tarball.sh#L215>. I will do assessment for debian/watch. Thanks, ytao If you have any questions please let me know. Regards chuck debian/watch - Debian Wiki<https://wiki.debian.org/debian/watch> debian/watch. The file named watch in the debian directory is used to check for newer versions of upstream software is available and to download it if necessary. The download itself will be performed with the uscan program from the devscripts package. It takes the path to the debian directory that uses the watch file as an argument or searches the directories underneath the current working ... wiki.debian.org ________________________________ From: ytao <yue.tao@windriver.com><mailto:yue.tao@windriver.com> Sent: July 26, 2021 3:33 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io>; Asselstine, Mark <Mark.Asselstine@windriver.com><mailto:Mark.Asselstine@windriver.com> Subject: [Starlingx-discuss] [Debian Build]: Layout of the debian folder Hello Everyone: You may have known Wind River team is working on the transition to Debian OS, more details of the project can be found at https://docs.starlingx.io/specs/specs/stx-6.0/. My purpose is launching a discussion about the userspace packages transition to Debian. The spec can be found at https://docs.starlingx.io/specs/specs/stx-6.0/approved/starlingx_2008704_deb... In order to inherit the existing userspace construction as much as possible, our proposal is creating a 'debian' folder in same directory with 'centos' folder for each package. For example, the dhcp package in integ repo, it's centos folder is at: https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/centos We will create a debian folder in same location https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/debian. All materials in this folder control building dhcp under Debian host. The "stx_deb_folder_layout.rst" is the layout of debian folder. I also attach a couple of samples to demonstrate how to fill the debian folder for a debian package and a 3rd package. This layout is not the final version, I'm appreciated for any suggestion from you. Thanks, ytao
On 2021-07-31 14:04:24 +0000 (+0000), Tao, Yue wrote: [...]
As I mentioned before, some urls of 3rd part packages can’t be converted to a regex expression, so I think uscan may not suitable for downloading 3rd part packages. But uscan is a very powerful tool, probably we use it in other places.
FYI the special urls. http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto/snapshot/linux-yocto-b... https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8... [...]
I recommend reviewing the (very extensive) uscan manpage as the tool has many options for working around such cases as well as convenience modes for things like directly accessing Git repositories or interfacing with popular but known-problematic sites (SourceForge, GitHub, PyPI, NPM, Google Code). Check the examples section for some watchfile options which may be exactly like what you need for these cases. -- Jeremy Stanley
On 2021-07-31 10:04 a.m., Tao, Yue wrote:
Hi
Exactly, --download-version specify a version to download, but the restriction of uscan is the url in debian/watch must be a regex expression
$cat isc-dhcp-4.4.1/debian/watch
version=3
opts="uversionmangle=s/(rc|a|b|c)/~$1/,pgpsigurlmangle=s/$/.sha512.asc/" \
http://ftp.isc.org/isc/dhcp/(\d.\d.\d*)/dhcp-(\d.*)\.(?:tgz|tbz2|txz|tar\.(?:gz|bz2|xz)) <http://ftp.isc.org/isc/dhcp/(/d./d./d*)/dhcp-(/d.*)/.(?:tgz|tbz2|txz|tar\.(?:gz|bz2|xz))>
As I mentioned before, some urls of 3^rd part packages can’t be converted to a regex expression, so I think uscan may not suitable for downloading 3^rd part packages. But uscan is a very powerful tool, probably we use it in other places.
Let's keep it in mind as we refine things. If it is possible to detect when it may not function as expected, or only allow its use when we are 100% sure it will function as expected it might help be a good assist to have in place. MarkA
FYI the special urls.
http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto/snapshot/linux-yocto-b... <http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto/snapshot/linux-yocto-b44437fb32fe50b5664afd12098d928e1aaee111.tar.bz2>
https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8... <https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8042c9354ccf7f31>
BTW, I attach a revision layout, which updates md5 checksum in dl_patch.
Thanks,
Ytao
*From:*Short, Charles <Charles.Short@windriver.com> *Sent:* Friday, July 30, 2021 9:07 PM *To:* Tao, Yue <Yue.Tao@windriver.com>; starlingx-discuss@lists.starlingx.io; Asselstine, Mark <Mark.Asselstine@windriver.com>; Bai, Haiqing <Haiqing.Bai@windriver.com>; Panech, Davlet <Davlet.Panech@windriver.com> *Subject:* RE: [Starlingx-discuss] [Debian Build]: Layout of the debian folder
Hi,
You can specify the version you wish to download, I did the following:
* apt-get source isc-dhcp * cd isc-dhcp-4.4.1 * uscan –download-version 4.3.5 *
Regards
chuck
*From:*Tao, Yue <Yue.Tao@windriver.com <mailto:Yue.Tao@windriver.com>> *Sent:* Friday, July 30, 2021 6:50 AM *To:* Tao, Yue <Yue.Tao@windriver.com <mailto:Yue.Tao@windriver.com>>; Short, Charles <Charles.Short@windriver.com <mailto:Charles.Short@windriver.com>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io>; Asselstine, Mark <Mark.Asselstine@windriver.com <mailto:Mark.Asselstine@windriver.com>>; Bai, Haiqing <Haiqing.Bai@windriver.com <mailto:Haiqing.Bai@windriver.com>>; Panech, Davlet <Davlet.Panech@windriver.com <mailto:Davlet.Panech@windriver.com>> *Subject:* RE: [Starlingx-discuss] [Debian Build]: Layout of the debian folder
Hi Charles:
I did some investigation of debian/watch file, which is the input file for uscan command. It’s purpose is to detect if upstream has a newer release. I tried the uscan, which is limited to run with a full debian folder, not only a dedbian/watch. I also try to the option “—watchfile” to specify a watch file, but it still checks the debian/changelog file.
$uscan –watchfile path/watch
uscan die: Are you in the source code tree?
Cannot find readable debian/changelog anywhere! at
And the url in watch file must be a regex expression, for example,
Allow
http://ftp.isc.org/isc/dhcp/(\d.\d.\d*)/dhcp-(\d.*)\.(?:tgz|tbz2|txz|tar\.(?:gz|bz2|xz)) <http://ftp.isc.org/isc/dhcp/(/d./d./d*)/dhcp-(/d.*)/.(?:tgz|tbz2|txz|tar\.(?:gz|bz2|xz))>
Not allow
http://ftp.isc.org/isc/dhcp/4.4.2/dhcp-4.4.2.tar.gz <http://ftp.isc.org/isc/dhcp/4.4.2/dhcp-4.4.2.tar.gz>
A developer only care about a special version, but he has to build a regex expression, of cause uscan has –download-version to specify a version, but watch file can’t meet a unregular url, like https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8... <https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8042c9354ccf7f31>, which is a downloading url in Starlingx repositories. I think it’s better not to use the uscan+watch to download 3^rd part packages.
Thanks,
ytao
*From:*Tao, Yue <yue.tao@windriver.com <mailto:yue.tao@windriver.com>> *Sent:* Tuesday, July 27, 2021 10:53 AM *To:* Short, Charles <Charles.Short@windriver.com <mailto:Charles.Short@windriver.com>>; starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io>; Asselstine, Mark <Mark.Asselstine@windriver.com <mailto:Mark.Asselstine@windriver.com>>; Bai, Haiqing <Haiqing.Bai@windriver.com <mailto:Haiqing.Bai@windriver.com>>; Panech, Davlet <Davlet.Panech@windriver.com <mailto:Davlet.Panech@windriver.com>> *Subject:* Re: [Starlingx-discuss] [Debian Build]: Layout of the debian folder
On 7/27/21 7:05 AM, Short, Charles wrote:
Hi,
First of all, I'm appreciated for your input.
I had a look through the document and it looks okay. I just have a couple of comments/questions about the folder content:
* **debver** - This is the package version of the debian package? This is generated from the debian/changelog when the package is built. I guess the problem is that you want to differentiate from a starlingx package and native package? My suggestion would be what the Ubuntu developers do. For example, if we have patches to apply for systemd, the debian version would be 247.3-3, while the StarlingX package would be 247.3-3stx0. Then a user or developer can quickly see that they are working with a native debian package or a starlingx modified package.
For a Debian native package, the *debver* is the debian version. We use it to download the src via "apt-source packagname=debver", for a starlingx package, it is up to the developer. I observed in CentOS version, they added a patch to change the package version, for example integ/base/lighttpd/centos/meta_patches/Update-package-versioning-for-TIS-format.patch, which appends a 'tis' to package version, that requests developer to update the version manually. After transiting to debian, OBS can achieve it automatically. It can append 'tis' or 'stx' to a package, so don't need developers to update it manually.
* **deb_patches** - As a developer, if the package has its debian/patches I would rather use the debian/patches directory. If the directory doesnt exist I am more likely to create a debian/patches directory and then send a patch to debian to fix a problem.
Exactly, we should send a fix to debian community and then uprev the package to get the fix, but we also need to consider local changes. The *patches" folder contains the local change for source codes, that will be copied to debian/patches (or create it if doesn't exist), and update the debian/patches/series. The *deb_patches* contains the patches to change debian folder, for example, we want to customize the installation of a package, we need to update override_dh_install stage in debian/rules. The patches will be applied to package/debian folder directly.
The *deb_patches* is similar with centos/meta_patches
The *patches* is similar with centos/patches
* * **dl_path** - Debian already does this for you already, the debian/watch file keeps track of where to download a tarball, or a zip file, or a python wheel for you. You can use uscan(1), which is a part of devscripts package. Debian/Ubuntu developers have been doing this for years. (https://wiki.debian.org/debian/watch <https://wiki.debian.org/debian/watch>)
Thanks you for pointing me the debian/watch. I will take some time to learn about debian/watch and go back soon. It looks a debian file for monitoring the upstream version change. I am not sure if it is suitable for a no debian package. In our internal discussion, Davlet told me a special case, that the url of a tar ball likes this https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8... <https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8042c9354ccf7f31>
As Davlet's suggestion, the *dl_path* refers to example file <https://opendev.org/starlingx/integ/src/branch/master/centos_tarball-dl.lst> and the script that parses it <https://opendev.org/starlingx/tools/src/branch/master/centos-mirror-tools/dl_tarball.sh#L215>. I will do assessment for debian/watch.
Thanks,
ytao
If you have any questions please let me know.
Regards
chuck
debian/watch - Debian Wiki <https://wiki.debian.org/debian/watch>
debian/watch. The file named watch in the debian directory is used to check for newer versions of upstream software is available and to download it if necessary. The download itself will be performed with the uscan program from the devscripts package. It takes the path to the debian directory that uses the watch file as an argument or searches the directories underneath the current working ...
wiki.debian.org
------------------------------------------------------------------------
*From:* ytao <yue.tao@windriver.com> <mailto:yue.tao@windriver.com> *Sent:* July 26, 2021 3:33 AM *To:* starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io> <mailto:starlingx-discuss@lists.starlingx.io>; Asselstine, Mark <Mark.Asselstine@windriver.com> <mailto:Mark.Asselstine@windriver.com> *Subject:* [Starlingx-discuss] [Debian Build]: Layout of the debian folder
Hello Everyone:
You may have known Wind River team is working on the transition to Debian OS, more details of the project can be found at https://docs.starlingx.io/specs/specs/stx-6.0/ <https://docs.starlingx.io/specs/specs/stx-6.0/>. My purpose is launching a discussion about the userspace packages transition to Debian. The spec can be found at https://docs.starlingx.io/specs/specs/stx-6.0/approved/starlingx_2008704_deb... <https://docs.starlingx.io/specs/specs/stx-6.0/approved/starlingx_2008704_debian_transition.html>
In order to inherit the existing userspace construction as much as possible, our proposal is creating a 'debian' folder in same directory with 'centos' folder for each package. For example, the dhcp package in integ repo, it's centos folder is at:
https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/centos <https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/centos>
We will create a debian folder in same location
https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/debian <https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/debian>.
All materials in this folder control building dhcp under Debian host. The "stx_deb_folder_layout.rst" is the layout of debian folder. I also attach a couple of samples to demonstrate how to fill the debian folder for a debian package and a 3rd package.
This layout is not the final version, I'm appreciated for any suggestion from you.
Thanks,
ytao
On 7/27/21 10:52 AM, ytao wrote:
On 7/27/21 7:05 AM, Short, Charles wrote:
Hi,
First of all, I'm appreciated for your input.
I had a look through the document and it looks okay. I just have a couple of comments/questions about the folder content:
* **debver** - This is the package version of the debian package? This is generated from the debian/changelog when the package is built. I guess the problem is that you want to differentiate from a starlingx package and native package? My suggestion would be what the Ubuntu developers do. For example, if we have patches to apply for systemd, the debian version would be 247.3-3, while the StarlingX package would be 247.3-3stx0. Then a user or developer can quickly see that they are working with a native debian package or a starlingx modified package.
For a Debian native package, the *debver* is the debian version. We use it to download the src via "apt-source packagname=debver", for a starlingx package, it is up to the developer. I observed in CentOS version, they added a patch to change the package version, for example integ/base/lighttpd/centos/meta_patches/Update-package-versioning-for-TIS-format.patch, which appends a 'tis' to package version, that requests developer to update the version manually. After transiting to debian, OBS can achieve it automatically. It can append 'tis' or 'stx' to a package, so don't need developers to update it manually.
Recently, we found an issue related to "--append-to-version" option of sbuild, via which we append the ".stx" to a built package version, so that we don't need to involve a developer to modify revision manually. A upstream commit https://salsa.debian.org/debian/sbuild/commit/9741ec6eb018d500a93e607f998dda... forces the "--append-to-version"implies "--no-arch-all", which causes the packages with --arch-all are skipped by sbuild, for example, most of the python modules, so we have to give up the "--append-to-version" option. After reviewing the function "srpm_source_build_data()" in|| https://opendev.org/starlingx/root/src/branch/master/build-tools/srpm-utils in depth, we decide to inherit the PKG_GITREVCOUNT/SRC_GITREVCOUNT/ PKG_BASE_SRCREV/SRC_BASE_SRCREV 4 macros from CentOS version to increase the revision automatically. Difference with CentOS version is using a yaml file instead of build_srpm.data. And we define a new environment variable $STX_DIST (default value is ".stx"), which is appended to built package version. A new yaml file is involved, why not to merge other files into the yaml file? Hence, we merge the dl_path/src_path/debver/debname into meta_data.yaml, a sample of the meta_data_yaml $cat integ/base/libfdt/debian/meta_data.yaml ================== debver: 1.4.4-1 debname: libfdt dl_path: name: dtc-1.4.4.tar.gz url: https://www.kernel.org/pub/software/utils/dtc/dtc-1.4.4.tar.gz md5sum: e558cd0e244f122bcb4ebaca166aea4e revision: dist: $STX_DIST PKG_GITREVCOUNT: ================== More details please see the stx_deb_folder_layout.rst file. Thanks, ytao
* **deb_patches** - As a developer, if the package has its debian/patches I would rather use the debian/patches directory. If the directory doesnt exist I am more likely to create a debian/patches directory and then send a patch to debian to fix a problem.
Exactly, we should send a fix to debian community and then uprev the package to get the fix, but we also need to consider local changes. The *patches" folder contains the local change for source codes, that will be copied to debian/patches (or create it if doesn't exist), and update the debian/patches/series. The *deb_patches* contains the patches to change debian folder, for example, we want to customize the installation of a package, we need to update override_dh_install stage in debian/rules. The patches will be applied to package/debian folder directly.
The *deb_patches* is similar with centos/meta_patches
The *patches* is similar with centos/patches
*
* **dl_path** - Debian already does this for you already, the debian/watch file keeps track of where to download a tarball, or a zip file, or a python wheel for you. You can use uscan(1), which is a part of devscripts package. Debian/Ubuntu developers have been doing this for years. (https://wiki.debian.org/debian/watch)
Thanks you for pointing me the debian/watch. I will take some time to learn about debian/watch and go back soon. It looks a debian file for monitoring the upstream version change. I am not sure if it is suitable for a no debian package. In our internal discussion, Davlet told me a special case, that the url of a tar ball likes this https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8...
As Davlet's suggestion, the *dl_path* refers to example file <https://opendev.org/starlingx/integ/src/branch/master/centos_tarball-dl.lst> and the script that parses it <https://opendev.org/starlingx/tools/src/branch/master/centos-mirror-tools/dl_tarball.sh#L215>. I will do assessment for debian/watch.
Thanks,
ytao
If you have any questions please let me know.
Regards chuck
debian/watch - Debian Wiki <https://wiki.debian.org/debian/watch> debian/watch. The file named watch in the debian directory is used to check for newer versions of upstream software is available and to download it if necessary. The download itself will be performed with the uscan program from the devscripts package. It takes the path to the debian directory that uses the watch file as an argument or searches the directories underneath the current working ... wiki.debian.org
------------------------------------------------------------------------ *From:* ytao <yue.tao@windriver.com> *Sent:* July 26, 2021 3:33 AM *To:* starlingx-discuss@lists.starlingx.io <starlingx-discuss@lists.starlingx.io>; Asselstine, Mark <Mark.Asselstine@windriver.com> *Subject:* [Starlingx-discuss] [Debian Build]: Layout of the debian folder Hello Everyone:
You may have known Wind River team is working on the transition to Debian OS, more details of the project can be found at https://docs.starlingx.io/specs/specs/stx-6.0/. My purpose is launching a discussion about the userspace packages transition to Debian. The spec can be found at https://docs.starlingx.io/specs/specs/stx-6.0/approved/starlingx_2008704_deb...
In order to inherit the existing userspace construction as much as possible, our proposal is creating a 'debian' folder in same directory with 'centos' folder for each package. For example, the dhcp package in integ repo, it's centos folder is at:
https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/centos
We will create a debian folder in same location
https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/debian.
All materials in this folder control building dhcp under Debian host. The "stx_deb_folder_layout.rst" is the layout of debian folder. I also attach a couple of samples to demonstrate how to fill the debian folder for a debian package and a 3rd package.
This layout is not the final version, I'm appreciated for any suggestion from you.
Thanks,
ytao
Hi, Comments in line From: Tao, Yue <Yue.Tao@windriver.com> Sent: Sunday, August 15, 2021 11:18 PM To: Short, Charles <Charles.Short@windriver.com>; starlingx-discuss@lists.starlingx.io; Asselstine, Mark <Mark.Asselstine@windriver.com>; Bai, Haiqing <Haiqing.Bai@windriver.com>; Panech, Davlet <Davlet.Panech@windriver.com> Subject: Re: [Starlingx-discuss] [Debian Build]: Layout of the debian folder On 7/27/21 10:52 AM, ytao wrote: On 7/27/21 7:05 AM, Short, Charles wrote: Hi, First of all, I'm appreciated for your input. I had a look through the document and it looks okay. I just have a couple of comments/questions about the folder content: * **debver** - This is the package version of the debian package? This is generated from the debian/changelog when the package is built. I guess the problem is that you want to differentiate from a starlingx package and native package? My suggestion would be what the Ubuntu developers do. For example, if we have patches to apply for systemd, the debian version would be 247.3-3, while the StarlingX package would be 247.3-3stx0. Then a user or developer can quickly see that they are working with a native debian package or a starlingx modified package. For a Debian native package, the *debver* is the debian version. We use it to download the src via "apt-source packagname=debver", for a starlingx package, it is up to the developer. I observed in CentOS version, they added a patch to change the package version, for example integ/base/lighttpd/centos/meta_patches/Update-package-versioning-for-TIS-format.patch, which appends a 'tis' to package version, that requests developer to update the version manually. After transiting to debian, OBS can achieve it automatically. It can append 'tis' or 'stx' to a package, so don't need developers to update it manually. Recently, we found an issue related to "--append-to-version" option of sbuild, via which we append the ".stx" to a built package version, so that we don't need to involve a developer to modify revision manually. A upstream commit https://salsa.debian.org/debian/sbuild/commit/9741ec6eb018d500a93e607f998dda... forces the "--append-to-version"implies "--no-arch-all", which causes the packages with --arch-all are skipped by sbuild, for example, most of the python modules, so we have to give up the "--append-to-version" option. Yeah I don’t think we should be using sbuild in this manner. To change the version of the package you should be using the “dch” command. Sbuild is akin to mock for fedora, and it should be really only used for building packages. After reviewing the function "srpm_source_build_data()" in https://opendev.org/starlingx/root/src/branch/master/build-tools/srpm-utils in depth, we decide to inherit the PKG_GITREVCOUNT/SRC_GITREVCOUNT/ PKG_BASE_SRCREV/SRC_BASE_SRCREV 4 macros from CentOS version to increase the revision automatically. Difference with CentOS version is using a yaml file instead of build_srpm.data. And we define a new environment variable $STX_DIST (default value is ".stx"), which is appended to built package version. A new yaml file is involved, why not to merge other files into the yaml file? Hence, we merge the dl_path/src_path/debver/debname into meta_data.yaml, a sample of the meta_data_yaml $cat integ/base/libfdt/debian/meta_data.yaml ================== debver: 1.4.4-1 debname: libfdt dl_path: name: dtc-1.4.4.tar.gz url: https://www.kernel.org/pub/software/utils/dtc/dtc-1.4.4.tar.gz md5sum: e558cd0e244f122bcb4ebaca166aea4e revision: dist: $STX_DIST PKG_GITREVCOUNT: ================== More details please see the stx_deb_folder_layout.rst file. Thanks, ytao * **deb_patches** - As a developer, if the package has its debian/patches I would rather use the debian/patches directory. If the directory doesnt exist I am more likely to create a debian/patches directory and then send a patch to debian to fix a problem. Exactly, we should send a fix to debian community and then uprev the package to get the fix, but we also need to consider local changes. The *patches" folder contains the local change for source codes, that will be copied to debian/patches (or create it if doesn't exist), and update the debian/patches/series. The *deb_patches* contains the patches to change debian folder, for example, we want to customize the installation of a package, we need to update override_dh_install stage in debian/rules. The patches will be applied to package/debian folder directly. The *deb_patches* is similar with centos/meta_patches The *patches* is similar with centos/patches * * **dl_path** - Debian already does this for you already, the debian/watch file keeps track of where to download a tarball, or a zip file, or a python wheel for you. You can use uscan(1), which is a part of devscripts package. Debian/Ubuntu developers have been doing this for years. (https://wiki.debian.org/debian/watch) Thanks you for pointing me the debian/watch. I will take some time to learn about debian/watch and go back soon. It looks a debian file for monitoring the upstream version change. I am not sure if it is suitable for a no debian package. In our internal discussion, Davlet told me a special case, that the url of a tar ball likes this https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8... As Davlet's suggestion, the *dl_path* refers to example file<https://opendev.org/starlingx/integ/src/branch/master/centos_tarball-dl.lst> and the script that parses it<https://opendev.org/starlingx/tools/src/branch/master/centos-mirror-tools/dl_tarball.sh#L215>. I will do assessment for debian/watch. Thanks, ytao If you have any questions please let me know. Regards chuck debian/watch - Debian Wiki<https://wiki.debian.org/debian/watch> debian/watch. The file named watch in the debian directory is used to check for newer versions of upstream software is available and to download it if necessary. The download itself will be performed with the uscan program from the devscripts package. It takes the path to the debian directory that uses the watch file as an argument or searches the directories underneath the current working ... wiki.debian.org ________________________________ From: ytao <yue.tao@windriver.com><mailto:yue.tao@windriver.com> Sent: July 26, 2021 3:33 AM To: starlingx-discuss@lists.starlingx.io<mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io><mailto:starlingx-discuss@lists.starlingx.io>; Asselstine, Mark <Mark.Asselstine@windriver.com><mailto:Mark.Asselstine@windriver.com> Subject: [Starlingx-discuss] [Debian Build]: Layout of the debian folder Hello Everyone: You may have known Wind River team is working on the transition to Debian OS, more details of the project can be found at https://docs.starlingx.io/specs/specs/stx-6.0/. My purpose is launching a discussion about the userspace packages transition to Debian. The spec can be found at https://docs.starlingx.io/specs/specs/stx-6.0/approved/starlingx_2008704_deb... In order to inherit the existing userspace construction as much as possible, our proposal is creating a 'debian' folder in same directory with 'centos' folder for each package. For example, the dhcp package in integ repo, it's centos folder is at: https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/centos We will create a debian folder in same location https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/debian. All materials in this folder control building dhcp under Debian host. The "stx_deb_folder_layout.rst" is the layout of debian folder. I also attach a couple of samples to demonstrate how to fill the debian folder for a debian package and a 3rd package. This layout is not the final version, I'm appreciated for any suggestion from you. Thanks, ytao
On 8/16/21 8:44 PM, Short, Charles wrote:
Hi,
Comments in line
*From:*Tao, Yue <Yue.Tao@windriver.com> *Sent:* Sunday, August 15, 2021 11:18 PM *To:* Short, Charles <Charles.Short@windriver.com>; starlingx-discuss@lists.starlingx.io; Asselstine, Mark <Mark.Asselstine@windriver.com>; Bai, Haiqing <Haiqing.Bai@windriver.com>; Panech, Davlet <Davlet.Panech@windriver.com> *Subject:* Re: [Starlingx-discuss] [Debian Build]: Layout of the debian folder
On 7/27/21 10:52 AM, ytao wrote:
On 7/27/21 7:05 AM, Short, Charles wrote:
Hi,
First of all, I'm appreciated for your input.
I had a look through the document and it looks okay. I just have a couple of comments/questions about the folder content:
* **debver** - This is the package version of the debian package? This is generated from the debian/changelog when the package is built. I guess the problem is that you want to differentiate from a starlingx package and native package? My suggestion would be what the Ubuntu developers do. For example, if we have patches to apply for systemd, the debian version would be 247.3-3, while the StarlingX package would be 247.3-3stx0. Then a user or developer can quickly see that they are working with a native debian package or a starlingx modified package.
For a Debian native package, the *debver* is the debian version. We use it to download the src via "apt-source packagname=debver", for a starlingx package, it is up to the developer. I observed in CentOS version, they added a patch to change the package version, for example integ/base/lighttpd/centos/meta_patches/Update-package-versioning-for-TIS-format.patch, which appends a 'tis' to package version, that requests developer to update the version manually. After transiting to debian, OBS can achieve it automatically. It can append 'tis' or 'stx' to a package, so don't need developers to update it manually.
Recently, we found an issue related to "--append-to-version" option of sbuild, via which we append the ".stx" to a built package version, so that we don't need to involve a developer to modify revision manually. A upstream commit https://salsa.debian.org/debian/sbuild/commit/9741ec6eb018d500a93e607f998dda... forces the "--append-to-version"implies "--no-arch-all", which causes the packages with --arch-all are skipped by sbuild, for example, most of the python modules, so we have to give up the "--append-to-version" option.
Yeah I don’t think we should be using sbuild in this manner. To change the version of the package you should be using the “dch” command. Sbuild is akin to mock for fedora, and it should be really only used for building packages.
Yeah, new method is using "dch" command to update changelog by parsing the "revision" of meta_data.yaml. ytao
After reviewing the function "srpm_source_build_data()" in https://opendev.org/starlingx/root/src/branch/master/build-tools/srpm-utils in depth, we decide to inherit the PKG_GITREVCOUNT/SRC_GITREVCOUNT/
PKG_BASE_SRCREV/SRC_BASE_SRCREV 4 macros from CentOS version to increase the revision automatically. Difference with CentOS version is using a yaml file instead of build_srpm.data. And we define a new environment variable $STX_DIST (default value is ".stx"), which is appended to built package version.
A new yaml file is involved, why not to merge other files into the yaml file? Hence, we merge the dl_path/src_path/debver/debname into meta_data.yaml, a sample of the meta_data_yaml
$cat integ/base/libfdt/debian/meta_data.yaml
==================
debver: 1.4.4-1 debname: libfdt dl_path: name: dtc-1.4.4.tar.gz url: https://www.kernel.org/pub/software/utils/dtc/dtc-1.4.4.tar.gz md5sum: e558cd0e244f122bcb4ebaca166aea4e revision: dist: $STX_DIST PKG_GITREVCOUNT:
==================
More details please see the stx_deb_folder_layout.rst file.
Thanks,
ytao
* **deb_patches** - As a developer, if the package has its debian/patches I would rather use the debian/patches directory. If the directory doesnt exist I am more likely to create a debian/patches directory and then send a patch to debian to fix a problem.
Exactly, we should send a fix to debian community and then uprev the package to get the fix, but we also need to consider local changes. The *patches" folder contains the local change for source codes, that will be copied to debian/patches (or create it if doesn't exist), and update the debian/patches/series. The *deb_patches* contains the patches to change debian folder, for example, we want to customize the installation of a package, we need to update override_dh_install stage in debian/rules. The patches will be applied to package/debian folder directly.
The *deb_patches* is similar with centos/meta_patches
The *patches* is similar with centos/patches
* * **dl_path** - Debian already does this for you already, the debian/watch file keeps track of where to download a tarball, or a zip file, or a python wheel for you. You can use uscan(1), which is a part of devscripts package. Debian/Ubuntu developers have been doing this for years. (https://wiki.debian.org/debian/watch)
Thanks you for pointing me the debian/watch. I will take some time to learn about debian/watch and go back soon. It looks a debian file for monitoring the upstream version change. I am not sure if it is suitable for a no debian package. In our internal discussion, Davlet told me a special case, that the url of a tar ball likes this https://api.github.com/repos/ceph/jerasure/tarball/96c76b89d661c163f65a014b8...
As Davlet's suggestion, the *dl_path* refers to example file <https://opendev.org/starlingx/integ/src/branch/master/centos_tarball-dl.lst> and the script that parses it <https://opendev.org/starlingx/tools/src/branch/master/centos-mirror-tools/dl_tarball.sh#L215>. I will do assessment for debian/watch.
Thanks,
ytao
If you have any questions please let me know.
Regards
chuck
debian/watch - Debian Wiki <https://wiki.debian.org/debian/watch>
debian/watch. The file named watch in the debian directory is used to check for newer versions of upstream software is available and to download it if necessary. The download itself will be performed with the uscan program from the devscripts package. It takes the path to the debian directory that uses the watch file as an argument or searches the directories underneath the current working ...
wiki.debian.org
------------------------------------------------------------------------
*From:* ytao <yue.tao@windriver.com> <mailto:yue.tao@windriver.com> *Sent:* July 26, 2021 3:33 AM *To:* starlingx-discuss@lists.starlingx.io <mailto:starlingx-discuss@lists.starlingx.io> <starlingx-discuss@lists.starlingx.io> <mailto:starlingx-discuss@lists.starlingx.io>; Asselstine, Mark <Mark.Asselstine@windriver.com> <mailto:Mark.Asselstine@windriver.com> *Subject:* [Starlingx-discuss] [Debian Build]: Layout of the debian folder
Hello Everyone:
You may have known Wind River team is working on the transition to Debian OS, more details of the project can be found at https://docs.starlingx.io/specs/specs/stx-6.0/. My purpose is launching a discussion about the userspace packages transition to Debian. The spec can be found at https://docs.starlingx.io/specs/specs/stx-6.0/approved/starlingx_2008704_deb...
In order to inherit the existing userspace construction as much as possible, our proposal is creating a 'debian' folder in same directory with 'centos' folder for each package. For example, the dhcp package in integ repo, it's centos folder is at:
https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/centos
We will create a debian folder in same location
https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/debian.
All materials in this folder control building dhcp under Debian host. The "stx_deb_folder_layout.rst" is the layout of debian folder. I also attach a couple of samples to demonstrate how to fill the debian folder for a debian package and a 3rd package.
This layout is not the final version, I'm appreciated for any suggestion from you.
Thanks,
ytao
debname: The text is unclear. I think you are saying that the deb built out of directory "integ/base/dhcp" would get the name "dhcp" by defult. However the Debian convention is to call this package "isc-dhcp" so we must include a "integ/base/dhcp/debian/debname" file to force the package we build to receive the name "isc-dhcp" deb_folder: This overrides the upstream folder entirely? It can't be used to add files or override specific upstream files? If it does override the entire folder, perhaps we need a warning from the package builder that the override has occurred. src_path: Should support some common macros rather than relying strictly on relative paths. This aids in code restructuring, both within and across gits. e.g. we have macros like... PKG_BASE, GIT_BASE, STX_BASE general: At some point it would be desirable to have a tool to aid in patch creation. It should use git. Currently build-pkg --edit tries to do this, but it's not very reliable as rpm's approach to patching fairly free-form and hard to automate. e.g. stx edit --pkg-type=debian --package=dhcp = it copies 'debian' to a working directory and does git init, git add --all, git commit = It copies tarball source into a second working directory and does git init, git add --all, git commit = It applies 'patches' and 'deb_patches' to the correct gits, in the correct order, and creates a git commit for each. The user can then edit files, cherry-pick a fix from elsewhere, re-order or delete a patch... etc. Finally the user uses git format-patch to regenerate the patch and deb-patch series ... or perhaps this step is covered by the 'stx' tool ... stx edit --publish --pkg-type=debian --package=dhcp One major use of tis is to 'de-fuzz' patches when moving to a new upstream version/tarball On 2021-07-26 3:33 a.m., ytao wrote:
Hello Everyone:
You may have known Wind River team is working on the transition to Debian OS, more details of the project can be found at https://docs.starlingx.io/specs/specs/stx-6.0/. My purpose is launching a discussion about the userspace packages transition to Debian. The spec can be found at https://docs.starlingx.io/specs/specs/stx-6.0/approved/starlingx_2008704_deb...
In order to inherit the existing userspace construction as much as possible, our proposal is creating a 'debian' folder in same directory with 'centos' folder for each package. For example, the dhcp package in integ repo, it's centos folder is at:
https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/centos
We will create a debian folder in same location
https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/debian.
All materials in this folder control building dhcp under Debian host. The "stx_deb_folder_layout.rst" is the layout of debian folder. I also attach a couple of samples to demonstrate how to fill the debian folder for a debian package and a 3rd package.
This layout is not the final version, I'm appreciated for any suggestion from you.
Thanks,
ytao
_______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss
Comments inline From: Scott Little <scott.little@windriver.com> Sent: Wednesday, July 28, 2021 11:42 AM To: starlingx-discuss@lists.starlingx.io Subject: Re: [Starlingx-discuss] [Debian Build]: Layout of the debian folder debname: The text is unclear. I think you are saying that the deb built out of directory "integ/base/dhcp" would get the name "dhcp" by defult. However the Debian convention is to call this package "isc-dhcp" so we must include a "integ/base/dhcp/debian/debname" file to force the package we build to receive the name "isc-dhcp" deb_folder: This overrides the upstream folder entirely? It can't be used to add files or override specific upstream files? If it does override the entire folder, perhaps we need a warning from the package builder that the override has occurred. src_path: Should support some common macros rather than relying strictly on relative paths. This aids in code restructuring, both within and across gits. e.g. we have macros like... PKG_BASE, GIT_BASE, STX_BASE general: At some point it would be desirable to have a tool to aid in patch creation. It should use git. Currently build-pkg --edit tries to do this, but it's not very reliable as rpm's approach to patching fairly free-form and hard to automate. e.g. stx edit --pkg-type=debian --package=dhcp = it copies 'debian' to a working directory and does git init, git add --all, git commit = It copies tarball source into a second working directory and does git init, git add --all, git commit = It applies 'patches' and 'deb_patches' to the correct gits, in the correct order, and creates a git commit for each. The user can then edit files, cherry-pick a fix from elsewhere, re-order or delete a patch... etc. Finally the user uses git format-patch to regenerate the patch and deb-patch series ... or perhaps this step is covered by the 'stx' tool ... stx edit --publish --pkg-type=debian --package=dhcp One major use of tis is to 'de-fuzz' patches when moving to a new upstream version/tarball We could leverage git-buildpackage to do this for us: https://wiki.debian.org/PackagingWithGit chuck On 2021-07-26 3:33 a.m., ytao wrote: Hello Everyone: You may have known Wind River team is working on the transition to Debian OS, more details of the project can be found at https://docs.starlingx.io/specs/specs/stx-6.0/<https://urldefense.com/v3/__https:/docs.starlingx.io/specs/specs/stx-6.0/__;!!AjveYdw8EvQ!KKiGaR4c4k8o5E8Q-ZSn5OkHt-mu7NxuhOFFuLzDepL6GNUCtUYeB1Xz05Bs-qOvYRDU$>. My purpose is launching a discussion about the userspace packages transition to Debian. The spec can be found at https://docs.starlingx.io/specs/specs/stx-6.0/approved/starlingx_2008704_debian_transition.html<https://urldefense.com/v3/__https:/docs.starlingx.io/specs/specs/stx-6.0/approved/starlingx_2008704_debian_transition.html__;!!AjveYdw8EvQ!KKiGaR4c4k8o5E8Q-ZSn5OkHt-mu7NxuhOFFuLzDepL6GNUCtUYeB1Xz05Bs-v7vIRqm$> In order to inherit the existing userspace construction as much as possible, our proposal is creating a 'debian' folder in same directory with 'centos' folder for each package. For example, the dhcp package in integ repo, it's centos folder is at: https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/centos<https://urldefense.com/v3/__https:/opendev.org/starlingx/integ/src/branch/master/base/dhcp/centos__;!!AjveYdw8EvQ!KKiGaR4c4k8o5E8Q-ZSn5OkHt-mu7NxuhOFFuLzDepL6GNUCtUYeB1Xz05Bs-lbvuutm$> We will create a debian folder in same location https://opendev.org/starlingx/integ/src/branch/master/base/dhcp/debian<https://urldefense.com/v3/__https:/opendev.org/starlingx/integ/src/branch/master/base/dhcp/debian__;!!AjveYdw8EvQ!KKiGaR4c4k8o5E8Q-ZSn5OkHt-mu7NxuhOFFuLzDepL6GNUCtUYeB1Xz05Bs-haePFKA$>. All materials in this folder control building dhcp under Debian host. The "stx_deb_folder_layout.rst" is the layout of debian folder. I also attach a couple of samples to demonstrate how to fill the debian folder for a debian package and a 3rd package. This layout is not the final version, I'm appreciated for any suggestion from you. Thanks, ytao _______________________________________________ Starlingx-discuss mailing list Starlingx-discuss@lists.starlingx.io<mailto:Starlingx-discuss@lists.starlingx.io> http://lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss<https://urldefense.com/v3/__http:/lists.starlingx.io/cgi-bin/mailman/listinfo/starlingx-discuss__;!!AjveYdw8EvQ!KKiGaR4c4k8o5E8Q-ZSn5OkHt-mu7NxuhOFFuLzDepL6GNUCtUYeB1Xz05Bs-jdZ04zJ$>
participants (6)
-
Jeremy Stanley
-
Mark Asselstine
-
Scott Little
-
Short, Charles
-
Tao, Yue
-
ytao