[Starlingx-discuss] Role of zuul in build failures related to PLATFORM_RELEASE

[Penney, Don]

Sorry... I think I'm wrong on that... I thought all our repos were in the same queue, but I think I spoke too soon.

-----Original Message-----
From: Penney, Don 
Sent: Wednesday, May 27, 2020 12:39 PM
To: Jeremy Stanley; starlingx-discuss at lists.starlingx.io
Subject: RE: [Starlingx-discuss] Role of zuul in build failures related to PLATFORM_RELEASE

They're in the same "starlingx" queue, but we often see that multi-repo updates like this end up having some reviews "missed" by Zuul. In some cases, we've seen a chain of multiple reviews where a couple of the dependent reviews get picked up / noticed by Zuul, while others don't. It seems to be an intermittent issue.

-----Original Message-----
From: Jeremy Stanley [mailto:fungi at yuggoth.org] 
Sent: Wednesday, May 27, 2020 11:33 AM
To: starlingx-discuss at lists.starlingx.io
Subject: Re: [Starlingx-discuss] Role of zuul in build failures related to PLATFORM_RELEASE

On 2020-05-27 11:02:17 -0400 (-0400), Scott Little wrote:
> The two updates in question:
> 
> utilities: https://review.opendev.org/#/c/730108/
> 
> ansible-playbooks:  https://review.opendev.org/#/c/730113/
> 
> 
> 730113 depends in 730108
> 
> 730108 was WF-1 to prevent merge until 730113 was reviewd
> 
> 730113 received it's +2/+1 ... Zuul didn't merge because 730108 was 
> not merged.
> 
> 730108 is given it's WF+1 late Friday and merged.
> 
> Expected behavior was for 730113 to 'wake up' within zuul and merge.
> 
> It did not, and the builds produced unusable loads until the issue was 
> noticed on Monday.  730113 had to be given a new WF+1 to wake it up 
> and merge.

Zuul has logic to enqueue dependent changes together *if* their projects share a change queue. It does not have any feature to find changes which were blocked on other changes they don't share a change queue with, so doesn't know to automatically enqueue them once their dependencies merge. Zuul's pipeline enqueuing logic is event-driven, and only "wakes up" changes on events for related changes if they can be enqueued and tested together. Declaring cross-project dependencies between changes in projects which do not share a change queue serves only to block changes from merging, but will need a new enqueuing event for the depending change once its dependencies are satisfied, for example by removing and re-adding a Workflow +1 approval or a different reviewer adding a second Workflow +1.

The note at
https://zuul-ci.org/docs/zuul/discussion/gating.html#dependent-pipeline
hints at this, but I've just now proposed
https://review.opendev.org/731246 to make it more clear that the second change in this scenario is not automatically enqueued.
--
Jeremy Stanley