[Starlingx-discuss] DockerHub rate limit issues

[Scott.Little]

Hi Ildiko

The majority of StarlingX developmers are employed by WindRiver. In 
addition, the daily StarlingX builds and PV effort have both moved to 
WindRiver hosted machines recently.  Now the documented StarlingX 
working model was to use anonymous pulls. So it was WindRiver's external 
IP address, rather than user accounts, that was getting 'billed' for all 
this activity.

Scott


On 2023-08-29 16:15, Ildiko Vancsa wrote:
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
> Hi Scott,
>
> Thank you for the additional details.
>
> Another quick question to clarify. When you say "drove our quota down”, what does that mean exactly? What is “our” in this context? And is that quota assigned to the IP address, or to a DockerHub user, or the images themselves?
>
> Thanks,
> Ildikó
>
> ———
>
> Ildikó Váncsa
> Director of Community
> Open Infrastructure Foundation
>
>
>
>
>
>> On Aug 29, 2023, at 12:48, Scott Little <scott.little at windriver.com> wrote:
>>
>> It was hitting many images, not just golang.  I just don't have any logs left from those failures.
>>
>> I don't see the OSS mark on the golang images, so perhaps those images drove our quota down to zero. However once the quota hit zero, then any anonymous download of an image could trigger the download failure, including OSS ones. The strange part was that the OSS download failures are only seen within build containers.  The same image could be anonymously download on the containers host just fine.  Both container and host were being billed to the same IP address.
>>
>> Scott
>>
>>
>> On 2023-08-21 11:36, Ildiko Vancsa wrote:
>>> CAUTION: This email comes from a non Wind River email account!
>>> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>>>
>>> Hi Scott,
>>>
>>> I also just recognized, that the snipped that you copied in refers to a golang image: “Step 1/9 : FROMgolang:1.19 toomanyrequests: You have reached your pull rate limit.”. DO you have the same issue with StarlingX images as well? Or is it always the golang step that fails?
>>>
>>> Thanks,
>>> Ildikó
>>>
>>> ———
>>>
>>> Ildikó Váncsa
>>> Director of Community
>>> Open Infrastructure Foundation
>>>
>>>
>>>
>>>
>>>
>>>> On Aug 18, 2023, at 12:13, Ildiko Vancsa <ildiko at openinfra.dev> wrote:
>>>>
>>>> Hi Scott,
>>>>
>>>> Great, thank you for the pointers.
>>>>
>>>> I reached out to Docker on the channel that is dedicated for their open source program, and also included the links that you added below so they have the full context. I will get back with updates on this thread, if I receive anything beyond what you got on the forums already.
>>>>
>>>> Thanks and Best Regards,
>>>> Ildikó
>>>>
>>>> ———
>>>>
>>>> Ildikó Váncsa
>>>> Director of Community
>>>> Open Infrastructure Foundation
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> On Aug 17, 2023, at 07:40, Scott Little <scott.little at windriver.com> wrote:
>>>>>
>>>>> Here is a sample from a failed build ...
>>>>>
>>>>> docker build . --no-cache ...
>>>>> Sending build context to Docker daemon  37.68MB
>>>>>
>>>>> Step 1/9 : FROMgolang:1.19
>>>>> toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading:https://www.docker.com/increase-rate-limit
>>>>> make: *** [builder-build] Error 1
>>>>>
>>>>> One thing to note, It's pulls from within our build container that hits the problem.  We don't see it when pulling from bare metal hardware.
>>>>>
>>>>> I've tried raising the issue with the docker community here ...
>>>>>
>>>>> https://github.com/moby/moby/issues/46092#issuecomment-1654243988
>>>>>
>>>>> https://forums.docker.com/t/inconsistent-behaviour-of-docker-pull-rate-limit/136896/2
>>>>>
>>>>> We seem to be avoiding the rate limit by using docker login. Formal StarlingX builds are using a couple of service accounts. Designers were asked establish there own personal accounts with dockerhub.
>>>>>
>>>>>
>>>>>
>>>>> On 2023-08-16 16:17, Ildiko Vancsa wrote:
>>>>>> CAUTION: This email comes from a non Wind River email account!
>>>>>> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>>>>>>
>>>>>> Hi Scott,
>>>>>>
>>>>>> I looked into the DockerHub download issue again, to the capacity that I could. To confirm what was mentioned already, I have not found anything that would explain what the community is experiencing in terms of rate limits:
>>>>>>
>>>>>>   - The StarlingX project is still listed as ’Sponsored OSS’: https://hub.docker.com/u/starlingx
>>>>>>   - The DockerHub docs point to a blog post that says “we will suspend data pull rate restrictions, where no egress restrictions will apply to any Docker users pulling images from the approved OSS namespaces”: https://www.docker.com/blog/expanded-support-for-open-source-software-projects/
>>>>>>   - I haven’t received any notifications from DockerHub recently about not meeting the criteria to participate in their open source program anymore
>>>>>>
>>>>>> Can you please share a screenshot or the error message that you are getting about hitting a pull rate limit? Also, are you getting that on every repo, or just specific ones?
>>>>>>
>>>>>> Once having more background information, I will reach out to the DockerHub team to see what the background issue might be.
>>>>>>
>>>>>> Thanks and Best Regards,
>>>>>> Ildikó
>>>>>>
>>>>>> ———
>>>>>>
>>>>>> Ildikó Váncsa
>>>>>> Director of Community
>>>>>> Open Infrastructure Foundation
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Starlingx-discuss mailing list
>>>>> Starlingx-discuss at lists.starlingx.io
>>>> _______________________________________________
>>>> Starlingx-discuss mailing list
>>>> Starlingx-discuss at lists.starlingx.io
>>