Resolve the latest release image to a published version - #7786
Open
CharlieTLe wants to merge 3 commits into
Open
Resolve the latest release image to a published version#7786CharlieTLe wants to merge 3 commits into
CharlieTLe wants to merge 3 commits into
Conversation
CharlieTLe
requested review from
a team and
danielblando
and removed request for
a team
August 20, 2026 19:09
Member
|
Thanks for catching this. I think the issue with this PR is: Can we ask the registry what is actually published? The registry is the only source of truth for "published".
|
CharlieTLe
added a commit
to CharlieTLe/cortex
that referenced
this pull request
Sep 8, 2026
Deriving the previous release from VERSION alone left one case broken, as SungJin1212 pointed out on cortexproject#7786: on the GA tag push VERSION is 1.22.0 with no pre-release suffix, so it resolves to v1.22.0 — an image that only `deploy` publishes, and `deploy` needs `integration` to pass first. Ask quay.io what actually exists instead. The registry is the only source of truth for "published", so list the GA tags (^v\d+\.\d+\.\d+$) and take the highest one that does not exceed VERSION, then export it as CORTEX_LATEST_RELEASE_IMAGE for the preload and test steps. The <= bound only changes the result when a newer release already exists on quay than the branch being tested, e.g. preparing 1.21.2 on release-1.21 after v1.22.0 has shipped. This also drops the bash mirror of the Go derivation, and stops guessing at the previous minor's .0: with the registry answering, 1.19.5-rc.0 resolves to the v1.19.1 that exists rather than a v1.19.4 that never shipped, and a major pre-release such as 2.0.0-rc.0 no longer needs a manual override. The derivation stays in integration/util.go as the offline fallback for local runs, where no env var is set and no network call is wanted. Signed-off-by: Charlie Le <charlie_le@apple.com>
SungJin1212
reviewed
Sep 8, 2026
| @@ -0,0 +1,94 @@ | |||
| //go:build integration | |||
Member
There was a problem hiding this comment.
The discover-tags job excludes integration so util.test cannot be tested. We need to use something like an integration_query_fuzz.
done < <(grep -hE "^//go:build " integration/*.go \
| sed -E 's|^//go:build ||' \
| sort -u | grep -v '^integration$')
Member
Author
There was a problem hiding this comment.
Good catch, and it was worse than dead weight — thank you.
You're right that nothing ran. The binary did compile them, since the compile step sets ALL_TAGS="slicelabels,integration,${tags_csv}", but Generate Run-Pattern Manifest iterates tags_csv alone, so no -test.run regex ever named them. I reproduced the pattern generation across all 12 matrix tags and every one selected zero of the three functions.
Moved to integration_query_fuzz in 58b5278, since that's the only leg calling getLatestReleaseImage().
Retagging then exposed a second bug that your comment saved me from shipping: that leg is exactly where the new resolve step exports CORTEX_LATEST_RELEASE_IMAGE, which short-circuits getLatestReleaseImage(). So the moment the test started running, it asserted against the override instead of the derivation it exists to cover:
--- FAIL: TestGetLatestReleaseImage
expected: "quay.io/cortexproject/cortex:v1.21.0"
actual : "quay.io/cortexproject/cortex:v1.21.1"
Same commit clears the variable for that test. I verified by reverting the guard that the failure is real, then re-ran the way CI does — compiled with the full ALL_TAGS, regenerated the run-pattern (all three now selected), and ran the binary with CORTEX_LATEST_RELEASE_IMAGE and CORTEX_CHECKOUT_DIR set as the query_fuzz leg sets them. All three pass.
I should be straight about one thing: I'd said earlier this was verified, but that run used -tags integration — the one tag CI never sets. It proved nothing about CI, which is exactly the gap you found.
The registry lookup you suggested is in f4182fd; the PR description has the resolution table.
integration/util.go derived the "latest release" image straight from the VERSION file. That holds on master, where VERSION is the last GA, but not on a release branch: VERSION is bumped to the version being prepared (e.g. 1.22.0-rc.0) long before the deploy job publishes that tag, and the integration job is a dependency of deploy. So the query fuzz leg would pull an image that does not exist yet. Resolve a pre-release version to the release preceding it instead, and add CORTEX_LATEST_RELEASE_IMAGE as an escape hatch for the cases the version math cannot cover (a major pre-release). The preload step in test-build-deploy.yml mirrors the same rule. Signed-off-by: Charlie Le <charlie_le@apple.com>
Deriving the previous release from VERSION alone left one case broken, as SungJin1212 pointed out on cortexproject#7786: on the GA tag push VERSION is 1.22.0 with no pre-release suffix, so it resolves to v1.22.0 — an image that only `deploy` publishes, and `deploy` needs `integration` to pass first. Ask quay.io what actually exists instead. The registry is the only source of truth for "published", so list the GA tags (^v\d+\.\d+\.\d+$) and take the highest one that does not exceed VERSION, then export it as CORTEX_LATEST_RELEASE_IMAGE for the preload and test steps. The <= bound only changes the result when a newer release already exists on quay than the branch being tested, e.g. preparing 1.21.2 on release-1.21 after v1.22.0 has shipped. This also drops the bash mirror of the Go derivation, and stops guessing at the previous minor's .0: with the registry answering, 1.19.5-rc.0 resolves to the v1.19.1 that exists rather than a v1.19.4 that never shipped, and a major pre-release such as 2.0.0-rc.0 no longer needs a manual override. The derivation stays in integration/util.go as the offline fallback for local runs, where no env var is set and no network call is wanted. Signed-off-by: Charlie Le <charlie_le@apple.com>
util_test.go was tagged `integration`, which discover-tags filters out of the
matrix (`grep -v '^integration$'`). The binary still compiled the tests, since
the compile step sets ALL_TAGS="slicelabels,integration,${tags_csv}", but the
run-pattern manifest is built from tags_csv alone, so no -test.run regex ever
named them and they never executed.
Move them to `integration_query_fuzz`, the only leg that calls
getLatestReleaseImage().
Running them then exposed a second problem: that leg now exports
CORTEX_LATEST_RELEASE_IMAGE, which short-circuits getLatestReleaseImage() and
made TestGetLatestReleaseImage assert against the override instead of the
derivation it is meant to cover. Clear the variable for that test.
Signed-off-by: Charlie Le <charlie_le@apple.com>
CharlieTLe
force-pushed
the
release-1.22-pr-a-fix-latest-release-image
branch
from
September 12, 2026 18:58
58b5278 to
f9f15ee
Compare
CharlieTLe
added a commit
that referenced
this pull request
Sep 12, 2026
* Pull minio from quay.io instead of Docker Hub Every integration leg is failing at Preload Images: Error response from daemon: pull access denied for minio/minio, repository does not exist or may require 'docker login': denied: requested access to the resource is denied minio/minio is the first Docker Hub pull in the step, so no leg gets past it and all 24 fail in about 30 seconds. A CHANGELOG-only pull request reproduces it, so this is not specific to any change under test. master was last green at 4061a3d. This is not a rate limit: the same pull fails right after a successful 'docker login' with the repository credentials. The docker.io/minio/minio repository is simply no longer accessible. MinIO still publishes the identical image to quay.io. quay.io/minio/minio :RELEASE.2024-05-28T17-19-04Z is public and is a manifest list with 8 children, so it covers both the amd64 and arm64 runners. Point the integration tests, the CI preload list, and the three development docker-compose stacks at quay.io. The tag is unchanged, so no behaviour changes. Signed-off-by: Charlie Le <charlie_le@apple.com> * Authenticate the integration job's remaining Docker Hub pulls Preload Images still pulls consul, memcached, redis and postgres from Docker Hub. #7464 removed the Install Docker Client step from this job, and that script is where 'docker login' runs, so those pulls have been anonymous since and are subject to the anonymous rate limit. Log in explicitly, matching what the build job does. Pull requests from forks have no secrets, so skip the login there and leave those pulls anonymous instead of failing the step. This is hardening, not the fix for the current breakage: minio failed even when authenticated. Signed-off-by: Charlie Le <charlie_le@apple.com> * Resolve the latest release image to a published version Backport of #7786 onto release-1.22. integration/util.go derived the query fuzz comparison image straight from VERSION. The moment VERSION becomes 1.22.0-rc.0 on this branch, integration_query_fuzz tries to pull quay.io/cortexproject/cortex:v1.22.0-rc.0, which does not exist: that image is pushed by the tag build's deploy job, and deploy is gated on integration passing first. Ask quay.io which GA tags are actually published instead, and take the highest one at or below VERSION. The CI step mirrors the same resolution and exports CORTEX_LATEST_RELEASE_IMAGE for the preload step. Set the CORTEX_LATEST_RELEASE_IMAGE repository variable to bypass the lookup. Signed-off-by: Charlie Le <charlie_le@apple.com> * Mark release 1.22.0 in progress Add a '## 1.22.0 in progress' section below an empty '## master / unreleased', move the existing unreleased entries into it, and order them [CHANGE] -> [FEATURE] -> [ENHANCEMENT] -> [BUGFIX] per RELEASE.md. Also fill the gaps reported by ./tools/release/check-changelog.sh v1.21.1...master: - new entries for #7513, #7514 and #7559 - fold #7323, #7434, #7458, #7463, #7487, #7505, #7687, #7691, #7716, #7726, #7775 and #7807 into the entries they belong to Signed-off-by: Charlie Le <charlie_le@apple.com> * Update version to 1.22.0-rc.0 Signed-off-by: Charlie Le <charlie_le@apple.com> --------- Signed-off-by: Charlie Le <charlie_le@apple.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
getLatestReleaseImage()inintegration/util.goderives the "latest release" imagestraight from the
VERSIONfile, and.github/workflows/test-build-deploy.ymlmirrors itas
docker pull quay.io/cortexproject/cortex:v$(cat testdata/VERSION).That holds on
master, whereVERSIONis the last GA and its image exists. It does nothold on a release branch:
VERSIONis bumped to the version being prepared (e.g.1.22.0-rc.0) long before anything publishes that tag, and thedeployjob that wouldpublish it has
needs: [build, test, lint, integration]. So theintegration_query_fuzzleg would try to pull an image that does not exist yet, and
release-1.22would be redfrom the moment
VERSIONgains its-rc.0suffix.The same applies on the GA tag push, where
VERSIONis1.22.0with no suffix: thev1.22.0image is only pushed bydeploy, which again runs afterintegration.This mechanism landed in #7737 (2026-07-30), after v1.21.1, so it has never been through
a release.
The fix
VERSIONnames the version being prepared, so it cannot answer "what is published".The registry can, so ask it — thanks @SungJin1212 for the suggestion.
A new
Resolve Latest Release Imagestep lists the GA tags (^v\d+\.\d+\.\d+$) publishedto quay.io, takes the highest one that does not exceed
VERSION, and exports it asCORTEX_LATEST_RELEASE_IMAGEfor the preload and test steps. It runs only on theintegration_query_fuzzmatrix leg, which is the only caller ofgetLatestReleaseImage().Against a registry whose newest GA is
v1.21.1:VERSION1.21.1v1.21.11.22.0-rc.0v1.21.11.22.0v1.21.11.21.2-rc.0v1.21.12.0.0-rc.0v1.21.11.19.0v1.19.01.19.5-rc.0v1.19.11.19.4The
<=bound (rather than simply "the newest published GA tag") only changes the resultwhen a newer release already exists on quay than the branch being tested, e.g. preparing
1.21.2onrelease-1.21afterv1.22.0has shipped.Setting the
CORTEX_LATEST_RELEASE_IMAGErepository variable bypasses the lookupentirely.
The bash mirror of the Go derivation is gone.
latestReleaseVersion()stays inintegration/util.goas the offline fallback for local runs, where nothing sets the envvar and a network call is unwelcome; CI always resolves against the registry.
Verification
bash -e(what Actionsactually uses — no
pipefail, nonounset) against the live registry, producing everyrow in the table above, plus errors for a
VERSIONbelow everything published and for amalformed
VERSION. The repository-variable override was exercised too.shellcheckis clean on the step.actionlintreports the same 13 pre-existing SC2086findings as
masterand no new ones.latestReleaseVersion()keeps its table test;getLatestReleaseImage()is tested end toend against a scratch checkout and for the env override.
go vetpasses with every integration build tag set.