pacman-helper: maintain per-package .versions.json index files#695
Open
pacman-helper: maintain per-package .versions.json index files#695
Conversation
The Pacman repository for Git for Windows migrated from Azure Blob Storage to branches in git-for-windows/pacman-repo. Unlike the previous setup where all package versions accumulated in the same container, the Git-based repository only retains the latest version of each package in its branch, replacing older files when new versions are deployed. The package files are still available as assets of GitHub Releases (each tagged with a UTC timestamp), but without an index there is no way to discover which Release holds a particular version. To solve this, each architecture branch now carries per-package .versions.json files (e.g. mingw-w64-x86_64-curl-winssl-pdb.versions.json) that map version-pkgrel strings to the Release tag where the asset can be downloaded. The per-package granularity was chosen over a single monolithic index to keep individual files small and, more importantly, to make merge conflicts during concurrent deployments essentially impossible since two deployments updating different packages touch different files. The Release tag name is now generated before the per-architecture loop rather than after the push, so that it is available when writing the .versions.json entries into the branch commit. The tag is a UTC timestamp, so moving its generation earlier has no functional effect on the GitHub Release created later. In the concurrent-deployment rebase path, .versions.json changes are stripped from the commit before rebasing to avoid conflicts with the other deployment's .versions.json updates, then re-inserted afterwards via the idempotent update_versions_json helper which reads the rebased state, adds our entries, and re-sorts the keys. Assisted-by: Claude Opus 4.6 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Member
Author
|
We should merge this before the next |
mjcheetham
reviewed
May 7, 2026
| msys_sf_source_url=$sf_repos_url/MSYS2/Sources | ||
| mingw_sf_source_url=$sf_repos_url/MINGW/Sources | ||
| azure_blobs_source_url=https://wingit.blob.core.windows.net/sources | ||
| pacman_repo_url=https://raw.githubusercontent.com/git-for-windows/pacman-repo/refs/heads |
Member
There was a problem hiding this comment.
Are we concerned about hitting usage limits unauthenticated?
Member
Author
There was a problem hiding this comment.
Hrm. That's a good point. Raw endpoint is rate-limited without a token.
How about using a GITHUB_TOKEN if it is found in the environment?
mjcheetham
approved these changes
May 7, 2026
The bundle_pdbs function downloads .pdb packages from the Pacman repository to include debug symbols in the installer. When the requested version is no longer the latest (because a newer version has been deployed), the package file is absent from the branch and the download fails with curl error 22, as seen at https://github.com/git-for-windows/git-sdk-64/actions/runs/25404775441/job/74515324990#step:7:28 Previously this was papered over with hard-coded case statements mapping specific versions to their Release tags. Replace that with a lookup in the per-package .versions.json index that pacman-helper.sh now maintains in each architecture branch. The index maps every version-pkgrel to the GitHub Release tag where the .pkg.tar asset is available. If the index does not exist or does not contain the requested version, the code falls back to the branch URL, preserving backwards compatibility with branches that have not been seeded yet. Assisted-by: Claude Opus 4.6 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The source tarballs that get-sources.sh downloads for GPL compliance are uploaded as GitHub Release assets alongside the binary packages, sharing the same Release tag. Now that the per-package .versions.json index exists in the pacman-repo branches, we can look up the Release tag for any package version and download the source tarball directly from the Release. The lookup uses the x86_64 binary package's .versions.json to deduce the Release tag, relying on the fact that source and binary packages are always deployed together in the same Release. This is admittedly fragile, but no more so than the existing assumption that the Azure Blob "sources" container has the file. The existing URLs (Azure Blobs, repo.msys2.org, SourceForge) remain as fallbacks in the download chain. This is a prerequisite for turning off the Azure Blob Storage deployment, which currently serves as the only download source for Git-for-Windows-specific source packages. Assisted-by: Claude Opus 4.6 Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Pacman repository for Git for Windows migrated from Azure Blob Storage to branches in git-for-windows/pacman-repo. Unlike the previous setup where all package versions accumulated in the same container, the Git-based repository only retains the latest version of each package in its branch. Older versions are still available as GitHub Release assets, but there was no way to discover which Release holds a particular version.
This has already caused CI failures when a job needed a package version that had been superseded, e.g. https://github.com/git-for-windows/git-sdk-64/actions/runs/25404775441/job/74515324990#step:7:28
This PR adds per-package
.versions.jsonindex files to each architecture branch of pacman-repo, mapping every version-pkgrel to the GitHub Release tag where the.pkg.tarasset can be downloaded.Changes
pacman-helper.sh: Maintains
.versions.jsonduring deployment. Theupdate_versions_jsonhelper usessed/sort(no perl) to read, insert, sort, and write entries. In the concurrent-deployment rebase path,.versions.jsonchanges are stripped before rebase and re-inserted afterwards to avoid conflicts.please.sh: Replaces hard-coded curl 8.18.0-1 workarounds with a
.versions.jsonlookup for PDB package downloads. Falls back to the branch URL when the index is unavailable.get-sources.sh: Uses the x86_64 binary package's
.versions.jsonto deduce the Release tag for source tarballs, making source packages downloadable from GitHub Releases ahead of turning off Azure Blob Storage.pacman-repo seeding
The three architecture branches have been seeded with
.versions.jsonfiles built from all 1172 existing releases (97 packages / 2796 versions for x86_64, 112 / 2696 for i686, 46 / 932 for aarch64): git-for-windows/pacman-repo@c5122e6 git-for-windows/pacman-repo@84eb68d git-for-windows/pacman-repo@cfcc281