Setup and Config
Getting and Creating Projects
Basic Snapshotting
Branching and Merging
Sharing and Updating Projects
Inspection and Comparison
Patching
Debugging
External Systems
Server Admin
Guides
- gitattributes
- Command-line interface conventions
- Everyday Git
- Frequently Asked Questions (FAQ)
- Glossary
- Hooks
- gitignore
- gitmodules
- Revisions
- Submodules
- Tutorial
- Workflows
- All guides...
Administration
Plumbing Commands
- 2.37.1 → 2.37.7 no changes
- 2.37.0 06/27/22
The cruft packs feature offer an alternative to Git’s traditional mechanism of removing unreachable objects. This document provides an overview of Git’s pruning mechanism, and how a cruft pack can be used instead to accomplish the same.
Background
To remove unreachable objects from your repository, Git offers git repack -Ad
(see git-repack[1]). Quoting from the documentation:
[…] unreachable objects in a previous pack become loose, unpacked objects, instead of being left in the old pack. […] loose unreachable objects will be pruned according to normal expiry rules with the next git gc invocation.
Unreachable objects aren’t removed immediately, since doing so could race with an incoming push which may reference an object which is about to be deleted. Instead, those unreachable objects are stored as loose objects and stay that way until they are older than the expiration window, at which point they are removed by git-prune[1].
Git must store these unreachable objects loose in order to keep track of their per-object mtimes. If these unreachable objects were written into one big pack, then either freshening that pack (because an object contained within it was re-written) or creating a new pack of unreachable objects would cause the pack’s mtime to get updated, and the objects within it would never leave the expiration window. Instead, objects are stored loose in order to keep track of the individual object mtimes and avoid a situation where all cruft objects are freshened at once.
This can lead to undesirable situations when a repository contains many
unreachable objects which have not yet left the grace period. Having large
directories in the shards of .git/objects
can lead to decreased performance in
the repository. But given enough unreachable objects, this can lead to inode
starvation and degrade the performance of the whole system. Since we
can never pack those objects, these repositories often take up a large amount of
disk space, since we can only zlib compress them, but not store them in delta
chains.
Cruft packs
A cruft pack eliminates the need for storing unreachable objects in a loose state by including the per-object mtimes in a separate file alongside a single pack containing all loose objects.
A cruft pack is written by git repack --cruft
when generating a new pack.
git-pack-objects[1]'s --cruft
option. Note that git repack --cruft
is a classic all-into-one repack, meaning that everything in the resulting pack is
reachable, and everything else is unreachable. Once written, the --cruft
option instructs git repack
to generate another pack containing only objects
not packed in the previous step (which equates to packing all unreachable
objects together). This progresses as follows:
-
Enumerate every object, marking any object which is (a) not contained in a kept-pack, and (b) whose mtime is within the grace period as a traversal tip.
-
Perform a reachability traversal based on the tips gathered in the previous step, adding every object along the way to the pack.
-
Write the pack out, along with a
.mtimes
file that records the per-object timestamps.
This mode is invoked internally by git-repack[1] when instructed to write a cruft pack. Crucially, the set of in-core kept packs is exactly the set of packs which will not be deleted by the repack; in other words, they contain all of the repository’s reachable objects.
When a repository already has a cruft pack, git repack --cruft
typically only
adds objects to it. An exception to this is when git repack
is given the
--cruft-expiration
option, which allows the generated cruft pack to omit
expired objects instead of waiting for git-gc[1] to expire those objects
later on.
It is git-gc[1] that is typically responsible for removing expired unreachable objects.
Caution for mixed-version environments
Repositories that have cruft packs in them will continue to work with any older
version of Git. Note, however, that previous versions of Git which do not
understand the .mtimes
file will use the cruft pack’s mtime as the mtime for
all of the objects in it. In other words, do not expect older (pre-cruft pack)
versions of Git to interpret or even read the contents of the .mtimes
file.
Note that having mixed versions of Git GC-ing the same repository can lead to unreachable objects never being completely pruned. This can happen under the following circumstances:
-
An older version of Git running GC explodes the contents of an existing cruft pack loose, using the cruft pack’s mtime.
-
A newer version running GC collects those loose objects into a cruft pack, where the .mtime file reflects the loose object’s actual mtimes, but the cruft pack mtime is "now".
Repeating this process will lead to unreachable objects not getting pruned as a result of repeatedly resetting the objects' mtimes to the present time.
If you are GC-ing repositories in a mixed version environment, consider omitting
the --cruft
option when using git-repack[1] and git-gc[1], and
leaving the gc.cruftPacks
configuration unset until all writers understand
cruft packs.
Alternatives
Notable alternatives to this design include:
-
The location of the per-object mtime data, and
-
Storing unreachable objects in multiple cruft packs.
On the location of mtime data, a new auxiliary file tied to the pack was chosen
to avoid complicating the .idx
format. If the .idx
format were ever to gain
support for optional chunks of data, it may make sense to consolidate the
.mtimes
format into the .idx
itself.
Storing unreachable objects among multiple cruft packs (e.g., creating a new cruft pack during each repacking operation including only unreachable objects which aren’t already stored in an earlier cruft pack) is significantly more complicated to construct, and so aren’t pursued here. The obvious drawback to the current implementation is that the entire cruft pack must be re-written from scratch.