mirror of
https://github.com/git/git.git
synced 2024-05-06 12:16:14 +02:00
2f4ba2a867
Specify the format of the on-disk reverse index 'pack-*.rev' file, as well as prepare the code for the existence of such files. The reverse index maps from pack relative positions (i.e., an index into the array of object which is sorted by their offsets within the packfile) to their position within the 'pack-*.idx' file. Today, this is done by building up a list of (off_t, uint32_t) tuples for each object (the off_t corresponding to that object's offset, and the uint32_t corresponding to its position in the index). To convert between pack and index position quickly, this array of tuples is radix sorted based on its offset. This has two major drawbacks: First, the in-memory cost scales linearly with the number of objects in a pack. Each 'struct revindex_entry' is sizeof(off_t) + sizeof(uint32_t) + padding bytes for a total of 16. To observe this, force Git to load the reverse index by, for e.g., running 'git cat-file --batch-check="%(objectsize:disk)"'. When asking for a single object in a fresh clone of the kernel, Git needs to allocate 120+ MB of memory in order to hold the reverse index in memory. Second, the cost to sort also scales with the size of the pack. Luckily, this is a linear function since 'load_pack_revindex()' uses a radix sort, but this cost still must be paid once per pack per process. As an example, it takes ~60x longer to print the _size_ of an object as it does to print that entire object's _contents_: Benchmark #1: git.compile cat-file --batch <obj Time (mean ± σ): 3.4 ms ± 0.1 ms [User: 3.3 ms, System: 2.1 ms] Range (min … max): 3.2 ms … 3.7 ms 726 runs Benchmark #2: git.compile cat-file --batch-check="%(objectsize:disk)" <obj Time (mean ± σ): 210.3 ms ± 8.9 ms [User: 188.2 ms, System: 23.2 ms] Range (min … max): 193.7 ms … 224.4 ms 13 runs Instead, avoid computing and sorting the revindex once per process by writing it to a file when the pack itself is generated. The format is relatively straightforward. It contains an array of uint32_t's, the length of which is equal to the number of objects in the pack. The ith entry in this table contains the index position of the ith object in the pack, where "ith object in the pack" is determined by pack offset. One thing that the on-disk format does _not_ contain is the full (up to) eight-byte offset corresponding to each object. This is something that the in-memory revindex contains (it stores an off_t in 'struct revindex_entry' along with the same uint32_t that the on-disk format has). Omit it in the on-disk format, since knowing the index position for some object is sufficient to get a constant-time lookup in the pack-*.idx file to ask for an object's offset within the pack. This trades off between the on-disk size of the 'pack-*.rev' file for runtime to chase down the offset for some object. Even though the lookup is constant time, the constant is heavier, since it can potentially involve two pointer walks in v2 indexes (one to access the 4-byte offset table, and potentially a second to access the double wide offset table). Consider trying to map an object's pack offset to a relative position within that pack. In a cold-cache scenario, more page faults occur while switching between binary searching through the reverse index and searching through the *.idx file for an object's offset. Sure enough, with a cold cache (writing '3' into '/proc/sys/vm/drop_caches' after 'sync'ing), printing out the entire object's contents is still marginally faster than printing its size: Benchmark #1: git.compile cat-file --batch-check="%(objectsize:disk)" <obj >/dev/null Time (mean ± σ): 22.6 ms ± 0.5 ms [User: 2.4 ms, System: 7.9 ms] Range (min … max): 21.4 ms … 23.5 ms 41 runs Benchmark #2: git.compile cat-file --batch <obj >/dev/null Time (mean ± σ): 17.2 ms ± 0.7 ms [User: 2.8 ms, System: 5.5 ms] Range (min … max): 15.6 ms … 18.2 ms 45 runs (Numbers taken in the kernel after cheating and using the next patch to generate a reverse index). There are a couple of approaches to improve cold cache performance not pursued here: - We could include the object offsets in the reverse index format. Predictably, this does result in fewer page faults, but it triples the size of the file, while simultaneously duplicating a ton of data already available in the .idx file. (This was the original way I implemented the format, and it did show `--batch-check='%(objectsize:disk)'` winning out against `--batch`.) On the other hand, this increase in size also results in a large block-cache footprint, which could potentially hurt other workloads. - We could store the mapping from pack to index position in more cache-friendly way, like constructing a binary search tree from the table and writing the values in breadth-first order. This would result in much better locality, but the price you pay is trading O(1) lookup in 'pack_pos_to_index()' for an O(log n) one (since you can no longer directly index the table). So, neither of these approaches are taken here. (Thankfully, the format is versioned, so we are free to pursue these in the future.) But, cold cache performance likely isn't interesting outside of one-off cases like asking for the size of an object directly. In real-world usage, Git is often performing many operations in the revindex (i.e., asking about many objects rather than a single one). The trade-off is worth it, since we will avoid the vast majority of the cost of generating the revindex that the extra pointer chase will look like noise in the following patch's benchmarks. This patch describes the format and prepares callers (like in pack-revindex.c) to be able to read *.rev files once they exist. An implementation of the writer will appear in the next patch, and callers will gradually begin to start using the writer in the patches that follow after that. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
71 lines
2.4 KiB
C
71 lines
2.4 KiB
C
#ifndef PACK_REVINDEX_H
|
|
#define PACK_REVINDEX_H
|
|
|
|
/**
|
|
* A revindex allows converting efficiently between three properties
|
|
* of an object within a pack:
|
|
*
|
|
* - index position: the numeric position within the list of sorted object ids
|
|
* found in the .idx file
|
|
*
|
|
* - pack position: the numeric position within the list of objects in their
|
|
* order within the actual .pack file (i.e., 0 is the first object in the
|
|
* .pack, 1 is the second, and so on)
|
|
*
|
|
* - offset: the byte offset within the .pack file at which the object contents
|
|
* can be found
|
|
*/
|
|
|
|
#define RIDX_SIGNATURE 0x52494458 /* "RIDX" */
|
|
#define RIDX_VERSION 1
|
|
|
|
struct packed_git;
|
|
|
|
/*
|
|
* load_pack_revindex populates the revindex's internal data-structures for the
|
|
* given pack, returning zero on success and a negative value otherwise.
|
|
*
|
|
* If a '.rev' file is present it is mmap'd, and pointers are assigned into it
|
|
* (instead of using the in-memory variant).
|
|
*/
|
|
int load_pack_revindex(struct packed_git *p);
|
|
|
|
/*
|
|
* offset_to_pack_pos converts an object offset to a pack position. This
|
|
* function returns zero on success, and a negative number otherwise. The
|
|
* parameter 'pos' is usable only on success.
|
|
*
|
|
* If the reverse index has not yet been loaded, this function loads it lazily,
|
|
* and returns an negative number if an error was encountered.
|
|
*
|
|
* This function runs in time O(log N) with the number of objects in the pack.
|
|
*/
|
|
int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos);
|
|
|
|
/*
|
|
* pack_pos_to_index converts the given pack-relative position 'pos' by
|
|
* returning an index-relative position.
|
|
*
|
|
* If the reverse index has not yet been loaded, or the position is out of
|
|
* bounds, this function aborts.
|
|
*
|
|
* This function runs in constant time.
|
|
*/
|
|
uint32_t pack_pos_to_index(struct packed_git *p, uint32_t pos);
|
|
|
|
/*
|
|
* pack_pos_to_offset converts the given pack-relative position 'pos' into a
|
|
* pack offset. For a pack with 'N' objects, asking for position 'N' will return
|
|
* the total size (in bytes) of the pack.
|
|
*
|
|
* If the reverse index has not yet been loaded, or the position is out of
|
|
* bounds, this function aborts.
|
|
*
|
|
* This function runs in constant time under both in-memory and on-disk reverse
|
|
* indexes, but an additional step is taken to consult the corresponding .idx
|
|
* file when using the on-disk format.
|
|
*/
|
|
off_t pack_pos_to_offset(struct packed_git *p, uint32_t pos);
|
|
|
|
#endif
|