The recent safety check to trust only the commits we have made
things impossibly slow and turn out to waste a lot of memory.
This commit fixes it with the following improvements:
- mark already scanned objects and avoid rescanning the same
object again;
- free the tree entries when we have scanned the tree entries;
this is the same as b0d8923ec01fd91b75ab079034f89ced91500157
which reduced memory usage by rev-list;
- plug memory leak from the object_list dequeuing code;
- use the process_queue not just for fetching but for scanning,
to make things tail recursive to avoid deep recursion; the
deep recursion was especially prominent when we cloned a big
pack.
- avoid has_sha1_file() call when we already know we do not have
that object.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Using "git repack -a -d" can destroy your git archive if you use it
twice in succession, because the new pack can be called the same as
the old pack. Found by Linus.
Signed-off-by: Junio C Hamano <junkio@cox.net>
As a convenience measure, 'bisect bad' or 'bisect good' automatically
does 'bisect next' when it knows it can, but the result of that test
to see if it can was leaking through as the exit code from the whole
thing, which was bad. Noticed by Anton Blanchard.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This tries .../objects/info/http-alternates and then
.../objects/info/alternates, looking for a file which specifies where
else to download objects and packs from.
It currently only supports absolute paths, and doesn't support full URLs.
Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
I do not know why it was spelled git-rev-tree when I meant to say
git-read-tree, but the typo was left since day one.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Jason Riedy suggests that we should be able to use getdomainname
if we properly specify which libraries to link.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This is a nicer fix for git-shortlog being unable to handle the raw log
format. Just use a more permissive regexp instead of doing two nearly
identical ones.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
There were two bugs in there:
- if the range didn't end up working, we restored the '.' character in
the wrong place.
- an empty end-of-range should be interpreted as HEAD.
See rev-parse.c for the reference implementation of this.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
For local operations and downloading and uploading via git aware protocols,
use of $GIT_OBJECT_DIRECTORY/info/alternates is recommended on the server
side for big projects that are derived from another one (like Linux kernel).
However, dumb protocols and rsync transport needs to resolve this on the
client end, which we did not bother doing until this week.
I noticed we use "rsync -z" but most of our payload is already compressed,
which was not quite right. This commit also fixes it.
Signed-off-by: Junio C Hamano <junkio@cox.net>
The object parsing code builds a generic "this object references that
object" because doing a full connectivity check for fsck requires it.
However, nothing else really needs it, and it's quite expensive for
git-rev-list that can have tons of objects in flight.
So, exactly like the commit buffer save thing, add a global flag to
disable it, and use it in git-rev-list.
Before:
$ /usr/bin/time git-rev-list --objects v2.6.12..HEAD | wc -l
12.28user 0.29system 0:12.57elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+26718minor)pagefaults 0swaps
59124
After this change:
$ /usr/bin/time git-rev-list --objects v2.6.12..HEAD | wc -l
10.33user 0.18system 0:10.54elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+18509minor)pagefaults 0swaps
59124
and note how the number of pages touched by git-rev-list for this
particular object list has shrunk from 26,718 (104 MB) to 18,509 (72 MB).
Calculating the total object difference between two git revisions is still
clearly the most expensive git operation (both in memory and CPU time),
but it's now less than 40% of what it used to be.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This avoids keeping tree entries around, and free's them as it traverses
the list. This avoids building up a huge memory footprint just for these
small but very common allocations.
Before:
$ /usr/bin/time git-rev-list --objects v2.6.12..HEAD | wc -l
11.65user 0.38system 0:12.65elapsed 95%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+42934minor)pagefaults 0swaps
59124
After:
$ /usr/bin/time git-rev-list --objects v2.6.12..HEAD | wc -l
12.28user 0.29system 0:12.57elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+26718minor)pagefaults 0swaps
59124
Note how the minor fault numbers - which ends up being how many pages we
needed to map - go down from 42934 (167 MB) to 26718 (104 MB). That is:
Before:
42934 minor pagefaults
After:
26718 minor pagefaults
This is all in _addition_ to the previous fixes. It used to be
~48,000 pagefaults.
That's still a honking big memory footprint, but it's about half of what
it was just a day or two ago (and this is the object list for a pretty big
update - almost 60,000 objects. Smaller updates need less memory).
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Build systems should run tests. This patch adds the necessary
debian/control and debian/rules bits ("bc" was missing,
t/t4002-diff-basic.sh wants it).
Signed-off-by: Matthias Urlichs <smurf@smurf.noris.de>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Update git-core spec file based on feedback from Fedora Extras review.
- update BuildRoot to be more specific
- eliminate Requires that must be satisfied for base system install
- drop Vendor
- use dist tag to differentiate between branches
- own %{_datadir}/git-core/
- use RPM_OPT_FLAGS in spec file
Signed-off-by: Chris Wright <chrisw@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
An earlier commit causes a mismatch in <emphasis> and <superscript>
tags, one way of fixing it is having no more than one caret symbol per
line, which is the only solution I found in the asciidoc
documentation. Ugly, but it works.
[jc: ugly indeed but that is not Peter's fault.]
Signed-off-by: Peter Hagervall <hager@cs.umu.se>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Fix the "superscript" problem on the git-rev-list doc page.
Signed-off-by: Jon Loeliger <jdl@freescale.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
The logic to calculate the full object list used to be very inter-twined
with the logic that looked up the commits.
For no good reason - it's actually a lot simpler to just do that logic
as a separate pass.
This improves performance a bit, and uses slightly less memory in my
tests, but more importantly it makes the code simpler to work with and
follow what it does.
The performance win is less than I had hoped for, but I get:
Before:
[torvalds@g5 linux]$ /usr/bin/time git-rev-list --objects v2.6.12..HEAD | wc -l
13.64user 0.42system 0:14.13elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+47947minor)pagefaults 0swaps
58945
After:
[torvalds@g5 linux]$ /usr/bin/time git-rev-list --objects v2.6.12..HEAD | wc -l
11.80user 0.36system 0:12.16elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+42684minor)pagefaults 0swaps
58945
ie it improved by 2 seconds, and took a 5000+ fewer pages (hey, that's
20MB out of 174MB to go). And got the same number of objects (in theory,
the more expensive one might find some more shared objects to avoid. In
practice it obviously doesn't).
I know how to make it use _lots_ less memory, which will probably speed it
up. But that's for another time, and I'd prefer to see this go in first.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
It is a bit embarrassing that it took this long for a fix since the
problem was first reported on Aug 13th.
Message-ID: <87y876gl1r.wl@mail2.atmark-techno.com>
From: Yasushi SHOJI <yashi@atmark-techno.com>
Newsgroups: gmane.comp.version-control.git
Subject: [patch] possible memory leak in diff.c::diff_free_filepair()
Date: Sat, 13 Aug 2005 19:58:56 +0900
This time I used valgrind to make sure that it does not overeagerly
discard memory that is still being used.
Signed-off-by: Junio C Hamano <junkio@cox.net>
As pointed out on the list, git-rev-list can use a lot of memory.
One low-hanging fruit is to free the commit buffer for commits that we
parse. By default, parse_commit() will save away the buffer, since a lot
of cases do want it, and re-reading it continually would be unnecessary.
However, in many cases the buffer isn't actually necessary and saving it
just wastes memory.
We could just free the buffer ourselves, but especially in git-rev-list,
we actually end up using the helper functions that automatically add
parent commits to the commit lists, so we don't actually control the
commit parsing directly.
Instead, just make this behaviour of "parse_commit()" a global flag.
Maybe this is a bit tasteless, but it's very simple, and it makes a
noticable difference in memory usage.
Before the change:
[torvalds@g5 linux]$ /usr/bin/time git-rev-list v2.6.12..HEAD > /dev/null
0.26user 0.02system 0:00.28elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+3714minor)pagefaults 0swaps
after the change:
[torvalds@g5 linux]$ /usr/bin/time git-rev-list v2.6.12..HEAD > /dev/null
0.26user 0.00system 0:00.27elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+2433minor)pagefaults 0swaps
note how the minor faults have decreased from 3714 pages to 2433 pages.
That's all due to the fewer anonymous pages allocated to hold the comment
buffers and their metadata.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
HPA reminded me that these programs knows about the name of the
counterpart on the other end and simply symlinking the old name to
new name locally would not be enough.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This patch does proper quoting, and uses "env" to be compatible with
tcsh. As a side benefit, I believe the code is a lot cleaner to read.
[jc: I am accepting this not because I necessarily agree with the
quoting approach taken by it, but because (1) the code is only used
by ssh-fetch/ssh-upload pair which I do not care much about (if you
have ssh account on the remote end you should be using git-send-pack
git-fetch-pack pair over ssh anyway), and (2) HPA is one of the more
important customers belonging to the Linux kernel community and I
want to help his workflow -- which includes not wasting his time by
asking him to switch to git-send-pack/git-fetch-pack pair, nor to use
a better shell ;-). I might not have taken this patch if it mucked
with git_connect in connect.c in its current form.]
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Don't assume that any commit we have is complete; assume that any ref
we have is complete.
Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
The code did not catch the case where you removed an existing ref
without changing anything else. We are not talking about hundreds of
refs anyway, so remove that optimization.
Signed-off-by: Junio C Hamano <junkio@cox.net>
... in order to please Solaris 'install'. GNU install is not harmed
with this.
[jc: Documentation/Makefile also fixed.]
Signed-off-by: Junio C Hamano <junkio@cox.net>
By default the curl library adds "Pragma: no-cache" header to all
requests, which disables caching by proxy servers. However, most
files in a GIT repository are immutable, and caching them is safe and
could be useful.
This patch removes the "Pragma: no-cache" header from requests for all
files except the pack list (objects/info/packs) and references
(refs/*), which are really mutable and should not be cached.
Signed-off-by: Sergey Vlasov <vsu@altlinux.ru>
Signed-off-by: Junio C Hamano <junkio@cox.net>
(cherry picked from 3b2a4c46fd5093ec79fb60e1b14b8d4a58c74612 commit)
The new flag '-d' lets you delete a branch. For safety, it does not
lets you delete the branch you are currently on, nor a branch that
has been fully merged into your current branch.
The credit for the safety check idea goes to Daniel Barkalow.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This simplifies and fixes the initialization of a "diff_filespec" when
allocated.
The old code would not initialize "sha1_valid". Noticed by valgrind.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
We wanted to detect case #16 which should be rare, but botched the
case when some paths are missing, causing a segfault. My fault.
Signed-off-by: Junio C Hamano <junkio@cox.net>
With this change we can get rid of a call to 'git-update-index
--refresh'.
Signed-off-by: Fredrik Kuivinen <freku045@student.liu.se>
Signed-off-by: Junio C Hamano <junkio@cox.net>
If there are non-mergeable changes leave the head contents in the
cache and update the working directory with the output from merge(1).
In the add/add and delete/modify conflict cases leave unmerged cache
entries in the index.
Signed-off-by: Fredrik Kuivinen <freku045@student.liu.se>
Signed-off-by: Junio C Hamano <junkio@cox.net>
An entry in the alternates file can name a directory relative to
the object store it describes. A typical linux-2.6 maintainer
repository would have "../../../torvalds/linux-2.6.git/objects" there,
because the subsystem maintainer object store would live in
/pub/scm/linux/kernel/git/$u/$system.git/objects/
and the object store of Linus tree is in
/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects/
This unfortunately is different from GIT_ALTERNATE_OBJECT_DIRECTORIES
which is relative to the cwd of the running process, but there is no
way to make it consistent with the behaviour of the environment
variable. The process typically is run in $system.git/ directory for
a naked repository, or one level up for a repository with a working
tree, so we just define it to be relative to the objects/ directory
to be different from either ;-).
Later, the dumb transport could be updated to read from info/alternates
and make requests for the repository the repository borrows from.
Signed-off-by: Junio C Hamano <junkio@cox.net>
The part that can fail is before the pipe, so we need to propagate the
error properly to the main process.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Otherwise we would regret when Fredrik comes up with another merge
algorithm with different pros-and-cons with the current one.
Signed-off-by: Fredrik Kuivinen <freku045@student.liu.se>
Signed-off-by: Junio C Hamano <junkio@cox.net>
