Merge branch 'jn/merge-renormalize'

* jn/merge-renormalize:
  merge-recursive --renormalize
  rerere: never renormalize
  rerere: migrate to parse-options API
  t4200 (rerere): modernize style
  ll-merge: let caller decide whether to renormalize
  ll-merge: make flag easier to populate
  Documentation/technical: document ll_merge
  merge-trees: let caller decide whether to renormalize
  merge-trees: push choice to renormalize away from low level
  t6038 (merge.renormalize): check that it can be turned off
  t6038 (merge.renormalize): try checkout -m and cherry-pick
  t6038 (merge.renormalize): style nitpicks
  Don't expand CRLFs when normalizing text during merge
  Try normalizing files to avoid delete/modify conflicts when merging
  Avoid conflicts when merging branches with mixed normalization

Conflicts:
	builtin/rerere.c
	t/t4200-rerere.sh

Documentation/gitattributes.txt

@@ -317,6 +317,17 @@ command is "cat").
 	smudge = cat
 ------------------------
 
+For best results, `clean` should not alter its output further if it is
+run twice ("clean->clean" should be equivalent to "clean"), and
+multiple `smudge` commands should not alter `clean`'s output
+("smudge->smudge->clean" should be equivalent to "clean").  See the
+section on merging below.
+
+The "indent" filter is well-behaved in this regard: it will not modify
+input that is already correctly indented.  In this case, the lack of a
+smudge filter means that the clean filter _must_ accept its own output
+without modifying it.
+
 
 Interaction between checkin/checkout attributes
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -331,6 +342,29 @@ In the check-out codepath, the blob content is first converted
 with `text`, and then `ident` and fed to `filter`.
 
 
+Merging branches with differing checkin/checkout attributes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you have added attributes to a file that cause the canonical
+repository format for that file to change, such as adding a
+clean/smudge filter or text/eol/ident attributes, merging anything
+where the attribute is not in place would normally cause merge
+conflicts.
+
+To prevent these unnecessary merge conflicts, git can be told to run a
+virtual check-out and check-in of all three stages of a file when
+resolving a three-way merge by setting the `merge.renormalize`
+configuration variable.  This prevents changes caused by check-in
+conversion from causing spurious merge conflicts when a converted file
+is merged with an unconverted file.
+
+As long as a "smudge->clean" results in the same output as a "clean"
+even on files that are already smudged, this strategy will
+automatically resolve all filter-related conflicts.  Filters that do
+not act in this way may cause additional merge conflicts that must be
+resolved manually.
+
+
 Generating diff text
 ~~~~~~~~~~~~~~~~~~~~
 

Documentation/merge-config.txt

@@ -15,6 +15,16 @@ merge.renameLimit::
 	during a merge; if not specified, defaults to the value of
 	diff.renameLimit.
 
+merge.renormalize::
+	Tell git that canonical representation of files in the
+	repository has changed over time (e.g. earlier commits record
+	text files with CRLF line endings, but recent ones use LF line
+	endings).  In such a repository, git can convert the data
+	recorded in commits to a canonical form before performing a
+	merge to reduce unnecessary conflicts.  For more information,
+	see section "Merging branches with differing checkin/checkout
+	attributes" in linkgit:gitattributes[5].
+
 merge.stat::
 	Whether to print the diffstat between ORIG_HEAD and the merge result
 	at the end of the merge.  True by default.

Documentation/merge-strategies.txt

@@ -40,6 +40,18 @@ the other tree did, declaring 'our' history contains all that happened in it.
 theirs;;
 	This is opposite of 'ours'.
 
+renormalize;;
+	This runs a virtual check-out and check-in of all three stages
+	of a file when resolving a three-way merge.  This option is
+	meant to be used when merging branches with different clean
+	filters or end-of-line normalization rules.  See "Merging
+	branches with differing checkin/checkout attributes" in
+	linkgit:gitattributes[5] for details.
+
+no-renormalize;;
+	Disables the `renormalize` option.  This overrides the
+	`merge.renormalize` configuration variable.
+
 subtree[=path];;
 	This option is a more advanced form of 'subtree' strategy, where
 	the strategy makes a guess on how two trees must be shifted to

Documentation/technical/api-merge.txt

@@ -0,0 +1,73 @@
+merge API
+=========
+
+The merge API helps a program to reconcile two competing sets of
+improvements to some files (e.g., unregistered changes from the work
+tree versus changes involved in switching to a new branch), reporting
+conflicts if found.  The library called through this API is
+responsible for a few things.
+
+ * determining which trees to merge (recursive ancestor consolidation);
+
+ * lining up corresponding files in the trees to be merged (rename
+   detection, subtree shifting), reporting edge cases like add/add
+   and rename/rename conflicts to the user;
+
+ * performing a three-way merge of corresponding files, taking
+   path-specific merge drivers (specified in `.gitattributes`)
+   into account.
+
+Low-level (single file) merge
+-----------------------------
+
+`ll_merge`::
+
+	Perform a three-way single-file merge in core.  This is
+	a thin wrapper around `xdl_merge` that takes the path and
+	any merge backend specified in `.gitattributes` or
+	`.git/info/attributes` into account.  Returns 0 for a
+	clean merge.
+
+The caller:
+
+1. allocates an mmbuffer_t variable for the result;
+2. allocates and fills variables with the file's original content
+   and two modified versions (using `read_mmfile`, for example);
+3. calls ll_merge();
+4. reads the output from result_buf.ptr and result_buf.size;
+5. releases buffers when finished (free(ancestor.ptr); free(ours.ptr);
+   free(theirs.ptr); free(result_buf.ptr);).
+
+If the modifications do not merge cleanly, `ll_merge` will return a
+nonzero value and `result_buf` will generally include a description of
+the conflict bracketed by markers such as the traditional `<<<<<<<`
+and `>>>>>>>`.
+
+The `ancestor_label`, `our_label`, and `their_label` parameters are
+used to label the different sides of a conflict if the merge driver
+supports this.
+
+The `flag` parameter is a bitfield:
+
+ - The `LL_OPT_VIRTUAL_ANCESTOR` bit indicates whether this is an
+   internal merge to consolidate ancestors for a recursive merge.
+
+ - The `LL_OPT_FAVOR_MASK` bits allow local conflicts to be automatically
+   resolved in favor of one side or the other (as in 'git merge-file'
+   `--ours`/`--theirs`/`--union`).
+   They can be populated by `create_ll_flag`, whose argument can be
+   `XDL_MERGE_FAVOR_OURS`, `XDL_MERGE_FAVOR_THEIRS`, or
+   `XDL_MERGE_FAVOR_UNION`.
+
+Everything else
+---------------
+
+Talk about <merge-recursive.h> and merge_file():
+
+ - merge_trees() to merge with rename detection
+ - merge_recursive() for ancestor consolidation
+ - try_merge_command() for other strategies
+ - conflict format
+ - merge options
+
+(Daniel, Miklos, Stephan, JC)

builtin/checkout.c

@@ -154,6 +154,10 @@ static int checkout_merged(int pos, struct checkout *state)
 	read_mmblob(&ours, active_cache[pos+1]->sha1);
 	read_mmblob(&theirs, active_cache[pos+2]->sha1);
 
+	/*
+	 * NEEDSWORK: re-create conflicts from merges with
+	 * merge.renormalize set, too
+	 */
 	status = ll_merge(&result_buf, path, &ancestor, "base",
 			  &ours, "ours", &theirs, "theirs", 0);
 	free(ancestor.ptr);
@@ -437,6 +441,13 @@ static int merge_working_tree(struct checkout_opts *opts,
 			 */
 
 			add_files_to_cache(NULL, NULL, 0);
+			/*
+			 * NEEDSWORK: carrying over local changes
+			 * when branches have different end-of-line
+			 * normalization (or clean+smudge rules) is
+			 * a pain; plumb in an option to set
+			 * o.renormalize?
+			 */
 			init_merge_options(&o);
 			o.verbosity = 0;
 			work = write_tree_from_memory(&o);

builtin/merge-recursive.c

@@ -48,6 +48,10 @@ int cmd_merge_recursive(int argc, const char **argv, const char *prefix)
 				o.subtree_shift = "";
 			else if (!prefixcmp(arg+2, "subtree="))
 				o.subtree_shift = arg + 10;
+			else if (!strcmp(arg+2, "renormalize"))
+				o.renormalize = 1;
+			else if (!strcmp(arg+2, "no-renormalize"))
+				o.renormalize = 0;
 			else
 				die("Unknown option %s", arg);
 			continue;

builtin/merge.c

@@ -54,6 +54,7 @@ static size_t use_strategies_nr, use_strategies_alloc;
 static const char **xopts;
 static size_t xopts_nr, xopts_alloc;
 static const char *branch;
+static int option_renormalize;
 static int verbosity;
 static int allow_rerere_auto;
 
@@ -504,6 +505,8 @@ static int git_merge_config(const char *k, const char *v, void *cb)
 		return git_config_string(&pull_octopus, k, v);
 	else if (!strcmp(k, "merge.log") || !strcmp(k, "merge.summary"))
 		option_log = git_config_bool(k, v);
+	else if (!strcmp(k, "merge.renormalize"))
+		option_renormalize = git_config_bool(k, v);
 	return git_diff_ui_config(k, v, cb);
 }
 
@@ -625,6 +628,11 @@ static int try_merge_strategy(const char *strategy, struct commit_list *common,
 		if (!strcmp(strategy, "subtree"))
 			o.subtree_shift = "";
 
+		o.renormalize = option_renormalize;
+
+		/*
+		 * NEEDSWORK: merge with table in builtin/merge-recursive
+		 */
 		for (x = 0; x < xopts_nr; x++) {
 			if (!strcmp(xopts[x], "ours"))
 				o.recursive_variant = MERGE_RECURSIVE_OURS;
@@ -634,6 +642,10 @@ static int try_merge_strategy(const char *strategy, struct commit_list *common,
 				o.subtree_shift = "";
 			else if (!prefixcmp(xopts[x], "subtree="))
 				o.subtree_shift = xopts[x]+8;
+			else if (!strcmp(xopts[x], "renormalize"))
+				o.renormalize = 1;
+			else if (!strcmp(xopts[x], "no-renormalize"))
+				o.renormalize = 0;
 			else
 				die("Unknown option for merge-recursive: -X%s", xopts[x]);
 		}
@@ -818,7 +830,7 @@ static int finish_automerge(struct commit_list *common,
 	return 0;
 }
 
-static int suggest_conflicts(void)
+static int suggest_conflicts(int renormalizing)
 {
 	FILE *fp;
 	int pos;
@@ -1303,5 +1315,5 @@ int cmd_merge(int argc, const char **argv, const char *prefix)
 			"stopped before committing as requested\n");
 		return 0;
 	} else
-		return suggest_conflicts();
+		return suggest_conflicts(option_renormalize);
 }

builtin/rerere.c

@@ -1,13 +1,16 @@
 #include "builtin.h"
 #include "cache.h"
 #include "dir.h"
+#include "parse-options.h"
 #include "string-list.h"
 #include "rerere.h"
 #include "xdiff/xdiff.h"
 #include "xdiff-interface.h"
 
-static const char git_rerere_usage[] =
-"git rerere [clear | status | diff | gc]";
+static const char * const rerere_usage[] = {
+	"git rerere [clear | status | diff | gc]",
+	NULL,
+};
 
 /* these values are days */
 static int cutoff_noresolve = 15;
@@ -114,25 +117,26 @@ static int diff_two(const char *file1, const char *label1,
 int cmd_rerere(int argc, const char **argv, const char *prefix)
 {
 	struct string_list merge_rr = STRING_LIST_INIT_DUP;
-	int i, fd, flags = 0;
+	int i, fd, autoupdate = -1, flags = 0;
 
-	if (2 < argc) {
-		if (!strcmp(argv[1], "-h"))
-			usage(git_rerere_usage);
-		if (!strcmp(argv[1], "--rerere-autoupdate"))
-			flags = RERERE_AUTOUPDATE;
-		else if (!strcmp(argv[1], "--no-rerere-autoupdate"))
-			flags = RERERE_NOAUTOUPDATE;
-		if (flags) {
-			argc--;
-			argv++;
-		}
-	}
-	if (argc < 2)
+	struct option options[] = {
+		OPT_SET_INT(0, "rerere-autoupdate", &autoupdate,
+			"register clean resolutions in index", 1),
+		OPT_END(),
+	};
+
+	argc = parse_options(argc, argv, prefix, options, rerere_usage, 0);
+
+	if (autoupdate == 1)
+		flags = RERERE_AUTOUPDATE;
+	if (autoupdate == 0)
+		flags = RERERE_NOAUTOUPDATE;
+
+	if (argc < 1)
 		return rerere(flags);
 
-	if (!strcmp(argv[1], "forget")) {
-		const char **pathspec = get_pathspec(prefix, argv + 2);
+	if (!strcmp(argv[0], "forget")) {
+		const char **pathspec = get_pathspec(prefix, argv + 1);
 		return rerere_forget(pathspec);
 	}
 
@@ -140,26 +144,26 @@ int cmd_rerere(int argc, const char **argv, const char *prefix)
 	if (fd < 0)
 		return 0;
 
-	if (!strcmp(argv[1], "clear")) {
+	if (!strcmp(argv[0], "clear")) {
 		for (i = 0; i < merge_rr.nr; i++) {
 			const char *name = (const char *)merge_rr.items[i].util;
 			if (!has_rerere_resolution(name))
 				unlink_rr_item(name);
 		}
 		unlink_or_warn(git_path("MERGE_RR"));
-	} else if (!strcmp(argv[1], "gc"))
+	} else if (!strcmp(argv[0], "gc"))
 		garbage_collect(&merge_rr);
-	else if (!strcmp(argv[1], "status"))
+	else if (!strcmp(argv[0], "status"))
 		for (i = 0; i < merge_rr.nr; i++)
 			printf("%s\n", merge_rr.items[i].string);
-	else if (!strcmp(argv[1], "diff"))
+	else if (!strcmp(argv[0], "diff"))
 		for (i = 0; i < merge_rr.nr; i++) {
 			const char *path = merge_rr.items[i].string;
 			const char *name = (const char *)merge_rr.items[i].util;
 			diff_two(rerere_path(name, "preimage"), path, path, path);
 		}
 	else
-		usage(git_rerere_usage);
+		usage_with_options(rerere_usage, options);
 
 	string_list_clear(&merge_rr, 1);
 	return 0;

builtin/revert.c

@@ -316,6 +316,13 @@ static int do_recursive_merge(struct commit *base, struct commit *next,
 	index_fd = hold_locked_index(&index_lock, 1);
 
 	read_cache();
+
+	/*
+	 * NEEDSWORK: cherry-picking between branches with
+	 * different end-of-line normalization is a pain;
+	 * plumb in an option to set o.renormalize?
+	 * (or better: arbitrary -X options)
+	 */
 	init_merge_options(&o);
 	o.ancestor = base ? base_label : "(empty tree)";
 	o.branch1 = "HEAD";

cache.h

@@ -1057,6 +1057,7 @@ extern void trace_argv_printf(const char **argv, const char *format, ...);
 extern int convert_to_git(const char *path, const char *src, size_t len,
                           struct strbuf *dst, enum safe_crlf checksafe);
 extern int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst);
+extern int renormalize_buffer(const char *path, const char *src, size_t len, struct strbuf *dst);
 
 /* add */
 /*

convert.c

@@ -93,7 +93,8 @@ static int is_binary(unsigned long size, struct text_stat *stats)
 	return 0;
 }
 
-static enum eol determine_output_conversion(enum action action) {
+static enum eol determine_output_conversion(enum action action)
+{
 	switch (action) {
 	case CRLF_BINARY:
 		return EOL_UNSET;
@@ -693,7 +694,8 @@ static int git_path_check_ident(const char *path, struct git_attr_check *check)
 	return !!ATTR_TRUE(value);
 }
 
-enum action determine_action(enum action text_attr, enum eol eol_attr) {
+static enum action determine_action(enum action text_attr, enum eol eol_attr)
+{
 	if (text_attr == CRLF_BINARY)
 		return CRLF_BINARY;
 	if (eol_attr == EOL_LF)
@@ -739,7 +741,9 @@ int convert_to_git(const char *path, const char *src, size_t len,
 	return ret | ident_to_git(path, src, len, dst, ident);
 }
 
-int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst)
+static int convert_to_working_tree_internal(const char *path, const char *src,
+					    size_t len, struct strbuf *dst,
+					    int normalizing)
 {
 	struct git_attr_check check[5];
 	enum action action = CRLF_GUESS;
@@ -765,11 +769,32 @@ int convert_to_working_tree(const char *path, const char *src, size_t len, struc
 		src = dst->buf;
 		len = dst->len;
 	}
-	action = determine_action(action, eol_attr);
-	ret |= crlf_to_worktree(path, src, len, dst, action);
+	/*
+	 * CRLF conversion can be skipped if normalizing, unless there
+	 * is a smudge filter.  The filter might expect CRLFs.
+	 */
+	if (filter || !normalizing) {
+		action = determine_action(action, eol_attr);
+		ret |= crlf_to_worktree(path, src, len, dst, action);
+		if (ret) {
+			src = dst->buf;
+			len = dst->len;
+		}
+	}
+	return ret | apply_filter(path, src, len, dst, filter);
+}
+
+int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst)
+{
+	return convert_to_working_tree_internal(path, src, len, dst, 0);
+}
+
+int renormalize_buffer(const char *path, const char *src, size_t len, struct strbuf *dst)
+{
+	int ret = convert_to_working_tree_internal(path, src, len, dst, 1);
 	if (ret) {
 		src = dst->buf;
 		len = dst->len;
 	}
-	return ret | apply_filter(path, src, len, dst, filter);
+	return ret | convert_to_git(path, src, len, dst, 0);
 }

ll-merge.c

@@ -46,7 +46,7 @@ static int ll_binary_merge(const struct ll_merge_driver *drv_unused,
 	 * or common ancestor for an internal merge.  Still return
 	 * "conflicted merge" status.
 	 */
-	mmfile_t *stolen = (flag & 01) ? orig : src1;
+	mmfile_t *stolen = (flag & LL_OPT_VIRTUAL_ANCESTOR) ? orig : src1;
 
 	result->ptr = stolen->ptr;
 	result->size = stolen->size;
@@ -79,7 +79,7 @@ static int ll_xdl_merge(const struct ll_merge_driver *drv_unused,
 
 	memset(&xmp, 0, sizeof(xmp));
 	xmp.level = XDL_MERGE_ZEALOUS;
-	xmp.favor= (flag >> 1) & 03;
+	xmp.favor = ll_opt_favor(flag);
 	if (git_xmerge_style >= 0)
 		xmp.style = git_xmerge_style;
 	if (marker_size > 0)
@@ -99,7 +99,8 @@ static int ll_union_merge(const struct ll_merge_driver *drv_unused,
 			  int flag, int marker_size)
 {
 	/* Use union favor */
-	flag = (flag & 1) | (XDL_MERGE_FAVOR_UNION << 1);
+	flag &= ~LL_OPT_FAVOR_MASK;
+	flag |= create_ll_flag(XDL_MERGE_FAVOR_UNION);
 	return ll_xdl_merge(drv_unused, result, path_unused,
 			    orig, NULL, src1, NULL, src2, NULL,
 			    flag, marker_size);
@@ -321,6 +322,16 @@ static int git_path_check_merge(const char *path, struct git_attr_check check[2]
 	return git_checkattr(path, 2, check);
 }
 
+static void normalize_file(mmfile_t *mm, const char *path)
+{
+	struct strbuf strbuf = STRBUF_INIT;
+	if (renormalize_buffer(path, mm->ptr, mm->size, &strbuf)) {
+		free(mm->ptr);
+		mm->size = strbuf.len;
+		mm->ptr = strbuf_detach(&strbuf, NULL);
+	}
+}
+
 int ll_merge(mmbuffer_t *result_buf,
 	     const char *path,
 	     mmfile_t *ancestor, const char *ancestor_label,
@@ -332,8 +343,13 @@ int ll_merge(mmbuffer_t *result_buf,
 	const char *ll_driver_name = NULL;
 	int marker_size = DEFAULT_CONFLICT_MARKER_SIZE;
 	const struct ll_merge_driver *driver;
-	int virtual_ancestor = flag & 01;
+	int virtual_ancestor = flag & LL_OPT_VIRTUAL_ANCESTOR;
 
+	if (flag & LL_OPT_RENORMALIZE) {
+		normalize_file(ancestor, path);
+		normalize_file(ours, path);
+		normalize_file(theirs, path);
+	}
 	if (!git_path_check_merge(path, check)) {
 		ll_driver_name = check[0].value;
 		if (check[1].value) {

ll-merge.h

@@ -5,6 +5,21 @@
 #ifndef LL_MERGE_H
 #define LL_MERGE_H
 
+#define LL_OPT_VIRTUAL_ANCESTOR	(1 << 0)
+#define LL_OPT_FAVOR_MASK	((1 << 1) | (1 << 2))
+#define LL_OPT_FAVOR_SHIFT 1
+#define LL_OPT_RENORMALIZE	(1 << 3)
+
+static inline int ll_opt_favor(int flag)
+{
+	return (flag & LL_OPT_FAVOR_MASK) >> LL_OPT_FAVOR_SHIFT;
+}
+
+static inline int create_ll_flag(int favor)
+{
+	return ((favor << LL_OPT_FAVOR_SHIFT) & LL_OPT_FAVOR_MASK);
+}
+
 int ll_merge(mmbuffer_t *result_buf,
 	     const char *path,
 	     mmfile_t *ancestor, const char *ancestor_label,

merge-recursive.c

@@ -644,7 +644,9 @@ static int merge_3way(struct merge_options *o,
 
 	merge_status = ll_merge(result_buf, a->path, &orig, base_name,
 				&src1, name1, &src2, name2,
-				(!!o->call_depth) | (favor << 1));
+				((o->call_depth ? LL_OPT_VIRTUAL_ANCESTOR : 0) |
+				 (o->renormalize ? LL_OPT_RENORMALIZE : 0) |
+				 create_ll_flag(favor)));
 
 	free(name1);
 	free(name2);
@@ -1062,6 +1064,53 @@ static unsigned char *stage_sha(const unsigned char *sha, unsigned mode)
 	return (is_null_sha1(sha) || mode == 0) ? NULL: (unsigned char *)sha;
 }
 
+static int read_sha1_strbuf(const unsigned char *sha1, struct strbuf *dst)
+{
+	void *buf;
+	enum object_type type;
+	unsigned long size;
+	buf = read_sha1_file(sha1, &type, &size);
+	if (!buf)
+		return error("cannot read object %s", sha1_to_hex(sha1));
+	if (type != OBJ_BLOB) {
+		free(buf);
+		return error("object %s is not a blob", sha1_to_hex(sha1));
+	}
+	strbuf_attach(dst, buf, size, size + 1);
+	return 0;
+}
+
+static int blob_unchanged(const unsigned char *o_sha,
+			  const unsigned char *a_sha,
+			  int renormalize, const char *path)
+{
+	struct strbuf o = STRBUF_INIT;
+	struct strbuf a = STRBUF_INIT;
+	int ret = 0; /* assume changed for safety */
+
+	if (sha_eq(o_sha, a_sha))
+		return 1;
+	if (!renormalize)
+		return 0;
+
+	assert(o_sha && a_sha);
+	if (read_sha1_strbuf(o_sha, &o) || read_sha1_strbuf(a_sha, &a))
+		goto error_return;
+	/*
+	 * Note: binary | is used so that both renormalizations are
+	 * performed.  Comparison can be skipped if both files are
+	 * unchanged since their sha1s have already been compared.
+	 */
+	if (renormalize_buffer(path, o.buf, o.len, &o) |
+	    renormalize_buffer(path, a.buf, o.len, &a))
+		ret = (o.len == a.len && !memcmp(o.buf, a.buf, o.len));
+
+error_return:
+	strbuf_release(&o);
+	strbuf_release(&a);
+	return ret;
+}
+
 /* Per entry merge function */
 static int process_entry(struct merge_options *o,
 			 const char *path, struct stage_data *entry)
@@ -1071,6 +1120,7 @@ static int process_entry(struct merge_options *o,
 	print_index_entry("\tpath: ", entry);
 	*/
 	int clean_merge = 1;
+	int normalize = o->renormalize;
 	unsigned o_mode = entry->stages[1].mode;
 	unsigned a_mode = entry->stages[2].mode;
 	unsigned b_mode = entry->stages[3].mode;
@@ -1082,8 +1132,8 @@ static int process_entry(struct merge_options *o,
 	if (o_sha && (!a_sha || !b_sha)) {
 		/* Case A: Deleted in one */
 		if ((!a_sha && !b_sha) ||
-		    (sha_eq(a_sha, o_sha) && !b_sha) ||
-		    (!a_sha && sha_eq(b_sha, o_sha))) {
+		    (!b_sha && blob_unchanged(o_sha, a_sha, normalize, path)) ||
+		    (!a_sha && blob_unchanged(o_sha, b_sha, normalize, path))) {
 			/* Deleted in both or deleted in one and
 			 * unchanged in the other */
 			if (a_sha)
@@ -1525,6 +1575,7 @@ void init_merge_options(struct merge_options *o)
 	o->buffer_output = 1;
 	o->diff_rename_limit = -1;
 	o->merge_rename_limit = -1;
+	o->renormalize = 0;
 	git_config(merge_recursive_config, o);
 	if (getenv("GIT_MERGE_VERBOSITY"))
 		o->verbosity =

merge-recursive.h

@@ -14,6 +14,7 @@ struct merge_options {
 	} recursive_variant;
 	const char *subtree_shift;
 	unsigned buffer_output : 1;
+	unsigned renormalize : 1;
 	int verbosity;
 	int diff_rename_limit;
 	int merge_rename_limit;

rerere.c

@@ -319,6 +319,10 @@ static int handle_cache(const char *path, unsigned char *sha1, const char *outpu
 		if (!mmfile[i].ptr && !mmfile[i].size)
 			mmfile[i].ptr = xstrdup("");
 	}
+	/*
+	 * NEEDSWORK: handle conflicts from merges with
+	 * merge.renormalize set, too
+	 */
 	ll_merge(&result, path, &mmfile[0], NULL,
 		 &mmfile[1], "ours",
 		 &mmfile[2], "theirs", 0);

t/t4200-rerere.sh

@@ -4,245 +4,391 @@
 #
 
 test_description='git rerere
+
+! [fifth] version1
+ ! [first] first
+  ! [fourth] version1
+   ! [master] initial
+    ! [second] prefer first over second
+     ! [third] version2
+------
+     + [third] version2
++      [fifth] version1
+  +    [fourth] version1
++ +  + [third^] third
+    -  [second] prefer first over second
+ +  +  [first] first
+    +  [second^] second
+++++++ [master] initial
 '
 
 . ./test-lib.sh
 
-test_expect_success 'setup' "
-	cat > a1 <<- EOF &&
+test_expect_success 'setup' '
+	cat >a1 <<-\EOF &&
 	Some title
 	==========
-	Whether 'tis nobler in the mind to suffer
+	Whether '\''tis nobler in the mind to suffer
 	The slings and arrows of outrageous fortune,
 	Or to take arms against a sea of troubles,
 	And by opposing end them? To die: to sleep;
 	No more; and by a sleep to say we end
 	The heart-ache and the thousand natural shocks
-	That flesh is heir to, 'tis a consummation
-	Devoutly to be wish'd.
+	That flesh is heir to, '\''tis a consummation
+	Devoutly to be wish'\''d.
 	EOF
 
 	git add a1 &&
+	test_tick &&
 	git commit -q -a -m initial &&
 
-	git checkout -b first &&
-	cat >> a1 <<- EOF &&
+	cat >>a1 <<-\EOF &&
 	Some title
 	==========
 	To die, to sleep;
-	To sleep: perchance to dream: ay, there's the rub;
+	To sleep: perchance to dream: ay, there'\''s the rub;
 	For in that sleep of death what dreams may come
 	When we have shuffled off this mortal coil,
-	Must give us pause: there's the respect
+	Must give us pause: there'\''s the respect
 	That makes calamity of so long life;
 	EOF
+
+	git checkout -b first &&
+	test_tick &&
 	git commit -q -a -m first &&
 
 	git checkout -b second master &&
 	git show first:a1 |
-	sed -e 's/To die, t/To die! T/' -e 's/Some title/Some Title/' > a1 &&
-	echo '* END *' >>a1 &&
+	sed -e "s/To die, t/To die! T/" -e "s/Some title/Some Title/" >a1 &&
+	echo "* END *" >>a1 &&
+	test_tick &&
 	git commit -q -a -m second
-"
+'
 
 test_expect_success 'nothing recorded without rerere' '
-	(rm -rf .git/rr-cache; git config rerere.enabled false) &&
+	rm -rf .git/rr-cache &&
+	git config rerere.enabled false &&
 	test_must_fail git merge first &&
 	! test -d .git/rr-cache
 '
 
-# activate rerere, old style
-test_expect_success 'conflicting merge' '
+test_expect_success 'activate rerere, old style (conflicting merge)' '
 	git reset --hard &&
 	mkdir .git/rr-cache &&
-	git config --unset rerere.enabled &&
-	test_must_fail git merge first
-'
+	test_might_fail git config --unset rerere.enabled &&
+	test_must_fail git merge first &&
 
-sha1=$(perl -pe 's/	.*//' .git/MERGE_RR)
-rr=.git/rr-cache/$sha1
-test_expect_success 'recorded preimage' "grep ^=======$ $rr/preimage"
+	sha1=$(perl -pe "s/	.*//" .git/MERGE_RR) &&
+	rr=.git/rr-cache/$sha1 &&
+	grep "^=======\$" $rr/preimage &&
+	! test -f $rr/postimage &&
+	! test -f $rr/thisimage
+'
 
 test_expect_success 'rerere.enabled works, too' '
 	rm -rf .git/rr-cache &&
 	git config rerere.enabled true &&
 	git reset --hard &&
 	test_must_fail git merge first &&
+
+	sha1=$(perl -pe "s/	.*//" .git/MERGE_RR) &&
+	rr=.git/rr-cache/$sha1 &&
 	grep ^=======$ $rr/preimage
 '
 
-test_expect_success 'no postimage or thisimage yet' \
-	"test ! -f $rr/postimage -a ! -f $rr/thisimage"
+test_expect_success 'set up rr-cache' '
+	rm -rf .git/rr-cache &&
+	git config rerere.enabled true &&
+	git reset --hard &&
+	test_must_fail git merge first &&
+	sha1=$(perl -pe "s/	.*//" .git/MERGE_RR) &&
+	rr=.git/rr-cache/$sha1
+'
 
-test_expect_success 'preimage has right number of lines' '
+test_expect_success 'rr-cache looks sane' '
+	# no postimage or thisimage yet
+	! test -f $rr/postimage &&
+	! test -f $rr/thisimage &&
 
+	# preimage has right number of lines
 	cnt=$(sed -ne "/^<<<<<<</,/^>>>>>>>/p" $rr/preimage | wc -l) &&
+	echo $cnt &&
 	test $cnt = 13
-
 '
 
-git show first:a1 > a1
+test_expect_success 'rerere diff' '
+	git show first:a1 >a1 &&
+	cat >expect <<-\EOF &&
+	--- a/a1
+	+++ b/a1
+	@@ -1,4 +1,4 @@
+	-Some Title
+	+Some title
+	 ==========
+	 Whether '\''tis nobler in the mind to suffer
+	 The slings and arrows of outrageous fortune,
+	@@ -8,21 +8,11 @@
+	 The heart-ache and the thousand natural shocks
+	 That flesh is heir to, '\''tis a consummation
+	 Devoutly to be wish'\''d.
+	-<<<<<<<
+	-Some Title
+	-==========
+	-To die! To sleep;
+	-=======
+	 Some title
+	 ==========
+	 To die, to sleep;
+	->>>>>>>
+	 To sleep: perchance to dream: ay, there'\''s the rub;
+	 For in that sleep of death what dreams may come
+	 When we have shuffled off this mortal coil,
+	 Must give us pause: there'\''s the respect
+	 That makes calamity of so long life;
+	-<<<<<<<
+	-=======
+	-* END *
+	->>>>>>>
+	EOF
+	git rerere diff >out &&
+	test_cmp expect out
+'
 
-cat > expect << EOF
---- a/a1
-+++ b/a1
-@@ -1,4 +1,4 @@
--Some Title
-+Some title
- ==========
- Whether 'tis nobler in the mind to suffer
- The slings and arrows of outrageous fortune,
-@@ -8,21 +8,11 @@
- The heart-ache and the thousand natural shocks
- That flesh is heir to, 'tis a consummation
- Devoutly to be wish'd.
--<<<<<<<
--Some Title
--==========
--To die! To sleep;
--=======
- Some title
- ==========
- To die, to sleep;
-->>>>>>>
- To sleep: perchance to dream: ay, there's the rub;
- For in that sleep of death what dreams may come
- When we have shuffled off this mortal coil,
- Must give us pause: there's the respect
- That makes calamity of so long life;
--<<<<<<<
--=======
--* END *
-->>>>>>>
-EOF
-git rerere diff > out
+test_expect_success 'rerere status' '
+	echo a1 >expect &&
+	git rerere status >out &&
+	test_cmp expect out
+'
 
-test_expect_success 'rerere diff' 'test_cmp expect out'
+test_expect_success 'first postimage wins' '
+	git show first:a1 | sed "s/To die: t/To die! T/" >expect &&
 
-cat > expect << EOF
-a1
-EOF
+	git commit -q -a -m "prefer first over second" &&
+	test -f $rr/postimage &&
 
-git rerere status > out
+	oldmtimepost=$(test-chmtime -v -60 $rr/postimage | cut -f 1) &&
 
-test_expect_success 'rerere status' 'test_cmp expect out'
-
-test_expect_success 'commit succeeds' \
-	"git commit -q -a -m 'prefer first over second'"
-
-test_expect_success 'recorded postimage' "test -f $rr/postimage"
-
-oldmtimepost=$(test-chmtime -v -60 $rr/postimage |cut -f 1)
-
-test_expect_success 'another conflicting merge' '
 	git checkout -b third master &&
-	git show second^:a1 | sed "s/To die: t/To die! T/" > a1 &&
+	git show second^:a1 | sed "s/To die: t/To die! T/" >a1 &&
 	git commit -q -a -m third &&
-	test_must_fail git pull . first
+
+	test_must_fail git pull . first &&
+	# rerere kicked in
+	! grep "^=======\$" a1 &&
+	test_cmp expect a1
 '
 
-git show first:a1 | sed 's/To die: t/To die! T/' > expect
-test_expect_success 'rerere kicked in' "! grep ^=======$ a1"
-
-test_expect_success 'rerere prefers first change' 'test_cmp a1 expect'
-
 test_expect_success 'rerere updates postimage timestamp' '
-	newmtimepost=$(test-chmtime -v +0 $rr/postimage |cut -f 1) &&
+	newmtimepost=$(test-chmtime -v +0 $rr/postimage | cut -f 1) &&
 	test $oldmtimepost -lt $newmtimepost
 '
 
-rm $rr/postimage
-echo "$sha1	a1" | perl -pe 'y/\012/\000/' > .git/MERGE_RR
+test_expect_success 'rerere clear' '
+	rm $rr/postimage &&
+	echo "$sha1	a1" | perl -pe "y/\012/\000/" >.git/MERGE_RR &&
+	git rerere clear &&
+	! test -d $rr
+'
 
-test_expect_success 'rerere clear' 'git rerere clear'
+test_expect_success 'set up for garbage collection tests' '
+	mkdir -p $rr &&
+	echo Hello >$rr/preimage &&
+	echo World >$rr/postimage &&
 
-test_expect_success 'clear removed the directory' "test ! -d $rr"
+	sha2=4000000000000000000000000000000000000000 &&
+	rr2=.git/rr-cache/$sha2 &&
+	mkdir $rr2 &&
+	echo Hello >$rr2/preimage &&
 
-mkdir $rr
-echo Hello > $rr/preimage
-echo World > $rr/postimage
+	almost_15_days_ago=$((60-15*86400)) &&
+	just_over_15_days_ago=$((-1-15*86400)) &&
+	almost_60_days_ago=$((60-60*86400)) &&
+	just_over_60_days_ago=$((-1-60*86400)) &&
 
-sha2=4000000000000000000000000000000000000000
-rr2=.git/rr-cache/$sha2
-mkdir $rr2
-echo Hello > $rr2/preimage
+	test-chmtime =$just_over_60_days_ago $rr/preimage &&
+	test-chmtime =$almost_60_days_ago $rr/postimage &&
+	test-chmtime =$almost_15_days_ago $rr2/preimage
+'
 
-almost_15_days_ago=$((60-15*86400))
-just_over_15_days_ago=$((-1-15*86400))
-almost_60_days_ago=$((60-60*86400))
-just_over_60_days_ago=$((-1-60*86400))
+test_expect_success 'gc preserves young or recently used records' '
+	git rerere gc &&
+	test -f $rr/preimage &&
+	test -f $rr2/preimage
+'
 
-test-chmtime =$just_over_60_days_ago $rr/preimage
-test-chmtime =$almost_60_days_ago $rr/postimage
-test-chmtime =$almost_15_days_ago $rr2/preimage
+test_expect_success 'old records rest in peace' '
+	test-chmtime =$just_over_60_days_ago $rr/postimage &&
+	test-chmtime =$just_over_15_days_ago $rr2/preimage &&
+	git rerere gc &&
+	! test -f $rr/preimage &&
+	! test -f $rr2/preimage
+'
 
-test_expect_success 'garbage collection (part1)' 'git rerere gc'
-
-test_expect_success 'young or recently used records still live' \
-	"test -f $rr/preimage && test -f $rr2/preimage"
-
-test-chmtime =$just_over_60_days_ago $rr/postimage
-test-chmtime =$just_over_15_days_ago $rr2/preimage
-
-test_expect_success 'garbage collection (part2)' 'git rerere gc'
-
-test_expect_success 'old records rest in peace' \
-	"test ! -f $rr/preimage && test ! -f $rr2/preimage"
-
-test_expect_success 'file2 added differently in two branches' '
+test_expect_success 'setup: file2 added differently in two branches' '
 	git reset --hard &&
+
 	git checkout -b fourth &&
-	echo Hallo > file2 &&
+	echo Hallo >file2 &&
 	git add file2 &&
+	test_tick &&
 	git commit -m version1 &&
+
 	git checkout third &&
-	echo Bello > file2 &&
+	echo Bello >file2 &&
 	git add file2 &&
+	test_tick &&
 	git commit -m version2 &&
+
 	test_must_fail git merge fourth &&
-	echo Cello > file2 &&
+	echo Cello >file2 &&
 	git add file2 &&
 	git commit -m resolution
 '
 
 test_expect_success 'resolution was recorded properly' '
+	echo Cello >expected &&
+
 	git reset --hard HEAD~2 &&
 	git checkout -b fifth &&
-	echo Hallo > file3 &&
+
+	echo Hallo >file3 &&
 	git add file3 &&
+	test_tick &&
 	git commit -m version1 &&
+
 	git checkout third &&
-	echo Bello > file3 &&
+	echo Bello >file3 &&
 	git add file3 &&
+	test_tick &&
 	git commit -m version2 &&
 	git tag version2 &&
+
 	test_must_fail git merge fifth &&
-	test Cello = "$(cat file3)" &&
-	test 0 != $(git ls-files -u | wc -l)
+	test_cmp expected file3 &&
+	test_must_fail git update-index --refresh
 '
 
 test_expect_success 'rerere.autoupdate' '
-	git config rerere.autoupdate true
+	git config rerere.autoupdate true &&
 	git reset --hard &&
 	git checkout version2 &&
 	test_must_fail git merge fifth &&
-	test 0 = $(git ls-files -u | wc -l)
+	git update-index --refresh
 '
 
 test_expect_success 'merge --rerere-autoupdate' '
-	git config --unset rerere.autoupdate
+	test_might_fail git config --unset rerere.autoupdate &&
 	git reset --hard &&
 	git checkout version2 &&
 	test_must_fail git merge --rerere-autoupdate fifth &&
-	test 0 = $(git ls-files -u | wc -l)
+	git update-index --refresh
 '
 
 test_expect_success 'merge --no-rerere-autoupdate' '
-	git config rerere.autoupdate true
+	headblob=$(git rev-parse version2:file3) &&
+	mergeblob=$(git rev-parse fifth:file3) &&
+	cat >expected <<-EOF &&
+	100644 $headblob 2	file3
+	100644 $mergeblob 3	file3
+	EOF
+
+	git config rerere.autoupdate true &&
 	git reset --hard &&
 	git checkout version2 &&
 	test_must_fail git merge --no-rerere-autoupdate fifth &&
-	test 2 = $(git ls-files -u | wc -l)
+	git ls-files -u >actual &&
+	test_cmp expected actual
+'
+
+test_expect_success 'set up an unresolved merge' '
+	headblob=$(git rev-parse version2:file3) &&
+	mergeblob=$(git rev-parse fifth:file3) &&
+	cat >expected.unresolved <<-EOF &&
+	100644 $headblob 2	file3
+	100644 $mergeblob 3	file3
+	EOF
+
+	test_might_fail git config --unset rerere.autoupdate &&
+	git reset --hard &&
+	git checkout version2 &&
+	fifth=$(git rev-parse fifth) &&
+	echo "$fifth		branch 'fifth' of ." |
+	git fmt-merge-msg >msg &&
+	ancestor=$(git merge-base version2 fifth) &&
+	test_must_fail git merge-recursive "$ancestor" -- HEAD fifth &&
+
+	git ls-files --stage >failedmerge &&
+	cp file3 file3.conflict &&
+
+	git ls-files -u >actual &&
+	test_cmp expected.unresolved actual
+'
+
+test_expect_success 'explicit rerere' '
+	test_might_fail git config --unset rerere.autoupdate &&
+	git rm -fr --cached . &&
+	git update-index --index-info <failedmerge &&
+	cp file3.conflict file3 &&
+	test_must_fail git update-index --refresh -q &&
+
+	git rerere &&
+	git ls-files -u >actual &&
+	test_cmp expected.unresolved actual
+'
+
+test_expect_success 'explicit rerere with autoupdate' '
+	git config rerere.autoupdate true &&
+	git rm -fr --cached . &&
+	git update-index --index-info <failedmerge &&
+	cp file3.conflict file3 &&
+	test_must_fail git update-index --refresh -q &&
+
+	git rerere &&
+	git update-index --refresh
+'
+
+test_expect_success 'explicit rerere --rerere-autoupdate overrides' '
+	git config rerere.autoupdate false &&
+	git rm -fr --cached . &&
+	git update-index --index-info <failedmerge &&
+	cp file3.conflict file3 &&
+	git rerere &&
+	git ls-files -u >actual1 &&
+
+	git rm -fr --cached . &&
+	git update-index --index-info <failedmerge &&
+	cp file3.conflict file3 &&
+	git rerere --rerere-autoupdate &&
+	git update-index --refresh &&
+
+	git rm -fr --cached . &&
+	git update-index --index-info <failedmerge &&
+	cp file3.conflict file3 &&
+	git rerere --rerere-autoupdate --no-rerere-autoupdate &&
+	git ls-files -u >actual2 &&
+
+	git rm -fr --cached . &&
+	git update-index --index-info <failedmerge &&
+	cp file3.conflict file3 &&
+	git rerere --rerere-autoupdate --no-rerere-autoupdate --rerere-autoupdate &&
+	git update-index --refresh &&
+
+	test_cmp expected.unresolved actual1 &&
+	test_cmp expected.unresolved actual2
+'
+
+test_expect_success 'rerere --no-no-rerere-autoupdate' '
+	git rm -fr --cached . &&
+	git update-index --index-info <failedmerge &&
+	cp file3.conflict file3 &&
+	test_must_fail git rerere --no-no-rerere-autoupdate 2>err &&
+	grep [Uu]sage err &&
+	test_must_fail git update-index --refresh
+'
+
+test_expect_success 'rerere -h' '
+	test_must_fail git rerere -h >help &&
+	grep [Uu]sage help
 '
 
 test_done

t/t6038-merge-text-auto.sh

@@ -0,0 +1,189 @@
+#!/bin/sh
+
+test_description='CRLF merge conflict across text=auto change
+
+* [master] remove .gitattributes
+ ! [side] add line from b
+--
+ + [side] add line from b
+*  [master] remove .gitattributes
+*  [master^] add line from a
+*  [master~2] normalize file
+*+ [side^] Initial
+'
+
+. ./test-lib.sh
+
+test_expect_success setup '
+	git config core.autocrlf false &&
+
+	echo first line | append_cr >file &&
+	echo first line >control_file &&
+	echo only line >inert_file &&
+
+	git add file control_file inert_file &&
+	test_tick &&
+	git commit -m "Initial" &&
+	git tag initial &&
+	git branch side &&
+
+	echo "* text=auto" >.gitattributes &&
+	touch file &&
+	git add .gitattributes file &&
+	test_tick &&
+	git commit -m "normalize file" &&
+
+	echo same line | append_cr >>file &&
+	echo same line >>control_file &&
+	git add file control_file &&
+	test_tick &&
+	git commit -m "add line from a" &&
+	git tag a &&
+
+	git rm .gitattributes &&
+	rm file &&
+	git checkout file &&
+	test_tick &&
+	git commit -m "remove .gitattributes" &&
+	git tag c &&
+
+	git checkout side &&
+	echo same line | append_cr >>file &&
+	echo same line >>control_file &&
+	git add file control_file &&
+	test_tick &&
+	git commit -m "add line from b" &&
+	git tag b &&
+
+	git checkout master
+'
+
+test_expect_success 'set up fuzz_conflict() helper' '
+	fuzz_conflict() {
+		sed -e "s/^\([<>=]......\) .*/\1/" "$@"
+	}
+'
+
+test_expect_success 'Merge after setting text=auto' '
+	cat <<-\EOF >expected &&
+	first line
+	same line
+	EOF
+
+	git config merge.renormalize true &&
+	git rm -fr . &&
+	rm -f .gitattributes &&
+	git reset --hard a &&
+	git merge b &&
+	test_cmp expected file
+'
+
+test_expect_success 'Merge addition of text=auto' '
+	cat <<-\EOF >expected &&
+	first line
+	same line
+	EOF
+
+	git config merge.renormalize true &&
+	git rm -fr . &&
+	rm -f .gitattributes &&
+	git reset --hard b &&
+	git merge a &&
+	test_cmp expected file
+'
+
+test_expect_success 'Detect CRLF/LF conflict after setting text=auto' '
+	q_to_cr <<-\EOF >expected &&
+	<<<<<<<
+	first line
+	same line
+	=======
+	first lineQ
+	same lineQ
+	>>>>>>>
+	EOF
+
+	git config merge.renormalize false &&
+	rm -f .gitattributes &&
+	git reset --hard a &&
+	test_must_fail git merge b &&
+	fuzz_conflict file >file.fuzzy &&
+	test_cmp expected file.fuzzy
+'
+
+test_expect_success 'Detect LF/CRLF conflict from addition of text=auto' '
+	q_to_cr <<-\EOF >expected &&
+	<<<<<<<
+	first lineQ
+	same lineQ
+	=======
+	first line
+	same line
+	>>>>>>>
+	EOF
+
+	git config merge.renormalize false &&
+	rm -f .gitattributes &&
+	git reset --hard b &&
+	test_must_fail git merge a &&
+	fuzz_conflict file >file.fuzzy &&
+	test_cmp expected file.fuzzy
+'
+
+test_expect_failure 'checkout -m after setting text=auto' '
+	cat <<-\EOF >expected &&
+	first line
+	same line
+	EOF
+
+	git config merge.renormalize true &&
+	git rm -fr . &&
+	rm -f .gitattributes &&
+	git reset --hard initial &&
+	git checkout a -- . &&
+	git checkout -m b &&
+	test_cmp expected file
+'
+
+test_expect_failure 'checkout -m addition of text=auto' '
+	cat <<-\EOF >expected &&
+	first line
+	same line
+	EOF
+
+	git config merge.renormalize true &&
+	git rm -fr . &&
+	rm -f .gitattributes file &&
+	git reset --hard initial &&
+	git checkout b -- . &&
+	git checkout -m a &&
+	test_cmp expected file
+'
+
+test_expect_failure 'cherry-pick patch from after text=auto was added' '
+	append_cr <<-\EOF >expected &&
+	first line
+	same line
+	EOF
+
+	git config merge.renormalize true &&
+	git rm -fr . &&
+	git reset --hard b &&
+	test_must_fail git cherry-pick a >err 2>&1 &&
+	grep "[Nn]othing added" err &&
+	test_cmp expected file
+'
+
+test_expect_success 'Test delete/normalize conflict' '
+	git checkout -f side &&
+	git rm -fr . &&
+	rm -f .gitattributes &&
+	git reset --hard initial &&
+	git rm file &&
+	git commit -m "remove file" &&
+	git checkout master &&
+	git reset --hard a^ &&
+	git merge side
+'
+
+test_done