gitea/services/repository/files/upload.go

226 lines
6.1 KiB
Go
Raw Normal View History

Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
// Copyright 2019 The Gitea Authors. All rights reserved.
// SPDX-License-Identifier: MIT
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
package files
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
import (
"context"
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
"fmt"
"os"
"path"
"strings"
git_model "code.gitea.io/gitea/models/git"
repo_model "code.gitea.io/gitea/models/repo"
user_model "code.gitea.io/gitea/models/user"
"code.gitea.io/gitea/modules/git"
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
"code.gitea.io/gitea/modules/lfs"
"code.gitea.io/gitea/modules/setting"
)
// UploadRepoFileOptions contains the uploaded repository file options
type UploadRepoFileOptions struct {
LastCommitID string
OldBranch string
NewBranch string
TreePath string
Message string
Files []string // In UUID format.
Signoff bool
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
}
type uploadInfo struct {
upload *repo_model.Upload
lfsMetaObject *git_model.LFSMetaObject
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
}
func cleanUpAfterFailure(ctx context.Context, infos *[]uploadInfo, t *TemporaryUploadRepository, original error) error {
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
for _, info := range *infos {
if info.lfsMetaObject == nil {
continue
}
if !info.lfsMetaObject.Existing {
if _, err := git_model.RemoveLFSMetaObjectByOid(ctx, t.repo.ID, info.lfsMetaObject.Oid); err != nil {
original = fmt.Errorf("%w, %v", original, err) // We wrap the original error - as this is the underlying error that required the fallback
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
}
}
}
return original
}
// UploadRepoFiles uploads files to the given repository
func UploadRepoFiles(ctx context.Context, repo *repo_model.Repository, doer *user_model.User, opts *UploadRepoFileOptions) error {
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
if len(opts.Files) == 0 {
return nil
}
uploads, err := repo_model.GetUploadsByUUIDs(ctx, opts.Files)
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
if err != nil {
return fmt.Errorf("GetUploadsByUUIDs [uuids: %v]: %w", opts.Files, err)
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
}
names := make([]string, len(uploads))
infos := make([]uploadInfo, len(uploads))
for i, upload := range uploads {
// Check file is not lfs locked, will return nil if lock setting not enabled
filepath := path.Join(opts.TreePath, upload.Name)
lfsLock, err := git_model.GetTreePathLock(ctx, repo.ID, filepath)
if err != nil {
return err
}
if lfsLock != nil && lfsLock.OwnerID != doer.ID {
u, err := user_model.GetUserByID(ctx, lfsLock.OwnerID)
if err != nil {
return err
}
return git_model.ErrLFSFileLocked{RepoID: repo.ID, Path: filepath, UserName: u.Name}
}
names[i] = upload.Name
infos[i] = uploadInfo{upload: upload}
}
t, err := NewTemporaryUploadRepository(ctx, repo)
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
if err != nil {
return err
}
2019-06-12 21:41:28 +02:00
defer t.Close()
hasOldBranch := true
if err = t.Clone(opts.OldBranch); err != nil {
if !git.IsErrBranchNotExist(err) || !repo.IsEmpty {
return err
}
if err = t.Init(); err != nil {
return err
}
hasOldBranch = false
opts.LastCommitID = ""
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
}
if hasOldBranch {
if err = t.SetDefaultIndex(); err != nil {
return err
}
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
}
var filename2attribute2info map[string]map[string]string
if setting.LFS.StartServer {
filename2attribute2info, err = t.gitRepo.CheckAttribute(git.CheckAttributeOpts{
Refactor git command package to improve security and maintainability (#22678) This PR follows #21535 (and replace #22592) ## Review without space diff https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1 ## Purpose of this PR 1. Make git module command completely safe (risky user inputs won't be passed as argument option anymore) 2. Avoid low-level mistakes like https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918 3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg` type 4. Simplify code when using git command ## The main idea of this PR * Move the `git.CmdArg` to the `internal` package, then no other package except `git` could use it. Then developers could never do `AddArguments(git.CmdArg(userInput))` any more. * Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already trusted arguments. It's only used in a few cases, for example: use git arguments from config file, help unit test with some arguments. * Introduce `AddOptionValues` and `AddOptionFormat`, they make code more clear and simple: * Before: `AddArguments("-m").AddDynamicArguments(message)` * After: `AddOptionValues("-m", message)` * - * Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'", sig.Name, sig.Email)))` * After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)` ## FAQ ### Why these changes were not done in #21535 ? #21535 is mainly a search&replace, it did its best to not change too much logic. Making the framework better needs a lot of changes, so this separate PR is needed as the second step. ### The naming of `AddOptionXxx` According to git's manual, the `--xxx` part is called `option`. ### How can it guarantee that `internal.CmdArg` won't be not misused? Go's specification guarantees that. Trying to access other package's internal package causes compilation error. And, `golangci-lint` also denies the git/internal package. Only the `git/command.go` can use it carefully. ### There is still a `ToTrustedCmdArgs`, will it still allow developers to make mistakes and pass untrusted arguments? Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code will be very complex (see the changes for examples). Then developers and reviewers can know that something might be unreasonable. ### Why there was a `CmdArgCheck` and why it's removed? At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck` was introduced as a hacky patch. Now, almost all code could be written as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for `CmdArgCheck` anymore. ### Why many codes for `signArg == ""` is deleted? Because in the old code, `signArg` could never be empty string, it's either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just dead code. --------- Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-04 03:30:43 +01:00
Attributes: []string{"filter"},
Filenames: names,
CachedOnly: true,
})
if err != nil {
return err
}
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
}
// Copy uploaded files into repository.
for i := range infos {
if err := copyUploadedLFSFileIntoRepository(&infos[i], filename2attribute2info, t, opts.TreePath); err != nil {
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
return err
}
}
// Now write the tree
treeHash, err := t.WriteTree()
if err != nil {
return err
}
// make author and committer the doer
author := doer
committer := doer
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
// Now commit the tree
commitHash, err := t.CommitTree(opts.LastCommitID, author, committer, treeHash, opts.Message, opts.Signoff)
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
if err != nil {
return err
}
// Now deal with LFS objects
for i := range infos {
if infos[i].lfsMetaObject == nil {
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
continue
}
infos[i].lfsMetaObject, err = git_model.NewLFSMetaObject(ctx, infos[i].lfsMetaObject)
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
if err != nil {
// OK Now we need to cleanup
return cleanUpAfterFailure(ctx, &infos, t, err)
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
}
// Don't move the files yet - we need to ensure that
// everything can be inserted first
}
// OK now we can insert the data into the store - there's no way to clean up the store
// once it's in there, it's in there.
Add LFS Migration and Mirror (#14726) * Implemented LFS client. * Implemented scanning for pointer files. * Implemented downloading of lfs files. * Moved model-dependent code into services. * Removed models dependency. Added TryReadPointerFromBuffer. * Migrated code from service to module. * Centralised storage creation. * Removed dependency from models. * Moved ContentStore into modules. * Share structs between server and client. * Moved method to services. * Implemented lfs download on clone. * Implemented LFS sync on clone and mirror update. * Added form fields. * Updated templates. * Fixed condition. * Use alternate endpoint. * Added missing methods. * Fixed typo and make linter happy. * Detached pointer parser from gogit dependency. * Fixed TestGetLFSRange test. * Added context to support cancellation. * Use ReadFull to probably read more data. * Removed duplicated code from models. * Moved scan implementation into pointer_scanner_nogogit. * Changed method name. * Added comments. * Added more/specific log/error messages. * Embedded lfs.Pointer into models.LFSMetaObject. * Moved code from models to module. * Moved code from models to module. * Moved code from models to module. * Reduced pointer usage. * Embedded type. * Use promoted fields. * Fixed unexpected eof. * Added unit tests. * Implemented migration of local file paths. * Show an error on invalid LFS endpoints. * Hide settings if not used. * Added LFS info to mirror struct. * Fixed comment. * Check LFS endpoint. * Manage LFS settings from mirror page. * Fixed selector. * Adjusted selector. * Added more tests. * Added local filesystem migration test. * Fixed typo. * Reset settings. * Added special windows path handling. * Added unit test for HTTPClient. * Added unit test for BasicTransferAdapter. * Moved into util package. * Test if LFS endpoint is allowed. * Added support for git:// * Just use a static placeholder as the displayed url may be invalid. * Reverted to original code. * Added "Advanced Settings". * Updated wording. * Added discovery info link. * Implemented suggestion. * Fixed missing format parameter. * Added Pointer.IsValid(). * Always remove model on error. * Added suggestions. * Use channel instead of array. * Update routers/repo/migrate.go * fmt Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
2021-04-09 00:25:57 +02:00
contentStore := lfs.NewContentStore()
for _, info := range infos {
if err := uploadToLFSContentStore(info, contentStore); err != nil {
return cleanUpAfterFailure(ctx, &infos, t, err)
}
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
}
// Then push this tree to NewBranch
if err := t.Push(doer, commitHash, opts.NewBranch); err != nil {
return err
}
return repo_model.DeleteUploads(ctx, uploads...)
Refactor editor upload, update and delete to use git plumbing and add LFS support (#5702) * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFile * Use git plumbing for upload: #5621 repo_editor.go: GetDiffPreview * Use git plumbing for upload: #5621 repo_editor.go: DeleteRepoFile * Use git plumbing for upload: #5621 repo_editor.go: UploadRepoFiles * Move branch checkout functions out of repo_editor.go as they are no longer used there * BUGFIX: The default permissions should be 100644 This is a change from the previous code but is more in keeping with the default behaviour of git. Signed-off-by: Andrew Thornton <art27@cantab.net> * Standardise cleanUploadFilename to more closely match git See verify_path in: https://github.com/git/git/blob/7f4e64169352e03476b0ea64e7e2973669e491a2/read-cache.c#L951 Signed-off-by: Andrew Thornton <art27@cantab.net> * Redirect on bad paths Signed-off-by: Andrew Thornton <art27@cantab.net> * Refactor to move the uploading functions out to a module Signed-off-by: Andrew Thornton <art27@cantab.net> * Add LFS support Signed-off-by: Andrew Thornton <art27@cantab.net> * Update upload.go attribution header Upload.go is essentially the remnants of repo_editor.go. The remaining code is essentially unchanged from the Gogs code, hence the Gogs attribution. * Delete upload files after session committed * Ensure that GIT_AUTHOR_NAME etc. are valid for git see #5774 Signed-off-by: Andrew Thornton <art27@cantab.net> * Add in test cases per @lafriks comment * Add space between gitea and github imports Signed-off-by: Andrew Thornton <art27@cantab.net> * more examples in TestCleanUploadName Signed-off-by: Andrew Thornton <art27@cantab.net> * fix formatting Signed-off-by: Andrew Thornton <art27@cantab.net> * Set the SSH_ORIGINAL_COMMAND to ensure hooks are run Signed-off-by: Andrew Thornton <art27@cantab.net> * Switch off SSH_ORIGINAL_COMMAND Signed-off-by: Andrew Thornton <art27@cantab.net>
2019-02-12 14:07:31 +01:00
}
func copyUploadedLFSFileIntoRepository(info *uploadInfo, filename2attribute2info map[string]map[string]string, t *TemporaryUploadRepository, treePath string) error {
file, err := os.Open(info.upload.LocalPath())
if err != nil {
return err
}
defer file.Close()
var objectHash string
if setting.LFS.StartServer && filename2attribute2info[info.upload.Name] != nil && filename2attribute2info[info.upload.Name]["filter"] == "lfs" {
// Handle LFS
// FIXME: Inefficient! this should probably happen in models.Upload
Add LFS Migration and Mirror (#14726) * Implemented LFS client. * Implemented scanning for pointer files. * Implemented downloading of lfs files. * Moved model-dependent code into services. * Removed models dependency. Added TryReadPointerFromBuffer. * Migrated code from service to module. * Centralised storage creation. * Removed dependency from models. * Moved ContentStore into modules. * Share structs between server and client. * Moved method to services. * Implemented lfs download on clone. * Implemented LFS sync on clone and mirror update. * Added form fields. * Updated templates. * Fixed condition. * Use alternate endpoint. * Added missing methods. * Fixed typo and make linter happy. * Detached pointer parser from gogit dependency. * Fixed TestGetLFSRange test. * Added context to support cancellation. * Use ReadFull to probably read more data. * Removed duplicated code from models. * Moved scan implementation into pointer_scanner_nogogit. * Changed method name. * Added comments. * Added more/specific log/error messages. * Embedded lfs.Pointer into models.LFSMetaObject. * Moved code from models to module. * Moved code from models to module. * Moved code from models to module. * Reduced pointer usage. * Embedded type. * Use promoted fields. * Fixed unexpected eof. * Added unit tests. * Implemented migration of local file paths. * Show an error on invalid LFS endpoints. * Hide settings if not used. * Added LFS info to mirror struct. * Fixed comment. * Check LFS endpoint. * Manage LFS settings from mirror page. * Fixed selector. * Adjusted selector. * Added more tests. * Added local filesystem migration test. * Fixed typo. * Reset settings. * Added special windows path handling. * Added unit test for HTTPClient. * Added unit test for BasicTransferAdapter. * Moved into util package. * Test if LFS endpoint is allowed. * Added support for git:// * Just use a static placeholder as the displayed url may be invalid. * Reverted to original code. * Added "Advanced Settings". * Updated wording. * Added discovery info link. * Implemented suggestion. * Fixed missing format parameter. * Added Pointer.IsValid(). * Always remove model on error. * Added suggestions. * Use channel instead of array. * Update routers/repo/migrate.go * fmt Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
2021-04-09 00:25:57 +02:00
pointer, err := lfs.GeneratePointer(file)
if err != nil {
return err
}
info.lfsMetaObject = &git_model.LFSMetaObject{Pointer: pointer, RepositoryID: t.repo.ID}
Add LFS Migration and Mirror (#14726) * Implemented LFS client. * Implemented scanning for pointer files. * Implemented downloading of lfs files. * Moved model-dependent code into services. * Removed models dependency. Added TryReadPointerFromBuffer. * Migrated code from service to module. * Centralised storage creation. * Removed dependency from models. * Moved ContentStore into modules. * Share structs between server and client. * Moved method to services. * Implemented lfs download on clone. * Implemented LFS sync on clone and mirror update. * Added form fields. * Updated templates. * Fixed condition. * Use alternate endpoint. * Added missing methods. * Fixed typo and make linter happy. * Detached pointer parser from gogit dependency. * Fixed TestGetLFSRange test. * Added context to support cancellation. * Use ReadFull to probably read more data. * Removed duplicated code from models. * Moved scan implementation into pointer_scanner_nogogit. * Changed method name. * Added comments. * Added more/specific log/error messages. * Embedded lfs.Pointer into models.LFSMetaObject. * Moved code from models to module. * Moved code from models to module. * Moved code from models to module. * Reduced pointer usage. * Embedded type. * Use promoted fields. * Fixed unexpected eof. * Added unit tests. * Implemented migration of local file paths. * Show an error on invalid LFS endpoints. * Hide settings if not used. * Added LFS info to mirror struct. * Fixed comment. * Check LFS endpoint. * Manage LFS settings from mirror page. * Fixed selector. * Adjusted selector. * Added more tests. * Added local filesystem migration test. * Fixed typo. * Reset settings. * Added special windows path handling. * Added unit test for HTTPClient. * Added unit test for BasicTransferAdapter. * Moved into util package. * Test if LFS endpoint is allowed. * Added support for git:// * Just use a static placeholder as the displayed url may be invalid. * Reverted to original code. * Added "Advanced Settings". * Updated wording. * Added discovery info link. * Implemented suggestion. * Fixed missing format parameter. * Added Pointer.IsValid(). * Always remove model on error. * Added suggestions. * Use channel instead of array. * Update routers/repo/migrate.go * fmt Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
2021-04-09 00:25:57 +02:00
if objectHash, err = t.HashObject(strings.NewReader(pointer.StringContent())); err != nil {
return err
}
} else if objectHash, err = t.HashObject(file); err != nil {
return err
}
// Add the object to the index
return t.AddObjectToIndex("100644", objectHash, path.Join(treePath, info.upload.Name))
}
func uploadToLFSContentStore(info uploadInfo, contentStore *lfs.ContentStore) error {
if info.lfsMetaObject == nil {
return nil
}
Add LFS Migration and Mirror (#14726) * Implemented LFS client. * Implemented scanning for pointer files. * Implemented downloading of lfs files. * Moved model-dependent code into services. * Removed models dependency. Added TryReadPointerFromBuffer. * Migrated code from service to module. * Centralised storage creation. * Removed dependency from models. * Moved ContentStore into modules. * Share structs between server and client. * Moved method to services. * Implemented lfs download on clone. * Implemented LFS sync on clone and mirror update. * Added form fields. * Updated templates. * Fixed condition. * Use alternate endpoint. * Added missing methods. * Fixed typo and make linter happy. * Detached pointer parser from gogit dependency. * Fixed TestGetLFSRange test. * Added context to support cancellation. * Use ReadFull to probably read more data. * Removed duplicated code from models. * Moved scan implementation into pointer_scanner_nogogit. * Changed method name. * Added comments. * Added more/specific log/error messages. * Embedded lfs.Pointer into models.LFSMetaObject. * Moved code from models to module. * Moved code from models to module. * Moved code from models to module. * Reduced pointer usage. * Embedded type. * Use promoted fields. * Fixed unexpected eof. * Added unit tests. * Implemented migration of local file paths. * Show an error on invalid LFS endpoints. * Hide settings if not used. * Added LFS info to mirror struct. * Fixed comment. * Check LFS endpoint. * Manage LFS settings from mirror page. * Fixed selector. * Adjusted selector. * Added more tests. * Added local filesystem migration test. * Fixed typo. * Reset settings. * Added special windows path handling. * Added unit test for HTTPClient. * Added unit test for BasicTransferAdapter. * Moved into util package. * Test if LFS endpoint is allowed. * Added support for git:// * Just use a static placeholder as the displayed url may be invalid. * Reverted to original code. * Added "Advanced Settings". * Updated wording. * Added discovery info link. * Implemented suggestion. * Fixed missing format parameter. * Added Pointer.IsValid(). * Always remove model on error. * Added suggestions. * Use channel instead of array. * Update routers/repo/migrate.go * fmt Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
2021-04-09 00:25:57 +02:00
exist, err := contentStore.Exists(info.lfsMetaObject.Pointer)
if err != nil {
return err
}
if !exist {
file, err := os.Open(info.upload.LocalPath())
if err != nil {
return err
}
defer file.Close()
// FIXME: Put regenerates the hash and copies the file over.
// I guess this strictly ensures the soundness of the store but this is inefficient.
Add LFS Migration and Mirror (#14726) * Implemented LFS client. * Implemented scanning for pointer files. * Implemented downloading of lfs files. * Moved model-dependent code into services. * Removed models dependency. Added TryReadPointerFromBuffer. * Migrated code from service to module. * Centralised storage creation. * Removed dependency from models. * Moved ContentStore into modules. * Share structs between server and client. * Moved method to services. * Implemented lfs download on clone. * Implemented LFS sync on clone and mirror update. * Added form fields. * Updated templates. * Fixed condition. * Use alternate endpoint. * Added missing methods. * Fixed typo and make linter happy. * Detached pointer parser from gogit dependency. * Fixed TestGetLFSRange test. * Added context to support cancellation. * Use ReadFull to probably read more data. * Removed duplicated code from models. * Moved scan implementation into pointer_scanner_nogogit. * Changed method name. * Added comments. * Added more/specific log/error messages. * Embedded lfs.Pointer into models.LFSMetaObject. * Moved code from models to module. * Moved code from models to module. * Moved code from models to module. * Reduced pointer usage. * Embedded type. * Use promoted fields. * Fixed unexpected eof. * Added unit tests. * Implemented migration of local file paths. * Show an error on invalid LFS endpoints. * Hide settings if not used. * Added LFS info to mirror struct. * Fixed comment. * Check LFS endpoint. * Manage LFS settings from mirror page. * Fixed selector. * Adjusted selector. * Added more tests. * Added local filesystem migration test. * Fixed typo. * Reset settings. * Added special windows path handling. * Added unit test for HTTPClient. * Added unit test for BasicTransferAdapter. * Moved into util package. * Test if LFS endpoint is allowed. * Added support for git:// * Just use a static placeholder as the displayed url may be invalid. * Reverted to original code. * Added "Advanced Settings". * Updated wording. * Added discovery info link. * Implemented suggestion. * Fixed missing format parameter. * Added Pointer.IsValid(). * Always remove model on error. * Added suggestions. * Use channel instead of array. * Update routers/repo/migrate.go * fmt Signed-off-by: Andrew Thornton <art27@cantab.net> Co-authored-by: zeripath <art27@cantab.net>
2021-04-09 00:25:57 +02:00
if err := contentStore.Put(info.lfsMetaObject.Pointer, file); err != nil {
// OK Now we need to cleanup
// Can't clean up the store, once uploaded there they're there.
return err
}
}
return nil
}