2016-11-03 17:16:01 -05:00
// Copyright 2015 The Gogs Authors. All rights reserved.
2020-09-05 15:12:14 -05:00
// Copyright 2016 The Gitea Authors. All rights reserved.
2022-11-27 13:20:29 -05:00
// SPDX-License-Identifier: MIT
2016-11-03 17:16:01 -05:00
package git
import (
"bytes"
2017-04-07 21:23:39 -05:00
"context"
2022-06-09 20:57:49 -05:00
"errors"
2016-11-03 17:16:01 -05:00
"fmt"
"io"
2019-11-10 03:42:51 -05:00
"os"
2016-11-03 17:16:01 -05:00
"os/exec"
2024-03-02 08:38:34 -05:00
"runtime"
2016-11-03 17:16:01 -05:00
"strings"
"time"
2019-06-26 13:15:26 -05:00
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)
## Review without space diff
https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1
## Purpose of this PR
1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command
## The main idea of this PR
* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
* Before: `AddArguments("-m").AddDynamicArguments(message)`
* After: `AddOptionValues("-m", message)`
* -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`
## FAQ
### Why these changes were not done in #21535 ?
#21535 is mainly a search&replace, it did its best to not change too
much logic.
Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.
### The naming of `AddOptionXxx`
According to git's manual, the `--xxx` part is called `option`.
### How can it guarantee that `internal.CmdArg` won't be not misused?
Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.
And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.
### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?
Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.
### Why there was a `CmdArgCheck` and why it's removed?
At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.
### Why many codes for `signArg == ""` is deleted?
Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-03 21:30:43 -05:00
"code.gitea.io/gitea/modules/git/internal" //nolint:depguard // only this file can use the internal type CmdArg, other files and packages should use AddXxx functions
2021-06-25 11:54:08 -05:00
"code.gitea.io/gitea/modules/log"
2019-06-26 13:15:26 -05:00
"code.gitea.io/gitea/modules/process"
2022-03-27 06:54:09 -05:00
"code.gitea.io/gitea/modules/util"
2016-11-03 17:16:01 -05:00
)
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)
## Review without space diff
https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1
## Purpose of this PR
1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command
## The main idea of this PR
* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
* Before: `AddArguments("-m").AddDynamicArguments(message)`
* After: `AddOptionValues("-m", message)`
* -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`
## FAQ
### Why these changes were not done in #21535 ?
#21535 is mainly a search&replace, it did its best to not change too
much logic.
Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.
### The naming of `AddOptionXxx`
According to git's manual, the `--xxx` part is called `option`.
### How can it guarantee that `internal.CmdArg` won't be not misused?
Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.
And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.
### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?
Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.
### Why there was a `CmdArgCheck` and why it's removed?
At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.
### Why many codes for `signArg == ""` is deleted?
Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-03 21:30:43 -05:00
// TrustedCmdArgs returns the trusted arguments for git command.
// It's mainly for passing user-provided and trusted arguments to git command
// In most cases, it shouldn't be used. Use AddXxx function instead
type TrustedCmdArgs [ ] internal . CmdArg
2016-12-22 04:30:52 -05:00
var (
2022-01-25 13:15:58 -05:00
// globalCommandArgs global command args for external package setting
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)
## Review without space diff
https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1
## Purpose of this PR
1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command
## The main idea of this PR
* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
* Before: `AddArguments("-m").AddDynamicArguments(message)`
* After: `AddOptionValues("-m", message)`
* -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`
## FAQ
### Why these changes were not done in #21535 ?
#21535 is mainly a search&replace, it did its best to not change too
much logic.
Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.
### The naming of `AddOptionXxx`
According to git's manual, the `--xxx` part is called `option`.
### How can it guarantee that `internal.CmdArg` won't be not misused?
Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.
And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.
### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?
Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.
### Why there was a `CmdArgCheck` and why it's removed?
At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.
### Why many codes for `signArg == ""` is deleted?
Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-03 21:30:43 -05:00
globalCommandArgs TrustedCmdArgs
2018-01-07 08:10:20 -05:00
2021-06-26 06:28:55 -05:00
// defaultCommandExecutionTimeout default command execution timeout duration
defaultCommandExecutionTimeout = 360 * time . Second
2016-12-22 04:30:52 -05:00
)
2019-11-10 03:42:51 -05:00
// DefaultLocale is the default LC_ALL to run git commands in.
const DefaultLocale = "C"
2016-11-03 17:16:01 -05:00
// Command represents a command with its subcommands or arguments.
type Command struct {
2023-04-13 18:17:27 -05:00
prog string
2022-03-27 04:09:56 -05:00
args [ ] string
parentContext context . Context
desc string
globalArgsLength int
2022-10-15 07:18:31 -05:00
brokenArgs [ ] string
2016-11-03 17:16:01 -05:00
}
func ( c * Command ) String ( ) string {
2023-04-13 18:17:27 -05:00
return c . toString ( false )
}
func ( c * Command ) toString ( sanitizing bool ) string {
// WARNING: this function is for debugging purposes only. It's much better than old code (which only joins args with space),
// It's impossible to make a simple and 100% correct implementation of argument quoting for different platforms.
debugQuote := func ( s string ) string {
if strings . ContainsAny ( s , " `'\"\t\r\n" ) {
return fmt . Sprintf ( "%q" , s )
}
return s
2016-11-03 17:16:01 -05:00
}
2023-04-13 18:17:27 -05:00
a := make ( [ ] string , 0 , len ( c . args ) + 1 )
a = append ( a , debugQuote ( c . prog ) )
for _ , arg := range c . args {
if sanitizing && ( strings . Contains ( arg , "://" ) && strings . Contains ( arg , "@" ) ) {
a = append ( a , debugQuote ( util . SanitizeCredentialURLs ( arg ) ) )
} else {
a = append ( a , debugQuote ( arg ) )
}
}
return strings . Join ( a , " " )
2016-11-03 17:16:01 -05:00
}
// NewCommand creates and returns a new Git Command based on given command and arguments.
2022-10-15 07:18:31 -05:00
// Each argument should be safe to be trusted. User-provided arguments should be passed to AddDynamicArguments instead.
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)
## Review without space diff
https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1
## Purpose of this PR
1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command
## The main idea of this PR
* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
* Before: `AddArguments("-m").AddDynamicArguments(message)`
* After: `AddOptionValues("-m", message)`
* -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`
## FAQ
### Why these changes were not done in #21535 ?
#21535 is mainly a search&replace, it did its best to not change too
much logic.
Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.
### The naming of `AddOptionXxx`
According to git's manual, the `--xxx` part is called `option`.
### How can it guarantee that `internal.CmdArg` won't be not misused?
Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.
And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.
### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?
Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.
### Why there was a `CmdArgCheck` and why it's removed?
At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.
### Why many codes for `signArg == ""` is deleted?
Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-03 21:30:43 -05:00
func NewCommand ( ctx context . Context , args ... internal . CmdArg ) * Command {
2022-01-25 13:15:58 -05:00
// Make an explicit copy of globalCommandArgs, otherwise append might overwrite it
2022-10-23 09:44:45 -05:00
cargs := make ( [ ] string , 0 , len ( globalCommandArgs ) + len ( args ) )
for _ , arg := range globalCommandArgs {
cargs = append ( cargs , string ( arg ) )
}
for _ , arg := range args {
cargs = append ( cargs , string ( arg ) )
}
2016-11-03 17:16:01 -05:00
return & Command {
2023-04-13 18:17:27 -05:00
prog : GitExecutable ,
2022-10-23 09:44:45 -05:00
args : cargs ,
2022-03-27 04:09:56 -05:00
parentContext : ctx ,
globalArgsLength : len ( globalCommandArgs ) ,
2016-11-03 17:16:01 -05:00
}
}
2020-05-16 18:31:38 -05:00
// NewCommandContextNoGlobals creates and returns a new Git Command based on given command and arguments only with the specify args and don't care global command args
2022-10-15 07:18:31 -05:00
// Each argument should be safe to be trusted. User-provided arguments should be passed to AddDynamicArguments instead.
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)
## Review without space diff
https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1
## Purpose of this PR
1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command
## The main idea of this PR
* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
* Before: `AddArguments("-m").AddDynamicArguments(message)`
* After: `AddOptionValues("-m", message)`
* -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`
## FAQ
### Why these changes were not done in #21535 ?
#21535 is mainly a search&replace, it did its best to not change too
much logic.
Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.
### The naming of `AddOptionXxx`
According to git's manual, the `--xxx` part is called `option`.
### How can it guarantee that `internal.CmdArg` won't be not misused?
Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.
And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.
### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?
Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.
### Why there was a `CmdArgCheck` and why it's removed?
At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.
### Why many codes for `signArg == ""` is deleted?
Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-03 21:30:43 -05:00
func NewCommandContextNoGlobals ( ctx context . Context , args ... internal . CmdArg ) * Command {
2022-10-23 09:44:45 -05:00
cargs := make ( [ ] string , 0 , len ( args ) )
for _ , arg := range args {
cargs = append ( cargs , string ( arg ) )
}
2019-11-26 19:35:52 -05:00
return & Command {
2023-04-13 18:17:27 -05:00
prog : GitExecutable ,
2022-10-23 09:44:45 -05:00
args : cargs ,
2020-05-16 18:31:38 -05:00
parentContext : ctx ,
2019-11-26 19:35:52 -05:00
}
}
2019-11-30 09:40:22 -05:00
// SetParentContext sets the parent context for this command
func ( c * Command ) SetParentContext ( ctx context . Context ) * Command {
c . parentContext = ctx
return c
}
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)
## Review without space diff
https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1
## Purpose of this PR
1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command
## The main idea of this PR
* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
* Before: `AddArguments("-m").AddDynamicArguments(message)`
* After: `AddOptionValues("-m", message)`
* -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`
## FAQ
### Why these changes were not done in #21535 ?
#21535 is mainly a search&replace, it did its best to not change too
much logic.
Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.
### The naming of `AddOptionXxx`
According to git's manual, the `--xxx` part is called `option`.
### How can it guarantee that `internal.CmdArg` won't be not misused?
Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.
And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.
### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?
Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.
### Why there was a `CmdArgCheck` and why it's removed?
At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.
### Why many codes for `signArg == ""` is deleted?
Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-03 21:30:43 -05:00
// SetDescription sets the description for this command which be returned on c.String()
2019-11-30 09:40:22 -05:00
func ( c * Command ) SetDescription ( desc string ) * Command {
c . desc = desc
return c
}
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)
## Review without space diff
https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1
## Purpose of this PR
1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command
## The main idea of this PR
* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
* Before: `AddArguments("-m").AddDynamicArguments(message)`
* After: `AddOptionValues("-m", message)`
* -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`
## FAQ
### Why these changes were not done in #21535 ?
#21535 is mainly a search&replace, it did its best to not change too
much logic.
Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.
### The naming of `AddOptionXxx`
According to git's manual, the `--xxx` part is called `option`.
### How can it guarantee that `internal.CmdArg` won't be not misused?
Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.
And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.
### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?
Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.
### Why there was a `CmdArgCheck` and why it's removed?
At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.
### Why many codes for `signArg == ""` is deleted?
Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-03 21:30:43 -05:00
// isSafeArgumentValue checks if the argument is safe to be used as a value (not an option)
func isSafeArgumentValue ( s string ) bool {
return s == "" || s [ 0 ] != '-'
}
// isValidArgumentOption checks if the argument is a valid option (starting with '-').
// It doesn't check whether the option is supported or not
func isValidArgumentOption ( s string ) bool {
return s != "" && s [ 0 ] == '-'
}
// AddArguments adds new git arguments (option/value) to the command. It only accepts string literals, or trusted CmdArg.
// Type CmdArg is in the internal package, so it can not be used outside of this package directly,
// it makes sure that user-provided arguments won't cause RCE risks.
// User-provided arguments should be passed by other AddXxx functions
func ( c * Command ) AddArguments ( args ... internal . CmdArg ) * Command {
2022-10-23 09:44:45 -05:00
for _ , arg := range args {
c . args = append ( c . args , string ( arg ) )
}
2016-11-03 17:16:01 -05:00
return c
}
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)
## Review without space diff
https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1
## Purpose of this PR
1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command
## The main idea of this PR
* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
* Before: `AddArguments("-m").AddDynamicArguments(message)`
* After: `AddOptionValues("-m", message)`
* -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`
## FAQ
### Why these changes were not done in #21535 ?
#21535 is mainly a search&replace, it did its best to not change too
much logic.
Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.
### The naming of `AddOptionXxx`
According to git's manual, the `--xxx` part is called `option`.
### How can it guarantee that `internal.CmdArg` won't be not misused?
Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.
And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.
### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?
Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.
### Why there was a `CmdArgCheck` and why it's removed?
At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.
### Why many codes for `signArg == ""` is deleted?
Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-03 21:30:43 -05:00
// AddOptionValues adds a new option with a list of non-option values
// For example: AddOptionValues("--opt", val) means 2 arguments: {"--opt", val}.
// The values are treated as dynamic argument values. It equals to: AddArguments("--opt") then AddDynamicArguments(val).
func ( c * Command ) AddOptionValues ( opt internal . CmdArg , args ... string ) * Command {
if ! isValidArgumentOption ( string ( opt ) ) {
c . brokenArgs = append ( c . brokenArgs , string ( opt ) )
return c
}
c . args = append ( c . args , string ( opt ) )
c . AddDynamicArguments ( args ... )
return c
}
// AddOptionFormat adds a new option with a format string and arguments
// For example: AddOptionFormat("--opt=%s %s", val1, val2) means 1 argument: {"--opt=val1 val2"}.
func ( c * Command ) AddOptionFormat ( opt string , args ... any ) * Command {
if ! isValidArgumentOption ( opt ) {
c . brokenArgs = append ( c . brokenArgs , opt )
return c
}
// a quick check to make sure the format string matches the number of arguments, to find low-level mistakes ASAP
if strings . Count ( strings . ReplaceAll ( opt , "%%" , "" ) , "%" ) != len ( args ) {
c . brokenArgs = append ( c . brokenArgs , opt )
return c
}
s := fmt . Sprintf ( opt , args ... )
c . args = append ( c . args , s )
return c
}
// AddDynamicArguments adds new dynamic argument values to the command.
// The arguments may come from user input and can not be trusted, so no leading '-' is allowed to avoid passing options.
// TODO: in the future, this function can be renamed to AddArgumentValues
2022-10-15 05:49:26 -05:00
func ( c * Command ) AddDynamicArguments ( args ... string ) * Command {
for _ , arg := range args {
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)
## Review without space diff
https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1
## Purpose of this PR
1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command
## The main idea of this PR
* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
* Before: `AddArguments("-m").AddDynamicArguments(message)`
* After: `AddOptionValues("-m", message)`
* -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`
## FAQ
### Why these changes were not done in #21535 ?
#21535 is mainly a search&replace, it did its best to not change too
much logic.
Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.
### The naming of `AddOptionXxx`
According to git's manual, the `--xxx` part is called `option`.
### How can it guarantee that `internal.CmdArg` won't be not misused?
Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.
And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.
### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?
Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.
### Why there was a `CmdArgCheck` and why it's removed?
At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.
### Why many codes for `signArg == ""` is deleted?
Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-03 21:30:43 -05:00
if ! isSafeArgumentValue ( arg ) {
2022-10-15 07:18:31 -05:00
c . brokenArgs = append ( c . brokenArgs , arg )
2022-10-15 05:49:26 -05:00
}
}
2022-10-15 07:18:31 -05:00
if len ( c . brokenArgs ) != 0 {
return c
}
2022-10-15 05:49:26 -05:00
c . args = append ( c . args , args ... )
return c
}
2022-10-23 09:44:45 -05:00
// AddDashesAndList adds the "--" and then add the list as arguments, it's usually for adding file list
// At the moment, this function can be only called once, maybe in future it can be refactored to support multiple calls (if necessary)
func ( c * Command ) AddDashesAndList ( list ... string ) * Command {
c . args = append ( c . args , "--" )
// Some old code also checks `arg != ""`, IMO it's not necessary.
// If the check is needed, the list should be prepared before the call to this function
c . args = append ( c . args , list ... )
return c
}
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)
## Review without space diff
https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1
## Purpose of this PR
1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command
## The main idea of this PR
* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
* Before: `AddArguments("-m").AddDynamicArguments(message)`
* After: `AddOptionValues("-m", message)`
* -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`
## FAQ
### Why these changes were not done in #21535 ?
#21535 is mainly a search&replace, it did its best to not change too
much logic.
Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.
### The naming of `AddOptionXxx`
According to git's manual, the `--xxx` part is called `option`.
### How can it guarantee that `internal.CmdArg` won't be not misused?
Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.
And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.
### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?
Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.
### Why there was a `CmdArgCheck` and why it's removed?
At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.
### Why many codes for `signArg == ""` is deleted?
Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-03 21:30:43 -05:00
// ToTrustedCmdArgs converts a list of strings (trusted as argument) to TrustedCmdArgs
2023-03-09 10:48:52 -05:00
// In most cases, it shouldn't be used. Use NewCommand().AddXxx() function instead
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)
## Review without space diff
https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1
## Purpose of this PR
1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command
## The main idea of this PR
* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
* Before: `AddArguments("-m").AddDynamicArguments(message)`
* After: `AddOptionValues("-m", message)`
* -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`
## FAQ
### Why these changes were not done in #21535 ?
#21535 is mainly a search&replace, it did its best to not change too
much logic.
Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.
### The naming of `AddOptionXxx`
According to git's manual, the `--xxx` part is called `option`.
### How can it guarantee that `internal.CmdArg` won't be not misused?
Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.
And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.
### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?
Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.
### Why there was a `CmdArgCheck` and why it's removed?
At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.
### Why many codes for `signArg == ""` is deleted?
Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-03 21:30:43 -05:00
func ToTrustedCmdArgs ( args [ ] string ) TrustedCmdArgs {
ret := make ( TrustedCmdArgs , len ( args ) )
for i , arg := range args {
ret [ i ] = internal . CmdArg ( arg )
2022-10-23 09:44:45 -05:00
}
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)
## Review without space diff
https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1
## Purpose of this PR
1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command
## The main idea of this PR
* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
* Before: `AddArguments("-m").AddDynamicArguments(message)`
* After: `AddOptionValues("-m", message)`
* -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`
## FAQ
### Why these changes were not done in #21535 ?
#21535 is mainly a search&replace, it did its best to not change too
much logic.
Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.
### The naming of `AddOptionXxx`
According to git's manual, the `--xxx` part is called `option`.
### How can it guarantee that `internal.CmdArg` won't be not misused?
Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.
And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.
### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?
Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.
### Why there was a `CmdArgCheck` and why it's removed?
At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.
### Why many codes for `signArg == ""` is deleted?
Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-03 21:30:43 -05:00
return ret
2022-10-23 09:44:45 -05:00
}
2022-08-06 08:13:11 -05:00
// RunOpts represents parameters to run the command. If UseContextTimeout is specified, then Timeout is ignored.
2022-03-31 21:55:30 -05:00
type RunOpts struct {
2022-08-06 08:13:11 -05:00
Env [ ] string
Timeout time . Duration
UseContextTimeout bool
2023-04-19 08:40:42 -05:00
// Dir is the working dir for the git command, however:
// FIXME: this could be incorrect in many cases, for example:
// * /some/path/.git
// * /some/path/.git/gitea-data/data/repositories/user/repo.git
// If "user/repo.git" is invalid/broken, then running git command in it will use "/some/path/.git", and produce unexpected results
// The correct approach is to use `--git-dir" global argument
Dir string
Stdout , Stderr io . Writer
2023-09-08 08:20:38 -05:00
// Stdin is used for passing input to the command
// The caller must make sure the Stdin writer is closed properly to finish the Run function.
// Otherwise, the Run function may hang for long time or forever, especially when the Git's context deadline is not the same as the caller's.
// Some common mistakes:
// * `defer stdinWriter.Close()` then call `cmd.Run()`: the Run() would never return if the command is killed by timeout
// * `go { case <- parentContext.Done(): stdinWriter.Close() }` with `cmd.Run(DefaultTimeout)`: the command would have been killed by timeout but the Run doesn't return until stdinWriter.Close()
// * `go { if stdoutReader.Read() err != nil: stdinWriter.Close() }` with `cmd.Run()`: the stdoutReader may never return error if the command is killed by timeout
// In the future, ideally the git module itself should have full control of the stdin, to avoid such problems and make it easier to refactor to a better architecture.
Stdin io . Reader
PipelineFunc func ( context . Context , context . CancelFunc ) error
2021-08-18 08:10:39 -05:00
}
2022-07-08 03:09:07 -05:00
func commonBaseEnvs ( ) [ ] string {
2022-06-09 20:57:49 -05:00
// at the moment, do not set "GIT_CONFIG_NOSYSTEM", users may have put some configs like "receive.certNonceSeed" in it
2022-07-08 03:09:07 -05:00
envs := [ ] string {
2022-06-09 20:57:49 -05:00
"HOME=" + HomeDir ( ) , // make Gitea use internal git config only, to prevent conflicts with user's git config
2022-07-08 03:09:07 -05:00
"GIT_NO_REPLACE_OBJECTS=1" , // ignore replace references (https://git-scm.com/docs/git-replace)
}
// some environment variables should be passed to git command
passThroughEnvKeys := [ ] string {
"GNUPGHOME" , // git may call gnupg to do commit signing
}
for _ , key := range passThroughEnvKeys {
if val , ok := os . LookupEnv ( key ) ; ok {
envs = append ( envs , key + "=" + val )
}
2022-06-09 20:57:49 -05:00
}
2022-07-08 03:09:07 -05:00
return envs
}
// CommonGitCmdEnvs returns the common environment variables for a "git" command.
func CommonGitCmdEnvs ( ) [ ] string {
return append ( commonBaseEnvs ( ) , [ ] string {
"LC_ALL=" + DefaultLocale ,
"GIT_TERMINAL_PROMPT=0" , // avoid prompting for credentials interactively, supported since git v2.3
} ... )
2022-06-09 20:57:49 -05:00
}
2022-10-23 09:44:45 -05:00
// CommonCmdServEnvs is like CommonGitCmdEnvs, but it only returns minimal required environment variables for the "gitea serv" command
2022-06-09 20:57:49 -05:00
func CommonCmdServEnvs ( ) [ ] string {
2022-07-08 03:09:07 -05:00
return commonBaseEnvs ( )
2022-06-09 20:57:49 -05:00
}
2022-10-15 07:18:31 -05:00
var ErrBrokenCommand = errors . New ( "git command is broken" )
2022-03-31 21:55:30 -05:00
// Run runs the command with the RunOpts
func ( c * Command ) Run ( opts * RunOpts ) error {
2022-10-15 07:18:31 -05:00
if len ( c . brokenArgs ) != 0 {
log . Error ( "git command is broken: %s, broken args: %s" , c . String ( ) , strings . Join ( c . brokenArgs , " " ) )
return ErrBrokenCommand
}
2022-03-31 21:55:30 -05:00
if opts == nil {
opts = & RunOpts { }
}
2022-11-13 15:45:20 -05:00
// We must not change the provided options
timeout := opts . Timeout
if timeout <= 0 {
timeout = defaultCommandExecutionTimeout
2016-11-03 17:16:01 -05:00
}
2022-03-31 21:55:30 -05:00
if len ( opts . Dir ) == 0 {
2023-04-13 18:17:27 -05:00
log . Debug ( "git.Command.Run: %s" , c )
2016-11-03 17:16:01 -05:00
} else {
2023-04-13 18:17:27 -05:00
log . Debug ( "git.Command.RunDir(%s): %s" , opts . Dir , c )
2016-11-03 17:16:01 -05:00
}
2021-11-30 15:06:32 -05:00
desc := c . desc
if desc == "" {
2023-04-13 18:17:27 -05:00
if opts . Dir == "" {
desc = fmt . Sprintf ( "git: %s" , c . toString ( true ) )
} else {
desc = fmt . Sprintf ( "git(dir:%s): %s" , opts . Dir , c . toString ( true ) )
2022-03-27 06:54:09 -05:00
}
2021-11-30 15:06:32 -05:00
}
2022-08-06 08:13:11 -05:00
var ctx context . Context
var cancel context . CancelFunc
var finished context . CancelFunc
if opts . UseContextTimeout {
ctx , cancel , finished = process . GetManager ( ) . AddContext ( c . parentContext , desc )
} else {
2022-11-13 15:45:20 -05:00
ctx , cancel , finished = process . GetManager ( ) . AddContextTimeout ( c . parentContext , timeout , desc )
2022-08-06 08:13:11 -05:00
}
2021-11-30 15:06:32 -05:00
defer finished ( )
2017-04-07 21:23:39 -05:00
2023-07-04 11:22:37 -05:00
startTime := time . Now ( )
2023-04-13 18:17:27 -05:00
cmd := exec . CommandContext ( ctx , c . prog , c . args ... )
2022-03-31 21:55:30 -05:00
if opts . Env == nil {
2021-05-17 05:59:31 -05:00
cmd . Env = os . Environ ( )
2019-11-10 03:42:51 -05:00
} else {
2022-03-31 21:55:30 -05:00
cmd . Env = opts . Env
2019-11-10 03:42:51 -05:00
}
Disable new signal-based asynchronous goroutine preemption from GO 1.14 in git env (#11237)
As seen in trouble shooting #11032 the new feature of Go 1.14 is causing several second delays in startup in certain situations. Debugging shows it spending several seconds handling SIGURG commands during init:
```
6922:04:51.984234 trace init() ./modules/queue/unique_queue_wrapped.go
remote: ) = 69 <0.000012>
remote: [pid 15984] 22:04:51 write(1, "\ttime taken: 236.761\302\265s\n\n", 25 time taken: 236.761µs
remote:
remote: ) = 25 <0.000011>
remote: [pid 15984] 22:04:51 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=15984, si_uid=0} ---
remote: [pid 15984] 22:04:52 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=15984, si_uid=0} ---
remote: [pid 15984] 22:04:52 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=15984, si_uid=0} ---
```
This causes up to 20 seconds added to a push in some cases as it happens for each call of the gitea hook command. This is likely the cause of #10661 as well and would start to effect users once we release 1.12 which would be the first release compiled with Go 1.14. I suspect this is just a slight issue with the upstream implementatation as there have been a few very similar bugs fixed and reported:
https://github.com/golang/go/issues/37741
https://github.com/golang/go/issues/37942
We should revisit this in the future and see if a newer version of Go has solved it, but for now disable this option in the environment that gitea hook runs in to avoid it.
2020-04-28 10:45:32 -05:00
2022-06-03 09:36:18 -05:00
process . SetSysProcAttribute ( cmd )
2022-06-09 20:57:49 -05:00
cmd . Env = append ( cmd . Env , CommonGitCmdEnvs ( ) ... )
2022-03-31 21:55:30 -05:00
cmd . Dir = opts . Dir
cmd . Stdout = opts . Stdout
cmd . Stderr = opts . Stderr
cmd . Stdin = opts . Stdin
2016-11-03 17:16:01 -05:00
if err := cmd . Start ( ) ; err != nil {
return err
}
2022-03-31 21:55:30 -05:00
if opts . PipelineFunc != nil {
err := opts . PipelineFunc ( ctx , cancel )
2020-01-15 03:32:57 -05:00
if err != nil {
cancel ( )
2020-12-17 06:50:21 -05:00
_ = cmd . Wait ( )
2020-01-15 03:32:57 -05:00
return err
}
2019-11-11 06:46:28 -05:00
}
2023-07-04 11:22:37 -05:00
err := cmd . Wait ( )
elapsed := time . Since ( startTime )
if elapsed > time . Second {
log . Debug ( "slow git.Command.Run: %s (%s)" , c , elapsed )
}
2024-03-02 08:38:34 -05:00
// We need to check if the context is canceled by the program on Windows.
// This is because Windows does not have signal checking when terminating the process.
// It always returns exit code 1, unlike Linux, which has many exit codes for signals.
if runtime . GOOS == "windows" &&
err != nil &&
err . Error ( ) == "" &&
cmd . ProcessState . ExitCode ( ) == 1 &&
ctx . Err ( ) == context . Canceled {
return ctx . Err ( )
}
2023-07-04 11:22:37 -05:00
if err != nil && ctx . Err ( ) != context . DeadlineExceeded {
2017-05-30 04:32:01 -05:00
return err
}
return ctx . Err ( )
2016-11-03 17:16:01 -05:00
}
2022-03-31 21:55:30 -05:00
type RunStdError interface {
2022-03-31 06:56:22 -05:00
error
2022-06-09 20:57:49 -05:00
Unwrap ( ) error
2022-03-31 06:56:22 -05:00
Stderr ( ) string
2019-05-11 10:29:17 -05:00
}
2022-03-31 21:55:30 -05:00
type runStdError struct {
2022-03-31 06:56:22 -05:00
err error
stderr string
errMsg string
2019-05-11 10:29:17 -05:00
}
2022-03-31 21:55:30 -05:00
func ( r * runStdError ) Error ( ) string {
2022-03-31 06:56:22 -05:00
// the stderr must be in the returned error text, some code only checks `strings.Contains(err.Error(), "git error")`
if r . errMsg == "" {
r . errMsg = ConcatenateError ( r . err , r . stderr ) . Error ( )
}
return r . errMsg
2019-05-11 10:29:17 -05:00
}
2022-03-31 21:55:30 -05:00
func ( r * runStdError ) Unwrap ( ) error {
2022-03-31 06:56:22 -05:00
return r . err
}
2022-03-31 21:55:30 -05:00
func ( r * runStdError ) Stderr ( ) string {
2022-03-31 06:56:22 -05:00
return r . stderr
2016-11-03 17:16:01 -05:00
}
2024-03-24 11:05:00 -05:00
func IsErrorExitCode ( err error , code int ) bool {
2022-06-09 20:57:49 -05:00
var exitError * exec . ExitError
2024-03-24 11:05:00 -05:00
if errors . As ( err , & exitError ) {
2022-06-09 20:57:49 -05:00
return exitError . ExitCode ( ) == code
}
return false
}
2022-03-31 21:55:30 -05:00
// RunStdString runs the command with options and returns stdout/stderr as string. and store stderr to returned error (err combined with stderr).
func ( c * Command ) RunStdString ( opts * RunOpts ) ( stdout , stderr string , runErr RunStdError ) {
stdoutBytes , stderrBytes , err := c . RunStdBytes ( opts )
2023-12-17 09:38:54 -05:00
stdout = util . UnsafeBytesToString ( stdoutBytes )
stderr = util . UnsafeBytesToString ( stderrBytes )
2022-03-31 06:56:22 -05:00
if err != nil {
2022-03-31 21:55:30 -05:00
return stdout , stderr , & runStdError { err : err , stderr : stderr }
2022-03-31 06:56:22 -05:00
}
// even if there is no err, there could still be some stderr output, so we just return stdout/stderr as they are
return stdout , stderr , nil
}
2022-03-31 21:55:30 -05:00
// RunStdBytes runs the command with options and returns stdout/stderr as bytes. and store stderr to returned error (err combined with stderr).
func ( c * Command ) RunStdBytes ( opts * RunOpts ) ( stdout , stderr [ ] byte , runErr RunStdError ) {
if opts == nil {
opts = & RunOpts { }
}
if opts . Stdout != nil || opts . Stderr != nil {
2022-03-31 06:56:22 -05:00
// we must panic here, otherwise there would be bugs if developers set Stdin/Stderr by mistake, and it would be very difficult to debug
2022-03-31 21:55:30 -05:00
panic ( "stdout and stderr field must be nil when using RunStdBytes" )
2022-03-31 06:56:22 -05:00
}
stdoutBuf := & bytes . Buffer { }
stderrBuf := & bytes . Buffer { }
2022-11-13 15:45:20 -05:00
// We must not change the provided options as it could break future calls - therefore make a copy.
newOpts := & RunOpts {
Env : opts . Env ,
Timeout : opts . Timeout ,
UseContextTimeout : opts . UseContextTimeout ,
Dir : opts . Dir ,
Stdout : stdoutBuf ,
Stderr : stderrBuf ,
Stdin : opts . Stdin ,
PipelineFunc : opts . PipelineFunc ,
}
err := c . Run ( newOpts )
2022-03-31 06:56:22 -05:00
stderr = stderrBuf . Bytes ( )
if err != nil {
2023-12-17 09:38:54 -05:00
return nil , stderr , & runStdError { err : err , stderr : util . UnsafeBytesToString ( stderr ) }
2022-03-31 06:56:22 -05:00
}
// even if there is no err, there could still be some stderr output
return stdoutBuf . Bytes ( ) , stderr , nil
2016-11-03 17:16:01 -05:00
}
2022-01-25 13:15:58 -05:00
// AllowLFSFiltersArgs return globalCommandArgs with lfs filter, it should only be used for tests
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)
## Review without space diff
https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1
## Purpose of this PR
1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command
## The main idea of this PR
* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
* Before: `AddArguments("-m").AddDynamicArguments(message)`
* After: `AddOptionValues("-m", message)`
* -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`
## FAQ
### Why these changes were not done in #21535 ?
#21535 is mainly a search&replace, it did its best to not change too
much logic.
Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.
### The naming of `AddOptionXxx`
According to git's manual, the `--xxx` part is called `option`.
### How can it guarantee that `internal.CmdArg` won't be not misused?
Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.
And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.
### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?
Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.
### Why there was a `CmdArgCheck` and why it's removed?
At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.
### Why many codes for `signArg == ""` is deleted?
Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-03 21:30:43 -05:00
func AllowLFSFiltersArgs ( ) TrustedCmdArgs {
2022-01-25 13:15:58 -05:00
// Now here we should explicitly allow lfs filters to run
Refactor git command package to improve security and maintainability (#22678)
This PR follows #21535 (and replace #22592)
## Review without space diff
https://github.com/go-gitea/gitea/pull/22678/files?diff=split&w=1
## Purpose of this PR
1. Make git module command completely safe (risky user inputs won't be
passed as argument option anymore)
2. Avoid low-level mistakes like
https://github.com/go-gitea/gitea/pull/22098#discussion_r1045234918
3. Remove deprecated and dirty `CmdArgCheck` function, hide the `CmdArg`
type
4. Simplify code when using git command
## The main idea of this PR
* Move the `git.CmdArg` to the `internal` package, then no other package
except `git` could use it. Then developers could never do
`AddArguments(git.CmdArg(userInput))` any more.
* Introduce `git.ToTrustedCmdArgs`, it's for user-provided and already
trusted arguments. It's only used in a few cases, for example: use git
arguments from config file, help unit test with some arguments.
* Introduce `AddOptionValues` and `AddOptionFormat`, they make code more
clear and simple:
* Before: `AddArguments("-m").AddDynamicArguments(message)`
* After: `AddOptionValues("-m", message)`
* -
* Before: `AddArguments(git.CmdArg(fmt.Sprintf("--author='%s <%s>'",
sig.Name, sig.Email)))`
* After: `AddOptionFormat("--author='%s <%s>'", sig.Name, sig.Email)`
## FAQ
### Why these changes were not done in #21535 ?
#21535 is mainly a search&replace, it did its best to not change too
much logic.
Making the framework better needs a lot of changes, so this separate PR
is needed as the second step.
### The naming of `AddOptionXxx`
According to git's manual, the `--xxx` part is called `option`.
### How can it guarantee that `internal.CmdArg` won't be not misused?
Go's specification guarantees that. Trying to access other package's
internal package causes compilation error.
And, `golangci-lint` also denies the git/internal package. Only the
`git/command.go` can use it carefully.
### There is still a `ToTrustedCmdArgs`, will it still allow developers
to make mistakes and pass untrusted arguments?
Generally speaking, no. Because when using `ToTrustedCmdArgs`, the code
will be very complex (see the changes for examples). Then developers and
reviewers can know that something might be unreasonable.
### Why there was a `CmdArgCheck` and why it's removed?
At the moment of #21535, to reduce unnecessary changes, `CmdArgCheck`
was introduced as a hacky patch. Now, almost all code could be written
as `cmd := NewCommand(); cmd.AddXxx(...)`, then there is no need for
`CmdArgCheck` anymore.
### Why many codes for `signArg == ""` is deleted?
Because in the old code, `signArg` could never be empty string, it's
either `-S[key-id]` or `--no-gpg-sign`. So the `signArg == ""` is just
dead code.
---------
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-03 21:30:43 -05:00
filteredLFSGlobalArgs := make ( TrustedCmdArgs , len ( globalCommandArgs ) )
2022-01-25 13:15:58 -05:00
j := 0
for _ , arg := range globalCommandArgs {
2022-10-23 09:44:45 -05:00
if strings . Contains ( string ( arg ) , "lfs" ) {
2022-01-25 13:15:58 -05:00
j --
} else {
filteredLFSGlobalArgs [ j ] = arg
j ++
}
}
return filteredLFSGlobalArgs [ : j ]
}