Ramkumar Chinchani
b2a4388522
gc: add a unit test
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-03-24 12:02:52 -07:00
Petu Eusebiu
be910cf01c
lint: Move out config reloader context from controller struct
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-03-24 10:50:01 -07:00
Petu Eusebiu
353b0c6034
Move api constants in separate 'constants' package to avoid circular imports
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-03-24 10:50:01 -07:00
Petu Eusebiu
f53dc9eb8d
sync: Add a new flag to enforce syncing only signed images, closes #455
...
sync: When checking if a image is already synced also check for changes in upstream signatures.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-03-24 10:50:01 -07:00
laurentiuNiculae
0d148e1d6b
new config option for sync-destination
...
Signed-off-by: laurentiuNiculae <themelopeus@gmail.com>
2022-03-21 08:12:34 -07:00
Petu Eusebiu
6d04ab3cdc
sync: support reloading sync config when the config file changes
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-03-18 11:24:11 -07:00
laurentiuNiculae
63d94d4ac5
Update dist-spec version automatically
...
Warning if config has wrong dist-spec version
Signed-off-by: laurentiuNiculae <themelopeus@gmail.com>
2022-03-14 10:24:03 -07:00
Shivam Mishra
b8010e1ee4
routes: changes required to do browser authentication
...
whenever we make a request that contains header apart from CORS allowed header, browser sends a preflight request
and in response accept *Access-Control-Allow-Headers*.
preflight request is in form of OPTIONS method, added new http handler func to set headers
and returns HTTP status ok in case of OPTIONS method.
in case of authorization, request contains authorization header
added authorization header in Access-Control-Allow-Headers list
added AllowOrigin field in HTTPConfig this field value is set to Access-Control-Allow-Origin header and will give zot adminstrator to limit incoming request.
Signed-off-by: Shivam Mishra <shimish2@cisco.com>
2022-03-08 17:42:54 -08:00
Eng Zer Jun
0d77b60de7
test: use T.TempDir
to create temporary test directory
...
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-03-07 12:28:49 -08:00
Ramkumar Chinchani
4be2652085
conformance: fix cross-mount behavior when 'from' is missing
...
fixes issue #442
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-03-04 17:24:50 -08:00
Ramkumar Chinchani
95e4b2054b
upgrade module deps
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-03-04 13:10:58 -08:00
Petu Eusebiu
9cffbcaccb
s3: bugfix, use sync.Map instead of map for storing multi part uploads references
...
add storage lock in GetIndexContent
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-03-03 09:12:07 -08:00
Ramkumar Chinchani
8db3e1b192
CVE-2022-23649: fix dependabot alert
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-03-02 12:01:14 -08:00
Ramkumar Chinchani
3ada6af0de
tls: set min version to 1.2 and restrict cipher suites
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-03-02 10:03:50 -08:00
Petu Eusebiu
45968e0bb7
sync: fix inconsistent test, used inject error fw for hard to reach test cases
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-03-01 09:56:27 -08:00
Ramkumar Chinchani
38a110314b
gc: add a gcDelay param
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-14 14:03:22 -08:00
Petu Eusebiu
35eeedb22a
sync: pull only missing images, not everything, closes #335
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-02-14 08:09:18 -08:00
Petu Eusebiu
0ec39c0313
sync: make RetryDelay and MaxRetries optional
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-02-11 09:11:11 -08:00
Shivam Mishra
37d150e32f
search: graphql api to give detailed repo info
...
DetailedRepoInfo graphql api returns detailed repo info given repo name
repo contains its manifests info
Each manifest entry contains digest,signed, tag and layers info
Each layer info containes digest, size
Signed-off-by: Shivam Mishra <shimish2@cisco.com>
2022-02-10 16:34:13 -08:00
Ramkumar Chinchani
b2c8533719
test: fix ldap unit tests
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-09 14:48:34 -08:00
Alexei Dodon
47c9b6244e
Added config enable=true/false for extensions with default value as enabled closes #258
...
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2022-02-09 09:53:49 -08:00
Ramkumar Chinchani
730fe70f2f
coverage: improve code coverage
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-09 07:42:15 -08:00
Ramkumar Chinchani
d2aa016cdb
storage: flush/sync contents to disk on file close
...
Behavior controlled by configuration (default=off)
It is a trade-off between performance and consistency.
References:
[1] https://github.com/golang/go/issues/20599
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-07 12:08:01 -08:00
Ramkumar Chinchani
c73e71b018
ci/cd: add a basic push-pull testing using client tools
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-07 10:03:24 -08:00
Petu Eusebiu
f89925fb27
sync: periodically retry if on-demand fails inline, closes #281
...
sync: don't return error on sync signatures, just skip them, closes #375
sync: sync signatures on demand
sync on demand: in case of parallel requests pull image just once, closes #344
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-02-07 09:04:13 -08:00
Ramkumar Chinchani
87084f286b
storage: improve/fix oci image validation
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-02 13:31:41 -08:00
Petu Eusebiu
a0e65379c8
sync: for a prefix, allow multiple registries as a list instead of only one, closes #343
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-02-01 09:45:09 -08:00
Ramkumar Chinchani
45fe129c63
notaryv2: fix 'notation list'
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-01-31 14:33:21 -08:00
Petu Eusebiu
b9250a783a
Use InsecureSkipVerify only with https upstreams
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-01-29 20:00:27 -08:00
Ramkumar Chinchani
e0a1a82890
coverage: add failure injection framework
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-01-28 08:56:00 -08:00
Alexei Dodon
f47c8222c2
bug: Storage used per zot repo metric is broken
...
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2022-01-26 18:43:14 -08:00
Petu Eusebiu
89b143805e
Remove sync http handler, not needed anymore since added sync on demand
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-01-25 09:18:23 -08:00
Ramkumar Chinchani
4d576a4817
storage: return errors instead of panics
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-01-24 13:33:45 -08:00
Ramkumar Chinchani
1e5ea7e09c
controller: support rate-limiting incoming requests
...
helps constraining resource usage and against flood attacks.
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-01-24 12:48:13 -08:00
Alexei Dodon
d259ba6e4f
Fix scrub command crash
...
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2022-01-22 15:54:13 -08:00
Alexei Dodon
c9a81baa10
Renamed zot-exporter to zxp and added its image to zot release
...
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2022-01-19 10:31:37 -08:00
Petu Eusebiu
1109bb4dde
sync: Added support for syncing notary/cosign signatures, closes #261
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-01-13 08:45:59 -08:00
Petu Eusebiu
e6d6d5a7de
Fix cli version showing help message, closes #361
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-01-12 12:35:17 -08:00
Ramkumar Chinchani
4896adad1b
build: split functionality into separate binaries
...
zot: registry server
zli: zot cli to interact with the zot registry
zui: zot ui (proposed)
zb: zot benchmark (proposed)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-01-11 11:48:18 -08:00
Alexei Dodon
c4d34b7269
Added storage latency histogram metric
...
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2022-01-10 11:55:39 -08:00
Petu Eusebiu
4f825a5e2f
[Identity-based Authorization] Add an option to specify a global policy for all repositories
...
using regex.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-01-07 10:55:20 -08:00
Ramkumar Chinchani
cac7fe4854
storage: use sha256-simd from minio
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-28 22:25:11 -08:00
Ramkumar Chinchani
8183e1467c
lint: some more linter-related cleanup
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-23 22:01:40 -08:00
Ramkumar Chinchani
ac3801ea2d
lint: upgrade golangci-lint
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-20 17:20:35 -08:00
Petu Eusebiu
5f04092e71
Fix negative WaitGroup counter at runtime
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-12-17 16:52:34 -08:00
Petu Eusebiu
c86f44cc53
Disable sync periodically polling when pollInterval is not configured
...
Filtering out sync on demand images based on content configuration
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-12-14 08:59:50 -08:00
Andreea-Lupu
c61c3836db
implement scrub to check manifest/blob integrity
...
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2021-12-09 11:18:09 -08:00
Petu Eusebiu
627cb97ef1
Add wait group for graceful shutdown, closes #302
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-12-08 10:04:52 -08:00
Ramkumar Chinchani
d4307d96ac
build: fix build artifacts in binary
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-07 20:59:26 -08:00
Petu Eusebiu
63a75216ed
sync: allow for saving to a subpath, closes #307
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-12-07 10:13:20 -08:00
Ramkumar Chinchani
96226af869
move references to zotregistry.io and project-zot
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-05 10:52:27 -08:00
Ramkumar Chinchani
e42e42a2cc
artifacts: initial support for artifacts/notaryv2 spec
...
https://github.com/oras-project/artifacts-spec
https://github.com/notaryproject/notaryproject
Fixes issue #264
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-01 18:55:39 -08:00
Alexei Dodon
f99fa37623
ci/cd: unit test hangs for a long time intermittently closes #286
...
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2021-11-29 14:07:49 -08:00
Petu Eusebiu
fff6107310
Sync prefix can be an exact match or a glob pattern, closes #297
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-11-29 13:10:13 -08:00
Petu Eusebiu
f0ef10fa50
sync: cleanup the orphaned private download dir on failure, closes 282
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-11-25 10:31:45 -08:00
Petu Eusebiu
7ada50e9c8
sync: skip tls verify on /v2/_catalog when a registry is configured with tls-verify false
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-11-25 10:31:45 -08:00
Ramkumar Chinchani
5f8f61407e
routes: fix CheckManifest to return content length
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-11-23 14:09:36 -08:00
Alexei Dodon
e900b09cfb
Fix data races in tests, closes #255
...
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2021-11-17 13:23:59 -08:00
Petu Eusebiu
5c07e19c8d
Changed sync behaviour, it used to copy images over http interface
...
now it copies to a local cache and then it copies over storage APIs
- accept all images with or without signatures
- disable sync writing to stdout
- added more logs
- fixed switch statement in routes
- fixed enabling sync multiple times for storage subpaths
closes #266
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-11-15 09:32:43 -08:00
Petu Eusebiu
9c568c0ee2
storage: add s3 backend support (without GC and dedupe)
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-11-15 08:09:00 -08:00
Alexei Dodon
8e4d828867
Implement an API for performance monitoring
...
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2021-11-12 11:14:10 -08:00
Alexei Dodon
f76c76c2e6
Added a timeout option in db cache lock, closes #242
...
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2021-10-27 12:43:50 -07:00
Alexei Dodon
d8aa5b8bf3
Fixing ValidateHardLink, closes #256 ( #257 )
...
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2021-10-27 20:03:26 +03:00
Petu Eusebiu
f7ae491d22
Fix data race in sync tests
...
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-10-25 10:50:05 -07:00
Ramkumar Chinchani
8f3d7d3719
swagger: rename 'docs/' to 'swagger/'
...
Use 'docs/' for zot-related documentation.
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-10-21 13:46:14 -07:00
Petu Eusebiu
19003e8a71
Added new extension "sync"
...
Periodically poll registries and pull images according to sync's config
Added sync on demand, syncing when clients asks for an image which
zot doesn't have.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-10-21 10:32:46 -07:00
Shivam Mishra
d930adbd49
search: update trivy
...
trivy updated to v0.20.0
trivy-db updated to bec0c6a
fanal updated to f7efd1b
2021-10-13 16:37:31 -07:00
Petu Eusebiu
7d077eaf5a
Added storage interface
2021-10-11 17:38:46 -07:00
Petu Eusebiu
20f4051446
Clean blob uploads when clients interrupts uploading, closes #225
2021-10-08 14:55:57 -07:00
Shivam Mishra
63fef3e48c
search: added graphql api to return repository list with latest tag
2021-09-27 14:36:20 -07:00
Ramkumar Chinchani
008d382446
authn: serialize ldap authn calls
...
Some LDAP servers are not MT-safe in that when searches happen with binds
in flight leads to errors such as:
"comment: No other operations may be performed on the connection while a
bind is outstanding"
Add goroutine-id in logs to help debug MT bugs.
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-09-20 10:24:07 -07:00
Petu Eusebiu
62e724532a
Check if auth config is provided when using access control
2021-09-14 12:55:21 -07:00
Petu Eusebiu
609d85d875
Add identity-based access control, closes #51
...
Add a cli subcommand to verify config files validity
2021-08-30 13:56:27 -07:00
Ramkumar Chinchani
26926ad4c2
go.mod: update modules
2021-08-25 11:51:23 -07:00
Shivam Mishra
c6670b1329
api: implement delete by tag
2021-08-23 17:30:41 -07:00
Roxana Nemulescu
fed5c09b71
TLS certs in CLI client
...
resolve #194
2021-08-16 23:42:21 -07:00
Ramkumar Chinchani
63b88d0e57
pkg/storage: fix partially initialized repo storage
...
Thanks shimish2 for the unit test.
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-08-09 23:19:20 -07:00
Shivam Mishra
53b5fa6493
dedupe: stat blob path before creating link
2021-08-09 09:40:35 -07:00
Ramkumar Chinchani
f10c174c0e
routes: add Content-Type header for HEAD manifest response
...
With recent docker client-side changes, on 'docker pull' we see:
"Error response from daemon: missing or empty Content-Type header"
Hence, set Content-Type header.
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-07-23 10:00:32 -07:00
Shivam Mishra
af30c06aff
api: use blob cache path while making hard link
...
previously mount blob will look for blob that is provided in http request and try to hard link that path
but ideally we should look for path from our cache and do the hard link of that particular path.
this commit does the same.
2021-06-30 01:42:21 -07:00
Petu Eusebiu
3a59b9f487
Use freeport to get ports for zot servers in tests
2021-06-29 13:58:39 -07:00
Andrei Aaron
792e82cbdf
Add an 'enable' flag in the server configuration to enable gql-based searches
...
"extensions": {
"search": {
"enable": true
}
}
2021-06-24 12:15:25 -07:00
Andrei Aaron
c1dd7878e4
Add a '--verbose' flag to the 'zot images' output
...
- Show individual layers with size and digest under each image
- Include config digest for each image
See example below
```
IMAGE NAME TAG DIGEST CONFIG LAYERS SIZE
test/godev 0.4.7 7d38d8ca 05b9f86e 519MB
f824a027 65MB
a98af0f5 52MB
ba5b2bc4 163MB
58b1ca8d 228MB
67d798ee 12MB
test/cdev test 2292b4ae cf6f6c77 280MB
f824a027 65MB
a98af0f5 52MB
ba5b2bc4 163MB
test/cdev 0.4.7 2292b4ae cf6f6c77 280MB
f824a027 65MB
a98af0f5 52MB
ba5b2bc4 163MB
Note the new layers and config fields will be visible in the json/yaml format regardless of the value of the verbose flag
```
2021-06-24 12:15:25 -07:00
Andrei Aaron
519ea75d9a
Implement a way to search for an image by manifest, config or layer digest
...
```
Usage:
zot images [config-name] [flags]
Flags:
-d, --digest string List images containing a specific manifest, config, or layer digest
[...]
```
2021-06-24 12:15:25 -07:00
Roxana Nemulescu
97628e69c9
logs: add an audit log for API calls with unit tests
...
resolves #178
2021-06-24 10:53:27 -07:00
Shivam Mishra
28974e81dc
config: support multiple storage locations
...
added support to point multiple storage locations in zot by running multiple instance of zot in background.
see examples/config-multiple.json for more info about config.
Closes #181
2021-05-21 10:18:28 -07:00
Ramkumar Chinchani
9ca6eea940
routes: ignore media-type for PatchBlobUpload
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-05-19 10:53:49 -07:00
Ramkumar Chinchani
8f729820f5
controller: add default headers
...
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-05-10 12:47:53 -07:00
Shivam Mishra
cf25c6f3c8
ci/cd:inculde binary type in version information
2021-05-04 13:16:23 -07:00
Shivam Mishra
a7c17b7c16
spec: added support for mount request using hard link
2021-05-04 09:42:29 -07:00
Ramkumar Chinchani
f7829d6470
dedupe: optimize check-blob with hard links
...
In use cases, when there are large images with shared layers across
repositories, clients may benefit from not re-uploading the same blobs
over and over again.
We ensure this by hard linking when check-blob api is called.
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-05-04 09:42:29 -07:00
Shivam Mishra
3c88102870
server: add idle timeout in http server configuration
2021-04-29 11:00:12 -07:00
Shivam Mishra
2b7b57313a
conformance: fix http status code for cross-repository mounting
2021-01-29 09:35:15 -08:00
Shivam Mishra
9969ba0867
conformance: update README to display conformance results
2021-01-28 15:50:36 -08:00
Ramkumar Chinchani
affdd85986
build: remove bazel
...
The idea initially was to use bazel to do our builds, however golang
build system is now good enough and our code base is entirely go.
It is also slowing down our travis ci/cd pipeline.
2020-12-21 15:30:13 -08:00
Shivam Mishra
8787142d2c
test: pull test images from aws container registry
2020-12-21 15:24:26 -08:00
Ionut Costin Craciun
dad884ddeb
Raise error when adding a new zot config with an existed saved name
2020-11-04 10:25:34 +02:00
Shivam Mishra
b0ed625a2e
build: increase wait timeout for travis bazel build process
2020-10-27 19:30:06 -07:00
Shivam Mishra
46beb30fc1
build: add build tags to create customizable binaries
2020-10-22 17:20:07 -07:00
Shivam Mishra
7439feb1c2
build: set timeout in travis make build process to avoid timeout failure
2020-10-18 20:55:17 -07:00