0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2024-12-16 21:56:37 -05:00
Commit graph

88 commits

Author SHA1 Message Date
Petu Eusebiu
5c07e19c8d Changed sync behaviour, it used to copy images over http interface
now it copies to a local cache and then it copies over storage APIs

- accept all images with or without signatures
- disable sync writing to stdout
- added more logs
- fixed switch statement in routes
- fixed enabling sync multiple times for storage subpaths

closes #266

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-11-15 09:32:43 -08:00
Petu Eusebiu
9c568c0ee2 storage: add s3 backend support (without GC and dedupe)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-11-15 08:09:00 -08:00
Alexei Dodon
8e4d828867 Implement an API for performance monitoring
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2021-11-12 11:14:10 -08:00
Ramkumar Chinchani
8f3d7d3719 swagger: rename 'docs/' to 'swagger/'
Use 'docs/' for zot-related documentation.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-10-21 13:46:14 -07:00
Petu Eusebiu
19003e8a71 Added new extension "sync"
Periodically poll registries and pull images according to sync's config
Added sync on demand, syncing when clients asks for an image which
zot doesn't have.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-10-21 10:32:46 -07:00
Petu Eusebiu
7d077eaf5a Added storage interface 2021-10-11 17:38:46 -07:00
Petu Eusebiu
20f4051446 Clean blob uploads when clients interrupts uploading, closes #225 2021-10-08 14:55:57 -07:00
Ramkumar Chinchani
008d382446 authn: serialize ldap authn calls
Some LDAP servers are not MT-safe in that when searches happen with binds
in flight leads to errors such as:
"comment: No other operations may be performed on the connection while a
bind is outstanding"

Add goroutine-id in logs to help debug MT bugs.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-09-20 10:24:07 -07:00
Petu Eusebiu
62e724532a Check if auth config is provided when using access control 2021-09-14 12:55:21 -07:00
Petu Eusebiu
609d85d875 Add identity-based access control, closes #51
Add a cli subcommand to verify config files validity
2021-08-30 13:56:27 -07:00
Ramkumar Chinchani
26926ad4c2 go.mod: update modules 2021-08-25 11:51:23 -07:00
Ramkumar Chinchani
f10c174c0e routes: add Content-Type header for HEAD manifest response
With recent docker client-side changes, on 'docker pull' we see:
"Error response from daemon: missing or empty Content-Type header"

Hence, set Content-Type header.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-07-23 10:00:32 -07:00
Shivam Mishra
af30c06aff api: use blob cache path while making hard link
previously mount blob will look for blob that is provided in http request and try to hard link that path
but ideally we should look for path from our cache and do the hard link of that particular path.
this commit does the same.
2021-06-30 01:42:21 -07:00
Petu Eusebiu
3a59b9f487 Use freeport to get ports for zot servers in tests 2021-06-29 13:58:39 -07:00
Andrei Aaron
792e82cbdf Add an 'enable' flag in the server configuration to enable gql-based searches
"extensions": {
        "search": {
            "enable": true
        }
    }
2021-06-24 12:15:25 -07:00
Roxana Nemulescu
97628e69c9 logs: add an audit log for API calls with unit tests
resolves #178
2021-06-24 10:53:27 -07:00
Shivam Mishra
28974e81dc config: support multiple storage locations
added support to point multiple storage locations in zot by running multiple instance of zot in background.

see examples/config-multiple.json for more info about config.

Closes #181
2021-05-21 10:18:28 -07:00
Ramkumar Chinchani
9ca6eea940 routes: ignore media-type for PatchBlobUpload
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-05-19 10:53:49 -07:00
Ramkumar Chinchani
8f729820f5 controller: add default headers
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-05-10 12:47:53 -07:00
Shivam Mishra
cf25c6f3c8 ci/cd:inculde binary type in version information 2021-05-04 13:16:23 -07:00
Shivam Mishra
a7c17b7c16 spec: added support for mount request using hard link 2021-05-04 09:42:29 -07:00
Shivam Mishra
3c88102870 server: add idle timeout in http server configuration 2021-04-29 11:00:12 -07:00
Shivam Mishra
2b7b57313a conformance: fix http status code for cross-repository mounting 2021-01-29 09:35:15 -08:00
Shivam Mishra
9969ba0867 conformance: update README to display conformance results 2021-01-28 15:50:36 -08:00
Ramkumar Chinchani
affdd85986 build: remove bazel
The idea initially was to use bazel to do our builds, however golang
build system is now good enough and our code base is entirely go.
It is also slowing down our travis ci/cd pipeline.
2020-12-21 15:30:13 -08:00
Shivam Mishra
b0ed625a2e build: increase wait timeout for travis bazel build process 2020-10-27 19:30:06 -07:00
Shivam Mishra
46beb30fc1 build: add build tags to create customizable binaries 2020-10-22 17:20:07 -07:00
Shivam Mishra
7439feb1c2 build: set timeout in travis make build process to avoid timeout failure 2020-10-18 20:55:17 -07:00
Shivam Mishra
14214a5794 test: add unit test to verify lock changes 2020-10-16 14:58:45 -07:00
Ramkumar Chinchani
386c72d332 routes: refactor locks to handle large file uploads
The storage layer is protected with read-write locks.
However, we may be holding the locks over unnecessarily large critical
sections.

The typical workflow is that a blob is first uploaded via a per-client
private session-id meaning the blob is not publicly visible yet. When
the blob being uploaded is very large, the transfer takes a long time
while holding the lock.

Private session-id based uploads don't really need locks, and hold locks
only when blobs are published after the upload is complete.
2020-10-16 13:33:11 -07:00
Shivam Mishra
2cf2c16137 Added graphql api feature for image vulnerability scanning 2020-08-18 22:44:34 -07:00
Shivam Mishra
baa5d247ec Enable trivy db download and update 2020-08-18 21:46:17 -07:00
Shivam Mishra
e537f27f00 Added search extension and integrated trivy to support image vulnerability scanning 2020-08-18 21:03:48 -07:00
Ramkumar Chinchani
78be4cbe3c auth: support a read-only mode
This is useful if we want to roll out experimental versions of zot
pointing to some storage shared with another zot instance.

Also, when under storage full conditions, will be useful to turn on this
flag to prevent further writes.
2020-07-10 21:48:35 -07:00
Ramkumar Chinchani
324a517ea3 gc: add a policy to skip garbage collecting new blobs
We perform inline garbage collection of orphan blobs. However, the
dist-spec poses a problem because blobs begin their life as orphan blobs
and then a manifest is add which refers to these blobs.

We use umoci's GC() to perform garbage collection and policy support
has been added recently which can control whether a blob can be skipped
for GC.

In this patch, we use a time-based policy to skip blobs.
2020-07-06 15:52:35 -07:00
Shivam Mishra
af77876306 Upgraded build pipeline
Go version changed to 1.14.4
Golangci-lint changed to 1.26.0
Bazel version changed to 3.0.0
Bazel rules_go version changed to 0.23.3
Bazel gazelle version changed to v0.21.0
Bazel build tools version changed to 0.25.1
Bazel skylib version changed to 1.0.2
2020-06-25 23:43:31 -07:00
Shivam Mishra
85d3e1db4b Changed umoci import path 2020-06-25 17:04:32 -07:00
Tanmay Naik
3f3f7e3f8c tests: add better tests for 3cfb2b3 2020-06-17 20:17:49 -04:00
Tanmay Naik
904ae763d7 tests: add unit tests for fix 3cfb2b3 2020-06-09 19:18:33 -04:00
Tanmay Naik
3cfb2b30a6 fix: the bug when htpasswd has multiple creds
earlier, when you had more than one creds in htpasswd file separated by
newline, it used to only read the first cred in the file and ignore the
rest.
2020-06-09 17:19:01 -04:00
Ramkumar Chinchani
e83999ae0d helm: relax Content-Type checks
OCI registries are moving towards a more generic artifact storage
mechanism. In short-term, at the very least support helm charts.
2020-05-19 16:53:05 -07:00
Ramkumar Chinchani
026b009dbb compat: when in "world-readable" mode, return the WWW-Authenticate
header

containers/image is the dominant client library to interact with
registries.

It detects which authentication to use based on the WWW-Authenticate
header returned when pinging "/v2/" end-point. If we didn't return this
header, then creds are not used for other write-protected end-points.
Hence, the compatibility fix.
2020-05-19 13:54:45 -07:00
Ramkumar Chinchani
dd1fc1e866 config: add gc and dedupe as configurable params (default to enabled)
Since we want to conform to dist-spec, sometimes the gc and dedupe
optimizations conflict with the conformance tests that are being run.
So allow them to be turned off via configuration params.
2020-04-16 16:01:53 -07:00
Ramkumar Chinchani
b1f882e1b8 conformance: align with upstream conformance tests
Upstream conformance tests are being updated, so we need to align along
with our internal GC and dedupe features.

Add a new example config file which plays nice with conformance tests.

DeleteImageManifest() updated to deal with the case where the same
manifest can be created with multiple tags and deleted with the same
digest - so all entries must be deleted.

DeleteBlob() delete the digest key (bucket) when last reference is
dropped
2020-04-16 16:01:53 -07:00
Ramkumar Chinchani
25f5a45296 dedupe: use hard links to dedupe blobs
As the number of repos and layers increases, the greater the probability
that layers are duplicated. We dedupe using hard links when content is
the same. This is intended to be purely a storage layer optimization.
Access control when available is orthogonal this optimization.

Add a durable cache to help speed up layer lookups.

Update README.

Add more unit tests.
2020-04-03 09:29:12 -07:00
Ramkumar Chinchani
9fa185f2bb ldap: prune unused code
We don't use this method. Remove it so CI/CD coverage is better
reported.
2020-03-30 23:12:16 -07:00
Ramkumar Chinchani
8ff60f9138 conformance: fix error msg for DELETE MANIFEST
---
Ran 27 of 27 Specs in 0.120 seconds
SUCCESS! -- 27 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS
---
2020-03-25 12:53:15 -07:00
Ramkumar Chinchani
dfe023225f pkg/api: use a rwlock when accessing storage
The original patch used a mutex, however, the workload patterns are
likely to be read-heavy, so use a rwlock instead.
2020-03-20 12:23:36 -07:00
Ramkumar Chinchani
2fd87b6a86 pkg/api: use a rwlock when accessing storage
The original patch used a mutex, however, the workload patterns are
likely to be read-heavy, so use a rwlock instead.
2020-03-20 10:58:21 -07:00
Ramkumar Chinchani
404c83fbc1 bazel: fix bazel build failures 2020-03-20 10:58:21 -07:00