Thanks @jdolitsky et al for kicking off these changes at:
https://github.com/oci-playground/zot/commits/main
Thanks @sudo-bmitch for reviewing the patch
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Moved boltdb to a driver implementation for such interface
Added CreateCacheDatabaseDriver in controller
Fixed default directory creation (boltDB will only create the file, not the dir
Added coverage tests
Added example config for boltdb
Re-added caching on subpaths, rewrote CreateCacheDatabaseDriver
Fix tests
Made cacheDriver argument mandatory for NewImageStore, added more validation, added defaults
Moved cache interface to own file, removed useRelPaths from config
Got rid of cache config, refactored
Moved cache to own package and folder
Renamed + removed cache factory to backend, replaced CloudCache to RemoteCache
Moved storage constants back to storage package
moved cache interface and factory to storage package, changed remoteCache defaulting
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
- Digests were represented by different ways
- We needed a uniform way to represent the digests and enforce a format
- also replace usage of github.com/google/go-containerregistry/pkg/v1
with github.com/opencontainers/image-spec/specs-go/v1
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 96b2f29d6d57070a913ce419149cd481c0723815)
(cherry picked from commit 3d41b583daea654c98378ce3dcb78937d71538e8)
Co-authored-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* build: add commit hash to Config at build for proper discovery readme link
* fix: use tag instead of commit hash, add to release build
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Return this data as part of GlobalSearch and RepoListWithNewestImage
query results.
This commit also includes refactoring of the CVE scanning logic in
order to better encapsulate trivy specific logic, remove CVE scanning
logic from the graphql resolver.
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
currently different subpaths can only point to same root directory only
when one or both of the storage config does not enable dedupe
different subpath should be able to point to same root directory and in
that case their storage config should be same i.e GC,Dedupe, GC delay
and GC interval
Signed-off-by: Shivam Mishra <shimish2@cisco.com>
- AccessControlContext now resides in a separate package from where it can be imported,
along with the contextKey that will be used to set and retrieve this context value.
- AccessControlContext has a new field called Username, that will be of use for future
implementations in graphQL resolvers.
- GlobalSearch resolver now uses this context to filter repos available to the logged user.
- moved logic for uploading images in tests so that it can be used in every package
- tests were added for multiple request scenarios, when zot-server requires authz
on specific repos
- added tests with injected errors for extended coverage
- added tests for status code error injection utilities
Closes https://github.com/project-zot/zot/issues/615
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Conformance spec requires responding to PATCH requests with
response header 'Content-Range' and value <range>
Signed-off-by: Bogdan BIVOLARU <104334+bogdanbiv@users.noreply.github.com>
Files were added to be built whether an extension is on or off.
New build tags were added for each extension, while minimal and extended disappeared.
added custom binary naming depending on extensions used and changed references from binary to binary-extended
added automated blackbox tests for sync, search, scrub, metrics
added contributor guidelines
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
reuqest url also contains query parameter due to this in some scenarios
location header is setting up incorrectly, strip query parameter from
request url to correctly setup location header.
Closes#573#575
Signed-off-by: Shivam Mishra <shimish2@cisco.com>
Because s3 doesn't support hard links we store duplicated blobs
as empty files. When the original blob is deleted its content is
moved to the the next duplicated blob and so on.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
PR (linter: upgrade linter version #405) triggered lint job which failed
with many errors generated by various linters. Configurations were added to
golangcilint.yaml and several refactorings were made in order to improve the
results of the linter.
maintidx linter disabled
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
whenever we make a request that contains header apart from CORS allowed header, browser sends a preflight request
and in response accept *Access-Control-Allow-Headers*.
preflight request is in form of OPTIONS method, added new http handler func to set headers
and returns HTTP status ok in case of OPTIONS method.
in case of authorization, request contains authorization header
added authorization header in Access-Control-Allow-Headers list
added AllowOrigin field in HTTPConfig this field value is set to Access-Control-Allow-Origin header and will give zot adminstrator to limit incoming request.
Signed-off-by: Shivam Mishra <shimish2@cisco.com>
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Behavior controlled by configuration (default=off)
It is a trade-off between performance and consistency.
References:
[1] https://github.com/golang/go/issues/20599
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
sync: don't return error on sync signatures, just skip them, closes#375
sync: sync signatures on demand
sync on demand: in case of parallel requests pull image just once, closes#344
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
now it copies to a local cache and then it copies over storage APIs
- accept all images with or without signatures
- disable sync writing to stdout
- added more logs
- fixed switch statement in routes
- fixed enabling sync multiple times for storage subpaths
closes#266
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Periodically poll registries and pull images according to sync's config
Added sync on demand, syncing when clients asks for an image which
zot doesn't have.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Some LDAP servers are not MT-safe in that when searches happen with binds
in flight leads to errors such as:
"comment: No other operations may be performed on the connection while a
bind is outstanding"
Add goroutine-id in logs to help debug MT bugs.
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
With recent docker client-side changes, on 'docker pull' we see:
"Error response from daemon: missing or empty Content-Type header"
Hence, set Content-Type header.
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
previously mount blob will look for blob that is provided in http request and try to hard link that path
but ideally we should look for path from our cache and do the hard link of that particular path.
this commit does the same.
added support to point multiple storage locations in zot by running multiple instance of zot in background.
see examples/config-multiple.json for more info about config.
Closes#181
The idea initially was to use bazel to do our builds, however golang
build system is now good enough and our code base is entirely go.
It is also slowing down our travis ci/cd pipeline.
The storage layer is protected with read-write locks.
However, we may be holding the locks over unnecessarily large critical
sections.
The typical workflow is that a blob is first uploaded via a per-client
private session-id meaning the blob is not publicly visible yet. When
the blob being uploaded is very large, the transfer takes a long time
while holding the lock.
Private session-id based uploads don't really need locks, and hold locks
only when blobs are published after the upload is complete.
This is useful if we want to roll out experimental versions of zot
pointing to some storage shared with another zot instance.
Also, when under storage full conditions, will be useful to turn on this
flag to prevent further writes.
We perform inline garbage collection of orphan blobs. However, the
dist-spec poses a problem because blobs begin their life as orphan blobs
and then a manifest is add which refers to these blobs.
We use umoci's GC() to perform garbage collection and policy support
has been added recently which can control whether a blob can be skipped
for GC.
In this patch, we use a time-based policy to skip blobs.
Go version changed to 1.14.4
Golangci-lint changed to 1.26.0
Bazel version changed to 3.0.0
Bazel rules_go version changed to 0.23.3
Bazel gazelle version changed to v0.21.0
Bazel build tools version changed to 0.25.1
Bazel skylib version changed to 1.0.2
header
containers/image is the dominant client library to interact with
registries.
It detects which authentication to use based on the WWW-Authenticate
header returned when pinging "/v2/" end-point. If we didn't return this
header, then creds are not used for other write-protected end-points.
Hence, the compatibility fix.
Since we want to conform to dist-spec, sometimes the gc and dedupe
optimizations conflict with the conformance tests that are being run.
So allow them to be turned off via configuration params.
Upstream conformance tests are being updated, so we need to align along
with our internal GC and dedupe features.
Add a new example config file which plays nice with conformance tests.
DeleteImageManifest() updated to deal with the case where the same
manifest can be created with multiple tags and deleted with the same
digest - so all entries must be deleted.
DeleteBlob() delete the digest key (bucket) when last reference is
dropped
As the number of repos and layers increases, the greater the probability
that layers are duplicated. We dedupe using hard links when content is
the same. This is intended to be purely a storage layer optimization.
Access control when available is orthogonal this optimization.
Add a durable cache to help speed up layer lookups.
Update README.
Add more unit tests.
Now that we're GCing blobs on delete/update manifest, we should lock the
blob queries so that they don't race with each other.
This is a pretty coarse grained lock, there's probably a better way to do
this.
Signed-off-by: Tycho Andersen <tycho@tycho.ws>
Previously, CheckManifest() was not checking for repo not found
condition and would default to 500 status code.
Add the check now to return 404.
Fixes issue #74
- that errors be returned a certain way using the new NewErrorList()
method and the string enum constants
- allow for full blob upload without a session with repo name and digest
New options added to configuration file to reference a public key used
to validate authorization tokens signed by an auth server with
corresponding private key.
Resolves#24
Signed-off-by: Peter Engelbert <pmengelbert@gmail.com>
dist-spec compliance tests are now becoming a part of dist-spec repo
itself - we want to be compliant
pkg/api/regex.go:
* revert uppercasing in repository names
pkg/api/routes.go:
* ListTags() should support the URL params 'n' and 'last'
for pagination
* s/uuid/session_id/g to use the dist-spec's naming
* Fix off-by-one error in GetBlobUpload()'s http response "Range" header
* DeleteManifest() success status code is 202
* Fix PatchBlobUpload() to account for "streamed" use case
where neither "Content-Length" nor "Content-Range" headers are set
pkg/storage/storage.go:
* Add a "streamed" version of PutBlobChunk() called PutBlobChunkStreamed()
pkg/compliance/v1_0_0/check.go:
* fix unit tests to account for changed response status codes
We are noticing that LDAP connectivity issues and timeouts can affect
authN behavior from client side (which can timeout as well).
Instead, put local authN first so at least we have a reliable authN
method.
But, the caveat is that it is best if the local and LDAP user list
doesn't overlap.
zot ci/cd tests are too stict, so separate and relax them for compliance
tests.
Location header is set in some cases, but some clarification is needed
in URL construction.
Fix some incorrect compliance tests.
"mount blob" was the only feature we didn't handle.
We don't fully support it yet because we don't have access control
support, but at least handle it and return something sane to remain
compliant.
1) github.com/jtblin/go-ldap-client is an abandoned project and not keeping
up with go modules, so pull that code in (single file, BSD license)
2) minor refactor ldap as a result of 1)
3) update go.mod
