added support to point multiple storage locations in zot by running multiple instance of zot in background.
see examples/config-multiple.json for more info about config.
Closes#181
In use cases, when there are large images with shared layers across
repositories, clients may benefit from not re-uploading the same blobs
over and over again.
We ensure this by hard linking when check-blob api is called.
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
The idea initially was to use bazel to do our builds, however golang
build system is now good enough and our code base is entirely go.
It is also slowing down our travis ci/cd pipeline.
The storage layer is protected with read-write locks.
However, we may be holding the locks over unnecessarily large critical
sections.
The typical workflow is that a blob is first uploaded via a per-client
private session-id meaning the blob is not publicly visible yet. When
the blob being uploaded is very large, the transfer takes a long time
while holding the lock.
Private session-id based uploads don't really need locks, and hold locks
only when blobs are published after the upload is complete.
If image vulnerability scan does not support any media type, considering those images as an infected image and now this images will not be shown in fixed images list.
Fixes issue #130
Uses GraphQL API of zot to fetch CVE info
- Get all images affected by a CVE (input: CVEID)
- Get all CVEs of a layer (input: image:tag)
- Get all layers of an image which have resolved a CVE (input: image,
CVEID)
- Get all layers of an image affected by a CVE (input: image, CVEID)
dstRecord :- blob path stored in cache.
dst :- blob path that is trying to be uploaded.
Currently, if the actual blob on disk may have been removed by GC/delete, during syncing the cache dst is being passed to DeleteBlob function and retry section is being continuously called because DeleteBlob function never deletes dst path (doesn't exist in db), dstRecord should be passed into DeleteBlob function because dstRecord is actual blob path stored in db.
If dst and dstRecord path value is same then this issue will not be produced and DeleteBlob method will delete the blob info from cache but if both are different then DeleteBlob method will try to delete dst path which is not in cache.
Note:- boltdb delete method return nil even when value doesn't exist (https://godoc.org/github.com/boltdb/bolt#Bucket.Delete)
This is useful if we want to roll out experimental versions of zot
pointing to some storage shared with another zot instance.
Also, when under storage full conditions, will be useful to turn on this
flag to prevent further writes.
We perform inline garbage collection of orphan blobs. However, the
dist-spec poses a problem because blobs begin their life as orphan blobs
and then a manifest is add which refers to these blobs.
We use umoci's GC() to perform garbage collection and policy support
has been added recently which can control whether a blob can be skipped
for GC.
In this patch, we use a time-based policy to skip blobs.
Extends the existing zot CLI to add commands for listing all images and
their details on a zot server.
Listing all images introduces the need for configurations.
Each configuration has a name and URL at the least. Check 'zot config
-h' for more details.
The user can specify the URL of zot server explicitly while running the
command or configure a URL and pass it directly.
Adding a configuration:
zot config add aci-zot <zot-url>
Run 'zot config --help' for more.
Listing all images:
zot images --url <zot-url>
Pass a config instead of the url:
zot images <config-name>
Filter the list of images by image name:
zot images <config-name> --name <image-name>
Run 'zot images --help' for all details
- Stores configurations in '$HOME/.zot' file
Add CLI README
Go version changed to 1.14.4
Golangci-lint changed to 1.26.0
Bazel version changed to 3.0.0
Bazel rules_go version changed to 0.23.3
Bazel gazelle version changed to v0.21.0
Bazel build tools version changed to 0.25.1
Bazel skylib version changed to 1.0.2
An existing manifest descriptor in index.json can be updated with
different manifest contents for the same/existing tag. We were updating
the digest but not the size field causing GC to report an error.
Add a unit test case to cover this.
Add logs.
In a production use case we found that the actual rootdir can be moved.
Currently, cache entries for dedupe record the full blob path which
doesn't work in the move use case.
Only for dedupe cache entries, record relative blob paths.
header
containers/image is the dominant client library to interact with
registries.
It detects which authentication to use based on the WWW-Authenticate
header returned when pinging "/v2/" end-point. If we didn't return this
header, then creds are not used for other write-protected end-points.
Hence, the compatibility fix.
Since we want to conform to dist-spec, sometimes the gc and dedupe
optimizations conflict with the conformance tests that are being run.
So allow them to be turned off via configuration params.
Upstream conformance tests are being updated, so we need to align along
with our internal GC and dedupe features.
Add a new example config file which plays nice with conformance tests.
DeleteImageManifest() updated to deal with the case where the same
manifest can be created with multiple tags and deleted with the same
digest - so all entries must be deleted.
DeleteBlob() delete the digest key (bucket) when last reference is
dropped
As the number of repos and layers increases, the greater the probability
that layers are duplicated. We dedupe using hard links when content is
the same. This is intended to be purely a storage layer optimization.
Access control when available is orthogonal this optimization.
Add a durable cache to help speed up layer lookups.
Update README.
Add more unit tests.
Now that we're GCing blobs on delete/update manifest, we should lock the
blob queries so that they don't race with each other.
This is a pretty coarse grained lock, there's probably a better way to do
this.
Signed-off-by: Tycho Andersen <tycho@tycho.ws>
Clients today expect the repo to clean up if there are unused blobs, not to
manually delete things they think are unused. Let's do that, and use
umoci's code to do it since it's tested and works.
v2: also run GC on update as well as delete
v3: fix up error return paths needing two args
Signed-off-by: Serge Hallyn <shallyn@cisco.com>
Signed-off-by: Tycho Andersen <tycho@tycho.ws>
Previously, CheckManifest() was not checking for repo not found
condition and would default to 500 status code.
Add the check now to return 404.
Fixes issue #74
- that errors be returned a certain way using the new NewErrorList()
method and the string enum constants
- allow for full blob upload without a session with repo name and digest
Fixes issue #67.
As per dist spec, DELETE of a image manifest can only be done with
digest as <reference> param. Previously, tags were being allowed as
well. This is not conformant to the spec.
New options added to configuration file to reference a public key used
to validate authorization tokens signed by an auth server with
corresponding private key.
Resolves#24
Signed-off-by: Peter Engelbert <pmengelbert@gmail.com>
dist-spec community is now maintaining its own conformance suite hosted
at https://oci.bloodorange.io which over time will be more complete.
This makes zot's compliance cli superfluous. Hence, deprecating the cli.
However, we keep the unit tests itself though.
dist-spec compliance tests are now becoming a part of dist-spec repo
itself - we want to be compliant
pkg/api/regex.go:
* revert uppercasing in repository names
pkg/api/routes.go:
* ListTags() should support the URL params 'n' and 'last'
for pagination
* s/uuid/session_id/g to use the dist-spec's naming
* Fix off-by-one error in GetBlobUpload()'s http response "Range" header
* DeleteManifest() success status code is 202
* Fix PatchBlobUpload() to account for "streamed" use case
where neither "Content-Length" nor "Content-Range" headers are set
pkg/storage/storage.go:
* Add a "streamed" version of PutBlobChunk() called PutBlobChunkStreamed()
pkg/compliance/v1_0_0/check.go:
* fix unit tests to account for changed response status codes
We are noticing that LDAP connectivity issues and timeouts can affect
authN behavior from client side (which can timeout as well).
Instead, put local authN first so at least we have a reliable authN
method.
But, the caveat is that it is best if the local and LDAP user list
doesn't overlap.
zot ci/cd tests are too stict, so separate and relax them for compliance
tests.
Location header is set in some cases, but some clarification is needed
in URL construction.
Fix some incorrect compliance tests.
"mount blob" was the only feature we didn't handle.
We don't fully support it yet because we don't have access control
support, but at least handle it and return something sane to remain
compliant.
This adds a new --json flag to the compliance subcommand, which
will output the compliance test results as minified JSON to stdout.
Also a few other small additions:
- Exit 1 if compliance tests fail
- Use random port for test server using freeport library (added)
Signed-off-by: Josh Dolitsky <393494+jdolitsky@users.noreply.github.com>
1) github.com/jtblin/go-ldap-client is an abandoned project and not keeping
up with go modules, so pull that code in (single file, BSD license)
2) minor refactor ldap as a result of 1)
3) update go.mod
