* feat(cluster): initial commit for scale-out cluster
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* feat(cluster): support shared storage scale out
This change introduces support for shared storage backed
zot cluster scale out.
New feature
Multiple stateless zot instances can run using the same shared
storage backend where each instance looks at a specific set
of repositories based on a siphash of the repository name to improve
scale as the load is distributed across multiple instances.
For a given config, there will only be one instance that can perform
dist-spec read/write on a given repository.
What's changed?
- introduced a transparent request proxy for dist-spec endpoints based on
siphash of repository name.
- new config for scale out cluster that specifies list of
cluster members.
Signed-off-by: Vishwas Rajashekar <vrajashe@cisco.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Vishwas Rajashekar <vrajashe@cisco.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
* fix(scheduler): data race when pushing new tasks
the problem here is that scheduler can be closed in two ways:
- canceling the context given as argument to scheduler.RunScheduler()
- running scheduler.Shutdown()
because of this shutdown can trigger a data race between calling scheduler.inShutdown()
and actually pushing tasks into the pool workers
solved that by keeping a quit channel and listening on both quit channel and ctx.Done()
and closing the worker chan and scheduler afterwards.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
* refactor(scheduler): refactor into a single shutdown
before this we could stop scheduler either by closing the context
provided to RunScheduler(ctx) or by running Shutdown().
simplify things by getting rid of the external context in RunScheduler().
keep an internal context in the scheduler itself and pass it down to all tasks.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
---------
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
* feat(sync): local tmp store
Signed-off-by: a <a@tuxpa.in>
* fix(sync): various fixes for s3+remote storage feature
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
---------
Signed-off-by: a <a@tuxpa.in>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: a <a@tuxpa.in>
ci(notation): update to latest notation version
fix(sync): add layers info when syncing signatures
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
(cherry picked from commit 56ddb70f624e7070ad0d3531d498675f9f82c664)
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
New examples of running tests:
1. To run a specific bats file (with and without verbose output):
make run-blackbox-tests BATS_TEST_FILE_PATH=test/blackbox/delete_images.bats
make run-blackbox-tests BATS_TEST_FILE_PATH=test/blackbox/delete_images.bats BATS_VERBOSITY=2
2. To run the CI tests (with and without verbose output)
make run-blackbox-ci
make run-blackbox-ci BATS_VERBOSITY=2
BATS_TEST_FILE_PATH is used to pass on the test file to run using `run-blackbox-tests`
BATS_VERBOSITY controls the verbosity of the bats framework output, if unspecified the output only
contains test results and failure message in case of failures.
If BATS_VERBOSITY is 1, then also show commands as they are executed.
If BATS_VERBOSITY is 2, on top of the above it also shows output of passed tests.
Other changes in this PR:
- Update some of the tests to show logs after the run ends.
- Run the linters before the tests, as it saves time on failures when running in GH
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
fix(authz): fix isAdmin not using groups to determine if a user is admin.
fix(authz): return 401 instead of 403
403 is correct as per HTTP spec
However authz is not part of dist-spec and clients know only about 401
So this is a compromise.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
unified both local and s3 ImageStore logic into a single ImageStore
added a new driver interface for common file/dirs manipulations
to be implemented by different storage types
refactor(gc): drop umoci dependency, implemented internal gc
added retentionDelay config option that specifies
the garbage collect delay for images without tags
this will also clean manifests which are part of an index image
(multiarch) that no longer exist.
fix(dedupe): skip blobs under .sync/ directory
if startup dedupe is running while also syncing is running
ignore blobs under sync's temporary storage
fix(storage): do not allow image indexes modifications
when deleting a manifest verify that it is not part of a multiarch image
and throw a MethodNotAllowed error to the client if it is.
we don't want to modify multiarch images
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
