0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2024-12-16 21:56:37 -05:00
Commit graph

321 commits

Author SHA1 Message Date
Ramkumar Chinchani
cb2af94b0b
feat: add support for docker images (#2714)
* feat: add support for docker images

Issue #724

A new config section under "HTTP" called "Compat" is added which
currently takes a list of possible compatible legacy media-types.

https://github.com/opencontainers/image-spec/blob/main/media-types.md#compatibility-matrix

Only "docker2s2" (Docker Manifest V2 Schema V2) is currently supported.

Garbage collection also needs to be made aware of non-OCI compatible
layer types.
feat: add cve support for non-OCI compatible layer types

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

* 

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

* test: add more docker compat tests

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

* feat: add additional validation checks for non-OCI images

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

* ci: make "full" images docker-compatible

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

---------

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2024-10-31 09:44:04 +02:00
Andrei Aaron
d42ac4cd0d
fix(delete manifest): distinct behaviors for delete by tag vb delete by digest (#2626)
In case of delete by tag only the tag is removed, the manifest itself would continue to be accessible by digest.
In case of delete by digest the manifest would be completely removed (provided it is not used by an index or another reference).

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-10-03 09:06:41 -07:00
Ramkumar Chinchani
2a164cc56b
chore: update go.mod deps (#2609)
Fix outdated pkgs reported in issue #2522

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2024-08-13 11:28:51 -07:00
peusebiu
b461619682
fix(authn): make hashing/encryption keys used to secure cookies (#2536)
fix(authn): configurable hashing/encryption keys used to secure cookies

If they are not configured zot will generate a random hashing key at startup,
invalidating all cookies if zot is restarted. closes: #2526

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2024-08-12 15:11:53 -07:00
Andrei Aaron
2dea22f74a
chore: Update github.com/zitadel/oidc v1 to v3 (#2585)
Also removes dependency on gopkg.in/square/go-jose.v2

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-08-05 09:11:00 -07:00
Jan-Otto Kröpke
f618b1d4ef
ci(deps): upgrade golangci-lint (#2556)
* ci(deps): upgrade golangci-lint

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>

* build(deps): removed disabled linters

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>

* build(deps): go run github.com/daixiang0/gci@latest write .

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build(deps): go run golang.org/x/tools/cmd/goimports@latest -l -w .

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build(deps): go run github.com/bombsimon/wsl/v4/cmd...@latest -strict-append -test=true -fix ./...

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build(deps): go run github.com/catenacyber/perfsprint@latest -fix ./...

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build(deps): replace gomnd by mnd

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build(deps): make gqlgen

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build: Revert "build(deps): go run github.com/daixiang0/gci@latest write ."

This reverts commit 5bf8c42e1f.

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build(deps): go run github.com/daixiang0/gci@latest write -s 'standard' -s default -s 'prefix(zotregistry.dev/zot)' .

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build(deps): make gqlgen

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: check-log issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: gci issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: tests

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

---------

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>
2024-07-29 10:32:51 -07:00
Andrei Aaron
7d87558b7e
fix: various lint issues (#2571)
Originally found by @jkroepke in: https://github.com/project-zot/zot/actions/runs/10056774230/job/27796324933?pr=2556
And https://github.com/project-zot/zot/pull/2556 in general

This commit covers:
- (canonicalheader) capitalization of Docker-Content-Digest header
- (protogetter) the proto getters were not used, they check for nil pointers, we should switch to using them
- (zerologlint) fix the false positive

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-07-25 10:14:22 -07:00
Andrei Aaron
26be383aae
fix(digests): do not mandate sha256 as the only algorithm used for hashing blobs (#2075)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-07-19 09:56:31 -07:00
peusebiu
1c2736d970
fix(storage): handle dedupe disabled in GetAllDedupeReposCandidates() (#2533)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2024-07-09 15:33:11 +03:00
Ramkumar Chinchani
aaee0220e4
Merge pull request from GHSA-55r9-5mx9-qq7r
when a client pushes an image zot's inline dedupe
will try to find the blob path corresponding with the blob digest
that it's currently pushed and if it's found in the cache
then zot will make a symbolic link to that cache entry and report
to the client that the blob already exists on the location.

Before this patch authorization was not applied on this process meaning
that a user could copy blobs without having permissions on the source repo.

Added a rule which says that the client should have read permissions on the source repo
before deduping, otherwise just Stat() the blob and return the corresponding status code.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: Petu Eusebiu <peusebiu@cisco.com>
2024-07-08 11:35:44 -07:00
Vishwas Rajashekar
767f81d4f5
feat(sync): support for periodic repo sync in scale-out cluster (#2424)
This commit includes support for periodic repo sync in a scale-out
cluster.
Before this commit, all cluster members would sync all the repos as
the config is shared.

With this change, in periodic sync, the cluster member checks whether
it manages the repo. If it does not manage the repo, it will skip the
sync.

This commit also includes a unit test to test on-demand sync too, but
there are no logic changes for it as it is implicitly handled by the
proxying logic.

Signed-off-by: Vishwas Rajashekar <vrajashe@cisco.com>
2024-05-31 09:25:34 -07:00
Vishwas R
5ae7a028d9
feat(cluster): Add support for request proxying for scale out (#2385)
* feat(cluster): initial commit for scale-out cluster

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

* feat(cluster): support shared storage scale out

This change introduces support for shared storage backed
zot cluster scale out.

New feature
Multiple stateless zot instances can run using the same shared
storage backend where each instance looks at a specific set
of repositories based on a siphash of the repository name to improve
scale as the load is distributed across multiple instances.
For a given config, there will only be one instance that can perform
dist-spec read/write on a given repository.

What's changed?
- introduced a transparent request proxy for dist-spec endpoints based on
siphash of repository name.
- new config for scale out cluster that specifies list of
cluster members.

Signed-off-by: Vishwas Rajashekar <vrajashe@cisco.com>

---------

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Vishwas Rajashekar <vrajashe@cisco.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
2024-05-20 09:05:21 -07:00
Ramkumar Chinchani
6b3c160176
chore: fix dependabot alerts (#2377)
https://github.com/project-zot/zot/pull/2368
https://github.com/project-zot/zot/pull/2369
https://github.com/project-zot/zot/pull/2370
https://github.com/project-zot/zot/pull/2371
https://github.com/project-zot/zot/pull/2372
https://github.com/project-zot/zot/pull/2373
https://github.com/project-zot/zot/pull/2374
https://github.com/project-zot/zot/pull/2375
https://github.com/project-zot/zot/pull/2376

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2024-04-09 10:40:16 +03:00
Andrei Aaron
864cd00b9e
fix: Allow GET requests on repositories not found in metadb (#2351)
The issue was reported on Slack.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-03-26 18:38:58 +02:00
Ramkumar Chinchani
fdb401273c
fix: ignore metadb errors if tag not found (#2301) 2024-03-13 07:28:08 +02:00
Ramkumar Chinchani
18235ca254
fix(oras)!: remove ORAS artifact references support (#2294)
* fix(oras)!: remove ORAS artifact references support

ORAS artifacts/references predated OCI dist-spec 1.1.0 which now has the
same functionality and likely to see wider adoption.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

* test: update to released official images

So that they are unlikely to be deleted.
*-rc images may be cleaned up over time.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

---------

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2024-03-06 12:16:42 -08:00
Andrei Aaron
4e5db84cb1
chore: update image-spec and dist spec to 1.1.0 (#2255)
BREAKING CHANGE: the dist spec version in the config files needs to be bumped to 1.1.0
in order for the config verification to pass without warnings.

Also fix 1 dependabot alert for helm.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-02-20 13:27:21 +02:00
Andrei Aaron
6c953d6400
test: fix flacky coverage in cookiestore cleanup tests (#2257)
Refactor and add more coverage to test flacky coverage in case sessions
which are already deleted are flagged as expired/for deletion.

See coverage drop in pkg/api/cookiestore.go:
8e68255946/indirect-changes

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-02-19 14:43:30 -08:00
peusebiu
8e68255946
fix(sync): added bearer client for sync (#2222)
fixed ping function taking too much time

closes: #2213 #2212

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2024-02-14 09:18:10 -08:00
Andrei Aaron
60dc8569ec
build(go): switch to go 1.21 (#2049)
Also update to the latest swaggo version, as the previous one did not work with go 1.21

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-02-07 10:54:28 -08:00
Andrei Aaron
6a83dd47c0
fix(scheduler): the session cleanup generator is reset too often (#2220)
This causes the "fair" scheduler to run it too often in the detriment of other generators.
The intention was to run it every 2 hours but the measurement unit for 7200 was not specified.

Add more logs, including showing a generator name, in order to troubleshoot this kind of issues easier in the future.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-02-01 09:15:53 -08:00
Andrei Aaron
ce4924f841
refactor: rename go module from zotregistry.io/zot to zotregistry.dev/zot (#2187)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-01-31 20:34:07 -08:00
LaurentiuNiculae
1785688b7c
feat(ldap): hot reloading ldap credentials on change (#2167)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2024-01-25 09:15:22 -08:00
peusebiu
e9ab520905
fix(bearer): fixed /v2/ route not implementing token spec (#2176)
We use chartmuseum lib for handling bearer requests, which is not
implementing the token spec, mainly it expects "scope" parameter
to be given on every request, even for /v2/ route which doesn't represent
a resource.

Handle this /v2/ route inside our code.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2024-01-22 09:15:27 -08:00
Andrei Aaron
8467a80a50
refactor: update tests to use the newer API for creating test images (#2168)
- update cve tests
- update scrub tests
- update tests for parsing storage and loading into meta DB
- update controller tests

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-01-17 10:20:07 -08:00
peusebiu
ee9bbb0bf2
feat(log): print traceback when panics occur (#2166)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2024-01-16 09:08:14 -08:00
Ramkumar Chinchani
d685adb029
fix: npe if ldap query doesn't return attributes (#2151)
We cannot assume the LDAP server will have group attributes programmed
everytime. So handle it accordingly.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2024-01-12 14:08:35 -08:00
Ramkumar Chinchani
1c756b4db9
fix: enable panic backtraces (#2150)
We expect panics in the server/datapath to be few and far between.
So the backtraces are more valuable now.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2024-01-11 13:56:50 -08:00
peusebiu
a46e10269a
fix(shutdown): fix crash when shutting down before server and task scheduler have started. (#2148)
init shutdown routine after controller.Init()
check for nil values before stopping http server and task scheduler.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2024-01-05 17:50:48 -08:00
peusebiu
dbb1c3519f
feat(ui): let UI delete manifests if current user has permissions to do so (#2132)
- added a new field 'IsDeletable' for graphql ImageSummary struct.
- apply cors on DeleteManifest route

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-12-13 09:06:08 -08:00
peusebiu
7642e5af98
fix(scheduler): fix data race (#2085)
* fix(scheduler): data race when pushing new tasks

the problem here is that scheduler can be closed in two ways:
- canceling the context given as argument to scheduler.RunScheduler()
- running scheduler.Shutdown()

because of this shutdown can trigger a data race between calling scheduler.inShutdown()
and actually pushing tasks into the pool workers

solved that by keeping a quit channel and listening on both quit channel and ctx.Done()
and closing the worker chan and scheduler afterwards.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>

* refactor(scheduler): refactor into a single shutdown

before this we could stop scheduler either by closing the context
provided to RunScheduler(ctx) or by running Shutdown().

simplify things by getting rid of the external context in RunScheduler().
keep an internal context in the scheduler itself and pass it down to all tasks.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>

---------

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-12-11 10:00:34 -08:00
LaurentiuNiculae
79e14027ee
refactor(test): add lint rule for messages starting with the component (#2045)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-12-08 10:05:02 +02:00
Andrei Aaron
f321fa91fe
feat(pagination): make sure the URL to in the link header is inside angle brackets (#2116)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-12-06 09:54:47 -08:00
Alexei Dodon
2e733b3f4f
feat(metrics): add scheduler related metrics (#2076)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-12-04 14:13:50 -08:00
peusebiu
6222dae1f0
feat(scheduler): gracefully shutdown (#1951)
wait for workers to finish before exiting

should fix tests reporting they couldn't remove rootDir because it's being
written by tasks

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-11-24 00:40:10 -08:00
LaurentiuNiculae
92837c2bcb
refactor(log): replace panics with log fatal or log panic functions (#1723)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-11-24 00:38:36 -08:00
LaurentiuNiculae
4fb1e756c4
feat(startup): update logic for metadb update on startup, skip unmodified repos (#2024)
- MetaDB stores the time of the last update of a repo
- During startup we check if the layout has been updated after the last recorded change in the db
- If this is the case, the repo is parsed and updated in the DB otherwise it's skipped

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-11-16 10:39:27 -08:00
LaurentiuNiculae
272eb7cc43
feat(ldap): add option to load ldap from file (#1778)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-11-14 16:21:36 -08:00
peusebiu
7f52f58e3c
fix(routes): fix cors headers for api keys and logout route (#1984)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-11-07 10:11:40 -08:00
peusebiu
9074f8483b
feat(retention): added image retention policies (#1866)
feat(metaDB): add more image statistics info

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-11-01 09:16:18 -07:00
Alexei Dodon
a79d79a03a
fix: more accurate storage metrics after zot restart (#1972)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-11-01 09:09:21 -07:00
LaurentiuNiculae
56ad9e6707
refactor(metadb): improve UX by speeding up metadb serialize/deserialize (#1842)
Use protocol buffers and update the metadb interface to better suit our search needs

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-10-30 13:06:04 -07:00
Alexei Dodon
d2fbd273ba
fix: tests refactoring (#1950)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-10-26 11:20:39 +03:00
Ramkumar Chinchani
1675f30d4a
ci: update golangci-lint version (#1834)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-10-20 17:27:04 +03:00
Andrei Aaron
7ce5a74598
feat: use the "zot" namespace for the authentication url (#1947)
Some other minor fixes for swaggo comments (indentation and a bad description)

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-10-20 15:30:56 +03:00
Alexei Dodon
a345ba0823
fix: metrics should be protected behind authZ (#1895)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-10-20 10:33:26 +03:00
peusebiu
7f6534a52d
fix(sessions): periodically cleanup expired sessions (#1939)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-10-16 20:03:42 -07:00
Andreea Lupu
fc2380b57b
fix: add support for uploaded index when signing using notation (#1882)
ci(notation): update to latest notation version
fix(sync): add layers info when syncing signatures

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-10-12 18:45:20 -07:00
peusebiu
a91c0c5cfe
fix(authn): create sessions only if UI header value is supplied (#1919)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-10-12 16:37:55 +03:00
peusebiu
d1fcab421a
fix(authn): apply fail delay only if credentials/sessions are supplied (#1920)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-10-12 16:32:36 +03:00