0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2024-12-16 21:56:37 -05:00
Commit graph

256 commits

Author SHA1 Message Date
peusebiu
59dc4c3229
feat(scheduler): pass the shutdown/reload ctx to running tasks (#1671)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-05 09:48:56 -07:00
peusebiu
c6b822f3dd
refactor(authz): use a struct for user access control info operations (#1682)
fix(authz): fix isAdmin not using groups to determine if a user is admin.
fix(authz): return 401 instead of 403

403 is correct as per HTTP spec
However authz is not part of dist-spec and clients know only about 401
So this is a compromise.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-01 11:13:53 -07:00
peusebiu
b80deb9927
refactor(storage): refactor storage into a single ImageStore (#1656)
unified both local and s3 ImageStore logic into a single ImageStore
added a new driver interface for common file/dirs manipulations
to be implemented by different storage types

refactor(gc): drop umoci dependency, implemented internal gc

added retentionDelay config option that specifies
the garbage collect delay for images without tags

this will also clean manifests which are part of an index image
(multiarch) that no longer exist.

fix(dedupe): skip blobs under .sync/ directory

if startup dedupe is running while also syncing is running
ignore blobs under sync's temporary storage

fix(storage): do not allow image indexes modifications

when deleting a manifest verify that it is not part of a multiarch image
and throw a MethodNotAllowed error to the client if it is.
we don't want to modify multiarch images

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-01 10:54:39 -07:00
LaurentiuNiculae
112fbec5b6
refactor(cli): added equivalent subcommands for each flag combination under every command (#1674)
- image command is now deprecated in favor of 'images'
- cve command is now deprecated in favor of 'cves'

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-08-30 20:12:24 +03:00
peusebiu
6926bddd3a
feat(apikey): added route to list user api keys (#1708)
adding api key expiration date

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-08-29 09:38:38 -07:00
Andrei Aaron
28858f695f
feat(mgmt): mgmt extention no longer depends on UI being enabled (#1728)
It is now enabled based only on search configuration

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-08-26 13:32:41 -07:00
Damien Degois
289acfabbd
feat(authn): add generic oidc and allow customizable name (#1691)
Rebased and squashed

Signed-off-by: Damien Degois <damien@degois.info>
2023-08-24 12:33:35 +03:00
Alexei Dodon
247f6dcd3f
feat: propagate detailed error msgs to client (OCI dist-spec format) (#1681)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-08-23 20:59:52 +03:00
Alexei Dodon
ca2904762a
fix(ci/cd): detect uncommited swagger docs (#1724)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-08-23 19:21:43 +03:00
LaurentiuNiculae
0731fd3828
test(refactor): refactor tests that use zot-test to use smaller images (#1690)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-08-18 11:46:11 +03:00
Alexei Dodon
e510df7c22
fix: swagger Makefile target broken on darwin (#1701)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-08-15 13:07:56 -07:00
peusebiu
4d125d55ed
fix(authn): handle the case where zot with openid runs behind a proxy (#1675)
added a new config option under 'http' called externalURL which is used
by openid/oauth2 clients to redirect back to zot

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-08-09 09:11:21 -07:00
Andreea Lupu
76277f5ebd
fix: remove inline GC and schedule a background task instead (#1610)
* fix: remove inline GC and set a default value of gc interval

- remove inline GC
- add a default value of GC interval
- run the GC periodically by default with the default value if no interval provided
- generate GC tasks with a random delay(0-30s) between
- add IsReady() method to scheduler.TaskGenerator interface

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>

* ci: add test for gc with short interval

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>

---------

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-08-07 12:55:19 -07:00
Andrei Aaron
77149aa85c
refactor(extensions)!: refactor the extensions URLs and errors (#1636)
BREAKING CHANGE: The functionality provided by the mgmt endpoint has beed redesigned - see details below
BREAKING CHANGE: The API keys endpoint has been moved -  see details below
BREAKING CHANGE: The mgmt extension config has been removed - endpoint is now enabled by having both the search and the ui extensions enabled
BREAKING CHANGE: The API keys configuration has been moved from extensions to http>auth>apikey

mgmt and imagetrust extensions:
- separate the _zot/ext/mgmt into 3 separate endpoints: _zot/ext/auth, _zot/ext/notation, _zot/ext/cosign
- signature verification logic is in a separate `imagetrust` extension
- better hanling or errors in case of signature uploads: logging and error codes (more 400 and less 500 errors)
- add authz on signature uploads (and add a new middleware in common for this purpose)
- remove the mgmt extension configuration - it is now enabled if the UI and the search extensions are enabled

userprefs estension:
- userprefs are enabled if both search and ui extensions are enabled (as opposed to just search)

apikey extension is removed and logic moved into the api folder
- Move apikeys code out of pkg/extensions and into pkg/api
- Remove apikey configuration options from the extensions configuration and move it inside the http auth section
- remove the build label apikeys

other changes:
- move most of the logic adding handlers to the extensions endpoints out of routes.go and into the extensions files.
- add warnings in case the users are still using configurations with the obsolete settings for mgmt and api keys
- add a new function in the extension package which could be a single point of starting backgroud tasks for all extensions
- more clear methods for verifying specific extensions are enabled
- fix http methods paired with the UI handlers
- rebuild swagger docs

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-08-02 21:58:34 +03:00
Doug Rabson
20391a21c0
fix: freebsd build support (#1659)
- Avoid hard-coding the path for bash - this typically is installed as
  /usr/local/bin/bash

- Allow for the fact that FreeBSD's rlimit uses signed integers

Almost all of the test suite passes with three test failures that I
haven't tried to debug yet.

Signed-off-by: Doug Rabson <dfr@rabson.org>
2023-07-29 10:41:25 -07:00
Andrei Aaron
cb1b7ae9b8
fix(auth): fix anonymous auth for ui (#1662)
The ui sends the header X-ZOT-API-CLIENT=zot-ui regardless of session authentication status.
In case of new sessions zot would reject the unauthenticated call on /v2 (which is used to determine
if anonymous access is allowed by the server when the header was set) expecting all users sending
this header to be already authenticated.

Since the ui received 401 from the server, it would not show the option for anonymous login.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-07-29 09:18:33 -07:00
LaurentiuNiculae
a5c92bcac3
test(refactor): change upload image function signature (#1655)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-07-28 17:53:46 +03:00
peusebiu
635d71853e
fix(authn): session authn is skipped when anonymous policy is configured (#1647)
closes: #1642

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-27 09:55:25 -07:00
LaurentiuNiculae
abba6aa3cf
feat(test): added image builder for use in tests (#1583)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-07-26 13:08:04 +03:00
Ramkumar Chinchani
de24bd937f
fix(conformance): OCI-Filters-Applied should return a literal (#1640)
* fix(conformance): OCI-Filters-Applied should return a literal

https://github.com/opencontainers/distribution-spec/issues/448

Should only indicate what filter-type was applied and not what exact
values it was filtered on.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

* ci(conformance): point to temporary repo

Revert this commit once upstream changes have been merged

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

---------

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-07-22 16:25:02 -07:00
peusebiu
86a83ca6e3
fix(authn): fix several issues with authn, closes #1632 (#1633)
- apply Access-Control-Allow-Credentials only if authn is enabled
- enable Logout route for basic auth
- fixed Logout godoc
- fix Access-Control-Allow-Methods on Logout route
- added allowOrigin option in config example

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-19 09:27:04 -07:00
LaurentiuNiculae
28de980319
feat(refator): refactoring repodb into meta (#1626)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-07-18 10:27:26 -07:00
peusebiu
d3f27b4ba6
feat: add additional manifest validations (#1609)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-13 09:31:39 -07:00
peusebiu
94ba538e69
fix(test): consolidate api tests build tags (#1602)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-10 16:11:23 +03:00
peusebiu
cda6916b45
fix: don't allow blobs to be deleted if in use (#1559)
dist-spec APIs independently allow deletion of blobs and manifests.
Doing the former when in use by an image manifest or index is simply
error-prone. So disallow it.

Fixes issue #1509

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-07-10 12:24:45 +03:00
peusebiu
17d1338af1
feat: integrate openID auth logic and user profile management (#1381)
This change introduces OpenID authn by using providers such as Github,
Gitlab, Google and Dex.
User sessions are now used for web clients to identify
and persist an authenticated users session, thus not requiring every request to
use credentials.
Another change is apikey feature, users can create/revoke their api keys and use them
to authenticate when using cli clients such as skopeo.

eg:
login:
/auth/login?provider=github
/auth/login?provider=gitlab
and so on

logout:
/auth/logout

redirectURL:
/auth/callback/github
/auth/callback/gitlab
and so on

If network policy doesn't allow inbound connections, this callback wont work!

for more info read documentation added in this commit.

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
2023-07-07 09:27:10 -07:00
peusebiu
5494a1b8d6
fix(storage): do not open/download blobs when validating manifests (#1566)
when pushing manifests, zot will validate blobs (layers + config blob) are
present in repo, currently it opens(in case of filesystem storage) or download(
in case of cloud storage) each blob.

fixed that by adding a new method ImageStore.CheckBlobPresence() on storage
to check blobs presence without checking the cache like ImageStore.CheckBlob() method does.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-06 10:33:36 -07:00
Andreea Lupu
41b05c60dd
feat: upload certificates and public keys for verifying signatures (#1485)
In order to verify signatures, users could upload their certificates and public keys using these routes:
	-> for public keys:
		/v2/_zot/ext/mgmt?resource=signatures&tool=cosign
	-> for certificates:
		/v2/_zot/ext/mgmt?resource=signatures&tool=notation&truststoreType=ca&truststoreName=name
Then the public keys will be stored under $rootdir/_cosign and the certificates will be stored under
$rootdir/_notation/truststore/x509/$truststoreType/$truststoreName.
Also, for notation case, the "truststores" field of $rootir/_notation/trustpolicy.json file will be
updated with a new entry "$truststoreType:$truststoreName".
Also based on the uploaded files, the information about the signatures validity will be updated
periodically.

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-07-06 14:57:59 +03:00
peusebiu
49e4d93f42
fix(test): TestConfigReloader, wait for trivy db download (#1543)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-06 14:17:49 +03:00
alexstan12
62889c3cb1
refactor: split AuthZ mdw in 2 different parts, each for a specific purpose (#1542)
- AuthzHandler has now been split in BaseAuthzHandler and DistSpecAuthzHandler
The former populates context with user specific data needed in most handlers, while
the latter executes access logic specific to distribution-spec handlers.

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
2023-07-05 09:37:52 -07:00
Lisca Ana-Roberta
d4f200c2e1
fix: changing default numWorkers, making it customizable and refactoring scheduler (#1563)
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2023-07-04 11:03:29 +03:00
peusebiu
7881ce32b2
fix(extensions): setup UI extension as last one (#1572)
because UI routes will setup a http.FileServer on '/'
any router setup after UI will be ignored at runtime
becuase gorrilla will route it to http.Fileserver instead.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-03 17:13:15 +03:00
LaurentiuNiculae
620287c7a4
feat(cli): add referrers and search commands to cli (#1497)
* feat(cli): add referrers command to cli

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(cli): add global search command

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(cli): fix comments

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

---------

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-06-22 10:43:01 -07:00
alexstan12
ea7dbf9e5c
refactor: move helper functions under common, in usage specific named files (#1540)
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
2023-06-22 14:29:45 +03:00
peusebiu
d5487d53e3
fix(authz): assign identity to authz context in tls mutual authentication (#1541)
this causes a bug in extensions by not having the identity for the
authenticated user and couldn't apply his permissions, just the default ones.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-06-21 16:06:53 +03:00
peusebiu
fc6d6356fb
feat(sync): sync references(signatures/artifacts) recursively (#1500)
sync now also pulls chained artifacts recursively
eg:
 image->sbom->sbom signature
 image->artifact->artifact

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-06-16 10:27:33 -07:00
Andrei Aaron
f9f9422d13
ci(disk usage): disk related fixes and improvements (#1524)
ci(workflow): show disk usage and free up disk space used by unneeded tooling
ci(tests): routes tests: do not copy large images if they are not used later
ci(trivy): update a test: download trivy.db to a temporary folder

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-06-15 15:07:28 -07:00
Lisca Ana-Roberta
622dde9193
fix: referrers now appears in swagger generated docs (#1488)
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2023-06-12 10:32:11 -07:00
peusebiu
612a12e5a8
refactor(sync): use task scheduler (#1301)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-05-31 10:26:23 -07:00
LaurentiuNiculae
a3f355c278
refactor(storage): refactoring storage (#1459)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-26 11:08:19 -07:00
LaurentiuNiculae
2b8479f7f2
feat(userprefs): update documentation and list extensions endpoint (#1456)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-25 14:46:52 +03:00
LaurentiuNiculae
c0170b0811
feat(routes): move the cors handler from /v2 to only where it's needed (#1457)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-19 21:22:51 -07:00
LaurentiuNiculae
7bf40e7308
fix(sync): fixed way of updating repodb when syncing a signature (#1439)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-15 12:02:23 +03:00
LaurentiuNiculae
7d7bc9d5e4
feat(api): added oci-subject header when pushing an image with subject field (#1415)
- as requested by the latest version of the oci distribution spec

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-12 09:32:01 -07:00
LaurentiuNiculae
ea79be64da
refactor(artifact): remove oci artifact support (#1359)
* refactor(artifact): remove oci artifact support
- add header to referrers call to indicated applied artifact type filters

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(gc): simplify gc logic to increase coverage

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

---------

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-10 10:15:33 -07:00
LaurentiuNiculae
3be690c2ac
feat(userpreferences): update allowed methods header for user preferences routes (#1430)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-10 10:09:53 -07:00
Nicol
c169698c95
feat: remove usage of zerolog.Logger.Msgf() from zot code (#1382)
Signed-off-by: Nicol Draghici <idraghic@cisco.com>
2023-04-27 19:44:22 -07:00
Andreea Lupu
40bf76add5
chore(go.mod): upgrade trivy and cosign (#1387)
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-04-27 09:35:10 -07:00
peusebiu
07bfc8ab95
fix(authz): get username from authn.go request context (#1383)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-04-27 08:13:06 -07:00
LaurentiuNiculae
9cc990d7ca
feat(repodb): add user related information to repodb (#1317)
Initial code was contributed by Bogdan BIVOLARU <104334+bogdanbiv@users.noreply.github.com>
Moved implementation from a separate db to repodb by Andrei Aaron <aaaron@luxoft.com>

Not done yet:
- run/test dynamodb implementation, only boltdb was tested
- add additional coverage for existing functionality
- add web-based APIs to toggle the stars/bookmarks on/off

Initially graphql mutation was discussed for the missing API but
we decided REST endpoints would be better suited for configuration



feat(userdb): complete functionality for userdb integration

- dynamodb rollback changes to user starred repos in case increasing the total star count fails
- dynamodb increment/decrement repostars in repometa when user stars/unstars a repo
- dynamodb check anonymous user permissions are working as intendend
- common test handle anonymous users
- RepoMeta2RepoSummary set IsStarred and IsBookmarked



feat(userdb): rest api calls for toggling stars/bookmarks on/off



test(userdb): blackbox tests



test(userdb): move preferences tests in a different file with specific build tags



feat(repodb): add is-starred and is-bookmarked fields to repo-meta

- removed duplicated logic for determining if a repo is starred/bookmarked

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
2023-04-24 11:13:15 -07:00