0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2024-12-16 21:56:37 -05:00
Commit graph

142 commits

Author SHA1 Message Date
Andrei Aaron
7ce5a74598
feat: use the "zot" namespace for the authentication url (#1947)
Some other minor fixes for swaggo comments (indentation and a bad description)

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-10-20 15:30:56 +03:00
Alexei Dodon
a345ba0823
fix: metrics should be protected behind authZ (#1895)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-10-20 10:33:26 +03:00
Alexei Dodon
75085dcff5
fix: errors returned by zot should match the dist-spec errors (#1868)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-09-29 13:45:19 -07:00
Andrei Aaron
7c78f80a96
feat(cve): implement CVE scanning as background tasks (#1833)
1. Move existing CVE DB download generator/task login under the cve package
2. Add a new CVE scanner task generator and task type to run in the background, as well as tests for it
3. Move the CVE cache in its own package
4. Add a CVE scanner methods to check if an entry is present in the cache, and to retreive the results
5. Modify the FilterTags MetaDB method to not exit on first error
This is needed in order to pass all tags to the generator,
instead of the generator stopping at the first set of invalid data
6. Integrate the new scanning task generator with the existing zot code.
7. Fix an issue where the CVE scan results for multiarch images was not cached
8. Rewrite some of the older CVE tests to use the new image-utils test package
9. Use the CVE scanner as attribute of the controller instead of CveInfo.
Remove functionality of CVE DB update from CveInfo, it is now responsible,
as the name states, only for providing CVE information.
10. The logic to get maximum severity and cve count for image sumaries now uses only the scanner cache.
11. Removed the GetCVESummaryForImage method from CveInfo as it was only used in tests

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-09-22 11:49:17 -07:00
Andrei Aaron
a11fe2d195
feat(pprof): add profiling route handler to debug runtime (#1818)
(cherry picked from commit 56ddb70f624e7070ad0d3531d498675f9f82c664)

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
2023-09-18 14:05:41 -07:00
Alexei Dodon
14206dd6f3
refactor: Review metrics endpoints (#1770)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-09-15 14:49:34 +03:00
Alexei Dodon
48bf7f69f8
refactor: Reduce zb binary size (#1783)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-09-13 10:28:14 +03:00
Andrei Aaron
24e37eb68b
fix(api): Fix 'last' query param for <repo>/tags/list to work without param 'n' (#1777)
Also fix additional issues:
- sorting of tags on calls without pagination parameters ('n' or 'last')
- if 'n' is 0 we should return an empty list and not error

Added tests accordingly

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-09-08 19:46:17 +03:00
Andreea Lupu
5a3fac40db
feat: upload cosign public key and notation certificates to cloud (#1744)
- using secrets manager for storing public keys and certificates
- adding a default truststore for notation verification and upload all certificates to this default truststore
- removig `truststoreName` query param from notation api for uploading certificates


(cherry picked from commit eafcc1a213)

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-09-08 10:03:58 +03:00
peusebiu
59dc4c3229
feat(scheduler): pass the shutdown/reload ctx to running tasks (#1671)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-05 09:48:56 -07:00
peusebiu
c6b822f3dd
refactor(authz): use a struct for user access control info operations (#1682)
fix(authz): fix isAdmin not using groups to determine if a user is admin.
fix(authz): return 401 instead of 403

403 is correct as per HTTP spec
However authz is not part of dist-spec and clients know only about 401
So this is a compromise.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-01 11:13:53 -07:00
peusebiu
b80deb9927
refactor(storage): refactor storage into a single ImageStore (#1656)
unified both local and s3 ImageStore logic into a single ImageStore
added a new driver interface for common file/dirs manipulations
to be implemented by different storage types

refactor(gc): drop umoci dependency, implemented internal gc

added retentionDelay config option that specifies
the garbage collect delay for images without tags

this will also clean manifests which are part of an index image
(multiarch) that no longer exist.

fix(dedupe): skip blobs under .sync/ directory

if startup dedupe is running while also syncing is running
ignore blobs under sync's temporary storage

fix(storage): do not allow image indexes modifications

when deleting a manifest verify that it is not part of a multiarch image
and throw a MethodNotAllowed error to the client if it is.
we don't want to modify multiarch images

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-01 10:54:39 -07:00
LaurentiuNiculae
112fbec5b6
refactor(cli): added equivalent subcommands for each flag combination under every command (#1674)
- image command is now deprecated in favor of 'images'
- cve command is now deprecated in favor of 'cves'

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-08-30 20:12:24 +03:00
peusebiu
6926bddd3a
feat(apikey): added route to list user api keys (#1708)
adding api key expiration date

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-08-29 09:38:38 -07:00
Alexei Dodon
247f6dcd3f
feat: propagate detailed error msgs to client (OCI dist-spec format) (#1681)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-08-23 20:59:52 +03:00
Alexei Dodon
ca2904762a
fix(ci/cd): detect uncommited swagger docs (#1724)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-08-23 19:21:43 +03:00
Alexei Dodon
e510df7c22
fix: swagger Makefile target broken on darwin (#1701)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-08-15 13:07:56 -07:00
Andrei Aaron
77149aa85c
refactor(extensions)!: refactor the extensions URLs and errors (#1636)
BREAKING CHANGE: The functionality provided by the mgmt endpoint has beed redesigned - see details below
BREAKING CHANGE: The API keys endpoint has been moved -  see details below
BREAKING CHANGE: The mgmt extension config has been removed - endpoint is now enabled by having both the search and the ui extensions enabled
BREAKING CHANGE: The API keys configuration has been moved from extensions to http>auth>apikey

mgmt and imagetrust extensions:
- separate the _zot/ext/mgmt into 3 separate endpoints: _zot/ext/auth, _zot/ext/notation, _zot/ext/cosign
- signature verification logic is in a separate `imagetrust` extension
- better hanling or errors in case of signature uploads: logging and error codes (more 400 and less 500 errors)
- add authz on signature uploads (and add a new middleware in common for this purpose)
- remove the mgmt extension configuration - it is now enabled if the UI and the search extensions are enabled

userprefs estension:
- userprefs are enabled if both search and ui extensions are enabled (as opposed to just search)

apikey extension is removed and logic moved into the api folder
- Move apikeys code out of pkg/extensions and into pkg/api
- Remove apikey configuration options from the extensions configuration and move it inside the http auth section
- remove the build label apikeys

other changes:
- move most of the logic adding handlers to the extensions endpoints out of routes.go and into the extensions files.
- add warnings in case the users are still using configurations with the obsolete settings for mgmt and api keys
- add a new function in the extension package which could be a single point of starting backgroud tasks for all extensions
- more clear methods for verifying specific extensions are enabled
- fix http methods paired with the UI handlers
- rebuild swagger docs

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-08-02 21:58:34 +03:00
Ramkumar Chinchani
de24bd937f
fix(conformance): OCI-Filters-Applied should return a literal (#1640)
* fix(conformance): OCI-Filters-Applied should return a literal

https://github.com/opencontainers/distribution-spec/issues/448

Should only indicate what filter-type was applied and not what exact
values it was filtered on.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

* ci(conformance): point to temporary repo

Revert this commit once upstream changes have been merged

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

---------

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-07-22 16:25:02 -07:00
peusebiu
86a83ca6e3
fix(authn): fix several issues with authn, closes #1632 (#1633)
- apply Access-Control-Allow-Credentials only if authn is enabled
- enable Logout route for basic auth
- fixed Logout godoc
- fix Access-Control-Allow-Methods on Logout route
- added allowOrigin option in config example

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-19 09:27:04 -07:00
LaurentiuNiculae
28de980319
feat(refator): refactoring repodb into meta (#1626)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-07-18 10:27:26 -07:00
peusebiu
cda6916b45
fix: don't allow blobs to be deleted if in use (#1559)
dist-spec APIs independently allow deletion of blobs and manifests.
Doing the former when in use by an image manifest or index is simply
error-prone. So disallow it.

Fixes issue #1509

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-07-10 12:24:45 +03:00
peusebiu
17d1338af1
feat: integrate openID auth logic and user profile management (#1381)
This change introduces OpenID authn by using providers such as Github,
Gitlab, Google and Dex.
User sessions are now used for web clients to identify
and persist an authenticated users session, thus not requiring every request to
use credentials.
Another change is apikey feature, users can create/revoke their api keys and use them
to authenticate when using cli clients such as skopeo.

eg:
login:
/auth/login?provider=github
/auth/login?provider=gitlab
and so on

logout:
/auth/logout

redirectURL:
/auth/callback/github
/auth/callback/gitlab
and so on

If network policy doesn't allow inbound connections, this callback wont work!

for more info read documentation added in this commit.

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
2023-07-07 09:27:10 -07:00
alexstan12
62889c3cb1
refactor: split AuthZ mdw in 2 different parts, each for a specific purpose (#1542)
- AuthzHandler has now been split in BaseAuthzHandler and DistSpecAuthzHandler
The former populates context with user specific data needed in most handlers, while
the latter executes access logic specific to distribution-spec handlers.

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
2023-07-05 09:37:52 -07:00
peusebiu
7881ce32b2
fix(extensions): setup UI extension as last one (#1572)
because UI routes will setup a http.FileServer on '/'
any router setup after UI will be ignored at runtime
becuase gorrilla will route it to http.Fileserver instead.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-03 17:13:15 +03:00
alexstan12
ea7dbf9e5c
refactor: move helper functions under common, in usage specific named files (#1540)
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
2023-06-22 14:29:45 +03:00
peusebiu
fc6d6356fb
feat(sync): sync references(signatures/artifacts) recursively (#1500)
sync now also pulls chained artifacts recursively
eg:
 image->sbom->sbom signature
 image->artifact->artifact

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-06-16 10:27:33 -07:00
Lisca Ana-Roberta
622dde9193
fix: referrers now appears in swagger generated docs (#1488)
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2023-06-12 10:32:11 -07:00
peusebiu
612a12e5a8
refactor(sync): use task scheduler (#1301)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-05-31 10:26:23 -07:00
LaurentiuNiculae
a3f355c278
refactor(storage): refactoring storage (#1459)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-26 11:08:19 -07:00
LaurentiuNiculae
c0170b0811
feat(routes): move the cors handler from /v2 to only where it's needed (#1457)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-19 21:22:51 -07:00
LaurentiuNiculae
7bf40e7308
fix(sync): fixed way of updating repodb when syncing a signature (#1439)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-15 12:02:23 +03:00
LaurentiuNiculae
7d7bc9d5e4
feat(api): added oci-subject header when pushing an image with subject field (#1415)
- as requested by the latest version of the oci distribution spec

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-12 09:32:01 -07:00
LaurentiuNiculae
ea79be64da
refactor(artifact): remove oci artifact support (#1359)
* refactor(artifact): remove oci artifact support
- add header to referrers call to indicated applied artifact type filters

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(gc): simplify gc logic to increase coverage

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

---------

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-10 10:15:33 -07:00
LaurentiuNiculae
3be690c2ac
feat(userpreferences): update allowed methods header for user preferences routes (#1430)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-10 10:09:53 -07:00
Nicol
c169698c95
feat: remove usage of zerolog.Logger.Msgf() from zot code (#1382)
Signed-off-by: Nicol Draghici <idraghic@cisco.com>
2023-04-27 19:44:22 -07:00
LaurentiuNiculae
9cc990d7ca
feat(repodb): add user related information to repodb (#1317)
Initial code was contributed by Bogdan BIVOLARU <104334+bogdanbiv@users.noreply.github.com>
Moved implementation from a separate db to repodb by Andrei Aaron <aaaron@luxoft.com>

Not done yet:
- run/test dynamodb implementation, only boltdb was tested
- add additional coverage for existing functionality
- add web-based APIs to toggle the stars/bookmarks on/off

Initially graphql mutation was discussed for the missing API but
we decided REST endpoints would be better suited for configuration



feat(userdb): complete functionality for userdb integration

- dynamodb rollback changes to user starred repos in case increasing the total star count fails
- dynamodb increment/decrement repostars in repometa when user stars/unstars a repo
- dynamodb check anonymous user permissions are working as intendend
- common test handle anonymous users
- RepoMeta2RepoSummary set IsStarred and IsBookmarked



feat(userdb): rest api calls for toggling stars/bookmarks on/off



test(userdb): blackbox tests



test(userdb): move preferences tests in a different file with specific build tags



feat(repodb): add is-starred and is-bookmarked fields to repo-meta

- removed duplicated logic for determining if a repo is starred/bookmarked

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
2023-04-24 11:13:15 -07:00
peusebiu
3dd3c46ee3
test: stop task scheduler between test runs (#1311)
sync: remove sync WaitGroup, it's stopped with context
sync: onDemand will always try to sync newest image when a tag is used
if a digest is used then onDemand will serve local image
test(sync): fix flaky coverage in sync package
closes #1294

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-03-29 11:37:58 -07:00
LaurentiuNiculae
af819e7b76
refactor(repodb): moving common utilities under pkg/meta (#1292)
* refactor(repodb): moving common utilities under pkg/meta

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* refactor(repodb): moved update, version components under pkg/meta

- updated wrapper initialization to recieve a log object in constructor

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* refactor(repodb): moved repodb initialization from controller to pkg/meta/repodb

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

---------

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-03-28 10:20:09 -07:00
peusebiu
17a554b504
feat(routes): better error message in case of missing annotations (#1150)
putting this info into error detail would be ideal, but skopeo
doesn't print them, so overwrite the error message.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-03-16 20:09:30 -07:00
peusebiu
f04e66a5e2
feat(mgmt): added mgmt extension which returns current zot configuration (#1198)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-03-09 10:43:26 -08:00
LaurentiuNiculae
4c156234cb
feat(repodb): sync-repodb WIP (#1241)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-03-09 10:41:48 -08:00
Lisca Ana-Roberta
336526065f
feat(groups)!: added "groups" mechanism for authZ (#1123)
BREAKING CHANGE: repository paths are now specified under a new config key called "repositories" under "accessControl" section in order to handle "groups" feature. Previously the repository paths were specified directly under "accessControl".

This PR adds the ability to create groups of users which can be used for authZ policies, instead of just users.

{
"http": {
   "accessControl": {
       "groups": {

Just like the users, groups can be part of repository policies/default policies/admin policies. The 'groups' field in accessControl can be missing if there are no groups. The permissions priority is user>group>default>admin policy, verified in this order (in authz.go), and permissions are cumulative. It works with LDAP too, and the group attribute name is configurable. The DN of the group is used as the group name and the functionality is the same. All groups for the given user are added to the context in authn.go. Repository paths are now specified under a new keyword called "repositories" under "accessControl" section in order to handle "groups" feature.

Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
2023-03-08 11:47:15 -08:00
Andreea Lupu
ee95ab0ffc
fix: call notation-go libs instead of using notation binary (#1104)
fix: add loading notation path

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
Co-authored-by: Roxana Nemulescu <roxana.nemulescu@gmail.com>
2023-02-13 10:43:52 -08:00
Amit Marcus
2377d62344
feat(sync): onDemand check for updated manifest (#1190)
Signed-off-by: Amit Marcus <marxus@gmail.com>
2023-02-13 10:33:48 -08:00
Andrei Aaron
c0aaca8ed1
feat(ui): package zui within zot binary (#1161)
(cherry picked from commit d557da0baba819b7cd7e6b5941528776e125ac6d)

build(ui): fix stacker builds


(cherry picked from commit ba25daf02b4a9bc7ee1cb6f84b7a6b096ca7d61f)

build(ui): various fixes

- Fix metrics endpoint
- Fix unit tests unit tests
- Make the ui build optional in the makefile
before the linter lint runs in the golangci-lint workflow
- Do not attempt to include UI routes if search is enabled
- Fix authorization for search endpoint



fix: use zot tag in ui make target


(cherry picked from commit 2a6882fa23f06b2d68c6c299773a6ff50bf90e78)

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-02-10 14:52:54 -08:00
Andrei Aaron
d12836e69c
refactor(cve): improve CVE test time by mocking trivy (#1184)
- refactor(cve): remove the global of type cveinfo.CveInfo from the extensions package
  Replace it with an attribute on controller level
- refactor(controller): extract initialization logic from controller.Run()
- test(cve): mock cve scanner in cli tests

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-02-09 21:04:52 -08:00
Lisca Ana-Roberta
976ccfcf0d
fix: removed references to old dist-spec (#1128)
Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
2023-01-31 09:35:33 -08:00
Ramkumar Chinchani
e2c7a3c5ba
fix(referrers): fix some conformance issues (#1134)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-01-26 10:13:12 -08:00
Andrei Aaron
fac1d1d05d
chore(trivy): update trivy version and enforce OCI compliant repo names in local image storage (#1068)
1. chore(trivy): update trivy library version

The trivy team switched github.com/urfave/cli for viper so
there are some other code changes as well.

Since we don't use github.com/urfave/cli directly in our software
we needed to add a tools.go in order for "go mod tidy" to not delete it.
See this pattern explained in:
- https://github.com/99designs/gqlgen#quick-start
- https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module
- https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md#walk-through

The jobs using "go get -u" have been updated to use "go install", since go get
modifies the go.mod by upgrading some of the packages, but downgrading trivy to an older
version with broken dependencies

2. fix(storage) Update local storage to ignore folder names not compliant with dist spec
Also updated trivy to download the DB and cache results under the rootDir/_trivy folder

3. fix(s3): one of the s3 tests was missing the skipIt call
This caused a failure when running locally without s3 being available

4. make sure the offline scanning is enabled, and zot only downloads the trivy DB
on the regular schedule, and doesn't download the DB on every image scan

ci: increase build and test timeout as tests are reaching the limit more often

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-01-18 08:24:44 -08:00