unified both local and s3 ImageStore logic into a single ImageStore
added a new driver interface for common file/dirs manipulations
to be implemented by different storage types
refactor(gc): drop umoci dependency, implemented internal gc
added retentionDelay config option that specifies
the garbage collect delay for images without tags
this will also clean manifests which are part of an index image
(multiarch) that no longer exist.
fix(dedupe): skip blobs under .sync/ directory
if startup dedupe is running while also syncing is running
ignore blobs under sync's temporary storage
fix(storage): do not allow image indexes modifications
when deleting a manifest verify that it is not part of a multiarch image
and throw a MethodNotAllowed error to the client if it is.
we don't want to modify multiarch images
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
added a new config option under 'http' called externalURL which is used
by openid/oauth2 clients to redirect back to zot
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
* fix: remove inline GC and set a default value of gc interval
- remove inline GC
- add a default value of GC interval
- run the GC periodically by default with the default value if no interval provided
- generate GC tasks with a random delay(0-30s) between
- add IsReady() method to scheduler.TaskGenerator interface
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
* ci: add test for gc with short interval
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
---------
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
BREAKING CHANGE: The functionality provided by the mgmt endpoint has beed redesigned - see details below
BREAKING CHANGE: The API keys endpoint has been moved - see details below
BREAKING CHANGE: The mgmt extension config has been removed - endpoint is now enabled by having both the search and the ui extensions enabled
BREAKING CHANGE: The API keys configuration has been moved from extensions to http>auth>apikey
mgmt and imagetrust extensions:
- separate the _zot/ext/mgmt into 3 separate endpoints: _zot/ext/auth, _zot/ext/notation, _zot/ext/cosign
- signature verification logic is in a separate `imagetrust` extension
- better hanling or errors in case of signature uploads: logging and error codes (more 400 and less 500 errors)
- add authz on signature uploads (and add a new middleware in common for this purpose)
- remove the mgmt extension configuration - it is now enabled if the UI and the search extensions are enabled
userprefs estension:
- userprefs are enabled if both search and ui extensions are enabled (as opposed to just search)
apikey extension is removed and logic moved into the api folder
- Move apikeys code out of pkg/extensions and into pkg/api
- Remove apikey configuration options from the extensions configuration and move it inside the http auth section
- remove the build label apikeys
other changes:
- move most of the logic adding handlers to the extensions endpoints out of routes.go and into the extensions files.
- add warnings in case the users are still using configurations with the obsolete settings for mgmt and api keys
- add a new function in the extension package which could be a single point of starting backgroud tasks for all extensions
- more clear methods for verifying specific extensions are enabled
- fix http methods paired with the UI handlers
- rebuild swagger docs
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
This change introduces OpenID authn by using providers such as Github,
Gitlab, Google and Dex.
User sessions are now used for web clients to identify
and persist an authenticated users session, thus not requiring every request to
use credentials.
Another change is apikey feature, users can create/revoke their api keys and use them
to authenticate when using cli clients such as skopeo.
eg:
login:
/auth/login?provider=github
/auth/login?provider=gitlab
and so on
logout:
/auth/logout
redirectURL:
/auth/callback/github
/auth/callback/gitlab
and so on
If network policy doesn't allow inbound connections, this callback wont work!
for more info read documentation added in this commit.
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
Initial code was contributed by Bogdan BIVOLARU <104334+bogdanbiv@users.noreply.github.com>
Moved implementation from a separate db to repodb by Andrei Aaron <aaaron@luxoft.com>
Not done yet:
- run/test dynamodb implementation, only boltdb was tested
- add additional coverage for existing functionality
- add web-based APIs to toggle the stars/bookmarks on/off
Initially graphql mutation was discussed for the missing API but
we decided REST endpoints would be better suited for configuration
feat(userdb): complete functionality for userdb integration
- dynamodb rollback changes to user starred repos in case increasing the total star count fails
- dynamodb increment/decrement repostars in repometa when user stars/unstars a repo
- dynamodb check anonymous user permissions are working as intendend
- common test handle anonymous users
- RepoMeta2RepoSummary set IsStarred and IsBookmarked
feat(userdb): rest api calls for toggling stars/bookmarks on/off
test(userdb): blackbox tests
test(userdb): move preferences tests in a different file with specific build tags
feat(repodb): add is-starred and is-bookmarked fields to repo-meta
- removed duplicated logic for determining if a repo is starred/bookmarked
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
BREAKING CHANGE: repository paths are now specified under a new config key called "repositories" under "accessControl" section in order to handle "groups" feature. Previously the repository paths were specified directly under "accessControl".
This PR adds the ability to create groups of users which can be used for authZ policies, instead of just users.
{
"http": {
"accessControl": {
"groups": {
Just like the users, groups can be part of repository policies/default policies/admin policies. The 'groups' field in accessControl can be missing if there are no groups. The permissions priority is user>group>default>admin policy, verified in this order (in authz.go), and permissions are cumulative. It works with LDAP too, and the group attribute name is configurable. The DN of the group is used as the group name and the functionality is the same. All groups for the given user are added to the context in authn.go. Repository paths are now specified under a new keyword called "repositories" under "accessControl" section in order to handle "groups" feature.
Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
(cherry picked from commit d557da0baba819b7cd7e6b5941528776e125ac6d)
build(ui): fix stacker builds
(cherry picked from commit ba25daf02b4a9bc7ee1cb6f84b7a6b096ca7d61f)
build(ui): various fixes
- Fix metrics endpoint
- Fix unit tests unit tests
- Make the ui build optional in the makefile
before the linter lint runs in the golangci-lint workflow
- Do not attempt to include UI routes if search is enabled
- Fix authorization for search endpoint
fix: use zot tag in ui make target
(cherry picked from commit 2a6882fa23f06b2d68c6c299773a6ff50bf90e78)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
* feat(repodb): implement a DB for image specific information using boltdb
(cherry picked from commit e3cb60b856)
Some other fixes/improvements on top (Andrei)
Global search: The last updated attribute on repo level is now computed correctly.
Global search: Fix and enhance tests: validate more fields, and fix CVE verification logic
RepoListWithNewestImage: The vendors and platforms at repo level are no longer containing duplicate entries
CVE: scan OCIUncompressedLayer instead of skiping them (used in tests)
bug(repodb): do no try to increment download counters for signatures
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Add filtering to global search API (Laurentiu)
(cherry picked from commit a87976d635ea876fe8ced532e8adb7c3bb24098f)
Original work by Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Fix pagination bug
- when limit was bigger than the repo count result contained empty results
- now correctly returns only maximum available number of repo results
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Add history to the fields returned from RepoDB
Consolidate fields used in packages
- pkg/extensions/search/common/common_test
- pkg/extensions/search/common/common
Refactor duplicate code in GlobalSearch verification
Add vulnerability scan results to image:tag reply
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Refactor ExpandedRepoInfo to using RepoDB
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit fd7dc85c3a9d028fd8860d3791cad4df769ed005)
Init RepoDB at startup
- sync with storage
- ignore images without a tag
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 359898facd6541b2aa99ee95080f7aabf28c2650)
Update request to get image:tag to use repodb
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Sync RepoDB logging
- added logging for errors
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 2e128f4d01712b34c70b5468285100b0657001bb)
sync-repodb minor error checking fix
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Improve tests for syncing RepoDB with storage
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit b18408c6d64e01312849fc18b929e3a2a7931e9e)
Update scoring rule for repos
- now prioritize matches to the end of the repo name
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 6961346ccf02223132b3b12a2132c80bd1b6b33c)
Upgrade search filters to permit multiple values
- multiple values for os and arch
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 3ffb72c6fc0587ff827a03fe4f76a13b27b876a0)
feature(repodb): add pagination for RepoListWithNewestImage
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
(cherry picked from commit 32c917f2dc65363b0856345289353559a8027aee)
test(fix): fix tests failing since repodb is used for listing all repos
1. One of the tests was verifying disk/oci related erros and is not applicable
2. Another test was actually broken in an older PR, the default store and
the substore were using the same repo names (the substore ones were unprefixed),
which should not be the case, this was causing a single entry to show
in the RepoDB instead of two separate entries for each test image
Root cause in: b61aff62cd (diff-b86e11fa5a3102b336caebec3b30a9d35e26af554dd8658f124dba2404b7d24aR88)
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
chore: move code reponsible for transforming objects to gql_generated types to separate package
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Process input for global search
- Clean input: query, filter strings
- Add validation for global search input
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit f1ca8670fbe4a4a327ea25cf459237dbf23bb78a)
fix: only call cve scanning for data shown to the user
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
GQL omit scanning for CVE if field is not required
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 5479ce45d6cb2abcf5fbccadeaf6f3393c3f6bf1)
Fix filtering logic in RepoDB
- filter parameter was set to false instead of being calculator from the later image
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit a82d2327e34e5da617af0b7ca78a2dba90999f0a)
bug(repodb): Checking signature returns error if signed image is not found
- we considere a signature image orfan when the image it signs is not found
- we need this to ignore such signatures in certain cases
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit d0418505f76467accd8e1ee34fcc2b2a165efae5)
feat(repodb): CVE logic to use repoDB
Also update some method signatures to remove usage of:
github.com/google/go-containerregistry/pkg/v1
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
* feat(repodb): refactor repodb update logic
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(repodb): minor fixes
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): move repodb logic inside meta directory under pkg
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): change factory class for repodb initialization with factory metrod
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): simplify repodb configuration
- repodb now shares config parameters with the cache
- config taken directly from storage config
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(authors): fix authors information to work properly with repodb
Ideally this commit would be squshed in the repodb commit
but as-is it is easier to cherry-pick on other branches
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
* feat(repodb): dynamodb support for repodb
- clean-up repodb code + coverage improvements
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(dynamo): tables used by dynamo are created automatically if they don't exists
- if the table exists nothing happens
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* test(repodb): coverage tests
- minor fix for CVEListForImage to fix the tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): add descriptor with media type
- to represent images and multi-arch images
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): support signatures on repo level
- added to follow the behavior of signing and signature verification tools
that work on a manifest level for each repo
- all images with different tags but the same manifest will be signed at once
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): old repodb version migration support
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): tests for coverage
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): WIP fixing tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): work on patchRepoDB tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(repodb): create dynamo tables only for linux amd
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(ci): fix a typo in ci-cd.yml
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Andrei Aaron <andaaron@cisco.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
Suppose we push two identical manifests (sharing same digest) but with
different tags, then deleting by digest should throw an error otherwise
we end up deleting all image tags (with gc) or dangling references
(without gc)
This behaviour is controlled via Authorization, added a new policy
action named detectManifestsCollision which enables this behaviour
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
Moved boltdb to a driver implementation for such interface
Added CreateCacheDatabaseDriver in controller
Fixed default directory creation (boltDB will only create the file, not the dir
Added coverage tests
Added example config for boltdb
Re-added caching on subpaths, rewrote CreateCacheDatabaseDriver
Fix tests
Made cacheDriver argument mandatory for NewImageStore, added more validation, added defaults
Moved cache interface to own file, removed useRelPaths from config
Got rid of cache config, refactored
Moved cache to own package and folder
Renamed + removed cache factory to backend, replaced CloudCache to RemoteCache
Moved storage constants back to storage package
moved cache interface and factory to storage package, changed remoteCache defaulting
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
- AccessControlContext now resides in a separate package from where it can be imported,
along with the contextKey that will be used to set and retrieve this context value.
- AccessControlContext has a new field called Username, that will be of use for future
implementations in graphQL resolvers.
- GlobalSearch resolver now uses this context to filter repos available to the logged user.
- moved logic for uploading images in tests so that it can be used in every package
- tests were added for multiple request scenarios, when zot-server requires authz
on specific repos
- added tests with injected errors for extended coverage
- added tests for status code error injection utilities
Closes https://github.com/project-zot/zot/issues/615
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Files were added to be built whether an extension is on or off.
New build tags were added for each extension, while minimal and extended disappeared.
added custom binary naming depending on extensions used and changed references from binary to binary-extended
added automated blackbox tests for sync, search, scrub, metrics
added contributor guidelines
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Because s3 doesn't support hard links we store duplicated blobs
as empty files. When the original blob is deleted its content is
moved to the the next duplicated blob and so on.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
