fix(authz): fix isAdmin not using groups to determine if a user is admin.
fix(authz): return 401 instead of 403
403 is correct as per HTTP spec
However authz is not part of dist-spec and clients know only about 401
So this is a compromise.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
unified both local and s3 ImageStore logic into a single ImageStore
added a new driver interface for common file/dirs manipulations
to be implemented by different storage types
refactor(gc): drop umoci dependency, implemented internal gc
added retentionDelay config option that specifies
the garbage collect delay for images without tags
this will also clean manifests which are part of an index image
(multiarch) that no longer exist.
fix(dedupe): skip blobs under .sync/ directory
if startup dedupe is running while also syncing is running
ignore blobs under sync's temporary storage
fix(storage): do not allow image indexes modifications
when deleting a manifest verify that it is not part of a multiarch image
and throw a MethodNotAllowed error to the client if it is.
we don't want to modify multiarch images
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
- image command is now deprecated in favor of 'images'
- cve command is now deprecated in favor of 'cves'
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
- the size of the binary-minimal becomes 32MB
- "signatures" package is renamed into "imagetrust" and moved under extensions
- if the binary is not built using "imagetrust" tag then the signatures verification will
not be performed
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
added a new config option under 'http' called externalURL which is used
by openid/oauth2 clients to redirect back to zot
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
* fix: remove inline GC and set a default value of gc interval
- remove inline GC
- add a default value of GC interval
- run the GC periodically by default with the default value if no interval provided
- generate GC tasks with a random delay(0-30s) between
- add IsReady() method to scheduler.TaskGenerator interface
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
* ci: add test for gc with short interval
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
---------
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
BREAKING CHANGE: The functionality provided by the mgmt endpoint has beed redesigned - see details below
BREAKING CHANGE: The API keys endpoint has been moved - see details below
BREAKING CHANGE: The mgmt extension config has been removed - endpoint is now enabled by having both the search and the ui extensions enabled
BREAKING CHANGE: The API keys configuration has been moved from extensions to http>auth>apikey
mgmt and imagetrust extensions:
- separate the _zot/ext/mgmt into 3 separate endpoints: _zot/ext/auth, _zot/ext/notation, _zot/ext/cosign
- signature verification logic is in a separate `imagetrust` extension
- better hanling or errors in case of signature uploads: logging and error codes (more 400 and less 500 errors)
- add authz on signature uploads (and add a new middleware in common for this purpose)
- remove the mgmt extension configuration - it is now enabled if the UI and the search extensions are enabled
userprefs estension:
- userprefs are enabled if both search and ui extensions are enabled (as opposed to just search)
apikey extension is removed and logic moved into the api folder
- Move apikeys code out of pkg/extensions and into pkg/api
- Remove apikey configuration options from the extensions configuration and move it inside the http auth section
- remove the build label apikeys
other changes:
- move most of the logic adding handlers to the extensions endpoints out of routes.go and into the extensions files.
- add warnings in case the users are still using configurations with the obsolete settings for mgmt and api keys
- add a new function in the extension package which could be a single point of starting backgroud tasks for all extensions
- more clear methods for verifying specific extensions are enabled
- fix http methods paired with the UI handlers
- rebuild swagger docs
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
- Avoid hard-coding the path for bash - this typically is installed as
/usr/local/bin/bash
- Allow for the fact that FreeBSD's rlimit uses signed integers
Almost all of the test suite passes with three test failures that I
haven't tried to debug yet.
Signed-off-by: Doug Rabson <dfr@rabson.org>
The ui sends the header X-ZOT-API-CLIENT=zot-ui regardless of session authentication status.
In case of new sessions zot would reject the unauthenticated call on /v2 (which is used to determine
if anonymous access is allowed by the server when the header was set) expecting all users sending
this header to be already authenticated.
Since the ui received 401 from the server, it would not show the option for anonymous login.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* fix(conformance): OCI-Filters-Applied should return a literal
https://github.com/opencontainers/distribution-spec/issues/448
Should only indicate what filter-type was applied and not what exact
values it was filtered on.
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* ci(conformance): point to temporary repo
Revert this commit once upstream changes have been merged
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Resolves#1597
Fix missing properties in some calls OS/Arch/IsSigned/LastUpdated.
Since the properties were missing from the graphql requests, zli was showing default values.
Update the tests to generate a tets image with the created date different from
the go detault value for time.Time{}. The tests are now checking a non-default
value in the responses client side, in order to avoid missing issues like this in the future.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
https://github.com/project-zot/zot/issues/1591
- I will rename "IMAGE NAME" to "REPOSITORY" in order to make the header easier to parse
- The order of the images cannot be predicted if zot is getting them 1 by 1 using the REST API for manifests, so they cannot be sorted when printed. We could wait on all calls to return but that may take minutes, and printing partial results as they become available is better.
- The order of the images can be predicted when relying on the zot specific search API, but that is not available in all zot servers depending on build options. I added sorting ascending by default. We are planning to implement configurable sorting in a separate PR - see the work under https://github.com/project-zot/zot/pull/1577
- With regards to the column widths/alignments that was discussed before, and the issue is we don't know the values beforehand for the REST API based responses. As mentioned above printing partial results as they become available is better.
- The column widths/alignments are partially fixed in this PR for the search API, but we should properly fix this in - see https://github.com/project-zot/zot/pull/851https://github.com/project-zot/zot/issues/1592
- Fix missing space after help message
https://github.com/project-zot/zot/issues/1598
- Fix table headers showing for json/yaml format
- Fix spacing shown with json format, use 1 row per shown entry in order to be compatible with json lines format: https://jsonlines.org/
- Add document header `---` to every image shown in yaml format to separate the entries
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
sync now ignores self referencing urls, this will help
in clustering mode where we can have the same config
for multiple zots
closes#1335
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
dist-spec APIs independently allow deletion of blobs and manifests.
Doing the former when in use by an image manifest or index is simply
error-prone. So disallow it.
Fixes issue #1509
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
We use locks to protect OCI layouts. However, our critical sections have
too long and cover code instead of data.
Fixes issue #1595
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
This change introduces OpenID authn by using providers such as Github,
Gitlab, Google and Dex.
User sessions are now used for web clients to identify
and persist an authenticated users session, thus not requiring every request to
use credentials.
Another change is apikey feature, users can create/revoke their api keys and use them
to authenticate when using cli clients such as skopeo.
eg:
login:
/auth/login?provider=github
/auth/login?provider=gitlab
and so on
logout:
/auth/logout
redirectURL:
/auth/callback/github
/auth/callback/gitlab
and so on
If network policy doesn't allow inbound connections, this callback wont work!
for more info read documentation added in this commit.
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
when pushing manifests, zot will validate blobs (layers + config blob) are
present in repo, currently it opens(in case of filesystem storage) or download(
in case of cloud storage) each blob.
fixed that by adding a new method ImageStore.CheckBlobPresence() on storage
to check blobs presence without checking the cache like ImageStore.CheckBlob() method does.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
* fix: missing Oci-Subject header pushing index with subject
Signed-off-by: Josh Dolitsky <josh@dolit.ski>
* fix(s3): Add a test to cover handling pushing indexes with a subject
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
---------
Signed-off-by: Josh Dolitsky <josh@dolit.ski>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
In order to verify signatures, users could upload their certificates and public keys using these routes:
-> for public keys:
/v2/_zot/ext/mgmt?resource=signatures&tool=cosign
-> for certificates:
/v2/_zot/ext/mgmt?resource=signatures&tool=notation&truststoreType=ca&truststoreName=name
Then the public keys will be stored under $rootdir/_cosign and the certificates will be stored under
$rootdir/_notation/truststore/x509/$truststoreType/$truststoreName.
Also, for notation case, the "truststores" field of $rootir/_notation/trustpolicy.json file will be
updated with a new entry "$truststoreType:$truststoreName".
Also based on the uploaded files, the information about the signatures validity will be updated
periodically.
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
- AuthzHandler has now been split in BaseAuthzHandler and DistSpecAuthzHandler
The former populates context with user specific data needed in most handlers, while
the latter executes access logic specific to distribution-spec handlers.
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Also modify zli to retry in case of such errors,
assuming the trivyDB will eventually be downloaded by the scheduled task.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
because UI routes will setup a http.FileServer on '/'
any router setup after UI will be ignored at runtime
becuase gorrilla will route it to http.Fileserver instead.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
before syncing an image we first check if it's already present in our storage
to do that we get the manifest from remote and compare it with the local one
but in the case of syncing docker images, because the conversion to OCI format is done while
syncing, we get a docker manifest before conversion, so sync detects that local manifest and
remote one are different, so it starts syncing again.
to overcome this, convert remote docker manifests to OCI manifests and then compare.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
this causes a bug in extensions by not having the identity for the
authenticated user and couldn't apply his permissions, just the default ones.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
ci(workflow): show disk usage and free up disk space used by unneeded tooling
ci(tests): routes tests: do not copy large images if they are not used later
ci(trivy): update a test: download trivy.db to a temporary folder
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
fix(storage/local): also put deduped blobs in cache, not just origin blobs
this caused an error when trying to delete deduped blobs
from multiple repositories
fix(storage/s3): check blob is present in cache before deleting
this is an edge case where dedupe is false but cacheDriver is not nil
(because in s3 we open the cache.db if storage find it in rootDir)
it caused an error when trying to delete blobs uploaded with dedupe false
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Currently, when pushing an image, validation is performed to check that
a layer/blob in the manifest already exists. For non-distributable
layers, that check needs to be skipped.
Fixes issue #1394
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Initial code was contributed by Bogdan BIVOLARU <104334+bogdanbiv@users.noreply.github.com>
Moved implementation from a separate db to repodb by Andrei Aaron <aaaron@luxoft.com>
Not done yet:
- run/test dynamodb implementation, only boltdb was tested
- add additional coverage for existing functionality
- add web-based APIs to toggle the stars/bookmarks on/off
Initially graphql mutation was discussed for the missing API but
we decided REST endpoints would be better suited for configuration
feat(userdb): complete functionality for userdb integration
- dynamodb rollback changes to user starred repos in case increasing the total star count fails
- dynamodb increment/decrement repostars in repometa when user stars/unstars a repo
- dynamodb check anonymous user permissions are working as intendend
- common test handle anonymous users
- RepoMeta2RepoSummary set IsStarred and IsBookmarked
feat(userdb): rest api calls for toggling stars/bookmarks on/off
test(userdb): blackbox tests
test(userdb): move preferences tests in a different file with specific build tags
feat(repodb): add is-starred and is-bookmarked fields to repo-meta
- removed duplicated logic for determining if a repo is starred/bookmarked
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
The zap scanner started to check the csp header, which is causing a warning.
We also need to ignore the rule, as both settings are read by the scanner.
Per https://w3c.github.io/webappsec-csp/#example-7bb4ce67 we can have multiple
Content-Security-Policy headers, and the most restrictive policies apply.
This rule doesn't seem to be applied by zap.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
upgrade to github.com/aws/aws-sdk-go@v1.44.237
upgrade to github.com/aquasecurity/trivy@v0.38.3
upgrade to oras.land/oras-go@v1.2.3
upgrade to github.com/google/go-containerregistry@v0.14.0
upgrade to github.com/moby/buildkit@v0.11.4
Note we can't switch to trivy 0.39.0 as well as some other updates
because they would also require upgrade of cosign to v2 with
breaking api changes
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
sync: remove sync WaitGroup, it's stopped with context
sync: onDemand will always try to sync newest image when a tag is used
if a digest is used then onDemand will serve local image
test(sync): fix flaky coverage in sync package
closes#1294
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
* refactor(repodb): moving common utilities under pkg/meta
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* refactor(repodb): moved update, version components under pkg/meta
- updated wrapper initialization to recieve a log object in constructor
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* refactor(repodb): moved repodb initialization from controller to pkg/meta/repodb
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
---------
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
- before, the download count for a manifest and repo star count were lost after reload
- now we are keeping these values when we reset the repo-meta structure
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Changed repodb to store more information about the referrer needed for the referrers query
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
putting this info into error detail would be ideal, but skopeo
doesn't print them, so overwrite the error message.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
If we push an artifact and give it a tag, repodb would crash because of the null pointer dereferencing
Now when iterating over the tags of a repo and stumbling upon a unsupported media type, it's being ignored
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
The condition to generate trivyDB download tasks was bugged,
and new tasks were generated in case the download had already been
successful (state `done`).
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
BREAKING CHANGE: repository paths are now specified under a new config key called "repositories" under "accessControl" section in order to handle "groups" feature. Previously the repository paths were specified directly under "accessControl".
This PR adds the ability to create groups of users which can be used for authZ policies, instead of just users.
{
"http": {
"accessControl": {
"groups": {
Just like the users, groups can be part of repository policies/default policies/admin policies. The 'groups' field in accessControl can be missing if there are no groups. The permissions priority is user>group>default>admin policy, verified in this order (in authz.go), and permissions are cumulative. It works with LDAP too, and the group attribute name is configurable. The DN of the group is used as the group name and the functionality is the same. All groups for the given user are added to the context in authn.go. Repository paths are now specified under a new keyword called "repositories" under "accessControl" section in order to handle "groups" feature.
Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
(cherry picked from commit 6d03ce5f2d)
Additional changes on top of: 6d03ce5f2d
- Build and use zot from the same branch
do not use a container image as scan target, use the binary
- Fix typo in rules filename
- Add the full rule list to the rules config file
- Ignore some of the specific rules and add reasons
- Add security-related headers to fix some of the issues identified by the scan
- Update UI it includes the latest fixes for zap scan issues
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
This is for consistency with the other calls, and should help in making
all ZUI handling of ZOT errors consistent
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Update the default value of the EXTENSIONS variable in the makefile.
Also cleanup binary-ui and other make targets assuming the UI was not included by default.
Enable the ui by default in the zot container image
Swith back to using the distroless images, as c3 only has amd64 images.
Fix updating security events in github (permission issue)
Add an integration test for the UI extension
Rename ui extension files to use _ instead of -
feat(ui): upgrade to zui v2.0.0-rc3
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
(cherry picked from commit d557da0baba819b7cd7e6b5941528776e125ac6d)
build(ui): fix stacker builds
(cherry picked from commit ba25daf02b4a9bc7ee1cb6f84b7a6b096ca7d61f)
build(ui): various fixes
- Fix metrics endpoint
- Fix unit tests unit tests
- Make the ui build optional in the makefile
before the linter lint runs in the golangci-lint workflow
- Do not attempt to include UI routes if search is enabled
- Fix authorization for search endpoint
fix: use zot tag in ui make target
(cherry picked from commit 2a6882fa23f06b2d68c6c299773a6ff50bf90e78)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
- refactor(cve): remove the global of type cveinfo.CveInfo from the extensions package
Replace it with an attribute on controller level
- refactor(controller): extract initialization logic from controller.Run()
- test(cve): mock cve scanner in cli tests
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Changes in this commit:
A.
There were nested conveys following the pattern:
- outer convey was a positive test
- inner conveys were negative tests
The positive and negative tests did not depend on one another, but the positive test was repeated for each negative test.
So for example if there was 1 positive test and 2 negative tests what go would do is:
1. execute the outer convey with the positive test
2. execute the 1st inner convey with the 1st negative test
3. execute the outer convey with the positive test
4. execute the 2nd inner convey with the 2nd negative test
Since there were no dependencies between these tests they can be moved on the same level.
B.
The test function bringing up the controller for `TestServerCVEResponseGQL` was the same for `TestServerCVEResponse`, so I removed `TestServerCVEResponseGQL` and moved the Conveys under `TestServerCVEResponse`. This will save time on the initial TrivyDB download, since it's done only once when the controller starts.
C. The `Convey("without flags", func() {` was actually the same as the test under which it was nested: `Convey("Test CVE invalid url port", t, func() {` no code differences so I removed it.
D. Use the new test functions to start/stop and wait for the zot contoller to be reachable
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
The 'test' makefile target runs the tests for both 'minimal' and equivalent of the former 'extended' build.
The trivy package tests were run twice, even if the trivy logic is unreachable if search is disabled.
With this update we should see a cut of about 150s of test time.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* docs(graphql): rewrote search.md
docs(graphql): added pagination and filter docs for gql
Squash of both commits:
(cherry picked from commit 2268fa0510b32b27f2c1f71e9889ec769877553b)
(cherry picked from commit c96adc88b2fb9edff90e7e4b01a8885511ceb0df)
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Some additional updates by Andrei Aaron to keep up with the changes on main.
Also add more comments to schema.graphql
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* style: run a graphql schemma linter as a github workflow
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
---------
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Catalin Hofnar <catalin.hofnar@gmail.com>
We are now using v2.0.1 in the cve cache logic.
Unfortunately we are also using v0.5.4 indirectly, as it is required for gqlgen, see:
e6114a2c6a/go.mod (L7)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
- derivedImageList and baseImageList now use FilterTags to obtain results,
each with its own filter function
- images that have the exact same manifest as the one provided as a
parameter are no longer considered base images or derived images
- both calls can be made with specific pagination parameters, and the
response will include PageInfo
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
fix(tests): fix one of the pagination tests
The results were not reliable as the 2 returned tags were sorted by created date/time
which was not set, resulting in an unpredictable order
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
(cherry picked from commit be504200a1127371422aeb0e5c0219e2a1ead20a)
(cherry picked from commit ed8d797e639f262a63840120afe92da7db9a7600)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
"If filtering is requested and applied, the response MUST include an
annotation (org.opencontainers.referrers.filtersApplied) denoting that
an artifactType filter was applied.""
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* feat(repodb): implement pagination for ImageList and integrate it with RepoDB
- it can now return all images from all repos, when provided repo parameter is ""
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
(cherry picked from commit c003dcec9f805564946935e7eb091632f605035e)
(cherry picked from commit 72feba979b9ddd452465a652bb31f439584a046c)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* ci(timeouts): increase ci-cd workflow timeout for the build and test step
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
add page info to dynamo-> feat(repodb): add PageInfo to GlobalSearch and RepoListWithNewestImage results
(cherry picked from commit 4fed42bb4bbc68199281d9d9a4e09b97fbd4759b)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
1. chore(trivy): update trivy library version
The trivy team switched github.com/urfave/cli for viper so
there are some other code changes as well.
Since we don't use github.com/urfave/cli directly in our software
we needed to add a tools.go in order for "go mod tidy" to not delete it.
See this pattern explained in:
- https://github.com/99designs/gqlgen#quick-start
- https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module
- https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md#walk-through
The jobs using "go get -u" have been updated to use "go install", since go get
modifies the go.mod by upgrading some of the packages, but downgrading trivy to an older
version with broken dependencies
2. fix(storage) Update local storage to ignore folder names not compliant with dist spec
Also updated trivy to download the DB and cache results under the rootDir/_trivy folder
3. fix(s3): one of the s3 tests was missing the skipIt call
This caused a failure when running locally without s3 being available
4. make sure the offline scanning is enabled, and zot only downloads the trivy DB
on the regular schedule, and doesn't download the DB on every image scan
ci: increase build and test timeout as tests are reaching the limit more often
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(repodb): add pagination for ImageListForDigest and implement FilterTags
ImageListForDigest can now return paginated results, directly from DB.
It uses FilterTags, a new method to filter tags (obviously) based on
the criteria provided in the filter function.
Pagination of tags is now slightly different, it shows all results if
no limit and offset are provided.
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
bug(tests): cli tests for digests expecting wrong size
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
(cherry picked from commit 369216df931a4053c18278a8d89f86d2e1e6a436)
fix(repodb): do not include repo metadata in search results if no matching tags are identified
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* fix(repodb): Fix an issue in FilterTags where repometa was not proceesed correctly
The filter function was called only once per manifest digest.
The function is supposed to also take into consideration repometa,
but only the first repometa-manifestmeta pair was processed.
Also increase code coverage.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit bea0eabcaa)
(cherry picked from commit 0f02d625331987126326a28731f14d3139061adc)
fix(repodb): use dynamodb sdk functions for waiting on tables to be created or deleted
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
fix(ci): make sure the dynamodb tables used in testing have unique names
There were cases of failures when tests were run for the entire repodb package
which did not reproduce when tests were run for each sub-folder individually
This change should make sure there are no collisions between the names
used in concurrent tests.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): implement a DB for image specific information using boltdb
(cherry picked from commit e3cb60b856)
Some other fixes/improvements on top (Andrei)
Global search: The last updated attribute on repo level is now computed correctly.
Global search: Fix and enhance tests: validate more fields, and fix CVE verification logic
RepoListWithNewestImage: The vendors and platforms at repo level are no longer containing duplicate entries
CVE: scan OCIUncompressedLayer instead of skiping them (used in tests)
bug(repodb): do no try to increment download counters for signatures
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Add filtering to global search API (Laurentiu)
(cherry picked from commit a87976d635ea876fe8ced532e8adb7c3bb24098f)
Original work by Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Fix pagination bug
- when limit was bigger than the repo count result contained empty results
- now correctly returns only maximum available number of repo results
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Add history to the fields returned from RepoDB
Consolidate fields used in packages
- pkg/extensions/search/common/common_test
- pkg/extensions/search/common/common
Refactor duplicate code in GlobalSearch verification
Add vulnerability scan results to image:tag reply
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Refactor ExpandedRepoInfo to using RepoDB
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit fd7dc85c3a9d028fd8860d3791cad4df769ed005)
Init RepoDB at startup
- sync with storage
- ignore images without a tag
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 359898facd6541b2aa99ee95080f7aabf28c2650)
Update request to get image:tag to use repodb
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Sync RepoDB logging
- added logging for errors
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 2e128f4d01712b34c70b5468285100b0657001bb)
sync-repodb minor error checking fix
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Improve tests for syncing RepoDB with storage
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit b18408c6d64e01312849fc18b929e3a2a7931e9e)
Update scoring rule for repos
- now prioritize matches to the end of the repo name
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 6961346ccf02223132b3b12a2132c80bd1b6b33c)
Upgrade search filters to permit multiple values
- multiple values for os and arch
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 3ffb72c6fc0587ff827a03fe4f76a13b27b876a0)
feature(repodb): add pagination for RepoListWithNewestImage
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
(cherry picked from commit 32c917f2dc65363b0856345289353559a8027aee)
test(fix): fix tests failing since repodb is used for listing all repos
1. One of the tests was verifying disk/oci related erros and is not applicable
2. Another test was actually broken in an older PR, the default store and
the substore were using the same repo names (the substore ones were unprefixed),
which should not be the case, this was causing a single entry to show
in the RepoDB instead of two separate entries for each test image
Root cause in: b61aff62cd (diff-b86e11fa5a3102b336caebec3b30a9d35e26af554dd8658f124dba2404b7d24aR88)
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
chore: move code reponsible for transforming objects to gql_generated types to separate package
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Process input for global search
- Clean input: query, filter strings
- Add validation for global search input
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit f1ca8670fbe4a4a327ea25cf459237dbf23bb78a)
fix: only call cve scanning for data shown to the user
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
GQL omit scanning for CVE if field is not required
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 5479ce45d6cb2abcf5fbccadeaf6f3393c3f6bf1)
Fix filtering logic in RepoDB
- filter parameter was set to false instead of being calculator from the later image
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit a82d2327e34e5da617af0b7ca78a2dba90999f0a)
bug(repodb): Checking signature returns error if signed image is not found
- we considere a signature image orfan when the image it signs is not found
- we need this to ignore such signatures in certain cases
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit d0418505f76467accd8e1ee34fcc2b2a165efae5)
feat(repodb): CVE logic to use repoDB
Also update some method signatures to remove usage of:
github.com/google/go-containerregistry/pkg/v1
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
* feat(repodb): refactor repodb update logic
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(repodb): minor fixes
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): move repodb logic inside meta directory under pkg
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): change factory class for repodb initialization with factory metrod
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): simplify repodb configuration
- repodb now shares config parameters with the cache
- config taken directly from storage config
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(authors): fix authors information to work properly with repodb
Ideally this commit would be squshed in the repodb commit
but as-is it is easier to cherry-pick on other branches
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
* feat(repodb): dynamodb support for repodb
- clean-up repodb code + coverage improvements
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(dynamo): tables used by dynamo are created automatically if they don't exists
- if the table exists nothing happens
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* test(repodb): coverage tests
- minor fix for CVEListForImage to fix the tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): add descriptor with media type
- to represent images and multi-arch images
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): support signatures on repo level
- added to follow the behavior of signing and signature verification tools
that work on a manifest level for each repo
- all images with different tags but the same manifest will be signed at once
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): old repodb version migration support
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): tests for coverage
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): WIP fixing tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): work on patchRepoDB tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(repodb): create dynamo tables only for linux amd
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(ci): fix a typo in ci-cd.yml
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Andrei Aaron <andaaron@cisco.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
Currently there is no push-back on queries that should contain image names but
have only the repo name. This commit adds a check that will return an error for images
w/o a reference(tag or digest).
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
UI can now make use of OCI artifacts and references using `Referrers` gQL query.
It returns a list of descriptors that refer on their `subject` field to another
digest.
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
