- Avoid hard-coding the path for bash - this typically is installed as
/usr/local/bin/bash
- Allow for the fact that FreeBSD's rlimit uses signed integers
Almost all of the test suite passes with three test failures that I
haven't tried to debug yet.
Signed-off-by: Doug Rabson <dfr@rabson.org>
The ui sends the header X-ZOT-API-CLIENT=zot-ui regardless of session authentication status.
In case of new sessions zot would reject the unauthenticated call on /v2 (which is used to determine
if anonymous access is allowed by the server when the header was set) expecting all users sending
this header to be already authenticated.
Since the ui received 401 from the server, it would not show the option for anonymous login.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* fix(conformance): OCI-Filters-Applied should return a literal
https://github.com/opencontainers/distribution-spec/issues/448
Should only indicate what filter-type was applied and not what exact
values it was filtered on.
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* ci(conformance): point to temporary repo
Revert this commit once upstream changes have been merged
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Resolves#1597
Fix missing properties in some calls OS/Arch/IsSigned/LastUpdated.
Since the properties were missing from the graphql requests, zli was showing default values.
Update the tests to generate a tets image with the created date different from
the go detault value for time.Time{}. The tests are now checking a non-default
value in the responses client side, in order to avoid missing issues like this in the future.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
https://github.com/project-zot/zot/issues/1591
- I will rename "IMAGE NAME" to "REPOSITORY" in order to make the header easier to parse
- The order of the images cannot be predicted if zot is getting them 1 by 1 using the REST API for manifests, so they cannot be sorted when printed. We could wait on all calls to return but that may take minutes, and printing partial results as they become available is better.
- The order of the images can be predicted when relying on the zot specific search API, but that is not available in all zot servers depending on build options. I added sorting ascending by default. We are planning to implement configurable sorting in a separate PR - see the work under https://github.com/project-zot/zot/pull/1577
- With regards to the column widths/alignments that was discussed before, and the issue is we don't know the values beforehand for the REST API based responses. As mentioned above printing partial results as they become available is better.
- The column widths/alignments are partially fixed in this PR for the search API, but we should properly fix this in - see https://github.com/project-zot/zot/pull/851https://github.com/project-zot/zot/issues/1592
- Fix missing space after help message
https://github.com/project-zot/zot/issues/1598
- Fix table headers showing for json/yaml format
- Fix spacing shown with json format, use 1 row per shown entry in order to be compatible with json lines format: https://jsonlines.org/
- Add document header `---` to every image shown in yaml format to separate the entries
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
sync now ignores self referencing urls, this will help
in clustering mode where we can have the same config
for multiple zots
closes#1335
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
dist-spec APIs independently allow deletion of blobs and manifests.
Doing the former when in use by an image manifest or index is simply
error-prone. So disallow it.
Fixes issue #1509
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
We use locks to protect OCI layouts. However, our critical sections have
too long and cover code instead of data.
Fixes issue #1595
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
This change introduces OpenID authn by using providers such as Github,
Gitlab, Google and Dex.
User sessions are now used for web clients to identify
and persist an authenticated users session, thus not requiring every request to
use credentials.
Another change is apikey feature, users can create/revoke their api keys and use them
to authenticate when using cli clients such as skopeo.
eg:
login:
/auth/login?provider=github
/auth/login?provider=gitlab
and so on
logout:
/auth/logout
redirectURL:
/auth/callback/github
/auth/callback/gitlab
and so on
If network policy doesn't allow inbound connections, this callback wont work!
for more info read documentation added in this commit.
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
when pushing manifests, zot will validate blobs (layers + config blob) are
present in repo, currently it opens(in case of filesystem storage) or download(
in case of cloud storage) each blob.
fixed that by adding a new method ImageStore.CheckBlobPresence() on storage
to check blobs presence without checking the cache like ImageStore.CheckBlob() method does.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
* fix: missing Oci-Subject header pushing index with subject
Signed-off-by: Josh Dolitsky <josh@dolit.ski>
* fix(s3): Add a test to cover handling pushing indexes with a subject
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
---------
Signed-off-by: Josh Dolitsky <josh@dolit.ski>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
In order to verify signatures, users could upload their certificates and public keys using these routes:
-> for public keys:
/v2/_zot/ext/mgmt?resource=signatures&tool=cosign
-> for certificates:
/v2/_zot/ext/mgmt?resource=signatures&tool=notation&truststoreType=ca&truststoreName=name
Then the public keys will be stored under $rootdir/_cosign and the certificates will be stored under
$rootdir/_notation/truststore/x509/$truststoreType/$truststoreName.
Also, for notation case, the "truststores" field of $rootir/_notation/trustpolicy.json file will be
updated with a new entry "$truststoreType:$truststoreName".
Also based on the uploaded files, the information about the signatures validity will be updated
periodically.
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
- AuthzHandler has now been split in BaseAuthzHandler and DistSpecAuthzHandler
The former populates context with user specific data needed in most handlers, while
the latter executes access logic specific to distribution-spec handlers.
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Also modify zli to retry in case of such errors,
assuming the trivyDB will eventually be downloaded by the scheduled task.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
because UI routes will setup a http.FileServer on '/'
any router setup after UI will be ignored at runtime
becuase gorrilla will route it to http.Fileserver instead.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
before syncing an image we first check if it's already present in our storage
to do that we get the manifest from remote and compare it with the local one
but in the case of syncing docker images, because the conversion to OCI format is done while
syncing, we get a docker manifest before conversion, so sync detects that local manifest and
remote one are different, so it starts syncing again.
to overcome this, convert remote docker manifests to OCI manifests and then compare.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
this causes a bug in extensions by not having the identity for the
authenticated user and couldn't apply his permissions, just the default ones.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
ci(workflow): show disk usage and free up disk space used by unneeded tooling
ci(tests): routes tests: do not copy large images if they are not used later
ci(trivy): update a test: download trivy.db to a temporary folder
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
fix(storage/local): also put deduped blobs in cache, not just origin blobs
this caused an error when trying to delete deduped blobs
from multiple repositories
fix(storage/s3): check blob is present in cache before deleting
this is an edge case where dedupe is false but cacheDriver is not nil
(because in s3 we open the cache.db if storage find it in rootDir)
it caused an error when trying to delete blobs uploaded with dedupe false
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Currently, when pushing an image, validation is performed to check that
a layer/blob in the manifest already exists. For non-distributable
layers, that check needs to be skipped.
Fixes issue #1394
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Initial code was contributed by Bogdan BIVOLARU <104334+bogdanbiv@users.noreply.github.com>
Moved implementation from a separate db to repodb by Andrei Aaron <aaaron@luxoft.com>
Not done yet:
- run/test dynamodb implementation, only boltdb was tested
- add additional coverage for existing functionality
- add web-based APIs to toggle the stars/bookmarks on/off
Initially graphql mutation was discussed for the missing API but
we decided REST endpoints would be better suited for configuration
feat(userdb): complete functionality for userdb integration
- dynamodb rollback changes to user starred repos in case increasing the total star count fails
- dynamodb increment/decrement repostars in repometa when user stars/unstars a repo
- dynamodb check anonymous user permissions are working as intendend
- common test handle anonymous users
- RepoMeta2RepoSummary set IsStarred and IsBookmarked
feat(userdb): rest api calls for toggling stars/bookmarks on/off
test(userdb): blackbox tests
test(userdb): move preferences tests in a different file with specific build tags
feat(repodb): add is-starred and is-bookmarked fields to repo-meta
- removed duplicated logic for determining if a repo is starred/bookmarked
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
The zap scanner started to check the csp header, which is causing a warning.
We also need to ignore the rule, as both settings are read by the scanner.
Per https://w3c.github.io/webappsec-csp/#example-7bb4ce67 we can have multiple
Content-Security-Policy headers, and the most restrictive policies apply.
This rule doesn't seem to be applied by zap.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
upgrade to github.com/aws/aws-sdk-go@v1.44.237
upgrade to github.com/aquasecurity/trivy@v0.38.3
upgrade to oras.land/oras-go@v1.2.3
upgrade to github.com/google/go-containerregistry@v0.14.0
upgrade to github.com/moby/buildkit@v0.11.4
Note we can't switch to trivy 0.39.0 as well as some other updates
because they would also require upgrade of cosign to v2 with
breaking api changes
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
sync: remove sync WaitGroup, it's stopped with context
sync: onDemand will always try to sync newest image when a tag is used
if a digest is used then onDemand will serve local image
test(sync): fix flaky coverage in sync package
closes#1294
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
* refactor(repodb): moving common utilities under pkg/meta
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* refactor(repodb): moved update, version components under pkg/meta
- updated wrapper initialization to recieve a log object in constructor
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* refactor(repodb): moved repodb initialization from controller to pkg/meta/repodb
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
---------
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
- before, the download count for a manifest and repo star count were lost after reload
- now we are keeping these values when we reset the repo-meta structure
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Changed repodb to store more information about the referrer needed for the referrers query
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
putting this info into error detail would be ideal, but skopeo
doesn't print them, so overwrite the error message.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
If we push an artifact and give it a tag, repodb would crash because of the null pointer dereferencing
Now when iterating over the tags of a repo and stumbling upon a unsupported media type, it's being ignored
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
The condition to generate trivyDB download tasks was bugged,
and new tasks were generated in case the download had already been
successful (state `done`).
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
BREAKING CHANGE: repository paths are now specified under a new config key called "repositories" under "accessControl" section in order to handle "groups" feature. Previously the repository paths were specified directly under "accessControl".
This PR adds the ability to create groups of users which can be used for authZ policies, instead of just users.
{
"http": {
"accessControl": {
"groups": {
Just like the users, groups can be part of repository policies/default policies/admin policies. The 'groups' field in accessControl can be missing if there are no groups. The permissions priority is user>group>default>admin policy, verified in this order (in authz.go), and permissions are cumulative. It works with LDAP too, and the group attribute name is configurable. The DN of the group is used as the group name and the functionality is the same. All groups for the given user are added to the context in authn.go. Repository paths are now specified under a new keyword called "repositories" under "accessControl" section in order to handle "groups" feature.
Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
(cherry picked from commit 6d03ce5f2d)
Additional changes on top of: 6d03ce5f2d
- Build and use zot from the same branch
do not use a container image as scan target, use the binary
- Fix typo in rules filename
- Add the full rule list to the rules config file
- Ignore some of the specific rules and add reasons
- Add security-related headers to fix some of the issues identified by the scan
- Update UI it includes the latest fixes for zap scan issues
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
This is for consistency with the other calls, and should help in making
all ZUI handling of ZOT errors consistent
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Update the default value of the EXTENSIONS variable in the makefile.
Also cleanup binary-ui and other make targets assuming the UI was not included by default.
Enable the ui by default in the zot container image
Swith back to using the distroless images, as c3 only has amd64 images.
Fix updating security events in github (permission issue)
Add an integration test for the UI extension
Rename ui extension files to use _ instead of -
feat(ui): upgrade to zui v2.0.0-rc3
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
(cherry picked from commit d557da0baba819b7cd7e6b5941528776e125ac6d)
build(ui): fix stacker builds
(cherry picked from commit ba25daf02b4a9bc7ee1cb6f84b7a6b096ca7d61f)
build(ui): various fixes
- Fix metrics endpoint
- Fix unit tests unit tests
- Make the ui build optional in the makefile
before the linter lint runs in the golangci-lint workflow
- Do not attempt to include UI routes if search is enabled
- Fix authorization for search endpoint
fix: use zot tag in ui make target
(cherry picked from commit 2a6882fa23f06b2d68c6c299773a6ff50bf90e78)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
- refactor(cve): remove the global of type cveinfo.CveInfo from the extensions package
Replace it with an attribute on controller level
- refactor(controller): extract initialization logic from controller.Run()
- test(cve): mock cve scanner in cli tests
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Changes in this commit:
A.
There were nested conveys following the pattern:
- outer convey was a positive test
- inner conveys were negative tests
The positive and negative tests did not depend on one another, but the positive test was repeated for each negative test.
So for example if there was 1 positive test and 2 negative tests what go would do is:
1. execute the outer convey with the positive test
2. execute the 1st inner convey with the 1st negative test
3. execute the outer convey with the positive test
4. execute the 2nd inner convey with the 2nd negative test
Since there were no dependencies between these tests they can be moved on the same level.
B.
The test function bringing up the controller for `TestServerCVEResponseGQL` was the same for `TestServerCVEResponse`, so I removed `TestServerCVEResponseGQL` and moved the Conveys under `TestServerCVEResponse`. This will save time on the initial TrivyDB download, since it's done only once when the controller starts.
C. The `Convey("without flags", func() {` was actually the same as the test under which it was nested: `Convey("Test CVE invalid url port", t, func() {` no code differences so I removed it.
D. Use the new test functions to start/stop and wait for the zot contoller to be reachable
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
The 'test' makefile target runs the tests for both 'minimal' and equivalent of the former 'extended' build.
The trivy package tests were run twice, even if the trivy logic is unreachable if search is disabled.
With this update we should see a cut of about 150s of test time.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* docs(graphql): rewrote search.md
docs(graphql): added pagination and filter docs for gql
Squash of both commits:
(cherry picked from commit 2268fa0510b32b27f2c1f71e9889ec769877553b)
(cherry picked from commit c96adc88b2fb9edff90e7e4b01a8885511ceb0df)
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Some additional updates by Andrei Aaron to keep up with the changes on main.
Also add more comments to schema.graphql
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* style: run a graphql schemma linter as a github workflow
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
---------
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Catalin Hofnar <catalin.hofnar@gmail.com>
We are now using v2.0.1 in the cve cache logic.
Unfortunately we are also using v0.5.4 indirectly, as it is required for gqlgen, see:
e6114a2c6a/go.mod (L7)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
- derivedImageList and baseImageList now use FilterTags to obtain results,
each with its own filter function
- images that have the exact same manifest as the one provided as a
parameter are no longer considered base images or derived images
- both calls can be made with specific pagination parameters, and the
response will include PageInfo
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
fix(tests): fix one of the pagination tests
The results were not reliable as the 2 returned tags were sorted by created date/time
which was not set, resulting in an unpredictable order
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
(cherry picked from commit be504200a1127371422aeb0e5c0219e2a1ead20a)
(cherry picked from commit ed8d797e639f262a63840120afe92da7db9a7600)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
"If filtering is requested and applied, the response MUST include an
annotation (org.opencontainers.referrers.filtersApplied) denoting that
an artifactType filter was applied.""
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* feat(repodb): implement pagination for ImageList and integrate it with RepoDB
- it can now return all images from all repos, when provided repo parameter is ""
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
(cherry picked from commit c003dcec9f805564946935e7eb091632f605035e)
(cherry picked from commit 72feba979b9ddd452465a652bb31f439584a046c)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* ci(timeouts): increase ci-cd workflow timeout for the build and test step
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
