BREAKING CHANGE: the dist spec version in the config files needs to be bumped to 1.1.0
in order for the config verification to pass without warnings.
Also fix 1 dependabot alert for helm.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Refactor and add more coverage to test flacky coverage in case sessions
which are already deleted are flagged as expired/for deletion.
See coverage drop in pkg/api/cookiestore.go:
8e68255946/indirect-changes
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Fallback to Created field and the History entries in the image config
only if the annotation "org.opencontainers.image.created" is not available
closes#2210
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
This causes the "fair" scheduler to run it too often in the detriment of other generators.
The intention was to run it every 2 hours but the measurement unit for 7200 was not specified.
Add more logs, including showing a generator name, in order to troubleshoot this kind of issues easier in the future.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Generators are now ordered by rank in the priority queue.
The rank computation formula is:
- 100/(1+generated_task_count) for high priority tasks
- 10/(1+generated_task_count) for medium priority tasks
- 1/(1+generated_task_count) for low priority tasks
Note the ranks are used when comparing generators both with the same priority and with different priority.
So now we are:
- giving an opportunity to all generators with the same priority to take turns generating tasks
- giving roughly 1 low priority and 10 medium priority tasks the opportunity to run for every 100 high priority tasks running.
After a generator generates a task, the generators are reordered in the priority queue based on rank.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
We use chartmuseum lib for handling bearer requests, which is not
implementing the token spec, mainly it expects "scope" parameter
to be given on every request, even for /v2/ route which doesn't represent
a resource.
Handle this /v2/ route inside our code.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
- update cve tests
- update scrub tests
- update tests for parsing storage and loading into meta DB
- update controller tests
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
instead of reading entire files before calculating their digests
stream them by using their Reader method.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
We cannot assume the LDAP server will have group attributes programmed
everytime. So handle it accordingly.
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
We expect panics in the server/datapath to be few and far between.
So the backtraces are more valuable now.
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
init shutdown routine after controller.Init()
check for nil values before stopping http server and task scheduler.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
For CLI output is similar to:
CRITICAL 0, HIGH 1, MEDIUM 1, LOW 0, UNKNOWN 0, TOTAL 2
ID SEVERITY TITLE
CVE-2023-0464 HIGH openssl: Denial of service by excessive resou...
CVE-2023-0465 MEDIUM openssl: Invalid certificate policies in leaf...
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
- added a new field 'IsDeletable' for graphql ImageSummary struct.
- apply cors on DeleteManifest route
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
* fix(scheduler): data race when pushing new tasks
the problem here is that scheduler can be closed in two ways:
- canceling the context given as argument to scheduler.RunScheduler()
- running scheduler.Shutdown()
because of this shutdown can trigger a data race between calling scheduler.inShutdown()
and actually pushing tasks into the pool workers
solved that by keeping a quit channel and listening on both quit channel and ctx.Done()
and closing the worker chan and scheduler afterwards.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
* refactor(scheduler): refactor into a single shutdown
before this we could stop scheduler either by closing the context
provided to RunScheduler(ctx) or by running Shutdown().
simplify things by getting rid of the external context in RunScheduler().
keep an internal context in the scheduler itself and pass it down to all tasks.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
---------
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
* feat(sync): local tmp store
Signed-off-by: a <a@tuxpa.in>
* fix(sync): various fixes for s3+remote storage feature
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
---------
Signed-off-by: a <a@tuxpa.in>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: a <a@tuxpa.in>
wait for workers to finish before exiting
should fix tests reporting they couldn't remove rootDir because it's being
written by tasks
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
- MetaDB stores the time of the last update of a repo
- During startup we check if the layout has been updated after the last recorded change in the db
- If this is the case, the repo is parsed and updated in the DB otherwise it's skipped
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
in the case of an already existing meta db without pushTimestamp field
its value would be 0 until image is updated, check for zero values and update them
with time.Now() so that retention logic won't remove them.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
- Cosign supports 2 types of signature formats:
1. Using tag -> each new signature of the same manifest is
added as a new layer of the signature manifest having that
specific tag("{alghoritm}-{digest_of_signed_manifest}.sig")
2. Using referrers -> each new signature of the same manifest is
added as a new manifest
- For adding these cosign signature to metadb, we reserved index 0 of the
list of cosign signatures for tag-based signatures. When a new tag-based
signature is added for the same manifest, the element on first position
in its list of cosign signatures(in metadb) will be updated/overwritten.
When a new cosign signature(using referrers) will be added for the same
manifest this new signature will be appended to the list of cosign
signatures.
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
- use secretsManagerMock and secretsManagerCacheMock to avoid failing
because of "already exists" error when running multiple times
image_trust_test on the same localstack instance
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
ci(notation): update to latest notation version
fix(sync): add layers info when syncing signatures
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
As part of this change searchConfig needed to be exported,
as it was passed as a parameter to exported functions
At this moment most of the tests remaining internal depend on the mock service.
The interface it implements has unexported methods.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Which could be imported independently. See more details:
1. "zotregistry.io/zot/pkg/test/common" - currently used as
tcommon "zotregistry.io/zot/pkg/test/common" - inside pkg/test
test "zotregistry.io/zot/pkg/test/common" - in tests
. "zotregistry.io/zot/pkg/test/common" - in tests
Decouple zb from code in test/pkg in order to keep the size small.
2. "zotregistry.io/zot/pkg/test/image-utils" - curently used as
. "zotregistry.io/zot/pkg/test/image-utils"
3. "zotregistry.io/zot/pkg/test/deprecated" - curently used as
"zotregistry.io/zot/pkg/test/deprecated"
This one will bre replaced gradually by image-utils in the future.
4. "zotregistry.io/zot/pkg/test/signature" - (cosign + notation) use as
"zotregistry.io/zot/pkg/test/signature"
5. "zotregistry.io/zot/pkg/test/auth" - (bearer + oidc) curently used as
authutils "zotregistry.io/zot/pkg/test/auth"
6. "zotregistry.io/zot/pkg/test/oci-utils" - curently used as
ociutils "zotregistry.io/zot/pkg/test/oci-utils"
Some unused functions were removed, some were replaced, and in
a few cases specific funtions were moved to the files they were used in.
Added an interface for the StoreController, this reduces the number of imports
of the entire image store, decreasing binary size for tests.
If the zb code was still coupled with pkg/test, this would have reflected in zb size.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
no need to run dedupe/restore blobs for images being pushed or synced while
running dedupe task, they are already deduped/restored inline.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
- implement scrub also for S3 storage by replacing umoci
- change scrub implementation for ImageIndex
- take the `Subject` into consideration when running scrub
- remove test code relying on the umoci library. Since we started
relying on images in test/data, and we create our own images using
go code we can obtain digests by other means. (cherry picked from commit 489d4e2d23c1b4e48799283f8281024bbef6123f)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
fix(gc): fix cleaning deduped blobs because they have the modTime of
the original blobs, fixed by updating the modTime when hard linking
the blobs.
fix(gc): failing to parse rootDir at zot startup when using s3 storage
because there are no files under rootDir and we can not create empty dirs
on s3, fixed by creating an empty file under rootDir.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
1. Move existing CVE DB download generator/task login under the cve package
2. Add a new CVE scanner task generator and task type to run in the background, as well as tests for it
3. Move the CVE cache in its own package
4. Add a CVE scanner methods to check if an entry is present in the cache, and to retreive the results
5. Modify the FilterTags MetaDB method to not exit on first error
This is needed in order to pass all tags to the generator,
instead of the generator stopping at the first set of invalid data
6. Integrate the new scanning task generator with the existing zot code.
7. Fix an issue where the CVE scan results for multiarch images was not cached
8. Rewrite some of the older CVE tests to use the new image-utils test package
9. Use the CVE scanner as attribute of the controller instead of CveInfo.
Remove functionality of CVE DB update from CveInfo, it is now responsible,
as the name states, only for providing CVE information.
10. The logic to get maximum severity and cve count for image sumaries now uses only the scanner cache.
11. Removed the GetCVESummaryForImage method from CveInfo as it was only used in tests
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
now gc stress on s3 storage is using minio for ci/cd builds
gc stress on s3 storage is using localstack for nightly builds
fixed(gc): make sure we don't remove repo if there are blobs
being uploaded or the number of blobs gc'ed is not 0
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
(cherry picked from commit 56ddb70f624e7070ad0d3531d498675f9f82c664)
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
1. Only scan CVEs for images returned by graphql calls
Since pagination was refactored to account for image indexes, we had started
to run the CVE scanner before pagination was applied, resulting in
decreased ZOT performance if CVE information was requested
2. Increase in medory-cache of cve results to 1m, from 10k digests.
3. Update CVE model to use CVSS severity values in our code.
Previously we relied upon the strings returned by trivy directly,
and the sorting they implemented.
Since CVE severities are standardized, we don't need to pass around
an adapter object just for pagination and sorting purposes anymore.
This also improves our testing since we don't mock the sorting functions anymore.
4. Fix a flaky CLI test not waiting for the zot service to start.
5. Add the search build label on search/cve tests which were missing it.
6. The boltdb update method was used in a few places where view was supposed to be called.
7. Add logs for start and finish of parsing MetaDB.
8. Avoid unmarshalling twice to obtain annotations for multiarch images.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
