0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2024-12-16 21:56:37 -05:00
Commit graph

207 commits

Author SHA1 Message Date
Andrei Aaron
8b4abc6ef6
Add a job to check zot config examples (and fix existing examples) (#2322)
* fix: Add credentials config verification

(cherry picked from commit e7fdfa0bcc)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

* fix: Update golang version to 1.21.x

Signed-off-by: onidoru <25552941+onidoru@users.noreply.github.com>
Signed-off-by: Nikita Kotikov <25552941+onidoru@users.noreply.github.com>
(cherry picked from commit cbc0f89dfb)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

* fix: LDAP credentials files are now required, add more tests

Signed-off-by: onidoru <25552941+onidoru@users.noreply.github.com>
Signed-off-by: Nikita Kotikov <25552941+onidoru@users.noreply.github.com>
(cherry picked from commit b74366d50b)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

* fix: Update error handling, add more tests

Signed-off-by: onidoru <25552941+onidoru@users.noreply.github.com>
Signed-off-by: Nikita Kotikov <25552941+onidoru@users.noreply.github.com>
(cherry picked from commit 8a61bbc2d4)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

* fix: Add coverage

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

---------

Signed-off-by: onidoru <25552941+onidoru@users.noreply.github.com>
Signed-off-by: Nikita Kotikov <25552941+onidoru@users.noreply.github.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: onidoru <onidoru@yahoo.com>
Co-authored-by: Nikita Kotikov <25552941+onidoru@users.noreply.github.com>
2024-03-21 10:23:37 -07:00
Vishwas R
c7472a2dda
feat: add verbose mode for cves for image listing (#2308)
Signed-off-by: Vishwas Rajashekar <vrajashe@cisco.com>
2024-03-12 13:38:48 -07:00
LaurentiuNiculae
5039128723
feat(cve): cli cve diff (#2242)
* feat(gql): add new query for diff of cves for 2 images

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(cli): add cli for cve diff

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

---------

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2024-03-06 10:40:29 +02:00
Andrei Aaron
4e5db84cb1
chore: update image-spec and dist spec to 1.1.0 (#2255)
BREAKING CHANGE: the dist spec version in the config files needs to be bumped to 1.1.0
in order for the config verification to pass without warnings.

Also fix 1 dependabot alert for helm.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-02-20 13:27:21 +02:00
Vishwas R
0aa6bf0fff
feat: include PackagePath data in CVEs for image queries (#2241)
Signed-off-by: Vishwas Rajashekar <vrajashe@cisco.com>
2024-02-15 13:19:49 -08:00
Andreea Lupu
55acce6923
feat(graphql): filter CVEs by severity (#2246)
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2024-02-14 09:11:57 -08:00
Andrei Aaron
ce4924f841
refactor: rename go module from zotregistry.io/zot to zotregistry.dev/zot (#2187)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-01-31 20:34:07 -08:00
LaurentiuNiculae
1785688b7c
feat(ldap): hot reloading ldap credentials on change (#2167)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2024-01-25 09:15:22 -08:00
LaurentiuNiculae
3f97f878fd
feat(cve): add option to exclude string from cve search (#2163)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2024-01-19 12:59:42 -08:00
peusebiu
0bed25dddd
feat(config): handle config files with no explicit extension (#2147)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2024-01-15 13:10:56 +02:00
peusebiu
a46e10269a
fix(shutdown): fix crash when shutting down before server and task scheduler have started. (#2148)
init shutdown routine after controller.Init()
check for nil values before stopping http server and task scheduler.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2024-01-05 17:50:48 -08:00
Andrei Aaron
18aa975ae2
feat(CVE): add CVE severity counters to returned images and CVE list calls (#2131)
For CLI output is similar to:

CRITICAL 0, HIGH 1, MEDIUM 1, LOW 0, UNKNOWN 0, TOTAL 2

ID                SEVERITY  TITLE
CVE-2023-0464     HIGH      openssl: Denial of service by excessive resou...
CVE-2023-0465     MEDIUM    openssl: Invalid certificate policies in leaf...

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-12-13 09:16:31 -08:00
peusebiu
7642e5af98
fix(scheduler): fix data race (#2085)
* fix(scheduler): data race when pushing new tasks

the problem here is that scheduler can be closed in two ways:
- canceling the context given as argument to scheduler.RunScheduler()
- running scheduler.Shutdown()

because of this shutdown can trigger a data race between calling scheduler.inShutdown()
and actually pushing tasks into the pool workers

solved that by keeping a quit channel and listening on both quit channel and ctx.Done()
and closing the worker chan and scheduler afterwards.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>

* refactor(scheduler): refactor into a single shutdown

before this we could stop scheduler either by closing the context
provided to RunScheduler(ctx) or by running Shutdown().

simplify things by getting rid of the external context in RunScheduler().
keep an internal context in the scheduler itself and pass it down to all tasks.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>

---------

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-12-11 10:00:34 -08:00
LaurentiuNiculae
79e14027ee
refactor(test): add lint rule for messages starting with the component (#2045)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-12-08 10:05:02 +02:00
peusebiu
3c8da6e6fc
Sync s3 (#2073)
* feat(sync): local tmp store

Signed-off-by: a <a@tuxpa.in>

* fix(sync): various fixes for s3+remote storage feature

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>

---------

Signed-off-by: a <a@tuxpa.in>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: a <a@tuxpa.in>
2023-11-28 12:08:15 -08:00
peusebiu
6222dae1f0
feat(scheduler): gracefully shutdown (#1951)
wait for workers to finish before exiting

should fix tests reporting they couldn't remove rootDir because it's being
written by tasks

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-11-24 00:40:10 -08:00
LaurentiuNiculae
92837c2bcb
refactor(log): replace panics with log fatal or log panic functions (#1723)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-11-24 00:38:36 -08:00
LaurentiuNiculae
83f287d1f6
feat(cli): add command to interogate the server version and other details (#1709)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-11-22 19:25:21 +02:00
peusebiu
60eaf7b5d9
fix(config): better configuration errors using viper.UnmarshalExact() (#2050)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-11-16 09:00:55 -08:00
LaurentiuNiculae
272eb7cc43
feat(ldap): add option to load ldap from file (#1778)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-11-14 16:21:36 -08:00
Andreea Lupu
d5065513f5
feat: add support for oci1.1 cosign signatures(using referrers) (#1963)
- Cosign supports 2 types of signature formats:

	1. Using tag -> each new signature of the same manifest is
	added as a new layer of the signature manifest having that
	specific tag("{alghoritm}-{digest_of_signed_manifest}.sig")

	2. Using referrers -> each new signature of the same manifest is
	added as a new manifest

- For adding these cosign signature to metadb, we reserved index 0 of the
list of cosign signatures for tag-based signatures. When a new tag-based
signature is added for the same manifest, the element on first position
in its list of cosign signatures(in metadb) will be updated/overwritten.
When a new cosign signature(using referrers) will be added for the same
manifest this new signature will be appended to the list of cosign
signatures.

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-11-06 14:09:39 -08:00
peusebiu
9074f8483b
feat(retention): added image retention policies (#1866)
feat(metaDB): add more image statistics info

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-11-01 09:16:18 -07:00
LaurentiuNiculae
56ad9e6707
refactor(metadb): improve UX by speeding up metadb serialize/deserialize (#1842)
Use protocol buffers and update the metadb interface to better suit our search needs

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-10-30 13:06:04 -07:00
Alexei Dodon
d2fbd273ba
fix: tests refactoring (#1950)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-10-26 11:20:39 +03:00
Ramkumar Chinchani
1675f30d4a
ci: update golangci-lint version (#1834)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-10-20 17:27:04 +03:00
Andreea Lupu
fc2380b57b
fix: add support for uploaded index when signing using notation (#1882)
ci(notation): update to latest notation version
fix(sync): add layers info when syncing signatures

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-10-12 18:45:20 -07:00
Alexei Dodon
044ea85279
fix: running tests locally fails (#1879)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-10-05 14:34:50 +03:00
Andrei Aaron
ca1c3288cf
refactor(test): make sure cli tests are not internal unless they need to be (#1878)
As part of this change searchConfig needed to be exported,
as it was passed as a parameter to exported functions

At this moment most of the tests remaining internal depend on the mock service.
The interface it implements has unexported methods.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-10-03 11:15:39 -07:00
Andrei Aaron
99e29c0f46
refactor(tests): Migrate some of the older tests to the new image-utils library (#1863)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-10-02 11:10:43 -07:00
Andrei Aaron
ba6f347d8d
refactor(pkg/test): split logic in pkg/test/common.go into multiple packages (#1861)
Which could be imported independently. See more details:
1. "zotregistry.io/zot/pkg/test/common" - currently used as
   tcommon "zotregistry.io/zot/pkg/test/common" - inside pkg/test
   test "zotregistry.io/zot/pkg/test/common" - in tests
   . "zotregistry.io/zot/pkg/test/common" - in tests
Decouple zb from code in test/pkg in order to keep the size small.

2. "zotregistry.io/zot/pkg/test/image-utils" - curently used as
   . "zotregistry.io/zot/pkg/test/image-utils"

3. "zotregistry.io/zot/pkg/test/deprecated" -  curently used as
   "zotregistry.io/zot/pkg/test/deprecated"
This one will bre replaced gradually by image-utils in the future.

4. "zotregistry.io/zot/pkg/test/signature" - (cosign + notation) use as
   "zotregistry.io/zot/pkg/test/signature"

5. "zotregistry.io/zot/pkg/test/auth" - (bearer + oidc)  curently used as
   authutils "zotregistry.io/zot/pkg/test/auth"

 6. "zotregistry.io/zot/pkg/test/oci-utils" -  curently used as
   ociutils "zotregistry.io/zot/pkg/test/oci-utils"

Some unused functions were removed, some were replaced, and in
a few cases specific funtions were moved to the files they were used in.

Added an interface for the StoreController, this reduces the number of imports
of the entire image store, decreasing binary size for tests.
If the zb code was still coupled with pkg/test, this would have reflected in zb size.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-09-27 11:34:48 -07:00
peusebiu
1df743f173
fix(gc): sync repodb when gc'ing manifests (#1819)
fix(gc): fix cleaning deduped blobs because they have the modTime of
the original blobs, fixed by updating the modTime when hard linking
the blobs.
fix(gc): failing to parse rootDir at zot startup when using s3 storage
because there are no files under rootDir and we can not create empty dirs
on s3, fixed by creating an empty file under rootDir.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-22 11:51:20 -07:00
Andrei Aaron
7c78f80a96
feat(cve): implement CVE scanning as background tasks (#1833)
1. Move existing CVE DB download generator/task login under the cve package
2. Add a new CVE scanner task generator and task type to run in the background, as well as tests for it
3. Move the CVE cache in its own package
4. Add a CVE scanner methods to check if an entry is present in the cache, and to retreive the results
5. Modify the FilterTags MetaDB method to not exit on first error
This is needed in order to pass all tags to the generator,
instead of the generator stopping at the first set of invalid data
6. Integrate the new scanning task generator with the existing zot code.
7. Fix an issue where the CVE scan results for multiarch images was not cached
8. Rewrite some of the older CVE tests to use the new image-utils test package
9. Use the CVE scanner as attribute of the controller instead of CveInfo.
Remove functionality of CVE DB update from CveInfo, it is now responsible,
as the name states, only for providing CVE information.
10. The logic to get maximum severity and cve count for image sumaries now uses only the scanner cache.
11. Removed the GetCVESummaryForImage method from CveInfo as it was only used in tests

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-09-22 11:49:17 -07:00
Alexei Dodon
4e04be420e
refactor(cli): Move cmdflags package under pkg/cli/client (#1840)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-09-22 16:33:18 +03:00
Andrei Aaron
bcdd9988f5
fix(cve): cummulative fixes and improvements for CVE scanning logic (#1810)
1. Only scan CVEs for images returned by graphql calls
Since pagination was refactored to account for image indexes, we had started
to run the CVE scanner before pagination was applied, resulting in
decreased ZOT performance if CVE information was requested

2. Increase in medory-cache of cve results to 1m, from 10k digests.

3. Update CVE model to use CVSS severity values in our code.
Previously we relied upon the strings returned by trivy directly,
and the sorting they implemented.
Since CVE severities are standardized, we don't need to pass around
an adapter object just for pagination and sorting purposes anymore.
This also improves our testing since we don't mock the sorting functions anymore.

4. Fix a flaky CLI test not waiting for the zot service to start.

5. Add the search build label on search/cve tests which were missing it.

6. The boltdb update method was used in a few places where view was supposed to be called.

7. Add logs for start and finish of parsing MetaDB.

8. Avoid unmarshalling twice to obtain annotations for multiarch images.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-09-17 15:12:20 -07:00
Alexei Dodon
f58597ade9
refactor: Reduce zli binary size (#1805)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-09-15 15:17:01 -07:00
LaurentiuNiculae
8e18917b07
refactor(test): move image utils for tests in a separate module (#1789)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-09-15 09:53:15 -07:00
LaurentiuNiculae
aae8b7b4e3
feat(cli): add sort-by flag to sub commands (#1768)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-09-14 10:51:17 -07:00
Alexei Dodon
48bf7f69f8
refactor: Reduce zb binary size (#1783)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-09-13 10:28:14 +03:00
LaurentiuNiculae
7b1e24c99e
refactor(cli): remove old cli commands (#1756)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-09-08 15:12:47 +03:00
Alexei Dodon
f5b63963be
refactor: Reduce binary size of zot-minimal; Added CI check for binary size (#1758)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-09-06 19:58:00 +03:00
peusebiu
59dc4c3229
feat(scheduler): pass the shutdown/reload ctx to running tasks (#1671)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-05 09:48:56 -07:00
peusebiu
b80deb9927
refactor(storage): refactor storage into a single ImageStore (#1656)
unified both local and s3 ImageStore logic into a single ImageStore
added a new driver interface for common file/dirs manipulations
to be implemented by different storage types

refactor(gc): drop umoci dependency, implemented internal gc

added retentionDelay config option that specifies
the garbage collect delay for images without tags

this will also clean manifests which are part of an index image
(multiarch) that no longer exist.

fix(dedupe): skip blobs under .sync/ directory

if startup dedupe is running while also syncing is running
ignore blobs under sync's temporary storage

fix(storage): do not allow image indexes modifications

when deleting a manifest verify that it is not part of a multiarch image
and throw a MethodNotAllowed error to the client if it is.
we don't want to modify multiarch images

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-01 10:54:39 -07:00
LaurentiuNiculae
112fbec5b6
refactor(cli): added equivalent subcommands for each flag combination under every command (#1674)
- image command is now deprecated in favor of 'images'
- cve command is now deprecated in favor of 'cves'

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-08-30 20:12:24 +03:00
LaurentiuNiculae
40b599cca8
refactor(log): replace default logger with config complient log in root.go (#1734)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-08-29 12:10:30 -07:00
Andrei Aaron
28858f695f
feat(mgmt): mgmt extention no longer depends on UI being enabled (#1728)
It is now enabled based only on search configuration

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-08-26 13:32:41 -07:00
Damien Degois
289acfabbd
feat(authn): add generic oidc and allow customizable name (#1691)
Rebased and squashed

Signed-off-by: Damien Degois <damien@degois.info>
2023-08-24 12:33:35 +03:00
LaurentiuNiculae
0731fd3828
test(refactor): refactor tests that use zot-test to use smaller images (#1690)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-08-18 11:46:11 +03:00
Andreea Lupu
76277f5ebd
fix: remove inline GC and schedule a background task instead (#1610)
* fix: remove inline GC and set a default value of gc interval

- remove inline GC
- add a default value of GC interval
- run the GC periodically by default with the default value if no interval provided
- generate GC tasks with a random delay(0-30s) between
- add IsReady() method to scheduler.TaskGenerator interface

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>

* ci: add test for gc with short interval

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>

---------

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-08-07 12:55:19 -07:00
LaurentiuNiculae
fce9a02ed5
fix(convert): now returned annotations for an index will fallback to annotations from a random manifest if the annotations field is not present on the index manifest (#1667)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-08-07 10:25:05 +03:00
Andrei Aaron
77149aa85c
refactor(extensions)!: refactor the extensions URLs and errors (#1636)
BREAKING CHANGE: The functionality provided by the mgmt endpoint has beed redesigned - see details below
BREAKING CHANGE: The API keys endpoint has been moved -  see details below
BREAKING CHANGE: The mgmt extension config has been removed - endpoint is now enabled by having both the search and the ui extensions enabled
BREAKING CHANGE: The API keys configuration has been moved from extensions to http>auth>apikey

mgmt and imagetrust extensions:
- separate the _zot/ext/mgmt into 3 separate endpoints: _zot/ext/auth, _zot/ext/notation, _zot/ext/cosign
- signature verification logic is in a separate `imagetrust` extension
- better hanling or errors in case of signature uploads: logging and error codes (more 400 and less 500 errors)
- add authz on signature uploads (and add a new middleware in common for this purpose)
- remove the mgmt extension configuration - it is now enabled if the UI and the search extensions are enabled

userprefs estension:
- userprefs are enabled if both search and ui extensions are enabled (as opposed to just search)

apikey extension is removed and logic moved into the api folder
- Move apikeys code out of pkg/extensions and into pkg/api
- Remove apikey configuration options from the extensions configuration and move it inside the http auth section
- remove the build label apikeys

other changes:
- move most of the logic adding handlers to the extensions endpoints out of routes.go and into the extensions files.
- add warnings in case the users are still using configurations with the obsolete settings for mgmt and api keys
- add a new function in the extension package which could be a single point of starting backgroud tasks for all extensions
- more clear methods for verifying specific extensions are enabled
- fix http methods paired with the UI handlers
- rebuild swagger docs

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-08-02 21:58:34 +03:00