0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2024-12-16 21:56:37 -05:00
Commit graph

118 commits

Author SHA1 Message Date
peusebiu
9074f8483b
feat(retention): added image retention policies (#1866)
feat(metaDB): add more image statistics info

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-11-01 09:16:18 -07:00
Alexei Dodon
a79d79a03a
fix: more accurate storage metrics after zot restart (#1972)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-11-01 09:09:21 -07:00
LaurentiuNiculae
56ad9e6707
refactor(metadb): improve UX by speeding up metadb serialize/deserialize (#1842)
Use protocol buffers and update the metadb interface to better suit our search needs

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-10-30 13:06:04 -07:00
Alexei Dodon
d2fbd273ba
fix: tests refactoring (#1950)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-10-26 11:20:39 +03:00
peusebiu
7ab2032a21
feat(api): repair corrupted blobs when pushed again (#1927)
CheckBlob() returns ErrBlobNotFound on corrupted blobs

closes #1922

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-10-20 13:02:31 -07:00
Ramkumar Chinchani
1675f30d4a
ci: update golangci-lint version (#1834)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-10-20 17:27:04 +03:00
Alexei Dodon
a345ba0823
fix: metrics should be protected behind authZ (#1895)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-10-20 10:33:26 +03:00
peusebiu
7f6534a52d
fix(sessions): periodically cleanup expired sessions (#1939)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-10-16 20:03:42 -07:00
Andreea Lupu
fc2380b57b
fix: add support for uploaded index when signing using notation (#1882)
ci(notation): update to latest notation version
fix(sync): add layers info when syncing signatures

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-10-12 18:45:20 -07:00
Alexei Dodon
044ea85279
fix: running tests locally fails (#1879)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-10-05 14:34:50 +03:00
Andrei Aaron
99e29c0f46
refactor(tests): Migrate some of the older tests to the new image-utils library (#1863)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-10-02 11:10:43 -07:00
Andrei Aaron
ba6f347d8d
refactor(pkg/test): split logic in pkg/test/common.go into multiple packages (#1861)
Which could be imported independently. See more details:
1. "zotregistry.io/zot/pkg/test/common" - currently used as
   tcommon "zotregistry.io/zot/pkg/test/common" - inside pkg/test
   test "zotregistry.io/zot/pkg/test/common" - in tests
   . "zotregistry.io/zot/pkg/test/common" - in tests
Decouple zb from code in test/pkg in order to keep the size small.

2. "zotregistry.io/zot/pkg/test/image-utils" - curently used as
   . "zotregistry.io/zot/pkg/test/image-utils"

3. "zotregistry.io/zot/pkg/test/deprecated" -  curently used as
   "zotregistry.io/zot/pkg/test/deprecated"
This one will bre replaced gradually by image-utils in the future.

4. "zotregistry.io/zot/pkg/test/signature" - (cosign + notation) use as
   "zotregistry.io/zot/pkg/test/signature"

5. "zotregistry.io/zot/pkg/test/auth" - (bearer + oidc)  curently used as
   authutils "zotregistry.io/zot/pkg/test/auth"

 6. "zotregistry.io/zot/pkg/test/oci-utils" -  curently used as
   ociutils "zotregistry.io/zot/pkg/test/oci-utils"

Some unused functions were removed, some were replaced, and in
a few cases specific funtions were moved to the files they were used in.

Added an interface for the StoreController, this reduces the number of imports
of the entire image store, decreasing binary size for tests.
If the zb code was still coupled with pkg/test, this would have reflected in zb size.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-09-27 11:34:48 -07:00
peusebiu
c3801dc3d3
fix(dedupe): run dedupe only for repositories found at startup (#1844)
no need to run dedupe/restore blobs for images being pushed or synced while
running dedupe task, they are already deduped/restored inline.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-27 11:15:08 -07:00
Andreea Lupu
92e382ce39
refactor(scrub): replace umoci logic in scrub implementation (#1845)
- implement scrub also for S3 storage by replacing umoci
- change scrub implementation for ImageIndex
- take the `Subject` into consideration when running scrub
- remove test code relying on the umoci library. Since we started
relying on images in test/data, and we create our own images using
go code we can obtain digests by other means. (cherry picked from commit 489d4e2d23c1b4e48799283f8281024bbef6123f)

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-09-26 11:02:11 -07:00
peusebiu
1df743f173
fix(gc): sync repodb when gc'ing manifests (#1819)
fix(gc): fix cleaning deduped blobs because they have the modTime of
the original blobs, fixed by updating the modTime when hard linking
the blobs.
fix(gc): failing to parse rootDir at zot startup when using s3 storage
because there are no files under rootDir and we can not create empty dirs
on s3, fixed by creating an empty file under rootDir.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-22 11:51:20 -07:00
Andrei Aaron
7c78f80a96
feat(cve): implement CVE scanning as background tasks (#1833)
1. Move existing CVE DB download generator/task login under the cve package
2. Add a new CVE scanner task generator and task type to run in the background, as well as tests for it
3. Move the CVE cache in its own package
4. Add a CVE scanner methods to check if an entry is present in the cache, and to retreive the results
5. Modify the FilterTags MetaDB method to not exit on first error
This is needed in order to pass all tags to the generator,
instead of the generator stopping at the first set of invalid data
6. Integrate the new scanning task generator with the existing zot code.
7. Fix an issue where the CVE scan results for multiarch images was not cached
8. Rewrite some of the older CVE tests to use the new image-utils test package
9. Use the CVE scanner as attribute of the controller instead of CveInfo.
Remove functionality of CVE DB update from CveInfo, it is now responsible,
as the name states, only for providing CVE information.
10. The logic to get maximum severity and cve count for image sumaries now uses only the scanner cache.
11. Removed the GetCVESummaryForImage method from CveInfo as it was only used in tests

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-09-22 11:49:17 -07:00
Andrei Aaron
bcdd9988f5
fix(cve): cummulative fixes and improvements for CVE scanning logic (#1810)
1. Only scan CVEs for images returned by graphql calls
Since pagination was refactored to account for image indexes, we had started
to run the CVE scanner before pagination was applied, resulting in
decreased ZOT performance if CVE information was requested

2. Increase in medory-cache of cve results to 1m, from 10k digests.

3. Update CVE model to use CVSS severity values in our code.
Previously we relied upon the strings returned by trivy directly,
and the sorting they implemented.
Since CVE severities are standardized, we don't need to pass around
an adapter object just for pagination and sorting purposes anymore.
This also improves our testing since we don't mock the sorting functions anymore.

4. Fix a flaky CLI test not waiting for the zot service to start.

5. Add the search build label on search/cve tests which were missing it.

6. The boltdb update method was used in a few places where view was supposed to be called.

7. Add logs for start and finish of parsing MetaDB.

8. Avoid unmarshalling twice to obtain annotations for multiarch images.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-09-17 15:12:20 -07:00
LaurentiuNiculae
8e18917b07
refactor(test): move image utils for tests in a separate module (#1789)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-09-15 09:53:15 -07:00
LaurentiuNiculae
aae8b7b4e3
feat(cli): add sort-by flag to sub commands (#1768)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-09-14 10:51:17 -07:00
LaurentiuNiculae
c210e3f377
fix(convert): fix the update rule of download count for images (#1802)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-09-14 10:48:23 -07:00
Alexei Dodon
48bf7f69f8
refactor: Reduce zb binary size (#1783)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-09-13 10:28:14 +03:00
Andreea Lupu
68a197d7cc
fix: add retry logic to recreate existing trustpolicy secret (#1776)
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-09-11 10:13:22 +03:00
Andreea Lupu
5a3fac40db
feat: upload cosign public key and notation certificates to cloud (#1744)
- using secrets manager for storing public keys and certificates
- adding a default truststore for notation verification and upload all certificates to this default truststore
- removig `truststoreName` query param from notation api for uploading certificates


(cherry picked from commit eafcc1a213)

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-09-08 10:03:58 +03:00
Alexei Dodon
f5b63963be
refactor: Reduce binary size of zot-minimal; Added CI check for binary size (#1758)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-09-06 19:58:00 +03:00
peusebiu
b80deb9927
refactor(storage): refactor storage into a single ImageStore (#1656)
unified both local and s3 ImageStore logic into a single ImageStore
added a new driver interface for common file/dirs manipulations
to be implemented by different storage types

refactor(gc): drop umoci dependency, implemented internal gc

added retentionDelay config option that specifies
the garbage collect delay for images without tags

this will also clean manifests which are part of an index image
(multiarch) that no longer exist.

fix(dedupe): skip blobs under .sync/ directory

if startup dedupe is running while also syncing is running
ignore blobs under sync's temporary storage

fix(storage): do not allow image indexes modifications

when deleting a manifest verify that it is not part of a multiarch image
and throw a MethodNotAllowed error to the client if it is.
we don't want to modify multiarch images

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-01 10:54:39 -07:00
LaurentiuNiculae
112fbec5b6
refactor(cli): added equivalent subcommands for each flag combination under every command (#1674)
- image command is now deprecated in favor of 'images'
- cve command is now deprecated in favor of 'cves'

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-08-30 20:12:24 +03:00
peusebiu
6926bddd3a
feat(apikey): added route to list user api keys (#1708)
adding api key expiration date

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-08-29 09:38:38 -07:00
LaurentiuNiculae
0731fd3828
test(refactor): refactor tests that use zot-test to use smaller images (#1690)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-08-18 11:46:11 +03:00
Andrei Aaron
77149aa85c
refactor(extensions)!: refactor the extensions URLs and errors (#1636)
BREAKING CHANGE: The functionality provided by the mgmt endpoint has beed redesigned - see details below
BREAKING CHANGE: The API keys endpoint has been moved -  see details below
BREAKING CHANGE: The mgmt extension config has been removed - endpoint is now enabled by having both the search and the ui extensions enabled
BREAKING CHANGE: The API keys configuration has been moved from extensions to http>auth>apikey

mgmt and imagetrust extensions:
- separate the _zot/ext/mgmt into 3 separate endpoints: _zot/ext/auth, _zot/ext/notation, _zot/ext/cosign
- signature verification logic is in a separate `imagetrust` extension
- better hanling or errors in case of signature uploads: logging and error codes (more 400 and less 500 errors)
- add authz on signature uploads (and add a new middleware in common for this purpose)
- remove the mgmt extension configuration - it is now enabled if the UI and the search extensions are enabled

userprefs estension:
- userprefs are enabled if both search and ui extensions are enabled (as opposed to just search)

apikey extension is removed and logic moved into the api folder
- Move apikeys code out of pkg/extensions and into pkg/api
- Remove apikey configuration options from the extensions configuration and move it inside the http auth section
- remove the build label apikeys

other changes:
- move most of the logic adding handlers to the extensions endpoints out of routes.go and into the extensions files.
- add warnings in case the users are still using configurations with the obsolete settings for mgmt and api keys
- add a new function in the extension package which could be a single point of starting backgroud tasks for all extensions
- more clear methods for verifying specific extensions are enabled
- fix http methods paired with the UI handlers
- rebuild swagger docs

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-08-02 21:58:34 +03:00
LaurentiuNiculae
9e38ca51e3
feat(pagination): move pagination and sorting image summary results after conversion (#1637)
fix(config): check for config media type when pushing to repodb

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-07-31 22:16:09 +03:00
LaurentiuNiculae
a5c92bcac3
test(refactor): change upload image function signature (#1655)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-07-28 17:53:46 +03:00
LaurentiuNiculae
abba6aa3cf
feat(test): added image builder for use in tests (#1583)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-07-26 13:08:04 +03:00
LaurentiuNiculae
28de980319
feat(refator): refactoring repodb into meta (#1626)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-07-18 10:27:26 -07:00
Andrei Aaron
0a5241b025
fix(ci): trivy test to expect 3 CVEs instead of 2 in the vulnerable test layer (#1623)
Trivy started detecting CVE-2023-2975, so one of the checks expecting only 2 CVEs was failing

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-07-16 08:27:59 -07:00
peusebiu
d3f27b4ba6
feat: add additional manifest validations (#1609)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-13 09:31:39 -07:00
Andrei Aaron
418a1a006c
feat(cve): ability to return CVEs per image os and architecture (#1607)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-07-11 09:29:04 -07:00
peusebiu
17d1338af1
feat: integrate openID auth logic and user profile management (#1381)
This change introduces OpenID authn by using providers such as Github,
Gitlab, Google and Dex.
User sessions are now used for web clients to identify
and persist an authenticated users session, thus not requiring every request to
use credentials.
Another change is apikey feature, users can create/revoke their api keys and use them
to authenticate when using cli clients such as skopeo.

eg:
login:
/auth/login?provider=github
/auth/login?provider=gitlab
and so on

logout:
/auth/logout

redirectURL:
/auth/callback/github
/auth/callback/gitlab
and so on

If network policy doesn't allow inbound connections, this callback wont work!

for more info read documentation added in this commit.

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
2023-07-07 09:27:10 -07:00
peusebiu
5494a1b8d6
fix(storage): do not open/download blobs when validating manifests (#1566)
when pushing manifests, zot will validate blobs (layers + config blob) are
present in repo, currently it opens(in case of filesystem storage) or download(
in case of cloud storage) each blob.

fixed that by adding a new method ImageStore.CheckBlobPresence() on storage
to check blobs presence without checking the cache like ImageStore.CheckBlob() method does.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-06 10:33:36 -07:00
LaurentiuNiculae
0a04b2a4ed
feat(cve): implemented trivy image scan for multiarch images (#1510)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-07-06 11:36:26 +03:00
LaurentiuNiculae
620287c7a4
feat(cli): add referrers and search commands to cli (#1497)
* feat(cli): add referrers command to cli

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(cli): add global search command

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(cli): fix comments

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

---------

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-06-22 10:43:01 -07:00
Ramkumar Chinchani
4d6ca493f2
chore: fix dependabot alerts (#1501)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-06-09 10:27:42 -07:00
Andrei Aaron
96d00cd0ef
fix(cve): Fix CVE scanning in images containing Jar files (#1475) 2023-06-01 00:37:46 +03:00
peusebiu
612a12e5a8
refactor(sync): use task scheduler (#1301)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-05-31 10:26:23 -07:00
LaurentiuNiculae
a3f355c278
refactor(storage): refactoring storage (#1459)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-26 11:08:19 -07:00
Lisca Ana-Roberta
6a7035c599
fix: removed duplicate structures from service.go and moved them to pkg/common (#1436)
Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
2023-05-25 11:27:49 -07:00
Andreea Lupu
970997f3a8
feat(graphql & repodb): add info about signature validity (#1344)
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-05-24 09:46:16 -07:00
LaurentiuNiculae
6e6ffe800c
chore(go.mod): upgrade to notation-go v1.0.0-rc.5 and image-spec v1.1.0-rc3 (#1468)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-23 15:16:33 +00:00
LaurentiuNiculae
7bf40e7308
fix(sync): fixed way of updating repodb when syncing a signature (#1439)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-15 12:02:23 +03:00
LaurentiuNiculae
7d7bc9d5e4
feat(api): added oci-subject header when pushing an image with subject field (#1415)
- as requested by the latest version of the oci distribution spec

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-12 09:32:01 -07:00
Ramkumar Chinchani
9534e0b88b
chore: fix dependabot alerts (#1409)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-05-11 16:39:21 -07:00