0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2024-12-16 21:56:37 -05:00
Commit graph

26 commits

Author SHA1 Message Date
peusebiu
6222dae1f0
feat(scheduler): gracefully shutdown (#1951)
wait for workers to finish before exiting

should fix tests reporting they couldn't remove rootDir because it's being
written by tasks

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-11-24 00:40:10 -08:00
Alexei Dodon
dd079bf9a3
fix: TestPopulateStorageMetrics fails occasionally in CI (#2042)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-11-14 16:22:24 -08:00
Andreea Lupu
d5065513f5
feat: add support for oci1.1 cosign signatures(using referrers) (#1963)
- Cosign supports 2 types of signature formats:

	1. Using tag -> each new signature of the same manifest is
	added as a new layer of the signature manifest having that
	specific tag("{alghoritm}-{digest_of_signed_manifest}.sig")

	2. Using referrers -> each new signature of the same manifest is
	added as a new manifest

- For adding these cosign signature to metadb, we reserved index 0 of the
list of cosign signatures for tag-based signatures. When a new tag-based
signature is added for the same manifest, the element on first position
in its list of cosign signatures(in metadb) will be updated/overwritten.
When a new cosign signature(using referrers) will be added for the same
manifest this new signature will be appended to the list of cosign
signatures.

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-11-06 14:09:39 -08:00
Alexei Dodon
a79d79a03a
fix: more accurate storage metrics after zot restart (#1972)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-11-01 09:09:21 -07:00
peusebiu
7ab2032a21
feat(api): repair corrupted blobs when pushed again (#1927)
CheckBlob() returns ErrBlobNotFound on corrupted blobs

closes #1922

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-10-20 13:02:31 -07:00
Andrei Aaron
ba6f347d8d
refactor(pkg/test): split logic in pkg/test/common.go into multiple packages (#1861)
Which could be imported independently. See more details:
1. "zotregistry.io/zot/pkg/test/common" - currently used as
   tcommon "zotregistry.io/zot/pkg/test/common" - inside pkg/test
   test "zotregistry.io/zot/pkg/test/common" - in tests
   . "zotregistry.io/zot/pkg/test/common" - in tests
Decouple zb from code in test/pkg in order to keep the size small.

2. "zotregistry.io/zot/pkg/test/image-utils" - curently used as
   . "zotregistry.io/zot/pkg/test/image-utils"

3. "zotregistry.io/zot/pkg/test/deprecated" -  curently used as
   "zotregistry.io/zot/pkg/test/deprecated"
This one will bre replaced gradually by image-utils in the future.

4. "zotregistry.io/zot/pkg/test/signature" - (cosign + notation) use as
   "zotregistry.io/zot/pkg/test/signature"

5. "zotregistry.io/zot/pkg/test/auth" - (bearer + oidc)  curently used as
   authutils "zotregistry.io/zot/pkg/test/auth"

 6. "zotregistry.io/zot/pkg/test/oci-utils" -  curently used as
   ociutils "zotregistry.io/zot/pkg/test/oci-utils"

Some unused functions were removed, some were replaced, and in
a few cases specific funtions were moved to the files they were used in.

Added an interface for the StoreController, this reduces the number of imports
of the entire image store, decreasing binary size for tests.
If the zb code was still coupled with pkg/test, this would have reflected in zb size.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-09-27 11:34:48 -07:00
peusebiu
c3801dc3d3
fix(dedupe): run dedupe only for repositories found at startup (#1844)
no need to run dedupe/restore blobs for images being pushed or synced while
running dedupe task, they are already deduped/restored inline.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-27 11:15:08 -07:00
peusebiu
1df743f173
fix(gc): sync repodb when gc'ing manifests (#1819)
fix(gc): fix cleaning deduped blobs because they have the modTime of
the original blobs, fixed by updating the modTime when hard linking
the blobs.
fix(gc): failing to parse rootDir at zot startup when using s3 storage
because there are no files under rootDir and we can not create empty dirs
on s3, fixed by creating an empty file under rootDir.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-22 11:51:20 -07:00
LaurentiuNiculae
8e18917b07
refactor(test): move image utils for tests in a separate module (#1789)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-09-15 09:53:15 -07:00
Alexei Dodon
f5b63963be
refactor: Reduce binary size of zot-minimal; Added CI check for binary size (#1758)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-09-06 19:58:00 +03:00
peusebiu
59dc4c3229
feat(scheduler): pass the shutdown/reload ctx to running tasks (#1671)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-05 09:48:56 -07:00
peusebiu
a0290b4b37
fix(gc): gc removes unknown manifests (#1762)
without removing its index.json reference

fix that by also reporting if manifests with unknown mediatypes
are referenced in index.json

this will make gc delete manifest blobs with deleteImageManifest() method
instead of deleteBlob(), which also removes index.json entries.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-05 09:42:12 -07:00
Ramkumar Chinchani
8e36bfd4d1
fix: add manifest validation checks (#1747)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-09-02 01:28:31 -07:00
peusebiu
b80deb9927
refactor(storage): refactor storage into a single ImageStore (#1656)
unified both local and s3 ImageStore logic into a single ImageStore
added a new driver interface for common file/dirs manipulations
to be implemented by different storage types

refactor(gc): drop umoci dependency, implemented internal gc

added retentionDelay config option that specifies
the garbage collect delay for images without tags

this will also clean manifests which are part of an index image
(multiarch) that no longer exist.

fix(dedupe): skip blobs under .sync/ directory

if startup dedupe is running while also syncing is running
ignore blobs under sync's temporary storage

fix(storage): do not allow image indexes modifications

when deleting a manifest verify that it is not part of a multiarch image
and throw a MethodNotAllowed error to the client if it is.
we don't want to modify multiarch images

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-01 10:54:39 -07:00
Alexei Dodon
247f6dcd3f
feat: propagate detailed error msgs to client (OCI dist-spec format) (#1681)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-08-23 20:59:52 +03:00
Andreea Lupu
cacf54e8cb
refactor: move /pkg/meta/signatures under /pkg/extensions/imagetrust (#1712)
- the size of the binary-minimal becomes 32MB
- "signatures" package is renamed into "imagetrust" and moved under extensions
- if the binary is not built using "imagetrust" tag then the signatures verification will
not be performed

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-08-19 08:52:03 +03:00
LaurentiuNiculae
0731fd3828
test(refactor): refactor tests that use zot-test to use smaller images (#1690)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-08-18 11:46:11 +03:00
Andreea Lupu
76277f5ebd
fix: remove inline GC and schedule a background task instead (#1610)
* fix: remove inline GC and set a default value of gc interval

- remove inline GC
- add a default value of GC interval
- run the GC periodically by default with the default value if no interval provided
- generate GC tasks with a random delay(0-30s) between
- add IsReady() method to scheduler.TaskGenerator interface

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>

* ci: add test for gc with short interval

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>

---------

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-08-07 12:55:19 -07:00
peusebiu
d3f27b4ba6
feat: add additional manifest validations (#1609)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-13 09:31:39 -07:00
peusebiu
cda6916b45
fix: don't allow blobs to be deleted if in use (#1559)
dist-spec APIs independently allow deletion of blobs and manifests.
Doing the former when in use by an image manifest or index is simply
error-prone. So disallow it.

Fixes issue #1509

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-07-10 12:24:45 +03:00
peusebiu
5494a1b8d6
fix(storage): do not open/download blobs when validating manifests (#1566)
when pushing manifests, zot will validate blobs (layers + config blob) are
present in repo, currently it opens(in case of filesystem storage) or download(
in case of cloud storage) each blob.

fixed that by adding a new method ImageStore.CheckBlobPresence() on storage
to check blobs presence without checking the cache like ImageStore.CheckBlob() method does.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-06 10:33:36 -07:00
LaurentiuNiculae
96d9d318df
feat(referrers): added index support for referrers queries (#1560)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-07-05 09:42:16 -07:00
Lisca Ana-Roberta
d4f200c2e1
fix: changing default numWorkers, making it customizable and refactoring scheduler (#1563)
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2023-07-04 11:03:29 +03:00
Ramkumar Chinchani
1300fdfa88
refactor: filenames should use _ not - (#1547)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-06-22 11:54:41 -07:00
Ramkumar Chinchani
2202d6dfd4
fix: revert "org.opencontainers.referrers.filtersApplied" (#1478)
As per latest dist-spec, this is now removed.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-05-26 13:56:44 -07:00
LaurentiuNiculae
a3f355c278
refactor(storage): refactoring storage (#1459)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-26 11:08:19 -07:00