0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2024-12-16 21:56:37 -05:00
Commit graph

63 commits

Author SHA1 Message Date
Alexei Dodon
ad519e2d3e Leave zot repositories in a consistent state after zot hits fd limit closes #359
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2022-03-31 13:25:15 -07:00
Ramkumar Chinchani
b2a4388522 gc: add a unit test
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-03-24 12:02:52 -07:00
Petu Eusebiu
f53dc9eb8d sync: Add a new flag to enforce syncing only signed images, closes #455
sync: When checking if a image is already synced also check for changes in upstream signatures.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-03-24 10:50:01 -07:00
Eng Zer Jun
0d77b60de7 test: use T.TempDir to create temporary test directory
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-03-07 12:28:49 -08:00
Petu Eusebiu
9cffbcaccb s3: bugfix, use sync.Map instead of map for storing multi part uploads references
add storage lock in GetIndexContent

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-03-03 09:12:07 -08:00
Petu Eusebiu
45968e0bb7 sync: fix inconsistent test, used inject error fw for hard to reach test cases
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-03-01 09:56:27 -08:00
Ramkumar Chinchani
38a110314b gc: add a gcDelay param
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-14 14:03:22 -08:00
Ramkumar Chinchani
d2aa016cdb storage: flush/sync contents to disk on file close
Behavior controlled by configuration (default=off)
It is a trade-off between performance and consistency.

References:
[1] https://github.com/golang/go/issues/20599

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-07 12:08:01 -08:00
Ramkumar Chinchani
c73e71b018 ci/cd: add a basic push-pull testing using client tools
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-07 10:03:24 -08:00
Ramkumar Chinchani
87084f286b storage: improve/fix oci image validation
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-02 13:31:41 -08:00
Ramkumar Chinchani
45fe129c63 notaryv2: fix 'notation list'
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-01-31 14:33:21 -08:00
Ramkumar Chinchani
4d576a4817 storage: return errors instead of panics
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-01-24 13:33:45 -08:00
Petu Eusebiu
1109bb4dde sync: Added support for syncing notary/cosign signatures, closes #261
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-01-13 08:45:59 -08:00
Alexei Dodon
c4d34b7269 Added storage latency histogram metric
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2022-01-10 11:55:39 -08:00
Ramkumar Chinchani
cac7fe4854 storage: use sha256-simd from minio
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-28 22:25:11 -08:00
Ramkumar Chinchani
ac3801ea2d lint: upgrade golangci-lint
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-20 17:20:35 -08:00
Andreea-Lupu
c61c3836db implement scrub to check manifest/blob integrity
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2021-12-09 11:18:09 -08:00
Ramkumar Chinchani
96226af869 move references to zotregistry.io and project-zot
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-05 10:52:27 -08:00
Ramkumar Chinchani
e42e42a2cc artifacts: initial support for artifacts/notaryv2 spec
https://github.com/oras-project/artifacts-spec
https://github.com/notaryproject/notaryproject

Fixes issue #264

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-01 18:55:39 -08:00
Alexei Dodon
e900b09cfb Fix data races in tests, closes #255
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2021-11-17 13:23:59 -08:00
Petu Eusebiu
9c568c0ee2 storage: add s3 backend support (without GC and dedupe)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-11-15 08:09:00 -08:00
Alexei Dodon
8e4d828867 Implement an API for performance monitoring
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2021-11-12 11:14:10 -08:00
Alexei Dodon
f76c76c2e6 Added a timeout option in db cache lock, closes #242
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2021-10-27 12:43:50 -07:00
Alexei Dodon
d8aa5b8bf3
Fixing ValidateHardLink, closes #256 (#257)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2021-10-27 20:03:26 +03:00
Petu Eusebiu
7d077eaf5a Added storage interface 2021-10-11 17:38:46 -07:00
Shivam Mishra
c6670b1329 api: implement delete by tag 2021-08-23 17:30:41 -07:00
Ramkumar Chinchani
63b88d0e57 pkg/storage: fix partially initialized repo storage
Thanks shimish2 for the unit test.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-08-09 23:19:20 -07:00
Shivam Mishra
53b5fa6493 dedupe: stat blob path before creating link 2021-08-09 09:40:35 -07:00
Shivam Mishra
af30c06aff api: use blob cache path while making hard link
previously mount blob will look for blob that is provided in http request and try to hard link that path
but ideally we should look for path from our cache and do the hard link of that particular path.
this commit does the same.
2021-06-30 01:42:21 -07:00
Shivam Mishra
28974e81dc config: support multiple storage locations
added support to point multiple storage locations in zot by running multiple instance of zot in background.

see examples/config-multiple.json for more info about config.

Closes #181
2021-05-21 10:18:28 -07:00
Shivam Mishra
a7c17b7c16 spec: added support for mount request using hard link 2021-05-04 09:42:29 -07:00
Ramkumar Chinchani
f7829d6470 dedupe: optimize check-blob with hard links
In use cases, when there are large images with shared layers across
repositories, clients may benefit from not re-uploading the same blobs
over and over again.

We ensure this by hard linking when check-blob api is called.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-05-04 09:42:29 -07:00
Shivam Mishra
2b7b57313a conformance: fix http status code for cross-repository mounting 2021-01-29 09:35:15 -08:00
Ramkumar Chinchani
affdd85986 build: remove bazel
The idea initially was to use bazel to do our builds, however golang
build system is now good enough and our code base is entirely go.
It is also slowing down our travis ci/cd pipeline.
2020-12-21 15:30:13 -08:00
Ramkumar Chinchani
386c72d332 routes: refactor locks to handle large file uploads
The storage layer is protected with read-write locks.
However, we may be holding the locks over unnecessarily large critical
sections.

The typical workflow is that a blob is first uploaded via a per-client
private session-id meaning the blob is not publicly visible yet. When
the blob being uploaded is very large, the transfer takes a long time
while holding the lock.

Private session-id based uploads don't really need locks, and hold locks
only when blobs are published after the upload is complete.
2020-10-16 13:33:11 -07:00
Shivam Mishra
3a30290e08 Using "destRecord" as a path in DeleteBlob function instead of "dst".
dstRecord :- blob path stored in cache.
dst :- blob path that is trying to be uploaded.

Currently, if the actual blob on disk may have been removed by GC/delete, during syncing the cache dst is being passed to DeleteBlob function and retry section is being continuously called because DeleteBlob function never deletes dst path (doesn't exist in db), dstRecord should be passed into DeleteBlob function because dstRecord is actual blob path stored in db.

If dst and dstRecord path value is same then this issue will not be produced and DeleteBlob method will delete the blob info from cache but if both are different then DeleteBlob method will try to delete dst path which is not in cache.

Note:- boltdb delete method return nil even when value doesn't exist (https://godoc.org/github.com/boltdb/bolt#Bucket.Delete)
2020-08-12 10:06:20 -07:00
Ramkumar Chinchani
adc6859cd6 stacker: fix stacker build 2020-07-14 13:31:57 -07:00
Ramkumar Chinchani
324a517ea3 gc: add a policy to skip garbage collecting new blobs
We perform inline garbage collection of orphan blobs. However, the
dist-spec poses a problem because blobs begin their life as orphan blobs
and then a manifest is add which refers to these blobs.

We use umoci's GC() to perform garbage collection and policy support
has been added recently which can control whether a blob can be skipped
for GC.

In this patch, we use a time-based policy to skip blobs.
2020-07-06 15:52:35 -07:00
Shivam Mishra
af77876306 Upgraded build pipeline
Go version changed to 1.14.4
Golangci-lint changed to 1.26.0
Bazel version changed to 3.0.0
Bazel rules_go version changed to 0.23.3
Bazel gazelle version changed to v0.21.0
Bazel build tools version changed to 0.25.1
Bazel skylib version changed to 1.0.2
2020-06-25 23:43:31 -07:00
Shivam Mishra
85d3e1db4b Changed umoci import path 2020-06-25 17:04:32 -07:00
Ramkumar Chinchani
3dc9885ee9 update the size field when existing manifest entry is updated
An existing manifest descriptor in index.json can be updated with
different manifest contents for the same/existing tag. We were updating
the digest but not the size field causing GC to report an error.

Add a unit test case to cover this.

Add logs.
2020-06-18 16:20:43 -07:00
Ramkumar Chinchani
cb9e773a3e dedupe: record relative path for cache entries
In a production use case we found that the actual rootdir can be moved.
Currently, cache entries for dedupe record the full blob path which
doesn't work in the move use case.

Only for dedupe cache entries, record relative blob paths.
2020-05-27 22:11:26 -07:00
Ramkumar Chinchani
dd1fc1e866 config: add gc and dedupe as configurable params (default to enabled)
Since we want to conform to dist-spec, sometimes the gc and dedupe
optimizations conflict with the conformance tests that are being run.
So allow them to be turned off via configuration params.
2020-04-16 16:01:53 -07:00
Ramkumar Chinchani
b1f882e1b8 conformance: align with upstream conformance tests
Upstream conformance tests are being updated, so we need to align along
with our internal GC and dedupe features.

Add a new example config file which plays nice with conformance tests.

DeleteImageManifest() updated to deal with the case where the same
manifest can be created with multiple tags and deleted with the same
digest - so all entries must be deleted.

DeleteBlob() delete the digest key (bucket) when last reference is
dropped
2020-04-16 16:01:53 -07:00
Ramkumar Chinchani
25f5a45296 dedupe: use hard links to dedupe blobs
As the number of repos and layers increases, the greater the probability
that layers are duplicated. We dedupe using hard links when content is
the same. This is intended to be purely a storage layer optimization.
Access control when available is orthogonal this optimization.

Add a durable cache to help speed up layer lookups.

Update README.

Add more unit tests.
2020-04-03 09:29:12 -07:00
Ramkumar Chinchani
8ff60f9138 conformance: fix error msg for DELETE MANIFEST
---
Ran 27 of 27 Specs in 0.120 seconds
SUCCESS! -- 27 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS
---
2020-03-25 12:53:15 -07:00
Ramkumar Chinchani
dfe023225f pkg/api: use a rwlock when accessing storage
The original patch used a mutex, however, the workload patterns are
likely to be read-heavy, so use a rwlock instead.
2020-03-20 12:23:36 -07:00
Ramkumar Chinchani
2fd87b6a86 pkg/api: use a rwlock when accessing storage
The original patch used a mutex, however, the workload patterns are
likely to be read-heavy, so use a rwlock instead.
2020-03-20 10:58:21 -07:00
Ramkumar Chinchani
fe471a3c35 gc: fix test cases since umoci GC is more strict
umoci GC enforces a valid index.json and current tests were a little
lax.
2020-03-20 10:58:21 -07:00
Ramkumar Chinchani
404c83fbc1 bazel: fix bazel build failures 2020-03-20 10:58:21 -07:00