0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2024-12-30 22:34:13 -05:00
Commit graph

101 commits

Author SHA1 Message Date
Andreea Lupu
ec05137eda
feat(scrub): add scrub logic for ImageIndex media type (#1031)
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2022-12-10 16:38:01 -08:00
peusebiu
cbf88bd668
fix(test): consolidate flaky scrub test (#1042)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-12-07 11:04:33 -08:00
peusebiu
c82748ab39
Upgrade oras cli (#1021)
* chore: upgrade oras cli

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>

* fix(oci references): referrers API should not return 404 if repo exists

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-11-28 10:43:53 -08:00
peusebiu
1d97d1e796
feat: gc untagged manifests and signatures without references (#948)
closes #906 #907

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-11-24 11:00:22 -08:00
Catalin-George Hofnar
31b9481713
feat(cache): dynamodb implementation (#953)
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
2022-11-22 10:29:57 -08:00
peusebiu
49c3d05706
fix(storage): sanitize storage locks (#1003)
removed all locks from common code
removed locks from GetBlobContent

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-11-22 10:13:08 -08:00
peusebiu
168d21da1e
fix(storage): deleting manifests with identical digests (#951)
Suppose we push two identical manifests (sharing same digest) but with
different tags, then deleting by digest should throw an error otherwise
we end up deleting all image tags (with gc) or dangling references
(without gc)

This behaviour is controlled via Authorization, added a new policy
action named detectManifestsCollision which enables this behaviour

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>

Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-11-18 09:35:28 -08:00
peusebiu
e96c80c344
feat(sync,s3): added s3 logic for ORAS and OCI artifacts (#985)
added sync logic for OCI artifacts

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-11-14 22:21:49 -08:00
Ramkumar Chinchani
c0f93caacb
feat(artifact): add OCI references support (#936)
Thanks @jdolitsky et al for kicking off these changes at:
https://github.com/oci-playground/zot/commits/main

Thanks @sudo-bmitch for reviewing the patch

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-11-08 00:38:16 -08:00
Catalin-George Hofnar
4170d2adbc
refactor(cache): rewrote/refactored cachedb functionality to use interface (#667)
Moved boltdb to a driver implementation for such interface
Added CreateCacheDatabaseDriver in controller
Fixed default directory creation (boltDB will only create the file, not the dir
Added coverage tests
Added example config for boltdb
Re-added caching on subpaths, rewrote CreateCacheDatabaseDriver
Fix tests
Made cacheDriver argument mandatory for NewImageStore, added more validation, added defaults
Moved cache interface to own file, removed useRelPaths from config
Got rid of cache config, refactored
Moved cache to own package and folder
Renamed + removed cache factory to backend, replaced CloudCache to RemoteCache
Moved storage constants back to storage package
moved cache interface and factory to storage package, changed remoteCache defaulting

Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
2022-11-02 15:53:08 -07:00
Andrei Aaron
ac6c6a844c
refactor(digests): standardise representation of digests to digest.Digest (#898)
- Digests were represented by different ways
  - We needed a uniform way to represent the digests and enforce a format
  - also replace usage of github.com/google/go-containerregistry/pkg/v1
    with github.com/opencontainers/image-spec/specs-go/v1

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 96b2f29d6d57070a913ce419149cd481c0723815)
(cherry picked from commit 3d41b583daea654c98378ce3dcb78937d71538e8)

Co-authored-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2022-10-22 13:46:13 -07:00
Andrei Aaron
38b00e3507
chore(lint): gci to separate zot from other imports (#870)
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-10-20 09:39:20 -07:00
peusebiu
92d97d48d6
fix(s3): remove tracking multipart uploads (#883)
Remove sticky sessions from clustering

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-10-20 09:36:58 -07:00
peusebiu
fd87a22829
fix(storage): resolve cache/storage inconsistencies on HEAD request (#794)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-10-11 10:05:39 -07:00
Nicol
33a431ef43
Update go version to 1.19 (#829)
* ci: Update go version to 1.19

Signed-off-by: Nicol Draghici <idraghic@cisco.com>

* ci: Fix lint issues

Signed-off-by: Nicol Draghici <idraghic@cisco.com>

* ci: Added needprivileges to lint, made needprivileges pass lint

Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Signed-off-by: Nicol Draghici <idraghic@cisco.com>

Signed-off-by: Nicol Draghici <idraghic@cisco.com>
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Co-authored-by: Catalin Hofnar <catalin.hofnar@gmail.com>
2022-10-05 13:21:14 +03:00
peusebiu
8237f8d20a
storage: Move common code in helper functions, closes #730 (#820)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-09-30 10:35:16 -07:00
Andreea Lupu
f686ab6bf6
initial design for task scheduler (#700)
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2022-09-22 22:27:56 -07:00
Ramkumar Chinchani
90c8390c29 routes: support resumable pull
Some embedded clients use the "Range" header for blob pulls in order to
resume downloads.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-09-15 15:51:47 -07:00
Petu Eusebiu
7912b6a3fb Refactor s3 dedupe
Under digest bucket create 2 buckets, one for storing origin blob
and one for storing deduped blobs.

PutBlob() - puts an origin blob in both buckets
          - puts a deduped blob in deduped bucket
GetBlob() - returns blobs only from origin bucket
DeleteBlob() - deletes an origin blob from both buckets
             and moves one deduped blob into origin bucket
             - deletes a deduped blob from deduped bucket

[storage] When deleting an origin blob, next time we GetBlob() we get a
deduped blob with no content and we will move the content from
the deleted origin blob to it (inside s3.go).

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-09-14 14:43:58 -07:00
peusebiu
5479e2c785
s3: fix dedupe failing to manage blobs correctly (#772)
in order to know which blob is 'real' (has content)
we need to know which was the first blob inserted in cache,
because that is always the real one.

because we can not modify the keys order in boltdb we'll do this
by marking the first blob inserted with a value

when GetBlob() return the blob which is marked
when PutBlob() if is the first one, mark it
when DeleteBlob() in case deleted is marked then mark the next blob

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-09-08 01:12:14 +03:00
slab713
8ffb053cec
Replaced deprecated io/ioutil functions (#768)
Signed-off-by: slab713 <109306207+slab713@users.noreply.github.com>
2022-09-02 15:56:02 +03:00
Ramkumar Chinchani
2ff8e8b7d2 fix dependabot alerts
https://github.com/project-zot/zot/pull/737
https://github.com/project-zot/zot/pull/738
https://github.com/project-zot/zot/pull/739
https://github.com/project-zot/zot/pull/740

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-08-23 09:38:30 -07:00
Ramkumar Chinchani
5c01c4eab4
support OCI image index at manifest endpoint (#638)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-08-20 11:18:48 +03:00
Andrei Aaron
bd9ad998cd Fix file handlers not being closed after calls to ImageStore.GetBlob
This is to fixes hitting the FD limit when reading blobs from the disk in the graphql API

Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-08-19 09:22:13 -07:00
alexstan12
16e9822c7f
Add fuzz tests for storage_fs (#601)
This commit uses native go fuzzing to fuzz test implementations
of storage in storage_fs.

moved fuzzing testdata for storage_fs in separate repo

added make target and script for importing fuzz data and running all fuzz tests

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
2022-07-27 20:37:55 +03:00
Lisca Ana-Roberta
87fc941b3c image level lint: enforce manifest mandatory annotations
closes #536

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2022-07-27 11:48:04 +03:00
Laurentiu Niculae
7e3d063319 freeform querry api
Signed-off-by: Laurentiu Niculae <themelopeus@gmail.com>
2022-07-20 10:03:11 -07:00
Alex Stan
a31869f270 fix GetReferrers function to be able to retrieve referrers of any specified artifactType
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
2022-07-19 09:44:23 -07:00
Ramkumar Chinchani
eed48c1715 refactor filenames to reflect functionality
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-06-21 21:42:54 -07:00
Andreea-Lupu
081ba0b2f2 fix periodic background tasks - gc and scrub
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2022-06-02 08:58:02 -07:00
Lisca Ana-Roberta
62775cc095 fixed failed tests for all skopeo versions
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo verifications

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo verifications modified makefile

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

modified how to get digest and fixed makefile

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

fixed failed tests for all skopeo versions

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

echo skopeo version

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo verifications

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo verifications modified makefile

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

modified how to get digest and fixed makefile

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo failed tests fixed

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

changed function name

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

fixed lost modifications

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

fixed code coverage and dead code

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2022-05-27 08:19:07 -07:00
Lisca Ana-Roberta
e5a14670db code coverage improvement
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2022-05-26 08:54:05 -07:00
Petu Eusebiu
5e22acbbc4 s3: added logic for deduping blobs
Because s3 doesn't support hard links we store duplicated blobs
as empty files. When the original blob is deleted its content is
moved to the the next duplicated blob and so on.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-05-24 17:00:10 -07:00
Alex Stan
d325c8b5f4 Fix problems signaled by new linter version v1.45.2
PR (linter: upgrade linter version #405) triggered lint job which failed
with many errors generated by various linters. Configurations were added to
golangcilint.yaml and several refactorings were made in order to improve the
results of the linter.

maintidx linter disabled

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
2022-04-27 09:55:44 -07:00
Andreea-Lupu
5e35dfa28f make gc periodic
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2022-04-18 10:25:29 -07:00
Petu Eusebiu
c62dae06c9 s3: fix initRepo not creating index.json in some edge cases
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-04-14 15:06:40 -07:00
Catalin Hofnar
475d97b1ad Separate make commands that require sudo
Reworked privileged cert test so it runs in Go by moving make logic to Go logic

Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
2022-04-05 10:12:43 -07:00
Andreea-Lupu
9454c77be2 make scrub inline and periodic
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2022-04-01 13:38:24 -07:00
Alexei Dodon
ad519e2d3e Leave zot repositories in a consistent state after zot hits fd limit closes #359
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2022-03-31 13:25:15 -07:00
Ramkumar Chinchani
b2a4388522 gc: add a unit test
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-03-24 12:02:52 -07:00
Petu Eusebiu
f53dc9eb8d sync: Add a new flag to enforce syncing only signed images, closes #455
sync: When checking if a image is already synced also check for changes in upstream signatures.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-03-24 10:50:01 -07:00
Eng Zer Jun
0d77b60de7 test: use T.TempDir to create temporary test directory
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-03-07 12:28:49 -08:00
Petu Eusebiu
9cffbcaccb s3: bugfix, use sync.Map instead of map for storing multi part uploads references
add storage lock in GetIndexContent

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-03-03 09:12:07 -08:00
Petu Eusebiu
45968e0bb7 sync: fix inconsistent test, used inject error fw for hard to reach test cases
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-03-01 09:56:27 -08:00
Ramkumar Chinchani
38a110314b gc: add a gcDelay param
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-14 14:03:22 -08:00
Ramkumar Chinchani
d2aa016cdb storage: flush/sync contents to disk on file close
Behavior controlled by configuration (default=off)
It is a trade-off between performance and consistency.

References:
[1] https://github.com/golang/go/issues/20599

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-07 12:08:01 -08:00
Ramkumar Chinchani
c73e71b018 ci/cd: add a basic push-pull testing using client tools
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-07 10:03:24 -08:00
Ramkumar Chinchani
87084f286b storage: improve/fix oci image validation
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-02 13:31:41 -08:00
Ramkumar Chinchani
45fe129c63 notaryv2: fix 'notation list'
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-01-31 14:33:21 -08:00
Ramkumar Chinchani
4d576a4817 storage: return errors instead of panics
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-01-24 13:33:45 -08:00