Infrastructure/zot
0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2025-01-06 22:40:28 -05:00
Code Issues Activity
545 commits 5 branches 109 tags 209 MiB
Commit graph

84 commits

Author SHA1 Message Date
Ramkumar Chinchani
90c8390c29 routes: support resumable pull 
Some embedded clients use the "Range" header for blob pulls in order to
resume downloads.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-09-15 15:51:47 -07:00
Petu Eusebiu
7912b6a3fb Refactor s3 dedupe 
Under digest bucket create 2 buckets, one for storing origin blob
and one for storing deduped blobs.

PutBlob() - puts an origin blob in both buckets
          - puts a deduped blob in deduped bucket
GetBlob() - returns blobs only from origin bucket
DeleteBlob() - deletes an origin blob from both buckets
             and moves one deduped blob into origin bucket
             - deletes a deduped blob from deduped bucket

[storage] When deleting an origin blob, next time we GetBlob() we get a
deduped blob with no content and we will move the content from
the deleted origin blob to it (inside s3.go).

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-09-14 14:43:58 -07:00
peusebiu
5479e2c785
s3: fix dedupe failing to manage blobs correctly (#772) 
in order to know which blob is 'real' (has content)
we need to know which was the first blob inserted in cache,
because that is always the real one.

because we can not modify the keys order in boltdb we'll do this
by marking the first blob inserted with a value

when GetBlob() return the blob which is marked
when PutBlob() if is the first one, mark it
when DeleteBlob() in case deleted is marked then mark the next blob

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-09-08 01:12:14 +03:00
slab713
8ffb053cec
Replaced deprecated io/ioutil functions (#768) 
Signed-off-by: slab713 <109306207+slab713@users.noreply.github.com>
 2022-09-02 15:56:02 +03:00
Ramkumar Chinchani
2ff8e8b7d2 fix dependabot alerts 
https://github.com/project-zot/zot/pull/737
https://github.com/project-zot/zot/pull/738
https://github.com/project-zot/zot/pull/739
https://github.com/project-zot/zot/pull/740

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-08-23 09:38:30 -07:00
Ramkumar Chinchani
5c01c4eab4
support OCI image index at manifest endpoint (#638) 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-08-20 11:18:48 +03:00
Andrei Aaron
bd9ad998cd Fix file handlers not being closed after calls to ImageStore.GetBlob 
This is to fixes hitting the FD limit when reading blobs from the disk in the graphql API

Signed-off-by: Andrei Aaron <andaaron@cisco.com>
 2022-08-19 09:22:13 -07:00
alexstan12
16e9822c7f
Add fuzz tests for storage_fs (#601) 
This commit uses native go fuzzing to fuzz test implementations
of storage in storage_fs.

moved fuzzing testdata for storage_fs in separate repo

added make target and script for importing fuzz data and running all fuzz tests

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
 2022-07-27 20:37:55 +03:00
Lisca Ana-Roberta
87fc941b3c image level lint: enforce manifest mandatory annotations 
closes #536

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
 2022-07-27 11:48:04 +03:00
Laurentiu Niculae
7e3d063319 freeform querry api 
Signed-off-by: Laurentiu Niculae <themelopeus@gmail.com>
 2022-07-20 10:03:11 -07:00
Alex Stan
a31869f270 fix GetReferrers function to be able to retrieve referrers of any specified artifactType 
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
 2022-07-19 09:44:23 -07:00
Ramkumar Chinchani
eed48c1715 refactor filenames to reflect functionality 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-06-21 21:42:54 -07:00
Andreea-Lupu
081ba0b2f2 fix periodic background tasks - gc and scrub 
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
 2022-06-02 08:58:02 -07:00
Lisca Ana-Roberta
62775cc095 fixed failed tests for all skopeo versions 
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo verifications

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo verifications modified makefile

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

modified how to get digest and fixed makefile

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

fixed failed tests for all skopeo versions

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

echo skopeo version

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo verifications

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo verifications modified makefile

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

modified how to get digest and fixed makefile

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo failed tests fixed

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

changed function name

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

fixed lost modifications

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

fixed code coverage and dead code

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
 2022-05-27 08:19:07 -07:00
Lisca Ana-Roberta
e5a14670db code coverage improvement 
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
 2022-05-26 08:54:05 -07:00
Petu Eusebiu
5e22acbbc4 s3: added logic for deduping blobs 
Because s3 doesn't support hard links we store duplicated blobs
as empty files. When the original blob is deleted its content is
moved to the the next duplicated blob and so on.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-05-24 17:00:10 -07:00
Alex Stan
d325c8b5f4 Fix problems signaled by new linter version v1.45.2 
PR (linter: upgrade linter version #405) triggered lint job which failed
with many errors generated by various linters. Configurations were added to
golangcilint.yaml and several refactorings were made in order to improve the
results of the linter.

maintidx linter disabled

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
 2022-04-27 09:55:44 -07:00
Andreea-Lupu
5e35dfa28f make gc periodic 
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
 2022-04-18 10:25:29 -07:00
Petu Eusebiu
c62dae06c9 s3: fix initRepo not creating index.json in some edge cases 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-04-14 15:06:40 -07:00
Catalin Hofnar
475d97b1ad Separate make commands that require sudo 
Reworked privileged cert test so it runs in Go by moving make logic to Go logic

Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
 2022-04-05 10:12:43 -07:00
Andreea-Lupu
9454c77be2 make scrub inline and periodic 
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
 2022-04-01 13:38:24 -07:00
Alexei Dodon
ad519e2d3e Leave zot repositories in a consistent state after zot hits fd limit closes #359 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2022-03-31 13:25:15 -07:00
Ramkumar Chinchani
b2a4388522 gc: add a unit test 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-03-24 12:02:52 -07:00
Petu Eusebiu
f53dc9eb8d sync: Add a new flag to enforce syncing only signed images, closes #455 
sync: When checking if a image is already synced also check for changes in upstream signatures.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-03-24 10:50:01 -07:00
Eng Zer Jun
0d77b60de7 test: use T.TempDir to create temporary test directory 
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2022-03-07 12:28:49 -08:00
Petu Eusebiu
9cffbcaccb s3: bugfix, use sync.Map instead of map for storing multi part uploads references 
add storage lock in GetIndexContent

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-03-03 09:12:07 -08:00
Petu Eusebiu
45968e0bb7 sync: fix inconsistent test, used inject error fw for hard to reach test cases 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-03-01 09:56:27 -08:00
Ramkumar Chinchani
38a110314b gc: add a gcDelay param 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-02-14 14:03:22 -08:00
Ramkumar Chinchani
d2aa016cdb storage: flush/sync contents to disk on file close 
Behavior controlled by configuration (default=off)
It is a trade-off between performance and consistency.

References:
[1] https://github.com/golang/go/issues/20599

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-02-07 12:08:01 -08:00
Ramkumar Chinchani
c73e71b018 ci/cd: add a basic push-pull testing using client tools 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-02-07 10:03:24 -08:00
Ramkumar Chinchani
87084f286b storage: improve/fix oci image validation 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-02-02 13:31:41 -08:00
Ramkumar Chinchani
45fe129c63 notaryv2: fix 'notation list' 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-01-31 14:33:21 -08:00
Ramkumar Chinchani
4d576a4817 storage: return errors instead of panics 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-01-24 13:33:45 -08:00
Petu Eusebiu
1109bb4dde sync: Added support for syncing notary/cosign signatures, closes #261 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-01-13 08:45:59 -08:00
Alexei Dodon
c4d34b7269 Added storage latency histogram metric 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2022-01-10 11:55:39 -08:00
Ramkumar Chinchani
cac7fe4854 storage: use sha256-simd from minio 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-12-28 22:25:11 -08:00
Ramkumar Chinchani
ac3801ea2d lint: upgrade golangci-lint 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-12-20 17:20:35 -08:00
Andreea-Lupu
c61c3836db implement scrub to check manifest/blob integrity 
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
 2021-12-09 11:18:09 -08:00
Ramkumar Chinchani
96226af869 move references to zotregistry.io and project-zot 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-12-05 10:52:27 -08:00
Ramkumar Chinchani
e42e42a2cc artifacts: initial support for artifacts/notaryv2 spec 
https://github.com/oras-project/artifacts-spec
https://github.com/notaryproject/notaryproject

Fixes issue #264

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-12-01 18:55:39 -08:00
Alexei Dodon
e900b09cfb Fix data races in tests, closes #255 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2021-11-17 13:23:59 -08:00
Petu Eusebiu
9c568c0ee2 storage: add s3 backend support (without GC and dedupe) 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2021-11-15 08:09:00 -08:00
Alexei Dodon
8e4d828867 Implement an API for performance monitoring 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2021-11-12 11:14:10 -08:00
Alexei Dodon
f76c76c2e6 Added a timeout option in db cache lock, closes #242 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2021-10-27 12:43:50 -07:00
Alexei Dodon
d8aa5b8bf3
Fixing ValidateHardLink, closes #256 (#257) 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2021-10-27 20:03:26 +03:00
Petu Eusebiu
7d077eaf5a Added storage interface 2021-10-11 17:38:46 -07:00
Shivam Mishra
c6670b1329 api: implement delete by tag 2021-08-23 17:30:41 -07:00
Ramkumar Chinchani
63b88d0e57 pkg/storage: fix partially initialized repo storage 
Thanks shimish2 for the unit test.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-08-09 23:19:20 -07:00
Shivam Mishra
53b5fa6493 dedupe: stat blob path before creating link 2021-08-09 09:40:35 -07:00
Shivam Mishra
af30c06aff api: use blob cache path while making hard link 
previously mount blob will look for blob that is provided in http request and try to hard link that path
but ideally we should look for path from our cache and do the hard link of that particular path.
this commit does the same.
 2021-06-30 01:42:21 -07:00