Infrastructure/zot
0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2024-12-30 22:34:13 -05:00
Code Issues Activity
638 commits 5 branches 109 tags 208 MiB
Commit graph

321 commits

Author SHA1 Message Date
Lisca Ana-Roberta
111b80625d added repos command to list repositories 
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
 2022-06-15 02:22:18 -07:00
Shivam Mishra
620bc7c517 routes: strip query parameter from request URL 
reuqest url also contains query parameter due to this in some scenarios
location header is setting up incorrectly, strip query parameter from
request url to correctly setup location header.

Closes #573 #575

Signed-off-by: Shivam Mishra <shimish2@cisco.com>
 2022-06-08 22:50:37 -07:00
Shivam Mishra
f52c950d04 fix sample request url in search extension README 
Signed-off-by: Shivam Mishra <shimish2@cisco.com>
 2022-06-07 11:24:19 -07:00
Catalin Hofnar
0b6fdc23ea Added sync onDemand test for ORAS artifact 
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
 2022-06-06 10:38:42 -07:00
Shivam Mishra
b61aff62cd check notary v2 signature while looking for available signatures 
expanded repo info also provides information if manifests of repo is signed or not
previously it was looking for only cosign signature.

Signed-off-by: Shivam Mishra <shimish2@cisco.com>
 2022-06-03 17:45:22 -07:00
laurentiuNiculae
c9b32c73ae added more types of severity 
Signed-off-by: laurentiuNiculae <themelopeus@gmail.com>
 2022-06-03 09:44:54 -07:00
Andreea-Lupu
081ba0b2f2 fix periodic background tasks - gc and scrub 
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
 2022-06-02 08:58:02 -07:00
Lisca Ana-Roberta
62775cc095 fixed failed tests for all skopeo versions 
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo verifications

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo verifications modified makefile

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

modified how to get digest and fixed makefile

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

fixed failed tests for all skopeo versions

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

echo skopeo version

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo verifications

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo verifications modified makefile

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

modified how to get digest and fixed makefile

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo failed tests fixed

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

changed function name

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

fixed lost modifications

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

fixed code coverage and dead code

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
 2022-05-27 08:19:07 -07:00
Lisca Ana-Roberta
e5a14670db code coverage improvement 
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
 2022-05-26 08:54:05 -07:00
Shivam Mishra
0dd00e7883 fix extension endpoints 
Signed-off-by: Shivam Mishra <shimish2@cisco.com>
 2022-05-25 13:46:43 -07:00
Ramkumar Chinchani
6b841809e3 fix CVE-2022-29173/GHSA-66x3-6cw3-v5gj 
https://github.com/project-zot/zot/security/dependabot/16

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-05-25 11:34:24 -07:00
Petu Eusebiu
da4acaf178 sync: preserve upstream digests after syncing images 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-05-25 10:19:36 -07:00
Petu Eusebiu
bd730150a8 sync: allow HTTP redirects when GETing signatures blobs 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-05-25 10:19:36 -07:00
Petu Eusebiu
aeb8a5da39 sync: specify contentType in headers when GETing cosign manifest 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-05-25 10:19:36 -07:00
Shivam Mishra
dcdeb935fd use zot as an extension name, ext as a component and search as a module 
add endpoints field in ext discover api

distribution spec extension discover api has endpoints field required.

https://github.com/opencontainers/distribution-spec/blob/main/extensions/_oci.md#extensions-property-descriptions

Signed-off-by: Shivam Mishra <shimish2@cisco.com>
 2022-05-24 19:12:40 -07:00
Petu Eusebiu
5e22acbbc4 s3: added logic for deduping blobs 
Because s3 doesn't support hard links we store duplicated blobs
as empty files. When the original blob is deleted its content is
moved to the the next duplicated blob and so on.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-05-24 17:00:10 -07:00
Shivam Mishra
36c9631000 ext: use distribution spec route prefix for extension api 
Following the spec defined here https://github.com/opencontainers/distribution-spec/tree/main/extensions

Signed-off-by: Shivam Mishra <shimish2@cisco.com>
 2022-05-22 16:35:16 -07:00
laurentiuNiculae
7d8af50aec mocked tests for routes 
Signed-off-by: laurentiuNiculae <themelopeus@gmail.com>
 2022-05-20 13:27:09 -07:00
Ramkumar Chinchani
287ac05ddc update linter version to 1.46.2 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-05-20 11:53:56 -07:00
Petu Eusebiu
7c3a8f9d07 Report unknown keys when parsing configuration files 
Report missing mandatory ldap keys

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-05-16 14:13:31 -07:00
Catalin Hofnar
20a60cbad4 Enhance sync logic - stop blob redownloads and re-pushes (#479 #480) 
Changed imagesToCopyFromUpstream to return a map[string][]types.ImageReference from just an array of refs
Rewrote some logic in sync.go to use the new signature of imagesToCopyFromUpstream
Split getLocalImageRef by adding function getLocalCachePath
Adapted tests for new changes, added some tests
Merged #481

Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
 2022-05-16 10:05:01 -07:00
Shivam Mishra
e04a9bf6e2 use TempDir instead of /tmp/zot in tests 
Closes #508

Signed-off-by: Shivam Mishra <shimish2@cisco.com>
 2022-05-05 10:38:53 -07:00
laurentiuNiculae
bb95af5b4d default policy only authorization 
unit tests for manifest integrity when updating

Signed-off-by: laurentiuNiculae <themelopeus@gmail.com>
 2022-05-03 11:46:55 -07:00
Alex Stan
d325c8b5f4 Fix problems signaled by new linter version v1.45.2 
PR (linter: upgrade linter version #405) triggered lint job which failed
with many errors generated by various linters. Configurations were added to
golangcilint.yaml and several refactorings were made in order to improve the
results of the linter.

maintidx linter disabled

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
 2022-04-27 09:55:44 -07:00
Andreea-Lupu
5e35dfa28f make gc periodic 
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
 2022-04-18 10:25:29 -07:00
Petu Eusebiu
89c5f4f604 sync: fix functions which compare signatures 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-04-15 10:31:37 -07:00
Petu Eusebiu
4e20ab8a5d go.mod: update dependencies 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-04-15 10:31:37 -07:00
Petu Eusebiu
c62dae06c9 s3: fix initRepo not creating index.json in some edge cases 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-04-14 15:06:40 -07:00
laurentiuNiculae
0d4cc8736d Target for cheking not commited config files. 
Signed-off-by: laurentiuNiculae <themelopeus@gmail.com>

Separated updateDistSpec functionality

Removed rewriting of config when distSpecVersion was wrong
 2022-04-14 10:28:38 -07:00
Catalin Hofnar
475d97b1ad Separate make commands that require sudo 
Reworked privileged cert test so it runs in Go by moving make logic to Go logic

Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
 2022-04-05 10:12:43 -07:00
Andreea-Lupu
9454c77be2 make scrub inline and periodic 
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
 2022-04-01 13:38:24 -07:00
Alexei Dodon
ad519e2d3e Leave zot repositories in a consistent state after zot hits fd limit closes #359 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2022-03-31 13:25:15 -07:00
Petu Eusebiu
03dd558ec6 sync: fix inconsistent test 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-03-31 09:19:07 -07:00
Ramkumar Chinchani
b2a4388522 gc: add a unit test 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-03-24 12:02:52 -07:00
Petu Eusebiu
be910cf01c lint: Move out config reloader context from controller struct 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-03-24 10:50:01 -07:00
Petu Eusebiu
353b0c6034 Move api constants in separate 'constants' package to avoid circular imports 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-03-24 10:50:01 -07:00
Petu Eusebiu
f53dc9eb8d sync: Add a new flag to enforce syncing only signed images, closes #455 
sync: When checking if a image is already synced also check for changes in upstream signatures.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-03-24 10:50:01 -07:00
laurentiuNiculae
0d148e1d6b new config option for sync-destination 
Signed-off-by: laurentiuNiculae <themelopeus@gmail.com>
 2022-03-21 08:12:34 -07:00
Petu Eusebiu
6d04ab3cdc sync: support reloading sync config when the config file changes 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-03-18 11:24:11 -07:00
laurentiuNiculae
63d94d4ac5 Update dist-spec version automatically 
Warning if config has wrong dist-spec version

Signed-off-by: laurentiuNiculae <themelopeus@gmail.com>
 2022-03-14 10:24:03 -07:00
Shivam Mishra
b8010e1ee4 routes: changes required to do browser authentication 
whenever we make a request that contains header apart from CORS allowed header, browser sends a preflight request
and in response accept *Access-Control-Allow-Headers*.

preflight request is in form of OPTIONS method, added new http handler func to set headers
and returns HTTP status ok in case of OPTIONS method.

in case of authorization, request contains authorization header
added authorization header in Access-Control-Allow-Headers list

added AllowOrigin field in HTTPConfig this field value is set to Access-Control-Allow-Origin header and will give zot adminstrator to limit incoming request.

Signed-off-by: Shivam Mishra <shimish2@cisco.com>
 2022-03-08 17:42:54 -08:00
Eng Zer Jun
0d77b60de7 test: use T.TempDir to create temporary test directory 
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2022-03-07 12:28:49 -08:00
Ramkumar Chinchani
4be2652085 conformance: fix cross-mount behavior when 'from' is missing 
fixes issue #442

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-03-04 17:24:50 -08:00
Ramkumar Chinchani
95e4b2054b upgrade module deps 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-03-04 13:10:58 -08:00
Petu Eusebiu
9cffbcaccb s3: bugfix, use sync.Map instead of map for storing multi part uploads references 
add storage lock in GetIndexContent

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-03-03 09:12:07 -08:00
Ramkumar Chinchani
8db3e1b192 CVE-2022-23649: fix dependabot alert 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-03-02 12:01:14 -08:00
Ramkumar Chinchani
3ada6af0de tls: set min version to 1.2 and restrict cipher suites 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-03-02 10:03:50 -08:00
Petu Eusebiu
45968e0bb7 sync: fix inconsistent test, used inject error fw for hard to reach test cases 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-03-01 09:56:27 -08:00
Ramkumar Chinchani
38a110314b gc: add a gcDelay param 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-02-14 14:03:22 -08:00
Petu Eusebiu
35eeedb22a sync: pull only missing images, not everything, closes #335 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-02-14 08:09:18 -08:00
Petu Eusebiu
0ec39c0313 sync: make RetryDelay and MaxRetries optional 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-02-11 09:11:11 -08:00
Shivam Mishra
37d150e32f search: graphql api to give detailed repo info 
DetailedRepoInfo graphql api returns detailed repo info given repo name
repo contains its manifests info
Each manifest entry contains digest,signed, tag and layers info
Each layer info containes digest, size

Signed-off-by: Shivam Mishra <shimish2@cisco.com>
 2022-02-10 16:34:13 -08:00
Ramkumar Chinchani
b2c8533719 test: fix ldap unit tests 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-02-09 14:48:34 -08:00
Alexei Dodon
47c9b6244e Added config enable=true/false for extensions with default value as enabled closes #258 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2022-02-09 09:53:49 -08:00
Ramkumar Chinchani
730fe70f2f coverage: improve code coverage 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-02-09 07:42:15 -08:00
Ramkumar Chinchani
d2aa016cdb storage: flush/sync contents to disk on file close 
Behavior controlled by configuration (default=off)
It is a trade-off between performance and consistency.

References:
[1] https://github.com/golang/go/issues/20599

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-02-07 12:08:01 -08:00
Ramkumar Chinchani
c73e71b018 ci/cd: add a basic push-pull testing using client tools 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-02-07 10:03:24 -08:00
Petu Eusebiu
f89925fb27 sync: periodically retry if on-demand fails inline, closes #281 
sync: don't return error on sync signatures, just skip them, closes #375
sync: sync signatures on demand
sync on demand: in case of parallel requests pull image just once, closes #344

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-02-07 09:04:13 -08:00
Ramkumar Chinchani
87084f286b storage: improve/fix oci image validation 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-02-02 13:31:41 -08:00
Petu Eusebiu
a0e65379c8 sync: for a prefix, allow multiple registries as a list instead of only one, closes #343 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-02-01 09:45:09 -08:00
Ramkumar Chinchani
45fe129c63 notaryv2: fix 'notation list' 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-01-31 14:33:21 -08:00
Petu Eusebiu
b9250a783a Use InsecureSkipVerify only with https upstreams 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-01-29 20:00:27 -08:00
Ramkumar Chinchani
e0a1a82890 coverage: add failure injection framework 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-01-28 08:56:00 -08:00
Alexei Dodon
f47c8222c2 bug: Storage used per zot repo metric is broken 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2022-01-26 18:43:14 -08:00
Petu Eusebiu
89b143805e Remove sync http handler, not needed anymore since added sync on demand 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-01-25 09:18:23 -08:00
Ramkumar Chinchani
4d576a4817 storage: return errors instead of panics 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-01-24 13:33:45 -08:00
Ramkumar Chinchani
1e5ea7e09c controller: support rate-limiting incoming requests 
helps constraining resource usage and against flood attacks.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-01-24 12:48:13 -08:00
Alexei Dodon
d259ba6e4f Fix scrub command crash 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2022-01-22 15:54:13 -08:00
Alexei Dodon
c9a81baa10 Renamed zot-exporter to zxp and added its image to zot release 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2022-01-19 10:31:37 -08:00
Petu Eusebiu
1109bb4dde sync: Added support for syncing notary/cosign signatures, closes #261 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-01-13 08:45:59 -08:00
Petu Eusebiu
e6d6d5a7de Fix cli version showing help message, closes #361 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-01-12 12:35:17 -08:00
Ramkumar Chinchani
4896adad1b build: split functionality into separate binaries 
zot: registry server
zli: zot cli to interact with the zot registry
zui: zot ui (proposed)
zb: zot benchmark (proposed)

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-01-11 11:48:18 -08:00
Alexei Dodon
c4d34b7269 Added storage latency histogram metric 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2022-01-10 11:55:39 -08:00
Petu Eusebiu
4f825a5e2f [Identity-based Authorization] Add an option to specify a global policy for all repositories 
using regex.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-01-07 10:55:20 -08:00
Ramkumar Chinchani
cac7fe4854 storage: use sha256-simd from minio 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-12-28 22:25:11 -08:00
Ramkumar Chinchani
8183e1467c lint: some more linter-related cleanup 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-12-23 22:01:40 -08:00
Ramkumar Chinchani
ac3801ea2d lint: upgrade golangci-lint 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-12-20 17:20:35 -08:00
Petu Eusebiu
5f04092e71 Fix negative WaitGroup counter at runtime 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2021-12-17 16:52:34 -08:00
Petu Eusebiu
c86f44cc53 Disable sync periodically polling when pollInterval is not configured 
Filtering out sync on demand images based on content configuration

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2021-12-14 08:59:50 -08:00
Andreea-Lupu
c61c3836db implement scrub to check manifest/blob integrity 
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
 2021-12-09 11:18:09 -08:00
Petu Eusebiu
627cb97ef1 Add wait group for graceful shutdown, closes #302 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2021-12-08 10:04:52 -08:00
Ramkumar Chinchani
d4307d96ac build: fix build artifacts in binary 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-12-07 20:59:26 -08:00
Petu Eusebiu
63a75216ed sync: allow for saving to a subpath, closes #307 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2021-12-07 10:13:20 -08:00
Ramkumar Chinchani
96226af869 move references to zotregistry.io and project-zot 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-12-05 10:52:27 -08:00
Ramkumar Chinchani
e42e42a2cc artifacts: initial support for artifacts/notaryv2 spec 
https://github.com/oras-project/artifacts-spec
https://github.com/notaryproject/notaryproject

Fixes issue #264

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-12-01 18:55:39 -08:00
Alexei Dodon
f99fa37623 ci/cd: unit test hangs for a long time intermittently closes #286 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2021-11-29 14:07:49 -08:00
Petu Eusebiu
fff6107310 Sync prefix can be an exact match or a glob pattern, closes #297 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2021-11-29 13:10:13 -08:00
Petu Eusebiu
f0ef10fa50 sync: cleanup the orphaned private download dir on failure, closes 282 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2021-11-25 10:31:45 -08:00
Petu Eusebiu
7ada50e9c8 sync: skip tls verify on /v2/_catalog when a registry is configured with tls-verify false 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2021-11-25 10:31:45 -08:00
Ramkumar Chinchani
5f8f61407e routes: fix CheckManifest to return content length 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-11-23 14:09:36 -08:00
Alexei Dodon
e900b09cfb Fix data races in tests, closes #255 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2021-11-17 13:23:59 -08:00
Petu Eusebiu
5c07e19c8d Changed sync behaviour, it used to copy images over http interface 
now it copies to a local cache and then it copies over storage APIs

- accept all images with or without signatures
- disable sync writing to stdout
- added more logs
- fixed switch statement in routes
- fixed enabling sync multiple times for storage subpaths

closes #266

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2021-11-15 09:32:43 -08:00
Petu Eusebiu
9c568c0ee2 storage: add s3 backend support (without GC and dedupe) 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2021-11-15 08:09:00 -08:00
Alexei Dodon
8e4d828867 Implement an API for performance monitoring 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2021-11-12 11:14:10 -08:00
Alexei Dodon
f76c76c2e6 Added a timeout option in db cache lock, closes #242 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2021-10-27 12:43:50 -07:00
Alexei Dodon
d8aa5b8bf3
Fixing ValidateHardLink, closes #256 (#257) 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2021-10-27 20:03:26 +03:00
Petu Eusebiu
f7ae491d22 Fix data race in sync tests 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2021-10-25 10:50:05 -07:00
Ramkumar Chinchani
8f3d7d3719 swagger: rename 'docs/' to 'swagger/' 
Use 'docs/' for zot-related documentation.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-10-21 13:46:14 -07:00
Petu Eusebiu
19003e8a71 Added new extension "sync" 
Periodically poll registries and pull images according to sync's config
Added sync on demand, syncing when clients asks for an image which
zot doesn't have.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2021-10-21 10:32:46 -07:00
Shivam Mishra
d930adbd49 search: update trivy 
trivy updated to v0.20.0
trivy-db updated to bec0c6a
fanal updated to f7efd1b
 2021-10-13 16:37:31 -07:00
Petu Eusebiu
7d077eaf5a Added storage interface 2021-10-11 17:38:46 -07:00
Petu Eusebiu
20f4051446 Clean blob uploads when clients interrupts uploading, closes #225 2021-10-08 14:55:57 -07:00
Shivam Mishra
63fef3e48c search: added graphql api to return repository list with latest tag 2021-09-27 14:36:20 -07:00
Ramkumar Chinchani
008d382446 authn: serialize ldap authn calls 
Some LDAP servers are not MT-safe in that when searches happen with binds
in flight leads to errors such as:
"comment: No other operations may be performed on the connection while a
bind is outstanding"

Add goroutine-id in logs to help debug MT bugs.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-09-20 10:24:07 -07:00
Petu Eusebiu
62e724532a Check if auth config is provided when using access control 2021-09-14 12:55:21 -07:00
Petu Eusebiu
609d85d875 Add identity-based access control, closes #51 
Add a cli subcommand to verify config files validity
 2021-08-30 13:56:27 -07:00
Ramkumar Chinchani
26926ad4c2 go.mod: update modules 2021-08-25 11:51:23 -07:00
Shivam Mishra
c6670b1329 api: implement delete by tag 2021-08-23 17:30:41 -07:00
Roxana Nemulescu
fed5c09b71 TLS certs in CLI client 
resolve #194
 2021-08-16 23:42:21 -07:00
Ramkumar Chinchani
63b88d0e57 pkg/storage: fix partially initialized repo storage 
Thanks shimish2 for the unit test.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-08-09 23:19:20 -07:00
Shivam Mishra
53b5fa6493 dedupe: stat blob path before creating link 2021-08-09 09:40:35 -07:00
Ramkumar Chinchani
f10c174c0e routes: add Content-Type header for HEAD manifest response 
With recent docker client-side changes, on 'docker pull' we see:
"Error response from daemon: missing or empty Content-Type header"

Hence, set Content-Type header.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-07-23 10:00:32 -07:00
Shivam Mishra
af30c06aff api: use blob cache path while making hard link 
previously mount blob will look for blob that is provided in http request and try to hard link that path
but ideally we should look for path from our cache and do the hard link of that particular path.
this commit does the same.
 2021-06-30 01:42:21 -07:00
Petu Eusebiu
3a59b9f487 Use freeport to get ports for zot servers in tests 2021-06-29 13:58:39 -07:00
Andrei Aaron
792e82cbdf Add an 'enable' flag in the server configuration to enable gql-based searches 
"extensions": {
        "search": {
            "enable": true
        }
    }
 2021-06-24 12:15:25 -07:00
Andrei Aaron
c1dd7878e4 Add a '--verbose' flag to the 'zot images' output 
- Show individual layers with size and digest under each image
- Include config digest for each image

See example below
```
IMAGE NAME                        TAG                       DIGEST    CONFIG    LAYERS    SIZE
test/godev                        0.4.7                     7d38d8ca  05b9f86e            519MB
                                                                                f824a027  65MB
                                                                                a98af0f5  52MB
                                                                                ba5b2bc4  163MB
                                                                                58b1ca8d  228MB
                                                                                67d798ee  12MB
test/cdev                         test                      2292b4ae  cf6f6c77            280MB
                                                                                f824a027  65MB
                                                                                a98af0f5  52MB
                                                                                ba5b2bc4  163MB
test/cdev                         0.4.7                     2292b4ae  cf6f6c77            280MB
                                                                                f824a027  65MB
                                                                                a98af0f5  52MB
                                                                                ba5b2bc4  163MB

Note the new layers and config fields will be visible in the json/yaml format regardless of the value of the verbose flag
```
 2021-06-24 12:15:25 -07:00
Andrei Aaron
519ea75d9a Implement a way to search for an image by manifest, config or layer digest 
```
Usage:
  zot images [config-name] [flags]

Flags:
  -d, --digest string   List images containing a specific manifest, config, or layer digest
[...]
```
 2021-06-24 12:15:25 -07:00
Roxana Nemulescu
97628e69c9 logs: add an audit log for API calls with unit tests 
resolves #178
 2021-06-24 10:53:27 -07:00
Shivam Mishra
28974e81dc config: support multiple storage locations 
added support to point multiple storage locations in zot by running multiple instance of zot in background.

see examples/config-multiple.json for more info about config.

Closes #181
 2021-05-21 10:18:28 -07:00
Ramkumar Chinchani
9ca6eea940 routes: ignore media-type for PatchBlobUpload 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-05-19 10:53:49 -07:00
Ramkumar Chinchani
8f729820f5 controller: add default headers 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-05-10 12:47:53 -07:00
Shivam Mishra
cf25c6f3c8 ci/cd:inculde binary type in version information 2021-05-04 13:16:23 -07:00
Shivam Mishra
a7c17b7c16 spec: added support for mount request using hard link 2021-05-04 09:42:29 -07:00
Ramkumar Chinchani
f7829d6470 dedupe: optimize check-blob with hard links 
In use cases, when there are large images with shared layers across
repositories, clients may benefit from not re-uploading the same blobs
over and over again.

We ensure this by hard linking when check-blob api is called.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-05-04 09:42:29 -07:00
Shivam Mishra
3c88102870 server: add idle timeout in http server configuration 2021-04-29 11:00:12 -07:00
Shivam Mishra
2b7b57313a conformance: fix http status code for cross-repository mounting 2021-01-29 09:35:15 -08:00
Shivam Mishra
9969ba0867 conformance: update README to display conformance results 2021-01-28 15:50:36 -08:00
Ramkumar Chinchani
affdd85986 build: remove bazel 
The idea initially was to use bazel to do our builds, however golang
build system is now good enough and our code base is entirely go.
It is also slowing down our travis ci/cd pipeline.
 2020-12-21 15:30:13 -08:00
Shivam Mishra
8787142d2c test: pull test images from aws container registry 2020-12-21 15:24:26 -08:00
Ionut Costin Craciun
dad884ddeb Raise error when adding a new zot config with an existed saved name 2020-11-04 10:25:34 +02:00
Shivam Mishra
b0ed625a2e build: increase wait timeout for travis bazel build process 2020-10-27 19:30:06 -07:00
Shivam Mishra
46beb30fc1 build: add build tags to create customizable binaries 2020-10-22 17:20:07 -07:00
Shivam Mishra
7439feb1c2 build: set timeout in travis make build process to avoid timeout failure 2020-10-18 20:55:17 -07:00
Shivam Mishra
14214a5794 test: add unit test to verify lock changes 2020-10-16 14:58:45 -07:00
Ramkumar Chinchani
386c72d332 routes: refactor locks to handle large file uploads 
The storage layer is protected with read-write locks.
However, we may be holding the locks over unnecessarily large critical
sections.

The typical workflow is that a blob is first uploaded via a per-client
private session-id meaning the blob is not publicly visible yet. When
the blob being uploaded is very large, the transfer takes a long time
while holding the lock.

Private session-id based uploads don't really need locks, and hold locks
only when blobs are published after the upload is complete.
 2020-10-16 13:33:11 -07:00
Shivam Mishra
25ad71787a test: minimize trivy db download tests to avoid api rate limit 2020-10-15 14:32:37 -07:00
Shivam Mishra
8075eadc1a test: add wait for trivy db download in test case 2020-10-02 16:47:54 -07:00
Shivam Mishra
971404f6ee search/cve: fix log messages 2020-09-23 12:47:50 -07:00
Shivam Mishra
d63f715fe5 search/cve: exclude unsupported images from fixed-tag list. 
If image vulnerability scan does not support any media type, considering those images as an infected image and now this images will not be shown in fixed images list.

Fixes issue #130
 2020-09-22 09:24:04 -07:00
Shivam Mishra
cd0206fe6c Fixes issue #132, if image does not have any fixed tags, empty list with no error should be returned 2020-09-08 16:41:06 -07:00
Tanmay Naik
f5867ce0b6 cli: group CVEs by severity 2020-09-04 13:56:47 -04:00
Tanmay Naik
c590b86d14 cli: add commands for CVE 
Uses GraphQL API of zot to fetch CVE info

- Get all images affected by a CVE (input: CVEID)
- Get all CVEs of a layer (input: image:tag)
- Get all layers of an image which have resolved a CVE (input: image,
CVEID)
- Get all layers of an image affected by a CVE (input: image, CVEID)
 2020-08-21 12:42:01 -04:00
Shivam Mishra
5f230bd8ff Added unit test cases 2020-08-19 00:19:35 -07:00
Shivam Mishra
ed254159a0 Added support for searching fixed tag given cve and an image 2020-08-18 23:53:04 -07:00
Shivam Mishra
72ae02ca4b Added image format validation to validate media type because squashfs image media type not supported for vulnerability scanning 2020-08-18 23:05:52 -07:00
Shivam Mishra
2cf2c16137 Added graphql api feature for image vulnerability scanning 2020-08-18 22:44:34 -07:00
Shivam Mishra
baa5d247ec Enable trivy db download and update 2020-08-18 21:46:17 -07:00
Shivam Mishra
e537f27f00 Added search extension and integrated trivy to support image vulnerability scanning 2020-08-18 21:03:48 -07:00
Shivam Mishra
3a30290e08 Using "destRecord" as a path in DeleteBlob function instead of "dst". 
dstRecord :- blob path stored in cache.
dst :- blob path that is trying to be uploaded.

Currently, if the actual blob on disk may have been removed by GC/delete, during syncing the cache dst is being passed to DeleteBlob function and retry section is being continuously called because DeleteBlob function never deletes dst path (doesn't exist in db), dstRecord should be passed into DeleteBlob function because dstRecord is actual blob path stored in db.

If dst and dstRecord path value is same then this issue will not be produced and DeleteBlob method will delete the blob info from cache but if both are different then DeleteBlob method will try to delete dst path which is not in cache.

Note:- boltdb delete method return nil even when value doesn't exist (https://godoc.org/github.com/boltdb/bolt#Bucket.Delete)
 2020-08-12 10:06:20 -07:00
Tanmay Naik
6285a730a1 cli: add option to ignore TLS verification 
adds a property in config : "verify-tls"
 2020-07-17 17:48:42 -04:00