0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2024-12-23 22:27:35 -05:00
Commit graph

57 commits

Author SHA1 Message Date
Lisca Ana-Roberta
f264f26908
fix: error message when CVE search is disabled (#1100)
Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
2023-01-10 10:45:15 -08:00
LaurentiuNiculae
f408df0dac
feat(repodb): Implement RepoDB for image specific information using boltdb/dynamodb (#979)
* feat(repodb): implement a DB for image specific information using boltdb

(cherry picked from commit e3cb60b856)

Some other fixes/improvements on top (Andrei)

Global search: The last updated attribute on repo level is now computed correctly.
Global search: Fix and enhance tests: validate more fields, and fix CVE verification logic
RepoListWithNewestImage: The vendors and platforms at repo level are no longer containing duplicate entries
CVE: scan OCIUncompressedLayer instead of skiping them (used in tests)
bug(repodb): do no try to increment download counters for signatures

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

Add filtering to global search API (Laurentiu)

(cherry picked from commit a87976d635ea876fe8ced532e8adb7c3bb24098f)

Original work by Laurentiu Niculae <niculae.laurentiu1@gmail.com>

Fix pagination bug

 - when limit was bigger than the repo count result contained empty results
 - now correctly returns only maximum available number of repo results

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

Add history to the fields returned from RepoDB

Consolidate fields used in packages
- pkg/extensions/search/common/common_test
- pkg/extensions/search/common/common
Refactor duplicate code in GlobalSearch verification
Add vulnerability scan results to image:tag reply

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

Refactor ExpandedRepoInfo to using RepoDB

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit fd7dc85c3a9d028fd8860d3791cad4df769ed005)

Init RepoDB at startup
 - sync with storage
 - ignore images without a tag

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 359898facd6541b2aa99ee95080f7aabf28c2650)

Update request to get image:tag to use repodb

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

Sync RepoDB logging
 - added logging for errors

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 2e128f4d01712b34c70b5468285100b0657001bb)

sync-repodb minor error checking fix

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

Improve tests for syncing RepoDB with storage

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit b18408c6d64e01312849fc18b929e3a2a7931e9e)

Update scoring rule for repos
  - now prioritize matches to the end of the repo name

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 6961346ccf02223132b3b12a2132c80bd1b6b33c)

Upgrade search filters to permit multiple values
  - multiple values for os and arch

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 3ffb72c6fc0587ff827a03fe4f76a13b27b876a0)

feature(repodb): add pagination for RepoListWithNewestImage

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
(cherry picked from commit 32c917f2dc65363b0856345289353559a8027aee)

test(fix): fix tests failing since repodb is used for listing all repos

1. One of the tests was verifying disk/oci related erros and is not applicable
2. Another test was actually broken in an older PR, the default store and
the substore were using the same repo names (the substore ones were unprefixed),
which should not be the case, this was causing a single entry to show
in the RepoDB instead of two separate entries for each test image
Root cause in: b61aff62cd (diff-b86e11fa5a3102b336caebec3b30a9d35e26af554dd8658f124dba2404b7d24aR88)

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

chore: move code reponsible for transforming objects to gql_generated types to separate package

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

Process input for global search
  - Clean input: query, filter strings
  - Add validation for global search input

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit f1ca8670fbe4a4a327ea25cf459237dbf23bb78a)

fix: only call cve scanning for data shown to the user

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

GQL omit scanning for CVE if field is not required

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 5479ce45d6cb2abcf5fbccadeaf6f3393c3f6bf1)

Fix filtering logic in RepoDB
  - filter parameter was set to false instead of being calculator from the later image

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit a82d2327e34e5da617af0b7ca78a2dba90999f0a)

bug(repodb): Checking signature returns error if signed image is not found
  - we considere a signature image orfan when the image it signs is not found
  - we need this to ignore such signatures in certain cases

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit d0418505f76467accd8e1ee34fcc2b2a165efae5)

feat(repodb): CVE logic to use repoDB

Also update some method signatures to remove usage of:
github.com/google/go-containerregistry/pkg/v1

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

* feat(repodb): refactor repodb update logic

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* fix(repodb): minor fixes

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): move repodb logic inside meta directory under pkg

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): change factory class for repodb initialization with factory metrod

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): simplify repodb configuration
  - repodb now shares config parameters with the cache
  - config taken directly from storage config

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* fix(authors): fix authors information to work properly with repodb

Ideally this commit would be squshed in the repodb commit
but as-is it is easier to cherry-pick on other branches

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

* feat(repodb): dynamodb support for repodb
  - clean-up repodb code + coverage improvements

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(dynamo): tables used by dynamo are created automatically if they don't exists
  - if the table exists nothing happens

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* test(repodb): coverage tests
  - minor fix for CVEListForImage to fix the tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): add descriptor with media type

  - to represent images and multi-arch images

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): support signatures on repo level

  - added to follow the behavior of signing and signature verification tools
    that work on a manifest level for each repo
  - all images with different tags but the same manifest will be signed at once

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): old repodb version migration support

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): tests for coverage

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): WIP fixing tests

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): work on patchRepoDB tests

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* fix(repodb): create dynamo tables only for linux amd

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* fix(ci): fix a typo in ci-cd.yml

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Andrei Aaron <andaaron@cisco.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
2023-01-09 12:37:44 -08:00
alexstan12
8fb11180d4
fix: queries with images as arguments without a reference should return an error (#1040)
Currently there is no push-back on queries that should contain image names but
have only the repo name. This commit adds a check that will return an error for images
w/o a reference(tag or digest).

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
2022-12-09 11:40:06 -08:00
Catalin-George Hofnar
4170d2adbc
refactor(cache): rewrote/refactored cachedb functionality to use interface (#667)
Moved boltdb to a driver implementation for such interface
Added CreateCacheDatabaseDriver in controller
Fixed default directory creation (boltDB will only create the file, not the dir
Added coverage tests
Added example config for boltdb
Re-added caching on subpaths, rewrote CreateCacheDatabaseDriver
Fix tests
Made cacheDriver argument mandatory for NewImageStore, added more validation, added defaults
Moved cache interface to own file, removed useRelPaths from config
Got rid of cache config, refactored
Moved cache to own package and folder
Renamed + removed cache factory to backend, replaced CloudCache to RemoteCache
Moved storage constants back to storage package
moved cache interface and factory to storage package, changed remoteCache defaulting

Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
2022-11-02 15:53:08 -07:00
Andrei Aaron
92afd86cbb
feat(cve): better distinguish max severity on an image (#918)
Values returned now by GetCVESummaryForImage
// not scannable / error during scan  - max severity ""            - cve count 0   - Errors
// scannable no issues found          - max severity "NONE"        - cve count 0   - no Errors
// scannable issues found             - max severity from Scanner  - cve count >0  - no Errors

Before this change the max severity in case #1 and #2 was "UNKNOWN" which is also possible value
for case #3. To better distinguish them return different max severities.

This feature would be consumed by the UI.

Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-10-24 12:27:26 -07:00
Andrei Aaron
ac6c6a844c
refactor(digests): standardise representation of digests to digest.Digest (#898)
- Digests were represented by different ways
  - We needed a uniform way to represent the digests and enforce a format
  - also replace usage of github.com/google/go-containerregistry/pkg/v1
    with github.com/opencontainers/image-spec/specs-go/v1

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 96b2f29d6d57070a913ce419149cd481c0723815)
(cherry picked from commit 3d41b583daea654c98378ce3dcb78937d71538e8)

Co-authored-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2022-10-22 13:46:13 -07:00
Lisca Ana-Roberta
26d982becb
fix: replace time.sleep() with checking logs (#899)
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2022-10-21 11:17:06 -07:00
Ramkumar Chinchani
763287873e
fix(config): make all extension config consistent (#888)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-10-21 15:33:54 +03:00
Andrei Aaron
38b00e3507
chore(lint): gci to separate zot from other imports (#870)
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-10-20 09:39:20 -07:00
peusebiu
91dd5496fc
chore: rename search route prefix (#887)
* chore: rename search route prefix

* chore: use builtin time.Duration.Truncate() on latencies

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-10-18 20:46:06 -07:00
Nicol
33a431ef43
Update go version to 1.19 (#829)
* ci: Update go version to 1.19

Signed-off-by: Nicol Draghici <idraghic@cisco.com>

* ci: Fix lint issues

Signed-off-by: Nicol Draghici <idraghic@cisco.com>

* ci: Added needprivileges to lint, made needprivileges pass lint

Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Signed-off-by: Nicol Draghici <idraghic@cisco.com>

Signed-off-by: Nicol Draghici <idraghic@cisco.com>
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Co-authored-by: Catalin Hofnar <catalin.hofnar@gmail.com>
2022-10-05 13:21:14 +03:00
peusebiu
8237f8d20a
storage: Move common code in helper functions, closes #730 (#820)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-09-30 10:35:16 -07:00
Andrei Aaron
e0d808b196
Include image vulnerability information in ImageSummary (#798)
Return this data as part of GlobalSearch and RepoListWithNewestImage
query results.
This commit also includes refactoring of the CVE scanning logic in
order to better encapsulate trivy specific logic, remove CVE scanning
logic from the graphql resolver.

Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-09-28 11:39:54 -07:00
slab713
8ffb053cec
Replaced deprecated io/ioutil functions (#768)
Signed-off-by: slab713 <109306207+slab713@users.noreply.github.com>
2022-09-02 15:56:02 +03:00
Roxana Nemulescu
ab9a20c1ae Add GraphQL API for getting the information necessary to list images in the zot cli without download manifests.
If this GraphQL API is available, try that first, else fallback to the slowpath.

Signed-off-by: Roxana Nemulescu <roxana.nemulescu@gmail.com>
2022-08-23 16:32:00 +03:00
Lisca Ana-Roberta
87fc941b3c image level lint: enforce manifest mandatory annotations
closes #536

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2022-07-27 11:48:04 +03:00
Laurentiu Niculae
7e3d063319 freeform querry api
Signed-off-by: Laurentiu Niculae <themelopeus@gmail.com>
2022-07-20 10:03:11 -07:00
Andrei Aaron
43160dcc43 Update to graphql 1.17.13
We encountered some problems with using the existing folder structure,
but it looks like running the tooling with the latest versions works after
we regenerated the project using 'gql init' and refactoring to separate
the login previously in resolvers.go.

- the autogenerated code is now under the gql_generated folder
- the file resolvers.go now contains only the code which is not
rewritten by the gqlgen framework
- the file schema.resolvers.go is rewritten when gqlgen runs,
and we'll only keep there the actual resolvers matching query names
Changes we observed to schema.resolvers.go when gqlgen runs include
reordering methods, and renaming function parameters to match the
names used in schema.graphql
- we now have a gqlgen.yaml config file which governs the behavior of
gqlgen (can be tweaked to restructure the folder structure of the
generated code in the future)

Looks like the new graphql server has better validation
1 Returns 422 instead of 200 for missing query string - had to update tests
2 Correctly uncovered an error in a test for a bad `%` in query string.

As as result of 2, a `masked` bug was found in the way we check if images are
signed with Notary, the signatures were reasched for with the media type
of the image manifest itself instead of the media type for notation.
Fixed this bug, and improved error messages.
This bug would have also been reproducible with main branch if the bad `%`
in the test would have fixed.

Updated the linter to ignore some issues with the code which is
always rewritten when running:
`go run github.com/99designs/gqlgen@v0.17.13 generate`

Add a workflow to test gqlgen works and has no uncommitted changes

Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-18 12:55:40 -07:00
Alex Stan
ada21ed842 Manage builds with different combinations of extensions
Files were added to be built whether an extension is on or off.
New build tags were added for each extension, while minimal and extended disappeared.

added custom binary naming depending on extensions used and changed references from binary to binary-extended

added automated blackbox tests for sync, search, scrub, metrics

added contributor guidelines

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
2022-06-30 09:53:52 -07:00
Shivam Mishra
36c9631000 ext: use distribution spec route prefix for extension api
Following the spec defined here https://github.com/opencontainers/distribution-spec/tree/main/extensions

Signed-off-by: Shivam Mishra <shimish2@cisco.com>
2022-05-22 16:35:16 -07:00
Alex Stan
d325c8b5f4 Fix problems signaled by new linter version v1.45.2
PR (linter: upgrade linter version #405) triggered lint job which failed
with many errors generated by various linters. Configurations were added to
golangcilint.yaml and several refactorings were made in order to improve the
results of the linter.

maintidx linter disabled

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
2022-04-27 09:55:44 -07:00
Petu Eusebiu
be910cf01c lint: Move out config reloader context from controller struct
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-03-24 10:50:01 -07:00
Shivam Mishra
b8010e1ee4 routes: changes required to do browser authentication
whenever we make a request that contains header apart from CORS allowed header, browser sends a preflight request
and in response accept *Access-Control-Allow-Headers*.

preflight request is in form of OPTIONS method, added new http handler func to set headers
and returns HTTP status ok in case of OPTIONS method.

in case of authorization, request contains authorization header
added authorization header in Access-Control-Allow-Headers list

added AllowOrigin field in HTTPConfig this field value is set to Access-Control-Allow-Origin header and will give zot adminstrator to limit incoming request.

Signed-off-by: Shivam Mishra <shimish2@cisco.com>
2022-03-08 17:42:54 -08:00
Eng Zer Jun
0d77b60de7 test: use T.TempDir to create temporary test directory
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-03-07 12:28:49 -08:00
Ramkumar Chinchani
38a110314b gc: add a gcDelay param
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-14 14:03:22 -08:00
Alexei Dodon
47c9b6244e Added config enable=true/false for extensions with default value as enabled closes #258
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2022-02-09 09:53:49 -08:00
Ramkumar Chinchani
d2aa016cdb storage: flush/sync contents to disk on file close
Behavior controlled by configuration (default=off)
It is a trade-off between performance and consistency.

References:
[1] https://github.com/golang/go/issues/20599

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-07 12:08:01 -08:00
Ramkumar Chinchani
87084f286b storage: improve/fix oci image validation
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-02-02 13:31:41 -08:00
Ramkumar Chinchani
ac3801ea2d lint: upgrade golangci-lint
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-20 17:20:35 -08:00
Ramkumar Chinchani
96226af869 move references to zotregistry.io and project-zot
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-05 10:52:27 -08:00
Alexei Dodon
f99fa37623 ci/cd: unit test hangs for a long time intermittently closes #286
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2021-11-29 14:07:49 -08:00
Alexei Dodon
e900b09cfb Fix data races in tests, closes #255
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2021-11-17 13:23:59 -08:00
Alexei Dodon
8e4d828867 Implement an API for performance monitoring
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2021-11-12 11:14:10 -08:00
Petu Eusebiu
19003e8a71 Added new extension "sync"
Periodically poll registries and pull images according to sync's config
Added sync on demand, syncing when clients asks for an image which
zot doesn't have.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-10-21 10:32:46 -07:00
Shivam Mishra
d930adbd49 search: update trivy
trivy updated to v0.20.0
trivy-db updated to bec0c6a
fanal updated to f7efd1b
2021-10-13 16:37:31 -07:00
Petu Eusebiu
7d077eaf5a Added storage interface 2021-10-11 17:38:46 -07:00
Shivam Mishra
63fef3e48c search: added graphql api to return repository list with latest tag 2021-09-27 14:36:20 -07:00
Petu Eusebiu
3a59b9f487 Use freeport to get ports for zot servers in tests 2021-06-29 13:58:39 -07:00
Andrei Aaron
792e82cbdf Add an 'enable' flag in the server configuration to enable gql-based searches
"extensions": {
        "search": {
            "enable": true
        }
    }
2021-06-24 12:15:25 -07:00
Andrei Aaron
519ea75d9a Implement a way to search for an image by manifest, config or layer digest
```
Usage:
  zot images [config-name] [flags]

Flags:
  -d, --digest string   List images containing a specific manifest, config, or layer digest
[...]
```
2021-06-24 12:15:25 -07:00
Shivam Mishra
28974e81dc config: support multiple storage locations
added support to point multiple storage locations in zot by running multiple instance of zot in background.

see examples/config-multiple.json for more info about config.

Closes #181
2021-05-21 10:18:28 -07:00
Shivam Mishra
a7c17b7c16 spec: added support for mount request using hard link 2021-05-04 09:42:29 -07:00
Ramkumar Chinchani
affdd85986 build: remove bazel
The idea initially was to use bazel to do our builds, however golang
build system is now good enough and our code base is entirely go.
It is also slowing down our travis ci/cd pipeline.
2020-12-21 15:30:13 -08:00
Shivam Mishra
b0ed625a2e build: increase wait timeout for travis bazel build process 2020-10-27 19:30:06 -07:00
Shivam Mishra
46beb30fc1 build: add build tags to create customizable binaries 2020-10-22 17:20:07 -07:00
Shivam Mishra
14214a5794 test: add unit test to verify lock changes 2020-10-16 14:58:45 -07:00
Shivam Mishra
25ad71787a test: minimize trivy db download tests to avoid api rate limit 2020-10-15 14:32:37 -07:00
Shivam Mishra
8075eadc1a test: add wait for trivy db download in test case 2020-10-02 16:47:54 -07:00
Shivam Mishra
971404f6ee search/cve: fix log messages 2020-09-23 12:47:50 -07:00
Shivam Mishra
d63f715fe5 search/cve: exclude unsupported images from fixed-tag list.
If image vulnerability scan does not support any media type, considering those images as an infected image and now this images will not be shown in fixed images list.

Fixes issue #130
2020-09-22 09:24:04 -07:00