0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2024-12-16 21:56:37 -05:00
Commit graph

215 commits

Author SHA1 Message Date
LaurentiuNiculae
112fbec5b6
refactor(cli): added equivalent subcommands for each flag combination under every command (#1674)
- image command is now deprecated in favor of 'images'
- cve command is now deprecated in favor of 'cves'

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-08-30 20:12:24 +03:00
LaurentiuNiculae
40b599cca8
refactor(log): replace default logger with config complient log in root.go (#1734)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-08-29 12:10:30 -07:00
Andrei Aaron
28858f695f
feat(mgmt): mgmt extention no longer depends on UI being enabled (#1728)
It is now enabled based only on search configuration

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-08-26 13:32:41 -07:00
Damien Degois
289acfabbd
feat(authn): add generic oidc and allow customizable name (#1691)
Rebased and squashed

Signed-off-by: Damien Degois <damien@degois.info>
2023-08-24 12:33:35 +03:00
LaurentiuNiculae
0731fd3828
test(refactor): refactor tests that use zot-test to use smaller images (#1690)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-08-18 11:46:11 +03:00
Andreea Lupu
76277f5ebd
fix: remove inline GC and schedule a background task instead (#1610)
* fix: remove inline GC and set a default value of gc interval

- remove inline GC
- add a default value of GC interval
- run the GC periodically by default with the default value if no interval provided
- generate GC tasks with a random delay(0-30s) between
- add IsReady() method to scheduler.TaskGenerator interface

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>

* ci: add test for gc with short interval

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>

---------

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-08-07 12:55:19 -07:00
LaurentiuNiculae
fce9a02ed5
fix(convert): now returned annotations for an index will fallback to annotations from a random manifest if the annotations field is not present on the index manifest (#1667)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-08-07 10:25:05 +03:00
Andrei Aaron
77149aa85c
refactor(extensions)!: refactor the extensions URLs and errors (#1636)
BREAKING CHANGE: The functionality provided by the mgmt endpoint has beed redesigned - see details below
BREAKING CHANGE: The API keys endpoint has been moved -  see details below
BREAKING CHANGE: The mgmt extension config has been removed - endpoint is now enabled by having both the search and the ui extensions enabled
BREAKING CHANGE: The API keys configuration has been moved from extensions to http>auth>apikey

mgmt and imagetrust extensions:
- separate the _zot/ext/mgmt into 3 separate endpoints: _zot/ext/auth, _zot/ext/notation, _zot/ext/cosign
- signature verification logic is in a separate `imagetrust` extension
- better hanling or errors in case of signature uploads: logging and error codes (more 400 and less 500 errors)
- add authz on signature uploads (and add a new middleware in common for this purpose)
- remove the mgmt extension configuration - it is now enabled if the UI and the search extensions are enabled

userprefs estension:
- userprefs are enabled if both search and ui extensions are enabled (as opposed to just search)

apikey extension is removed and logic moved into the api folder
- Move apikeys code out of pkg/extensions and into pkg/api
- Remove apikey configuration options from the extensions configuration and move it inside the http auth section
- remove the build label apikeys

other changes:
- move most of the logic adding handlers to the extensions endpoints out of routes.go and into the extensions files.
- add warnings in case the users are still using configurations with the obsolete settings for mgmt and api keys
- add a new function in the extension package which could be a single point of starting backgroud tasks for all extensions
- more clear methods for verifying specific extensions are enabled
- fix http methods paired with the UI handlers
- rebuild swagger docs

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-08-02 21:58:34 +03:00
LaurentiuNiculae
9e38ca51e3
feat(pagination): move pagination and sorting image summary results after conversion (#1637)
fix(config): check for config media type when pushing to repodb

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-07-31 22:16:09 +03:00
LaurentiuNiculae
a5c92bcac3
test(refactor): change upload image function signature (#1655)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-07-28 17:53:46 +03:00
LaurentiuNiculae
abba6aa3cf
feat(test): added image builder for use in tests (#1583)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-07-26 13:08:04 +03:00
peusebiu
4194489868
feat: signal handling (#1651)
gracefully shutdown app

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-25 10:39:13 -07:00
peusebiu
86a83ca6e3
fix(authn): fix several issues with authn, closes #1632 (#1633)
- apply Access-Control-Allow-Credentials only if authn is enabled
- enable Logout route for basic auth
- fixed Logout godoc
- fix Access-Control-Allow-Methods on Logout route
- added allowOrigin option in config example

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-19 09:27:04 -07:00
LaurentiuNiculae
28de980319
feat(refator): refactoring repodb into meta (#1626)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-07-18 10:27:26 -07:00
Andrei Aaron
74f769d2c1
fix(cli): make sure all needed ImageSummary properties are requested from the server (#1618)
Resolves #1597

Fix missing properties in some calls OS/Arch/IsSigned/LastUpdated.
Since the properties were missing from the graphql requests, zli was showing default values.

Update the tests to generate a tets image with the created date different from
the go detault value for time.Time{}. The tests are now checking a non-default
value in the responses client side, in order to avoid missing issues like this in the future.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-07-14 09:22:14 +03:00
Andrei Aaron
e3ac50b2a2
feat(zli): add new subcommand to remove a zli config (#1619)
https://github.com/project-zot/zot/issues/1594
- use indent '  ' in the config json

https://github.com/project-zot/zot/issues/1593
- add subcommand `zli config remove`

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-07-13 09:34:01 -07:00
peusebiu
d3f27b4ba6
feat: add additional manifest validations (#1609)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-13 09:31:39 -07:00
Andrei Aaron
faa410a0c3
feat(cli): Fix multiple issues with zli output (#1612)
https://github.com/project-zot/zot/issues/1591
    - I will rename "IMAGE NAME" to "REPOSITORY" in order to make the header easier to parse
    - The order of the images cannot be predicted if zot is getting them 1 by 1 using the REST API for manifests, so they cannot be sorted when printed. We could wait on all calls to return but that may take minutes, and printing partial results as they become available is better.
    - The order of the images can be predicted when relying on the zot specific search API, but that is not available in all zot servers depending on build options. I added sorting ascending by default. We are planning to implement configurable sorting in a separate PR - see the work under https://github.com/project-zot/zot/pull/1577
    - With regards to the column widths/alignments that was discussed before, and the issue is we don't know the values beforehand for the REST API based responses. As mentioned above printing partial results as they become available is better.
    - The column widths/alignments are partially fixed in this PR for the search API, but we should properly fix this in - see https://github.com/project-zot/zot/pull/851

https://github.com/project-zot/zot/issues/1592
    - Fix missing space after help message

https://github.com/project-zot/zot/issues/1598
    - Fix table headers showing for json/yaml format
    - Fix spacing shown with json format, use 1 row per shown entry in order to be compatible with json lines format: https://jsonlines.org/
    - Add document header `---` to every image shown in yaml format to separate the entries

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-07-12 10:21:12 -07:00
peusebiu
17d1338af1
feat: integrate openID auth logic and user profile management (#1381)
This change introduces OpenID authn by using providers such as Github,
Gitlab, Google and Dex.
User sessions are now used for web clients to identify
and persist an authenticated users session, thus not requiring every request to
use credentials.
Another change is apikey feature, users can create/revoke their api keys and use them
to authenticate when using cli clients such as skopeo.

eg:
login:
/auth/login?provider=github
/auth/login?provider=gitlab
and so on

logout:
/auth/logout

redirectURL:
/auth/callback/github
/auth/callback/gitlab
and so on

If network policy doesn't allow inbound connections, this callback wont work!

for more info read documentation added in this commit.

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
2023-07-07 09:27:10 -07:00
peusebiu
49e4d93f42
fix(test): TestConfigReloader, wait for trivy db download (#1543)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-06 14:17:49 +03:00
LaurentiuNiculae
0a04b2a4ed
feat(cve): implemented trivy image scan for multiarch images (#1510)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-07-06 11:36:26 +03:00
Andrei Aaron
7fee57e7cc
fix(CVE): attempt to scan now returns early with an error if trivyDB metadata json is missing (#1548)
Also modify zli to retry in case of such errors,
assuming the trivyDB will eventually be downloaded by the scheduled task.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-07-05 09:08:16 -07:00
LaurentiuNiculae
809529be18
fix(cli): add help message for searching referrers under search command (#1551)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-06-23 17:44:30 +03:00
LaurentiuNiculae
620287c7a4
feat(cli): add referrers and search commands to cli (#1497)
* feat(cli): add referrers command to cli

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(cli): add global search command

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(cli): fix comments

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

---------

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-06-22 10:43:01 -07:00
Andrei Aaron
96d00cd0ef
fix(cve): Fix CVE scanning in images containing Jar files (#1475) 2023-06-01 00:37:46 +03:00
peusebiu
612a12e5a8
refactor(sync): use task scheduler (#1301)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-05-31 10:26:23 -07:00
LaurentiuNiculae
a3f355c278
refactor(storage): refactoring storage (#1459)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-26 11:08:19 -07:00
Lisca Ana-Roberta
6a7035c599
fix: removed duplicate structures from service.go and moved them to pkg/common (#1436)
Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
2023-05-25 11:27:49 -07:00
Lisca Ana-Roberta
e262fbea64
feat: verifying and enabling necessary extensions for ui (#1369)
Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
2023-05-12 09:43:14 -07:00
LaurentiuNiculae
ea79be64da
refactor(artifact): remove oci artifact support (#1359)
* refactor(artifact): remove oci artifact support
- add header to referrers call to indicated applied artifact type filters

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(gc): simplify gc logic to increase coverage

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

---------

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-05-10 10:15:33 -07:00
Ramkumar Chinchani
3dd228e3e3
fix(serve): gracefully exit with error instead of panic (#1396) 2023-05-01 08:26:23 +03:00
Nicol
c169698c95
feat: remove usage of zerolog.Logger.Msgf() from zot code (#1382)
Signed-off-by: Nicol Draghici <idraghic@cisco.com>
2023-04-27 19:44:22 -07:00
Andreea Lupu
40bf76add5
chore(go.mod): upgrade trivy and cosign (#1387)
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-04-27 09:35:10 -07:00
Nicol
0586c6227e
refactor: remove pkg/extensions/search/common and move the code to the appropriate packages (#1358)
Signed-off-by: Nicol Draghici <idraghic@cisco.com>
2023-04-18 11:07:47 -07:00
Lisca Ana-Roberta
28ce1c7e71
fix: replaced printing the port with logging it when chosen dynamically (#1339)
Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
2023-04-06 13:33:56 +03:00
LaurentiuNiculae
21b7c69fd9
feat(cli): updated display format for multiarch images (#1268)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-03-21 10:16:00 -07:00
Lisca Ana-Roberta
eea6f3f85a
fix(cve): Search by CVE title/id (full or partial) when listing an image's CVEs (#1264)
Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
2023-03-16 12:13:07 -07:00
peusebiu
f04e66a5e2
feat(mgmt): added mgmt extension which returns current zot configuration (#1198)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-03-09 10:43:26 -08:00
Lisca Ana-Roberta
336526065f
feat(groups)!: added "groups" mechanism for authZ (#1123)
BREAKING CHANGE: repository paths are now specified under a new config key called "repositories" under "accessControl" section in order to handle "groups" feature. Previously the repository paths were specified directly under "accessControl".

This PR adds the ability to create groups of users which can be used for authZ policies, instead of just users.

{
"http": {
   "accessControl": {
       "groups": {

Just like the users, groups can be part of repository policies/default policies/admin policies. The 'groups' field in accessControl can be missing if there are no groups. The permissions priority is user>group>default>admin policy, verified in this order (in authz.go), and permissions are cumulative. It works with LDAP too, and the group attribute name is configurable. The DN of the group is used as the group name and the functionality is the same. All groups for the given user are added to the context in authn.go. Repository paths are now specified under a new keyword called "repositories" under "accessControl" section in order to handle "groups" feature.

Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
2023-03-08 11:47:15 -08:00
Nicol
23efe24c72
refactor(test): update cve tests to stop duplicating test/data if not needed (#1232)
Signed-off-by: Nicol Draghici <idraghic@cisco.com>
2023-03-02 09:45:23 -08:00
LaurentiuNiculae
d62c09e2cc
feat(repodb): Multiarch Image support (#1147)
* feat(repodb): index logic + tests

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(cli): printing indexes support using the rest api

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

---------

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-02-27 11:23:18 -08:00
Nicol
f00a9e6e48
perf: update the ImageList queries to return PaginatedImagesResult (#1182)
Signed-off-by: Nicol Draghici <idraghic@cisco.com>
2023-02-15 11:34:07 -08:00
Ramkumar Chinchani
d79ad863e4
fix: set GC delay defaults for storage subPaths (#1189)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-02-14 09:16:37 -08:00
Andreea Lupu
ee95ab0ffc
fix: call notation-go libs instead of using notation binary (#1104)
fix: add loading notation path

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
Co-authored-by: Roxana Nemulescu <roxana.nemulescu@gmail.com>
2023-02-13 10:43:52 -08:00
Andrei Aaron
c0aaca8ed1
feat(ui): package zui within zot binary (#1161)
(cherry picked from commit d557da0baba819b7cd7e6b5941528776e125ac6d)

build(ui): fix stacker builds


(cherry picked from commit ba25daf02b4a9bc7ee1cb6f84b7a6b096ca7d61f)

build(ui): various fixes

- Fix metrics endpoint
- Fix unit tests unit tests
- Make the ui build optional in the makefile
before the linter lint runs in the golangci-lint workflow
- Do not attempt to include UI routes if search is enabled
- Fix authorization for search endpoint



fix: use zot tag in ui make target


(cherry picked from commit 2a6882fa23f06b2d68c6c299773a6ff50bf90e78)

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-02-10 14:52:54 -08:00
Andrei Aaron
d12836e69c
refactor(cve): improve CVE test time by mocking trivy (#1184)
- refactor(cve): remove the global of type cveinfo.CveInfo from the extensions package
  Replace it with an attribute on controller level
- refactor(controller): extract initialization logic from controller.Run()
- test(cve): mock cve scanner in cli tests

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-02-09 21:04:52 -08:00
Andrei Aaron
af222de108
test: refactor CVE tests in CLI package (#1170)
Changes in this commit:

A.
There were nested conveys following the pattern:
- outer convey was a positive test
- inner conveys were negative tests
The positive and negative tests did not depend on one another, but the positive test was repeated for each negative test.

So for example if there was 1 positive test and 2 negative tests what go would do is:
1. execute the outer convey with the positive test
2. execute the 1st inner convey with the 1st negative test
3. execute the outer convey with the positive test
4. execute the 2nd inner convey with the 2nd negative test

Since there were no dependencies between these tests they can be moved on the same level.

B.
The test function bringing up the controller for `TestServerCVEResponseGQL` was the same for `TestServerCVEResponse`, so I removed `TestServerCVEResponseGQL` and moved the Conveys under `TestServerCVEResponse`. This will save time on the initial TrivyDB download, since it's done only once when the controller starts.

C. The `Convey("without flags", func() {` was actually the same as the test under which it was nested: `Convey("Test CVE invalid url port", t, func() {` no code differences so I removed it.

D. Use the new test functions to start/stop and wait for the zot contoller to be reachable

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-02-06 13:16:28 -08:00
Lisca Ana-Roberta
976ccfcf0d
fix: removed references to old dist-spec (#1128)
Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
2023-01-31 09:35:33 -08:00
Andrei Aaron
feb7328f50
feat(repodb): DerivedImageList and BaseImageList make use of RepoDB (#1135)
- derivedImageList and baseImageList now use FilterTags to obtain results,
each with its own filter function
- images that have the exact same manifest as the one provided as a
parameter are no longer considered base images or derived images
- both calls can be made with specific pagination parameters, and the
response will include PageInfo

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>

fix(tests): fix one of the pagination tests

The results were not reliable as the 2 returned tags were sorted by created date/time
which was not set, resulting in an unpredictable order

Signed-off-by: Andrei Aaron <andaaron@cisco.com>
(cherry picked from commit be504200a1127371422aeb0e5c0219e2a1ead20a)
(cherry picked from commit ed8d797e639f262a63840120afe92da7db9a7600)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
2023-01-25 14:06:02 -08:00
Andrei Aaron
08983a845a
feat(repodb): implement pagination for ImageList and integrate it with RepoDB (#1129)
* feat(repodb): implement pagination for ImageList and integrate it with RepoDB

- it can now return all images from all repos, when provided repo parameter is ""

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
(cherry picked from commit c003dcec9f805564946935e7eb091632f605035e)
(cherry picked from commit 72feba979b9ddd452465a652bb31f439584a046c)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

* ci(timeouts): increase ci-cd workflow timeout for the build and test step

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
2023-01-23 09:45:11 -08:00
Nicol
f0d947809b
test(refactor): cleanup/simplify testcases (#1124)
Signed-off-by: Nicol Draghici <idraghic@cisco.com>
2023-01-20 10:01:52 -08:00
Andrei Aaron
d960c2e23f
fix(ci): do not use "ghcr.io/aquasecurity/trivy-db" for ci (#1107)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-01-18 14:18:03 -08:00
Andrei Aaron
fac1d1d05d
chore(trivy): update trivy version and enforce OCI compliant repo names in local image storage (#1068)
1. chore(trivy): update trivy library version

The trivy team switched github.com/urfave/cli for viper so
there are some other code changes as well.

Since we don't use github.com/urfave/cli directly in our software
we needed to add a tools.go in order for "go mod tidy" to not delete it.
See this pattern explained in:
- https://github.com/99designs/gqlgen#quick-start
- https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module
- https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md#walk-through

The jobs using "go get -u" have been updated to use "go install", since go get
modifies the go.mod by upgrading some of the packages, but downgrading trivy to an older
version with broken dependencies

2. fix(storage) Update local storage to ignore folder names not compliant with dist spec
Also updated trivy to download the DB and cache results under the rootDir/_trivy folder

3. fix(s3): one of the s3 tests was missing the skipIt call
This caused a failure when running locally without s3 being available

4. make sure the offline scanning is enabled, and zot only downloads the trivy DB
on the regular schedule, and doesn't download the DB on every image scan

ci: increase build and test timeout as tests are reaching the limit more often

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-01-18 08:24:44 -08:00
Andrei Aaron
e8e7c343ad
feat(repodb): add pagination for ImageListForDigest and implement FilterTags (#1102)
* feat(repodb): add pagination for ImageListForDigest and implement FilterTags

ImageListForDigest can now return paginated results, directly from DB.
It uses FilterTags, a new method to filter tags (obviously) based on
the criteria provided in the filter function.
Pagination of tags is now slightly different, it shows all results if
no limit and offset are provided.

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>

bug(tests): cli tests for digests expecting wrong size

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
(cherry picked from commit 369216df931a4053c18278a8d89f86d2e1e6a436)

fix(repodb): do not include repo metadata in search results if no matching tags are identified

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

* fix(repodb): Fix an issue in FilterTags where repometa was not proceesed correctly

The filter function was called only once per manifest digest.
The function is supposed to also take into consideration repometa,
but only the first repometa-manifestmeta pair was processed.

Also increase code coverage.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
2023-01-18 00:31:54 +02:00
Nicol
c11c69c351
refactor: Cleanup/simplify testcases in /pkg/cli (#1103)
Signed-off-by: Nicol Draghici <idraghic@cisco.com>
2023-01-16 21:52:50 -08:00
Lisca Ana-Roberta
261fc2fb38
fix: added error message for missing CVEs (#1085)
Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
2023-01-11 15:02:29 -08:00
LaurentiuNiculae
f408df0dac
feat(repodb): Implement RepoDB for image specific information using boltdb/dynamodb (#979)
* feat(repodb): implement a DB for image specific information using boltdb

(cherry picked from commit e3cb60b856)

Some other fixes/improvements on top (Andrei)

Global search: The last updated attribute on repo level is now computed correctly.
Global search: Fix and enhance tests: validate more fields, and fix CVE verification logic
RepoListWithNewestImage: The vendors and platforms at repo level are no longer containing duplicate entries
CVE: scan OCIUncompressedLayer instead of skiping them (used in tests)
bug(repodb): do no try to increment download counters for signatures

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

Add filtering to global search API (Laurentiu)

(cherry picked from commit a87976d635ea876fe8ced532e8adb7c3bb24098f)

Original work by Laurentiu Niculae <niculae.laurentiu1@gmail.com>

Fix pagination bug

 - when limit was bigger than the repo count result contained empty results
 - now correctly returns only maximum available number of repo results

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

Add history to the fields returned from RepoDB

Consolidate fields used in packages
- pkg/extensions/search/common/common_test
- pkg/extensions/search/common/common
Refactor duplicate code in GlobalSearch verification
Add vulnerability scan results to image:tag reply

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

Refactor ExpandedRepoInfo to using RepoDB

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit fd7dc85c3a9d028fd8860d3791cad4df769ed005)

Init RepoDB at startup
 - sync with storage
 - ignore images without a tag

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 359898facd6541b2aa99ee95080f7aabf28c2650)

Update request to get image:tag to use repodb

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

Sync RepoDB logging
 - added logging for errors

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 2e128f4d01712b34c70b5468285100b0657001bb)

sync-repodb minor error checking fix

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

Improve tests for syncing RepoDB with storage

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit b18408c6d64e01312849fc18b929e3a2a7931e9e)

Update scoring rule for repos
  - now prioritize matches to the end of the repo name

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 6961346ccf02223132b3b12a2132c80bd1b6b33c)

Upgrade search filters to permit multiple values
  - multiple values for os and arch

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 3ffb72c6fc0587ff827a03fe4f76a13b27b876a0)

feature(repodb): add pagination for RepoListWithNewestImage

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
(cherry picked from commit 32c917f2dc65363b0856345289353559a8027aee)

test(fix): fix tests failing since repodb is used for listing all repos

1. One of the tests was verifying disk/oci related erros and is not applicable
2. Another test was actually broken in an older PR, the default store and
the substore were using the same repo names (the substore ones were unprefixed),
which should not be the case, this was causing a single entry to show
in the RepoDB instead of two separate entries for each test image
Root cause in: b61aff62cd (diff-b86e11fa5a3102b336caebec3b30a9d35e26af554dd8658f124dba2404b7d24aR88)

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

chore: move code reponsible for transforming objects to gql_generated types to separate package

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

Process input for global search
  - Clean input: query, filter strings
  - Add validation for global search input

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit f1ca8670fbe4a4a327ea25cf459237dbf23bb78a)

fix: only call cve scanning for data shown to the user

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

GQL omit scanning for CVE if field is not required

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 5479ce45d6cb2abcf5fbccadeaf6f3393c3f6bf1)

Fix filtering logic in RepoDB
  - filter parameter was set to false instead of being calculator from the later image

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit a82d2327e34e5da617af0b7ca78a2dba90999f0a)

bug(repodb): Checking signature returns error if signed image is not found
  - we considere a signature image orfan when the image it signs is not found
  - we need this to ignore such signatures in certain cases

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit d0418505f76467accd8e1ee34fcc2b2a165efae5)

feat(repodb): CVE logic to use repoDB

Also update some method signatures to remove usage of:
github.com/google/go-containerregistry/pkg/v1

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

* feat(repodb): refactor repodb update logic

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* fix(repodb): minor fixes

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): move repodb logic inside meta directory under pkg

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): change factory class for repodb initialization with factory metrod

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): simplify repodb configuration
  - repodb now shares config parameters with the cache
  - config taken directly from storage config

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* fix(authors): fix authors information to work properly with repodb

Ideally this commit would be squshed in the repodb commit
but as-is it is easier to cherry-pick on other branches

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

* feat(repodb): dynamodb support for repodb
  - clean-up repodb code + coverage improvements

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(dynamo): tables used by dynamo are created automatically if they don't exists
  - if the table exists nothing happens

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* test(repodb): coverage tests
  - minor fix for CVEListForImage to fix the tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): add descriptor with media type

  - to represent images and multi-arch images

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): support signatures on repo level

  - added to follow the behavior of signing and signature verification tools
    that work on a manifest level for each repo
  - all images with different tags but the same manifest will be signed at once

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): old repodb version migration support

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): tests for coverage

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): WIP fixing tests

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* feat(repodb): work on patchRepoDB tests

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* fix(repodb): create dynamo tables only for linux amd

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>

* fix(ci): fix a typo in ci-cd.yml

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Andrei Aaron <andaaron@cisco.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
2023-01-09 12:37:44 -08:00
Lisca Ana-Roberta
14238d4a8d
fix: removed resty calls from sync (#1016)
Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
2022-12-22 10:19:42 -08:00
Ramkumar Chinchani
69f0cf6bb4
fix(config): warn if cve is used with remote storage driver (#1034)
* fix(config): warn if cve is used with remote storage driver

Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>

* fix: also check if search is enabled

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Co-authored-by: Catalin Hofnar <catalin.hofnar@gmail.com>
2022-11-30 09:35:06 +02:00
Catalin-George Hofnar
31b9481713
feat(cache): dynamodb implementation (#953)
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
2022-11-22 10:29:57 -08:00
Lisca Ana-Roberta
4e13619dc8
fix: signatures now showing correctly (#954)
Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
2022-11-15 09:06:25 -08:00
Lisca Ana-Roberta
eb722905cb
fix: fixed image size not counting config and manifest sizes (#937)
Signed-off-by: Ana-Roberta Lisca <ana.kagome@yahoo.com>
2022-11-04 10:27:34 -07:00
Catalin-George Hofnar
4170d2adbc
refactor(cache): rewrote/refactored cachedb functionality to use interface (#667)
Moved boltdb to a driver implementation for such interface
Added CreateCacheDatabaseDriver in controller
Fixed default directory creation (boltDB will only create the file, not the dir
Added coverage tests
Added example config for boltdb
Re-added caching on subpaths, rewrote CreateCacheDatabaseDriver
Fix tests
Made cacheDriver argument mandatory for NewImageStore, added more validation, added defaults
Moved cache interface to own file, removed useRelPaths from config
Got rid of cache config, refactored
Moved cache to own package and folder
Renamed + removed cache factory to backend, replaced CloudCache to RemoteCache
Moved storage constants back to storage package
moved cache interface and factory to storage package, changed remoteCache defaulting

Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
2022-11-02 15:53:08 -07:00
Catalin-George Hofnar
c6ffbce6cf
refactor(tests): removed globals from digest test, removed some more hardcoded digests (#923)
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
2022-10-26 19:14:16 +03:00
Andrei Aaron
1d9c88c313
fix(cli): do not show signatures and fix tls verification client side (#904)
Issues fixed:
- the cli calls reaching out to the catalog endpoint used to request signature manifests
- resty was used instead of the cli http client to check if the discovery api was available
but it did not take into account TLS verification configuration
(testing locally withself-signed certificates did not work)

(cherry picked from commit ca42031ae9b1ceb459f5cd4f86cb82b3c9f78157)
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-10-22 23:44:20 -07:00
Andrei Aaron
ac6c6a844c
refactor(digests): standardise representation of digests to digest.Digest (#898)
- Digests were represented by different ways
  - We needed a uniform way to represent the digests and enforce a format
  - also replace usage of github.com/google/go-containerregistry/pkg/v1
    with github.com/opencontainers/image-spec/specs-go/v1

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 96b2f29d6d57070a913ce419149cd481c0723815)
(cherry picked from commit 3d41b583daea654c98378ce3dcb78937d71538e8)

Co-authored-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2022-10-22 13:46:13 -07:00
Lisca Ana-Roberta
26d982becb
fix: replace time.sleep() with checking logs (#899)
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2022-10-21 11:17:06 -07:00
Ramkumar Chinchani
763287873e
fix(config): make all extension config consistent (#888)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-10-21 15:33:54 +03:00
Andrei Aaron
38b00e3507
chore(lint): gci to separate zot from other imports (#870)
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-10-20 09:39:20 -07:00
Lisca Ana-Roberta
7f9052972d
fix: zli images show if signed instead of signature (#886)
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2022-10-20 09:35:24 -07:00
peusebiu
91dd5496fc
chore: rename search route prefix (#887)
* chore: rename search route prefix

* chore: use builtin time.Duration.Truncate() on latencies

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-10-18 20:46:06 -07:00
Lisca Ana-Roberta
4bc7a2c824
fix: images command not truncating image name/tag (#851)
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2022-10-11 08:56:03 -07:00
Andrei Aaron
98854337ff
build(tags): remove redundant build tag ui_base (#857)
It was not used for UI, it had become a CLI dependency
with the same functionality as search

Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-10-10 15:05:55 +03:00
Andrei Aaron
1afc5c8c3f
test(authz): add an extra test for authz (#859)
Should help with test flakiness
(the order in which the policies are read from the map impacted tested codepath)

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-10-07 15:31:18 +03:00
peusebiu
c146448f01
fix(sync): revert code which removed image destination feature (#840)
Added an end to end test for this feature, closes #793

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-10-05 11:03:24 -07:00
Nicol
33a431ef43
Update go version to 1.19 (#829)
* ci: Update go version to 1.19

Signed-off-by: Nicol Draghici <idraghic@cisco.com>

* ci: Fix lint issues

Signed-off-by: Nicol Draghici <idraghic@cisco.com>

* ci: Added needprivileges to lint, made needprivileges pass lint

Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Signed-off-by: Nicol Draghici <idraghic@cisco.com>

Signed-off-by: Nicol Draghici <idraghic@cisco.com>
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Co-authored-by: Catalin Hofnar <catalin.hofnar@gmail.com>
2022-10-05 13:21:14 +03:00
peusebiu
8237f8d20a
storage: Move common code in helper functions, closes #730 (#820)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-09-30 10:35:16 -07:00
Andreea Lupu
5ef023dbc1
add enable/disable option for scrub extension (#827)
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2022-09-27 18:06:50 -07:00
Lisca Ana-Roberta
1bad90bb9d
add debug flag for zli commands (#785)
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2022-09-23 09:24:01 -07:00
Lisca Ana-Roberta
0f7b174fc0
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com> (#713)
list all images that have are base images for the given image + zli command

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2022-09-23 09:23:31 -07:00
Ramkumar Chinchani
944ae66844
fix incorrect http transport to work with proxies (#802)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-09-23 11:16:35 +03:00
Andreea Lupu
f686ab6bf6
initial design for task scheduler (#700)
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2022-09-22 22:27:56 -07:00
Andrei Aaron
7517f2a5bb
list all images that have all layers of the base image included (2) (#813)
* list all images that are base images for the given image + zli command

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

* Fix a failing test

The test expected the image size to be the size of the layer, not the manifest+config+layer

Signed-off-by: Andrei Aaron <andaaron@cisco.com>

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Co-authored-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2022-09-22 22:08:58 +03:00
Ramkumar Chinchani
f3faae0e09
report listening port when chosen by kernel (#770)
Based off of the PR by @thesayyn
https://github.com/project-zot/zot/pull/720

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2022-09-09 08:41:13 +03:00
Nicol Draghici
6471add89d Read log path and verify content separately to avoid failed tests
Signed-off-by: Nicol Draghici <idraghic@cisco.com>
2022-09-05 12:41:33 -07:00
slab713
8ffb053cec
Replaced deprecated io/ioutil functions (#768)
Signed-off-by: slab713 <109306207+slab713@users.noreply.github.com>
2022-09-02 15:56:02 +03:00
Shivam Mishra
6c293719e3 storage: different subpaths can point to same root directory
currently different subpaths can only point to same root directory only
when one or both of the storage config does not enable dedupe

different subpath should be able to point to same root directory and in
that case their storage config should be same i.e GC,Dedupe, GC delay
and GC interval

Signed-off-by: Shivam Mishra <shimish2@cisco.com>
2022-08-31 15:43:43 -07:00
Roxana Nemulescu
ab9a20c1ae Add GraphQL API for getting the information necessary to list images in the zot cli without download manifests.
If this GraphQL API is available, try that first, else fallback to the slowpath.

Signed-off-by: Roxana Nemulescu <roxana.nemulescu@gmail.com>
2022-08-23 16:32:00 +03:00
Nicol Draghici
a702a2377e Remove AllowReadOnly and ReadOnly
Signed-off-by: Nicol Draghici <idraghic@cisco.com>

Remove check and set header every time

Signed-off-by: Nicol Draghici <idraghic@cisco.com>
2022-08-10 14:27:21 -07:00
Lisca Ana-Roberta
87fc941b3c image level lint: enforce manifest mandatory annotations
closes #536

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2022-07-27 11:48:04 +03:00
Andrei Aaron
10d9b1514b Fixes/Improvements to pkg/cli/stress_test.go
- Decrease RLIMIT_NOFILE and the number of goroutines used to reach this limit (from 512 to 100)
- Reset RLIMIT_NOFILE to the initial value before the test finishes
- Remove panic
- Use temporary dir managed by test framework
- Swith to using test logging in pkg/cli/stress_test.go
- Execute commands without `bash -c` in pkg/cli/stress_test.go

First item is needed as the GH runner seems to stop the test if stressed too much.
The lower number is still good enough to reproduce the test conditions

Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-26 13:01:16 +03:00
Andrei Aaron
43160dcc43 Update to graphql 1.17.13
We encountered some problems with using the existing folder structure,
but it looks like running the tooling with the latest versions works after
we regenerated the project using 'gql init' and refactoring to separate
the login previously in resolvers.go.

- the autogenerated code is now under the gql_generated folder
- the file resolvers.go now contains only the code which is not
rewritten by the gqlgen framework
- the file schema.resolvers.go is rewritten when gqlgen runs,
and we'll only keep there the actual resolvers matching query names
Changes we observed to schema.resolvers.go when gqlgen runs include
reordering methods, and renaming function parameters to match the
names used in schema.graphql
- we now have a gqlgen.yaml config file which governs the behavior of
gqlgen (can be tweaked to restructure the folder structure of the
generated code in the future)

Looks like the new graphql server has better validation
1 Returns 422 instead of 200 for missing query string - had to update tests
2 Correctly uncovered an error in a test for a bad `%` in query string.

As as result of 2, a `masked` bug was found in the way we check if images are
signed with Notary, the signatures were reasched for with the media type
of the image manifest itself instead of the media type for notation.
Fixed this bug, and improved error messages.
This bug would have also been reproducible with main branch if the bad `%`
in the test would have fixed.

Updated the linter to ignore some issues with the code which is
always rewritten when running:
`go run github.com/99designs/gqlgen@v0.17.13 generate`

Add a workflow to test gqlgen works and has no uncommitted changes

Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-18 12:55:40 -07:00
Petu Eusebiu
2496fef3c2 Fix data race on trivydb download in tests.
Multiple go routines downloading trivy db
triggers data race on trivy internal db.Path().
In each go routine wait for db download to start.
closes #636

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-07-14 09:31:15 -07:00
Petu Eusebiu
003de3a80a Fix config reloader in tests
config file may get removed before fsnotify starts watching it
make sure the config file gets removed when test ends, closes #608

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-07-14 09:31:15 -07:00
Petu Eusebiu
7954add73a Fix data races in tests closes #599, closes #598
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-06-30 13:33:47 -07:00
Alex Stan
ada21ed842 Manage builds with different combinations of extensions
Files were added to be built whether an extension is on or off.
New build tags were added for each extension, while minimal and extended disappeared.

added custom binary naming depending on extensions used and changed references from binary to binary-extended

added automated blackbox tests for sync, search, scrub, metrics

added contributor guidelines

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
2022-06-30 09:53:52 -07:00
Lisca Ana-Roberta
111b80625d added repos command to list repositories
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2022-06-15 02:22:18 -07:00
laurentiuNiculae
c9b32c73ae added more types of severity
Signed-off-by: laurentiuNiculae <themelopeus@gmail.com>
2022-06-03 09:44:54 -07:00
Andreea-Lupu
081ba0b2f2 fix periodic background tasks - gc and scrub
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2022-06-02 08:58:02 -07:00
Shivam Mishra
36c9631000 ext: use distribution spec route prefix for extension api
Following the spec defined here https://github.com/opencontainers/distribution-spec/tree/main/extensions

Signed-off-by: Shivam Mishra <shimish2@cisco.com>
2022-05-22 16:35:16 -07:00