0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2024-12-16 21:56:37 -05:00
Commit graph

177 commits

Author SHA1 Message Date
Alexei Dodon
48bf7f69f8
refactor: Reduce zb binary size (#1783)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-09-13 10:28:14 +03:00
Andrei Aaron
6011705931
chore(dependabot): increase the limit of dependabot PRs (#1788)
Given the default limit is 5 and some packages like AWS release daily,
having a weekly schedule results in the same packages being flagged for update
most of the time.

Let's increase to 10 to make sure there are no issues in other libraries

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-09-11 11:02:11 -07:00
Andrei Aaron
6115eed4ec
refactor(makefile): consolidate the make targets used for bats tests (#1746)
New examples of running tests:

1. To run a specific bats file (with and without verbose output):
make run-blackbox-tests BATS_TEST_FILE_PATH=test/blackbox/delete_images.bats
make run-blackbox-tests BATS_TEST_FILE_PATH=test/blackbox/delete_images.bats BATS_VERBOSITY=2

2. To run the CI tests (with and without verbose output)
make run-blackbox-ci
make run-blackbox-ci BATS_VERBOSITY=2

BATS_TEST_FILE_PATH is used to pass on the test file to run using `run-blackbox-tests`
BATS_VERBOSITY controls the verbosity of the bats framework output, if unspecified the output only
contains test results and failure message in case of failures.
If BATS_VERBOSITY is 1, then also show commands as they are executed.
If BATS_VERBOSITY is 2, on top of the above it also shows output of passed tests.

Other changes in this PR:
- Update some of the tests to show logs after the run ends.
- Run the linters before the tests, as it saves time on failures when running in GH

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-09-07 21:06:21 +03:00
Alexei Dodon
f5b63963be
refactor: Reduce binary size of zot-minimal; Added CI check for binary size (#1758)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-09-06 19:58:00 +03:00
Ramkumar Chinchani
75a76005b4
chore: fix dependabot alerts (#1763)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-09-05 22:30:49 +03:00
peusebiu
c6b822f3dd
refactor(authz): use a struct for user access control info operations (#1682)
fix(authz): fix isAdmin not using groups to determine if a user is admin.
fix(authz): return 401 instead of 403

403 is correct as per HTTP spec
However authz is not part of dist-spec and clients know only about 401
So this is a compromise.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-01 11:13:53 -07:00
peusebiu
b80deb9927
refactor(storage): refactor storage into a single ImageStore (#1656)
unified both local and s3 ImageStore logic into a single ImageStore
added a new driver interface for common file/dirs manipulations
to be implemented by different storage types

refactor(gc): drop umoci dependency, implemented internal gc

added retentionDelay config option that specifies
the garbage collect delay for images without tags

this will also clean manifests which are part of an index image
(multiarch) that no longer exist.

fix(dedupe): skip blobs under .sync/ directory

if startup dedupe is running while also syncing is running
ignore blobs under sync's temporary storage

fix(storage): do not allow image indexes modifications

when deleting a manifest verify that it is not part of a multiarch image
and throw a MethodNotAllowed error to the client if it is.
we don't want to modify multiarch images

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-01 10:54:39 -07:00
Alexei Dodon
41bbb23e30
fix: bats test refactoring (#1731)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-08-30 12:24:28 -07:00
Alexei Dodon
247f6dcd3f
feat: propagate detailed error msgs to client (OCI dist-spec format) (#1681)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-08-23 20:59:52 +03:00
Alexei Dodon
ca2904762a
fix(ci/cd): detect uncommited swagger docs (#1724)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-08-23 19:21:43 +03:00
Alexei Dodon
e510df7c22
fix: swagger Makefile target broken on darwin (#1701)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-08-15 13:07:56 -07:00
Ramkumar Chinchani
e129d4003b
chore: fix dependabot alerts (#1702)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-08-15 10:15:50 +03:00
LaurentiuNiculae
ed90e3bd24
test(annotations): print logs for annotations bats tests (#1680)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-08-07 13:51:57 -07:00
Andreea Lupu
76277f5ebd
fix: remove inline GC and schedule a background task instead (#1610)
* fix: remove inline GC and set a default value of gc interval

- remove inline GC
- add a default value of GC interval
- run the GC periodically by default with the default value if no interval provided
- generate GC tasks with a random delay(0-30s) between
- add IsReady() method to scheduler.TaskGenerator interface

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>

* ci: add test for gc with short interval

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>

---------

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-08-07 12:55:19 -07:00
Ramkumar Chinchani
cad564b9ee
ci(conformance): revert to upstream OCI checks (#1678)
Reverts part of de24bd937f

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-08-04 13:38:28 -07:00
Andrei Aaron
77149aa85c
refactor(extensions)!: refactor the extensions URLs and errors (#1636)
BREAKING CHANGE: The functionality provided by the mgmt endpoint has beed redesigned - see details below
BREAKING CHANGE: The API keys endpoint has been moved -  see details below
BREAKING CHANGE: The mgmt extension config has been removed - endpoint is now enabled by having both the search and the ui extensions enabled
BREAKING CHANGE: The API keys configuration has been moved from extensions to http>auth>apikey

mgmt and imagetrust extensions:
- separate the _zot/ext/mgmt into 3 separate endpoints: _zot/ext/auth, _zot/ext/notation, _zot/ext/cosign
- signature verification logic is in a separate `imagetrust` extension
- better hanling or errors in case of signature uploads: logging and error codes (more 400 and less 500 errors)
- add authz on signature uploads (and add a new middleware in common for this purpose)
- remove the mgmt extension configuration - it is now enabled if the UI and the search extensions are enabled

userprefs estension:
- userprefs are enabled if both search and ui extensions are enabled (as opposed to just search)

apikey extension is removed and logic moved into the api folder
- Move apikeys code out of pkg/extensions and into pkg/api
- Remove apikey configuration options from the extensions configuration and move it inside the http auth section
- remove the build label apikeys

other changes:
- move most of the logic adding handlers to the extensions endpoints out of routes.go and into the extensions files.
- add warnings in case the users are still using configurations with the obsolete settings for mgmt and api keys
- add a new function in the extension package which could be a single point of starting backgroud tasks for all extensions
- more clear methods for verifying specific extensions are enabled
- fix http methods paired with the UI handlers
- rebuild swagger docs

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-08-02 21:58:34 +03:00
Ramkumar Chinchani
42f9f78125
ci: add freebsd target for releases (#1660)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-07-31 13:51:03 -07:00
Ramkumar Chinchani
75b94a3c8b
chore: fix dependabot alerts (#1649) 2023-07-25 08:14:37 +03:00
Ramkumar Chinchani
de24bd937f
fix(conformance): OCI-Filters-Applied should return a literal (#1640)
* fix(conformance): OCI-Filters-Applied should return a literal

https://github.com/opencontainers/distribution-spec/issues/448

Should only indicate what filter-type was applied and not what exact
values it was filtered on.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

* ci(conformance): point to temporary repo

Revert this commit once upstream changes have been merged

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

---------

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-07-22 16:25:02 -07:00
Ramkumar Chinchani
fe9c9750b5
chore: fix dependabot alerts (#1631)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-07-17 23:37:54 +03:00
peusebiu
d3f27b4ba6
feat: add additional manifest validations (#1609)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-13 09:31:39 -07:00
Ramkumar Chinchani
730ef4aada
chore: fix dependabot alerts (#1613)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-07-13 10:14:03 +03:00
peusebiu
cda6916b45
fix: don't allow blobs to be deleted if in use (#1559)
dist-spec APIs independently allow deletion of blobs and manifests.
Doing the former when in use by an image manifest or index is simply
error-prone. So disallow it.

Fixes issue #1509

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-07-10 12:24:45 +03:00
peusebiu
cd6f679359
fix(ci/cd): stop localstack after ci/cd pipeline (#1590)
this should clean localstack persisted data in tests

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-07 12:40:06 -07:00
peusebiu
17d1338af1
feat: integrate openID auth logic and user profile management (#1381)
This change introduces OpenID authn by using providers such as Github,
Gitlab, Google and Dex.
User sessions are now used for web clients to identify
and persist an authenticated users session, thus not requiring every request to
use credentials.
Another change is apikey feature, users can create/revoke their api keys and use them
to authenticate when using cli clients such as skopeo.

eg:
login:
/auth/login?provider=github
/auth/login?provider=gitlab
and so on

logout:
/auth/logout

redirectURL:
/auth/callback/github
/auth/callback/gitlab
and so on

If network policy doesn't allow inbound connections, this callback wont work!

for more info read documentation added in this commit.

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
2023-07-07 09:27:10 -07:00
peusebiu
7d7e4416b5
fix(nightly): fix nightly builds (#1584)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-07-06 16:50:12 +03:00
Ramkumar Chinchani
137e5bd793
chore: fix dependabot alerts (#1581)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-07-05 11:42:24 +03:00
Ramkumar Chinchani
aad6db279b
chore: fix dependabot alerts (#1576)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-07-04 09:19:26 -07:00
Andreea Lupu
7ec748f58a
fix: change commit message for pushing changes to project-zot/helm-charts (#1564)
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-06-28 20:42:31 +03:00
Ramkumar Chinchani
d78ef0b1bd
ci: update oci dist-spec conformance test (#1552)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-06-24 18:51:47 -07:00
Ramkumar Chinchani
d30d7a9330
chore: fix dependabot alerts (#1537)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-06-19 12:34:50 -07:00
peusebiu
d45cfbf5d2
feat(ci): give minio container more time to start (#1527)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-06-16 12:24:38 +03:00
Ramkumar Chinchani
7dd17be96d
chore: fix dependabot alerts (#1517)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-06-16 10:28:43 +03:00
Andrei Aaron
f9f9422d13
ci(disk usage): disk related fixes and improvements (#1524)
ci(workflow): show disk usage and free up disk space used by unneeded tooling
ci(tests): routes tests: do not copy large images if they are not used later
ci(trivy): update a test: download trivy.db to a temporary folder

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-06-15 15:07:28 -07:00
Ramkumar Chinchani
d9e5f33e7e
chore: fix security alerts (#1493)
https://github.com/project-zot/zot/security/code-scanning/4293

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-06-01 16:53:50 -07:00
peusebiu
612a12e5a8
refactor(sync): use task scheduler (#1301)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-05-31 10:26:23 -07:00
Andrei Aaron
60148138d5
ci(cri-o): update cri-o installation steps (#1470)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-05-23 17:01:10 +03:00
Andreea Lupu
6269dbea0c
fix(ci-cd): trigger update-helm-chart after push-image is completed (#1435)
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-05-10 09:53:39 -07:00
Ramkumar Chinchani
42df4c505a
chore: fix dependabot alerts (#1403)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-05-01 12:49:10 -07:00
Ramkumar Chinchani
635d07ae04
chore: update golang (to 1.20.x) and golangci-linter (#1388)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-04-27 00:09:46 -07:00
LaurentiuNiculae
9cc990d7ca
feat(repodb): add user related information to repodb (#1317)
Initial code was contributed by Bogdan BIVOLARU <104334+bogdanbiv@users.noreply.github.com>
Moved implementation from a separate db to repodb by Andrei Aaron <aaaron@luxoft.com>

Not done yet:
- run/test dynamodb implementation, only boltdb was tested
- add additional coverage for existing functionality
- add web-based APIs to toggle the stars/bookmarks on/off

Initially graphql mutation was discussed for the missing API but
we decided REST endpoints would be better suited for configuration



feat(userdb): complete functionality for userdb integration

- dynamodb rollback changes to user starred repos in case increasing the total star count fails
- dynamodb increment/decrement repostars in repometa when user stars/unstars a repo
- dynamodb check anonymous user permissions are working as intendend
- common test handle anonymous users
- RepoMeta2RepoSummary set IsStarred and IsBookmarked



feat(userdb): rest api calls for toggling stars/bookmarks on/off



test(userdb): blackbox tests



test(userdb): move preferences tests in a different file with specific build tags



feat(repodb): add is-starred and is-bookmarked fields to repo-meta

- removed duplicated logic for determining if a repo is starred/bookmarked

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
2023-04-24 11:13:15 -07:00
Ramkumar Chinchani
9638899e6e
test: additional blackbox tests for client push/pull (#1371)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-04-13 13:47:38 -07:00
peusebiu
f35ff53146
feat(storage): rebuild storage(s3/local) dedupe index when switching dedupe status (#1062)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-04-07 09:49:24 -07:00
Ramkumar Chinchani
d9173e3ad3
chore(go.mod): fix dependabot alerts (#1330)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-04-05 09:39:15 +03:00
Ramkumar Chinchani
917159143c
chore: fix dependabot alerts (#1312)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-03-27 12:16:29 -07:00
Ramkumar Chinchani
906f8ce621
chore(deps): fix dependabot alerts (#1291)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-03-22 12:33:21 -07:00
LaurentiuNiculae
5d1f91a79f
feat(repodb): update referrers api to use repodb (#1230)
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-03-10 10:37:29 -08:00
peusebiu
f04e66a5e2
feat(mgmt): added mgmt extension which returns current zot configuration (#1198)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-03-09 10:43:26 -08:00
Andrei Aaron
73b1126bbf
chore(go.mod): fix dependabot alerts (#1247)
Supersedes:
- https://github.com/project-zot/zot/pull/1132
- https://github.com/project-zot/zot/pull/1243
- https://github.com/project-zot/zot/pull/1244
- https://github.com/project-zot/zot/pull/1245

Also update the AWS SDK libraries used

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-03-06 11:05:19 -08:00
Andrei Aaron
5968e7199f
test(ui): add owasp zap scanner in ci/cd (#1224)
(cherry picked from commit 6d03ce5f2d)

Additional changes on top of: 6d03ce5f2d
- Build and use zot from the same branch
do not use a container image as scan target, use the binary
- Fix typo in rules filename
- Add the full rule list to the rules config file
- Ignore some of the specific rules and add reasons
- Add security-related headers to fix some of the issues identified by the scan
- Update UI it includes the latest fixes for zap scan issues

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
2023-02-27 11:25:47 -08:00