0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2024-12-23 22:27:35 -05:00
Commit graph

107 commits

Author SHA1 Message Date
Andrei Aaron
792e82cbdf Add an 'enable' flag in the server configuration to enable gql-based searches
"extensions": {
        "search": {
            "enable": true
        }
    }
2021-06-24 12:15:25 -07:00
Andrei Aaron
c1dd7878e4 Add a '--verbose' flag to the 'zot images' output
- Show individual layers with size and digest under each image
- Include config digest for each image

See example below
```
IMAGE NAME                        TAG                       DIGEST    CONFIG    LAYERS    SIZE
test/godev                        0.4.7                     7d38d8ca  05b9f86e            519MB
                                                                                f824a027  65MB
                                                                                a98af0f5  52MB
                                                                                ba5b2bc4  163MB
                                                                                58b1ca8d  228MB
                                                                                67d798ee  12MB
test/cdev                         test                      2292b4ae  cf6f6c77            280MB
                                                                                f824a027  65MB
                                                                                a98af0f5  52MB
                                                                                ba5b2bc4  163MB
test/cdev                         0.4.7                     2292b4ae  cf6f6c77            280MB
                                                                                f824a027  65MB
                                                                                a98af0f5  52MB
                                                                                ba5b2bc4  163MB

Note the new layers and config fields will be visible in the json/yaml format regardless of the value of the verbose flag
```
2021-06-24 12:15:25 -07:00
Andrei Aaron
519ea75d9a Implement a way to search for an image by manifest, config or layer digest
```
Usage:
  zot images [config-name] [flags]

Flags:
  -d, --digest string   List images containing a specific manifest, config, or layer digest
[...]
```
2021-06-24 12:15:25 -07:00
Roxana Nemulescu
97628e69c9 logs: add an audit log for API calls with unit tests
resolves #178
2021-06-24 10:53:27 -07:00
Shivam Mishra
28974e81dc config: support multiple storage locations
added support to point multiple storage locations in zot by running multiple instance of zot in background.

see examples/config-multiple.json for more info about config.

Closes #181
2021-05-21 10:18:28 -07:00
Ramkumar Chinchani
9ca6eea940 routes: ignore media-type for PatchBlobUpload
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-05-19 10:53:49 -07:00
Ramkumar Chinchani
8f729820f5 controller: add default headers
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-05-10 12:47:53 -07:00
Shivam Mishra
cf25c6f3c8 ci/cd:inculde binary type in version information 2021-05-04 13:16:23 -07:00
Shivam Mishra
a7c17b7c16 spec: added support for mount request using hard link 2021-05-04 09:42:29 -07:00
Ramkumar Chinchani
f7829d6470 dedupe: optimize check-blob with hard links
In use cases, when there are large images with shared layers across
repositories, clients may benefit from not re-uploading the same blobs
over and over again.

We ensure this by hard linking when check-blob api is called.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-05-04 09:42:29 -07:00
Shivam Mishra
3c88102870 server: add idle timeout in http server configuration 2021-04-29 11:00:12 -07:00
Shivam Mishra
2b7b57313a conformance: fix http status code for cross-repository mounting 2021-01-29 09:35:15 -08:00
Shivam Mishra
9969ba0867 conformance: update README to display conformance results 2021-01-28 15:50:36 -08:00
Ramkumar Chinchani
affdd85986 build: remove bazel
The idea initially was to use bazel to do our builds, however golang
build system is now good enough and our code base is entirely go.
It is also slowing down our travis ci/cd pipeline.
2020-12-21 15:30:13 -08:00
Shivam Mishra
8787142d2c test: pull test images from aws container registry 2020-12-21 15:24:26 -08:00
Ionut Costin Craciun
dad884ddeb Raise error when adding a new zot config with an existed saved name 2020-11-04 10:25:34 +02:00
Shivam Mishra
b0ed625a2e build: increase wait timeout for travis bazel build process 2020-10-27 19:30:06 -07:00
Shivam Mishra
46beb30fc1 build: add build tags to create customizable binaries 2020-10-22 17:20:07 -07:00
Shivam Mishra
7439feb1c2 build: set timeout in travis make build process to avoid timeout failure 2020-10-18 20:55:17 -07:00
Shivam Mishra
14214a5794 test: add unit test to verify lock changes 2020-10-16 14:58:45 -07:00
Ramkumar Chinchani
386c72d332 routes: refactor locks to handle large file uploads
The storage layer is protected with read-write locks.
However, we may be holding the locks over unnecessarily large critical
sections.

The typical workflow is that a blob is first uploaded via a per-client
private session-id meaning the blob is not publicly visible yet. When
the blob being uploaded is very large, the transfer takes a long time
while holding the lock.

Private session-id based uploads don't really need locks, and hold locks
only when blobs are published after the upload is complete.
2020-10-16 13:33:11 -07:00
Shivam Mishra
25ad71787a test: minimize trivy db download tests to avoid api rate limit 2020-10-15 14:32:37 -07:00
Shivam Mishra
8075eadc1a test: add wait for trivy db download in test case 2020-10-02 16:47:54 -07:00
Shivam Mishra
971404f6ee search/cve: fix log messages 2020-09-23 12:47:50 -07:00
Shivam Mishra
d63f715fe5 search/cve: exclude unsupported images from fixed-tag list.
If image vulnerability scan does not support any media type, considering those images as an infected image and now this images will not be shown in fixed images list.

Fixes issue #130
2020-09-22 09:24:04 -07:00
Shivam Mishra
cd0206fe6c Fixes issue #132, if image does not have any fixed tags, empty list with no error should be returned 2020-09-08 16:41:06 -07:00
Tanmay Naik
f5867ce0b6 cli: group CVEs by severity 2020-09-04 13:56:47 -04:00
Tanmay Naik
c590b86d14 cli: add commands for CVE
Uses GraphQL API of zot to fetch CVE info

- Get all images affected by a CVE (input: CVEID)
- Get all CVEs of a layer (input: image:tag)
- Get all layers of an image which have resolved a CVE (input: image,
CVEID)
- Get all layers of an image affected by a CVE (input: image, CVEID)
2020-08-21 12:42:01 -04:00
Shivam Mishra
5f230bd8ff Added unit test cases 2020-08-19 00:19:35 -07:00
Shivam Mishra
ed254159a0 Added support for searching fixed tag given cve and an image 2020-08-18 23:53:04 -07:00
Shivam Mishra
72ae02ca4b Added image format validation to validate media type because squashfs image media type not supported for vulnerability scanning 2020-08-18 23:05:52 -07:00
Shivam Mishra
2cf2c16137 Added graphql api feature for image vulnerability scanning 2020-08-18 22:44:34 -07:00
Shivam Mishra
baa5d247ec Enable trivy db download and update 2020-08-18 21:46:17 -07:00
Shivam Mishra
e537f27f00 Added search extension and integrated trivy to support image vulnerability scanning 2020-08-18 21:03:48 -07:00
Shivam Mishra
3a30290e08 Using "destRecord" as a path in DeleteBlob function instead of "dst".
dstRecord :- blob path stored in cache.
dst :- blob path that is trying to be uploaded.

Currently, if the actual blob on disk may have been removed by GC/delete, during syncing the cache dst is being passed to DeleteBlob function and retry section is being continuously called because DeleteBlob function never deletes dst path (doesn't exist in db), dstRecord should be passed into DeleteBlob function because dstRecord is actual blob path stored in db.

If dst and dstRecord path value is same then this issue will not be produced and DeleteBlob method will delete the blob info from cache but if both are different then DeleteBlob method will try to delete dst path which is not in cache.

Note:- boltdb delete method return nil even when value doesn't exist (https://godoc.org/github.com/boltdb/bolt#Bucket.Delete)
2020-08-12 10:06:20 -07:00
Tanmay Naik
6285a730a1 cli: add option to ignore TLS verification
adds a property in config : "verify-tls"
2020-07-17 17:48:42 -04:00
Ramkumar Chinchani
adc6859cd6 stacker: fix stacker build 2020-07-14 13:31:57 -07:00
Tanmay Naik
bb9fbd2ef9 cli: move client-only code out of the server flow
earlier, some of the client exclusive code was being run on zot server
instance too.

cli: fix the bug: spinner is not stopped with -o
2020-07-14 13:35:56 -04:00
Ramkumar Chinchani
78be4cbe3c auth: support a read-only mode
This is useful if we want to roll out experimental versions of zot
pointing to some storage shared with another zot instance.

Also, when under storage full conditions, will be useful to turn on this
flag to prevent further writes.
2020-07-10 21:48:35 -07:00
Ramkumar Chinchani
324a517ea3 gc: add a policy to skip garbage collecting new blobs
We perform inline garbage collection of orphan blobs. However, the
dist-spec poses a problem because blobs begin their life as orphan blobs
and then a manifest is add which refers to these blobs.

We use umoci's GC() to perform garbage collection and policy support
has been added recently which can control whether a blob can be skipped
for GC.

In this patch, we use a time-based policy to skip blobs.
2020-07-06 15:52:35 -07:00
Tanmay Naik
ad684ac44b cli: add config and images command
Extends the existing zot CLI to add commands for listing all images and
their details on a zot server.
Listing all images introduces the need for configurations.

Each configuration has a name and URL at the least. Check 'zot config
-h' for more details.

The user can specify the URL of zot server explicitly while running the
command or configure a URL and pass it directly.

Adding a configuration:
zot config add aci-zot <zot-url>

Run 'zot config --help' for more.

Listing all images:
zot images --url <zot-url>

Pass a config instead of the url:
zot images <config-name>

Filter the list of images by image name:
zot images <config-name> --name <image-name>

Run 'zot images --help' for all details

- Stores configurations in '$HOME/.zot' file

Add CLI README
2020-07-02 14:30:35 -04:00
Shivam Mishra
af77876306 Upgraded build pipeline
Go version changed to 1.14.4
Golangci-lint changed to 1.26.0
Bazel version changed to 3.0.0
Bazel rules_go version changed to 0.23.3
Bazel gazelle version changed to v0.21.0
Bazel build tools version changed to 0.25.1
Bazel skylib version changed to 1.0.2
2020-06-25 23:43:31 -07:00
Shivam Mishra
85d3e1db4b Changed umoci import path 2020-06-25 17:04:32 -07:00
Ramkumar Chinchani
3dc9885ee9 update the size field when existing manifest entry is updated
An existing manifest descriptor in index.json can be updated with
different manifest contents for the same/existing tag. We were updating
the digest but not the size field causing GC to report an error.

Add a unit test case to cover this.

Add logs.
2020-06-18 16:20:43 -07:00
Tanmay Naik
3f3f7e3f8c tests: add better tests for 3cfb2b3 2020-06-17 20:17:49 -04:00
Tanmay Naik
904ae763d7 tests: add unit tests for fix 3cfb2b3 2020-06-09 19:18:33 -04:00
Tanmay Naik
3cfb2b30a6 fix: the bug when htpasswd has multiple creds
earlier, when you had more than one creds in htpasswd file separated by
newline, it used to only read the first cred in the file and ignore the
rest.
2020-06-09 17:19:01 -04:00
Ramkumar Chinchani
cb9e773a3e dedupe: record relative path for cache entries
In a production use case we found that the actual rootdir can be moved.
Currently, cache entries for dedupe record the full blob path which
doesn't work in the move use case.

Only for dedupe cache entries, record relative blob paths.
2020-05-27 22:11:26 -07:00
Ramkumar Chinchani
e83999ae0d helm: relax Content-Type checks
OCI registries are moving towards a more generic artifact storage
mechanism. In short-term, at the very least support helm charts.
2020-05-19 16:53:05 -07:00
Ramkumar Chinchani
026b009dbb compat: when in "world-readable" mode, return the WWW-Authenticate
header

containers/image is the dominant client library to interact with
registries.

It detects which authentication to use based on the WWW-Authenticate
header returned when pinging "/v2/" end-point. If we didn't return this
header, then creds are not used for other write-protected end-points.
Hence, the compatibility fix.
2020-05-19 13:54:45 -07:00