0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2024-12-23 22:27:35 -05:00
zot/pkg/extensions/search/schema.resolvers.go

289 lines
8.6 KiB
Go
Raw Normal View History

Update to graphql 1.17.13 We encountered some problems with using the existing folder structure, but it looks like running the tooling with the latest versions works after we regenerated the project using 'gql init' and refactoring to separate the login previously in resolvers.go. - the autogenerated code is now under the gql_generated folder - the file resolvers.go now contains only the code which is not rewritten by the gqlgen framework - the file schema.resolvers.go is rewritten when gqlgen runs, and we'll only keep there the actual resolvers matching query names Changes we observed to schema.resolvers.go when gqlgen runs include reordering methods, and renaming function parameters to match the names used in schema.graphql - we now have a gqlgen.yaml config file which governs the behavior of gqlgen (can be tweaked to restructure the folder structure of the generated code in the future) Looks like the new graphql server has better validation 1 Returns 422 instead of 200 for missing query string - had to update tests 2 Correctly uncovered an error in a test for a bad `%` in query string. As as result of 2, a `masked` bug was found in the way we check if images are signed with Notary, the signatures were reasched for with the media type of the image manifest itself instead of the media type for notation. Fixed this bug, and improved error messages. This bug would have also been reproducible with main branch if the bad `%` in the test would have fixed. Updated the linter to ignore some issues with the code which is always rewritten when running: `go run github.com/99designs/gqlgen@v0.17.13 generate` Add a workflow to test gqlgen works and has no uncommitted changes Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-15 06:10:51 -05:00
package search
// This file will be automatically regenerated based on the schema, any resolver implementations
// will be copied through when generating and any unknown code will be moved to the end.
// Code generated by github.com/99designs/gqlgen version v0.17.22
Update to graphql 1.17.13 We encountered some problems with using the existing folder structure, but it looks like running the tooling with the latest versions works after we regenerated the project using 'gql init' and refactoring to separate the login previously in resolvers.go. - the autogenerated code is now under the gql_generated folder - the file resolvers.go now contains only the code which is not rewritten by the gqlgen framework - the file schema.resolvers.go is rewritten when gqlgen runs, and we'll only keep there the actual resolvers matching query names Changes we observed to schema.resolvers.go when gqlgen runs include reordering methods, and renaming function parameters to match the names used in schema.graphql - we now have a gqlgen.yaml config file which governs the behavior of gqlgen (can be tweaked to restructure the folder structure of the generated code in the future) Looks like the new graphql server has better validation 1 Returns 422 instead of 200 for missing query string - had to update tests 2 Correctly uncovered an error in a test for a bad `%` in query string. As as result of 2, a `masked` bug was found in the way we check if images are signed with Notary, the signatures were reasched for with the media type of the image manifest itself instead of the media type for notation. Fixed this bug, and improved error messages. This bug would have also been reproducible with main branch if the bad `%` in the test would have fixed. Updated the linter to ignore some issues with the code which is always rewritten when running: `go run github.com/99designs/gqlgen@v0.17.13 generate` Add a workflow to test gqlgen works and has no uncommitted changes Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-15 06:10:51 -05:00
import (
"context"
"github.com/vektah/gqlparser/v2/gqlerror"
zerr "zotregistry.io/zot/errors"
Update to graphql 1.17.13 We encountered some problems with using the existing folder structure, but it looks like running the tooling with the latest versions works after we regenerated the project using 'gql init' and refactoring to separate the login previously in resolvers.go. - the autogenerated code is now under the gql_generated folder - the file resolvers.go now contains only the code which is not rewritten by the gqlgen framework - the file schema.resolvers.go is rewritten when gqlgen runs, and we'll only keep there the actual resolvers matching query names Changes we observed to schema.resolvers.go when gqlgen runs include reordering methods, and renaming function parameters to match the names used in schema.graphql - we now have a gqlgen.yaml config file which governs the behavior of gqlgen (can be tweaked to restructure the folder structure of the generated code in the future) Looks like the new graphql server has better validation 1 Returns 422 instead of 200 for missing query string - had to update tests 2 Correctly uncovered an error in a test for a bad `%` in query string. As as result of 2, a `masked` bug was found in the way we check if images are signed with Notary, the signatures were reasched for with the media type of the image manifest itself instead of the media type for notation. Fixed this bug, and improved error messages. This bug would have also been reproducible with main branch if the bad `%` in the test would have fixed. Updated the linter to ignore some issues with the code which is always rewritten when running: `go run github.com/99designs/gqlgen@v0.17.13 generate` Add a workflow to test gqlgen works and has no uncommitted changes Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-15 06:10:51 -05:00
"zotregistry.io/zot/pkg/extensions/search/common"
"zotregistry.io/zot/pkg/extensions/search/gql_generated"
)
// CVEListForImage is the resolver for the CVEListForImage field.
func (r *queryResolver) CVEListForImage(ctx context.Context, image string, requestedPage *gql_generated.PageInput) (*gql_generated.CVEResultForImage, error) {
if r.cveInfo == nil {
return &gql_generated.CVEResultForImage{}, zerr.ErrCVESearchDisabled
}
return getCVEListForImage(ctx, image, r.cveInfo, requestedPage, r.log)
Update to graphql 1.17.13 We encountered some problems with using the existing folder structure, but it looks like running the tooling with the latest versions works after we regenerated the project using 'gql init' and refactoring to separate the login previously in resolvers.go. - the autogenerated code is now under the gql_generated folder - the file resolvers.go now contains only the code which is not rewritten by the gqlgen framework - the file schema.resolvers.go is rewritten when gqlgen runs, and we'll only keep there the actual resolvers matching query names Changes we observed to schema.resolvers.go when gqlgen runs include reordering methods, and renaming function parameters to match the names used in schema.graphql - we now have a gqlgen.yaml config file which governs the behavior of gqlgen (can be tweaked to restructure the folder structure of the generated code in the future) Looks like the new graphql server has better validation 1 Returns 422 instead of 200 for missing query string - had to update tests 2 Correctly uncovered an error in a test for a bad `%` in query string. As as result of 2, a `masked` bug was found in the way we check if images are signed with Notary, the signatures were reasched for with the media type of the image manifest itself instead of the media type for notation. Fixed this bug, and improved error messages. This bug would have also been reproducible with main branch if the bad `%` in the test would have fixed. Updated the linter to ignore some issues with the code which is always rewritten when running: `go run github.com/99designs/gqlgen@v0.17.13 generate` Add a workflow to test gqlgen works and has no uncommitted changes Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-15 06:10:51 -05:00
}
// ImageListForCve is the resolver for the ImageListForCVE field.
func (r *queryResolver) ImageListForCve(ctx context.Context, id string, requestedPage *gql_generated.PageInput) ([]*gql_generated.ImageSummary, error) {
if r.cveInfo == nil {
return []*gql_generated.ImageSummary{}, zerr.ErrCVESearchDisabled
Update to graphql 1.17.13 We encountered some problems with using the existing folder structure, but it looks like running the tooling with the latest versions works after we regenerated the project using 'gql init' and refactoring to separate the login previously in resolvers.go. - the autogenerated code is now under the gql_generated folder - the file resolvers.go now contains only the code which is not rewritten by the gqlgen framework - the file schema.resolvers.go is rewritten when gqlgen runs, and we'll only keep there the actual resolvers matching query names Changes we observed to schema.resolvers.go when gqlgen runs include reordering methods, and renaming function parameters to match the names used in schema.graphql - we now have a gqlgen.yaml config file which governs the behavior of gqlgen (can be tweaked to restructure the folder structure of the generated code in the future) Looks like the new graphql server has better validation 1 Returns 422 instead of 200 for missing query string - had to update tests 2 Correctly uncovered an error in a test for a bad `%` in query string. As as result of 2, a `masked` bug was found in the way we check if images are signed with Notary, the signatures were reasched for with the media type of the image manifest itself instead of the media type for notation. Fixed this bug, and improved error messages. This bug would have also been reproducible with main branch if the bad `%` in the test would have fixed. Updated the linter to ignore some issues with the code which is always rewritten when running: `go run github.com/99designs/gqlgen@v0.17.13 generate` Add a workflow to test gqlgen works and has no uncommitted changes Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-15 06:10:51 -05:00
}
return getImageListForCVE(ctx, id, r.cveInfo, requestedPage, r.repoDB, r.log)
Update to graphql 1.17.13 We encountered some problems with using the existing folder structure, but it looks like running the tooling with the latest versions works after we regenerated the project using 'gql init' and refactoring to separate the login previously in resolvers.go. - the autogenerated code is now under the gql_generated folder - the file resolvers.go now contains only the code which is not rewritten by the gqlgen framework - the file schema.resolvers.go is rewritten when gqlgen runs, and we'll only keep there the actual resolvers matching query names Changes we observed to schema.resolvers.go when gqlgen runs include reordering methods, and renaming function parameters to match the names used in schema.graphql - we now have a gqlgen.yaml config file which governs the behavior of gqlgen (can be tweaked to restructure the folder structure of the generated code in the future) Looks like the new graphql server has better validation 1 Returns 422 instead of 200 for missing query string - had to update tests 2 Correctly uncovered an error in a test for a bad `%` in query string. As as result of 2, a `masked` bug was found in the way we check if images are signed with Notary, the signatures were reasched for with the media type of the image manifest itself instead of the media type for notation. Fixed this bug, and improved error messages. This bug would have also been reproducible with main branch if the bad `%` in the test would have fixed. Updated the linter to ignore some issues with the code which is always rewritten when running: `go run github.com/99designs/gqlgen@v0.17.13 generate` Add a workflow to test gqlgen works and has no uncommitted changes Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-15 06:10:51 -05:00
}
// ImageListWithCVEFixed is the resolver for the ImageListWithCVEFixed field.
func (r *queryResolver) ImageListWithCVEFixed(ctx context.Context, id string, image string, requestedPage *gql_generated.PageInput) ([]*gql_generated.ImageSummary, error) {
if r.cveInfo == nil {
return []*gql_generated.ImageSummary{}, zerr.ErrCVESearchDisabled
}
Update to graphql 1.17.13 We encountered some problems with using the existing folder structure, but it looks like running the tooling with the latest versions works after we regenerated the project using 'gql init' and refactoring to separate the login previously in resolvers.go. - the autogenerated code is now under the gql_generated folder - the file resolvers.go now contains only the code which is not rewritten by the gqlgen framework - the file schema.resolvers.go is rewritten when gqlgen runs, and we'll only keep there the actual resolvers matching query names Changes we observed to schema.resolvers.go when gqlgen runs include reordering methods, and renaming function parameters to match the names used in schema.graphql - we now have a gqlgen.yaml config file which governs the behavior of gqlgen (can be tweaked to restructure the folder structure of the generated code in the future) Looks like the new graphql server has better validation 1 Returns 422 instead of 200 for missing query string - had to update tests 2 Correctly uncovered an error in a test for a bad `%` in query string. As as result of 2, a `masked` bug was found in the way we check if images are signed with Notary, the signatures were reasched for with the media type of the image manifest itself instead of the media type for notation. Fixed this bug, and improved error messages. This bug would have also been reproducible with main branch if the bad `%` in the test would have fixed. Updated the linter to ignore some issues with the code which is always rewritten when running: `go run github.com/99designs/gqlgen@v0.17.13 generate` Add a workflow to test gqlgen works and has no uncommitted changes Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-15 06:10:51 -05:00
return getImageListWithCVEFixed(ctx, id, image, r.cveInfo, requestedPage, r.repoDB, r.log)
Update to graphql 1.17.13 We encountered some problems with using the existing folder structure, but it looks like running the tooling with the latest versions works after we regenerated the project using 'gql init' and refactoring to separate the login previously in resolvers.go. - the autogenerated code is now under the gql_generated folder - the file resolvers.go now contains only the code which is not rewritten by the gqlgen framework - the file schema.resolvers.go is rewritten when gqlgen runs, and we'll only keep there the actual resolvers matching query names Changes we observed to schema.resolvers.go when gqlgen runs include reordering methods, and renaming function parameters to match the names used in schema.graphql - we now have a gqlgen.yaml config file which governs the behavior of gqlgen (can be tweaked to restructure the folder structure of the generated code in the future) Looks like the new graphql server has better validation 1 Returns 422 instead of 200 for missing query string - had to update tests 2 Correctly uncovered an error in a test for a bad `%` in query string. As as result of 2, a `masked` bug was found in the way we check if images are signed with Notary, the signatures were reasched for with the media type of the image manifest itself instead of the media type for notation. Fixed this bug, and improved error messages. This bug would have also been reproducible with main branch if the bad `%` in the test would have fixed. Updated the linter to ignore some issues with the code which is always rewritten when running: `go run github.com/99designs/gqlgen@v0.17.13 generate` Add a workflow to test gqlgen works and has no uncommitted changes Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-15 06:10:51 -05:00
}
// ImageListForDigest is the resolver for the ImageListForDigest field.
func (r *queryResolver) ImageListForDigest(ctx context.Context, id string, requestedPage *gql_generated.PageInput) ([]*gql_generated.ImageSummary, error) {
Update to graphql 1.17.13 We encountered some problems with using the existing folder structure, but it looks like running the tooling with the latest versions works after we regenerated the project using 'gql init' and refactoring to separate the login previously in resolvers.go. - the autogenerated code is now under the gql_generated folder - the file resolvers.go now contains only the code which is not rewritten by the gqlgen framework - the file schema.resolvers.go is rewritten when gqlgen runs, and we'll only keep there the actual resolvers matching query names Changes we observed to schema.resolvers.go when gqlgen runs include reordering methods, and renaming function parameters to match the names used in schema.graphql - we now have a gqlgen.yaml config file which governs the behavior of gqlgen (can be tweaked to restructure the folder structure of the generated code in the future) Looks like the new graphql server has better validation 1 Returns 422 instead of 200 for missing query string - had to update tests 2 Correctly uncovered an error in a test for a bad `%` in query string. As as result of 2, a `masked` bug was found in the way we check if images are signed with Notary, the signatures were reasched for with the media type of the image manifest itself instead of the media type for notation. Fixed this bug, and improved error messages. This bug would have also been reproducible with main branch if the bad `%` in the test would have fixed. Updated the linter to ignore some issues with the code which is always rewritten when running: `go run github.com/99designs/gqlgen@v0.17.13 generate` Add a workflow to test gqlgen works and has no uncommitted changes Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-15 06:10:51 -05:00
r.log.Info().Msg("extracting repositories")
imgResultForDigest, err := getImageListForDigest(ctx, id, r.repoDB, r.cveInfo, requestedPage)
Update to graphql 1.17.13 We encountered some problems with using the existing folder structure, but it looks like running the tooling with the latest versions works after we regenerated the project using 'gql init' and refactoring to separate the login previously in resolvers.go. - the autogenerated code is now under the gql_generated folder - the file resolvers.go now contains only the code which is not rewritten by the gqlgen framework - the file schema.resolvers.go is rewritten when gqlgen runs, and we'll only keep there the actual resolvers matching query names Changes we observed to schema.resolvers.go when gqlgen runs include reordering methods, and renaming function parameters to match the names used in schema.graphql - we now have a gqlgen.yaml config file which governs the behavior of gqlgen (can be tweaked to restructure the folder structure of the generated code in the future) Looks like the new graphql server has better validation 1 Returns 422 instead of 200 for missing query string - had to update tests 2 Correctly uncovered an error in a test for a bad `%` in query string. As as result of 2, a `masked` bug was found in the way we check if images are signed with Notary, the signatures were reasched for with the media type of the image manifest itself instead of the media type for notation. Fixed this bug, and improved error messages. This bug would have also been reproducible with main branch if the bad `%` in the test would have fixed. Updated the linter to ignore some issues with the code which is always rewritten when running: `go run github.com/99designs/gqlgen@v0.17.13 generate` Add a workflow to test gqlgen works and has no uncommitted changes Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-15 06:10:51 -05:00
return imgResultForDigest, err
Update to graphql 1.17.13 We encountered some problems with using the existing folder structure, but it looks like running the tooling with the latest versions works after we regenerated the project using 'gql init' and refactoring to separate the login previously in resolvers.go. - the autogenerated code is now under the gql_generated folder - the file resolvers.go now contains only the code which is not rewritten by the gqlgen framework - the file schema.resolvers.go is rewritten when gqlgen runs, and we'll only keep there the actual resolvers matching query names Changes we observed to schema.resolvers.go when gqlgen runs include reordering methods, and renaming function parameters to match the names used in schema.graphql - we now have a gqlgen.yaml config file which governs the behavior of gqlgen (can be tweaked to restructure the folder structure of the generated code in the future) Looks like the new graphql server has better validation 1 Returns 422 instead of 200 for missing query string - had to update tests 2 Correctly uncovered an error in a test for a bad `%` in query string. As as result of 2, a `masked` bug was found in the way we check if images are signed with Notary, the signatures were reasched for with the media type of the image manifest itself instead of the media type for notation. Fixed this bug, and improved error messages. This bug would have also been reproducible with main branch if the bad `%` in the test would have fixed. Updated the linter to ignore some issues with the code which is always rewritten when running: `go run github.com/99designs/gqlgen@v0.17.13 generate` Add a workflow to test gqlgen works and has no uncommitted changes Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-15 06:10:51 -05:00
}
// RepoListWithNewestImage is the resolver for the RepoListWithNewestImage field.
func (r *queryResolver) RepoListWithNewestImage(ctx context.Context, requestedPage *gql_generated.PageInput) (*gql_generated.PaginatedReposResult, error) {
Update to graphql 1.17.13 We encountered some problems with using the existing folder structure, but it looks like running the tooling with the latest versions works after we regenerated the project using 'gql init' and refactoring to separate the login previously in resolvers.go. - the autogenerated code is now under the gql_generated folder - the file resolvers.go now contains only the code which is not rewritten by the gqlgen framework - the file schema.resolvers.go is rewritten when gqlgen runs, and we'll only keep there the actual resolvers matching query names Changes we observed to schema.resolvers.go when gqlgen runs include reordering methods, and renaming function parameters to match the names used in schema.graphql - we now have a gqlgen.yaml config file which governs the behavior of gqlgen (can be tweaked to restructure the folder structure of the generated code in the future) Looks like the new graphql server has better validation 1 Returns 422 instead of 200 for missing query string - had to update tests 2 Correctly uncovered an error in a test for a bad `%` in query string. As as result of 2, a `masked` bug was found in the way we check if images are signed with Notary, the signatures were reasched for with the media type of the image manifest itself instead of the media type for notation. Fixed this bug, and improved error messages. This bug would have also been reproducible with main branch if the bad `%` in the test would have fixed. Updated the linter to ignore some issues with the code which is always rewritten when running: `go run github.com/99designs/gqlgen@v0.17.13 generate` Add a workflow to test gqlgen works and has no uncommitted changes Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-15 06:10:51 -05:00
r.log.Info().Msg("extension api: finding image list")
paginatedReposResult, err := repoListWithNewestImage(ctx, r.cveInfo, r.log, requestedPage, r.repoDB)
if err != nil {
feat(repodb): Implement RepoDB for image specific information using boltdb/dynamodb (#979) * feat(repodb): implement a DB for image specific information using boltdb (cherry picked from commit e3cb60b8566d84158409815325083ed149175063) Some other fixes/improvements on top (Andrei) Global search: The last updated attribute on repo level is now computed correctly. Global search: Fix and enhance tests: validate more fields, and fix CVE verification logic RepoListWithNewestImage: The vendors and platforms at repo level are no longer containing duplicate entries CVE: scan OCIUncompressedLayer instead of skiping them (used in tests) bug(repodb): do no try to increment download counters for signatures Signed-off-by: Andrei Aaron <andaaron@cisco.com> Add filtering to global search API (Laurentiu) (cherry picked from commit a87976d635ea876fe8ced532e8adb7c3bb24098f) Original work by Laurentiu Niculae <niculae.laurentiu1@gmail.com> Fix pagination bug - when limit was bigger than the repo count result contained empty results - now correctly returns only maximum available number of repo results Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Add history to the fields returned from RepoDB Consolidate fields used in packages - pkg/extensions/search/common/common_test - pkg/extensions/search/common/common Refactor duplicate code in GlobalSearch verification Add vulnerability scan results to image:tag reply Signed-off-by: Andrei Aaron <andaaron@cisco.com> Refactor ExpandedRepoInfo to using RepoDB Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit fd7dc85c3a9d028fd8860d3791cad4df769ed005) Init RepoDB at startup - sync with storage - ignore images without a tag Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 359898facd6541b2aa99ee95080f7aabf28c2650) Update request to get image:tag to use repodb Signed-off-by: Andrei Aaron <andaaron@cisco.com> Sync RepoDB logging - added logging for errors Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 2e128f4d01712b34c70b5468285100b0657001bb) sync-repodb minor error checking fix Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Improve tests for syncing RepoDB with storage Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit b18408c6d64e01312849fc18b929e3a2a7931e9e) Update scoring rule for repos - now prioritize matches to the end of the repo name Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 6961346ccf02223132b3b12a2132c80bd1b6b33c) Upgrade search filters to permit multiple values - multiple values for os and arch Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 3ffb72c6fc0587ff827a03fe4f76a13b27b876a0) feature(repodb): add pagination for RepoListWithNewestImage Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro> (cherry picked from commit 32c917f2dc65363b0856345289353559a8027aee) test(fix): fix tests failing since repodb is used for listing all repos 1. One of the tests was verifying disk/oci related erros and is not applicable 2. Another test was actually broken in an older PR, the default store and the substore were using the same repo names (the substore ones were unprefixed), which should not be the case, this was causing a single entry to show in the RepoDB instead of two separate entries for each test image Root cause in: https://github.com/project-zot/zot/commit/b61aff62cd5088314cee39fae42cec0ad855bd7c#diff-b86e11fa5a3102b336caebec3b30a9d35e26af554dd8658f124dba2404b7d24aR88 Signed-off-by: Andrei Aaron <andaaron@cisco.com> chore: move code reponsible for transforming objects to gql_generated types to separate package Signed-off-by: Andrei Aaron <andaaron@cisco.com> Process input for global search - Clean input: query, filter strings - Add validation for global search input Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit f1ca8670fbe4a4a327ea25cf459237dbf23bb78a) fix: only call cve scanning for data shown to the user Signed-off-by: Andrei Aaron <andaaron@cisco.com> GQL omit scanning for CVE if field is not required Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 5479ce45d6cb2abcf5fbccadeaf6f3393c3f6bf1) Fix filtering logic in RepoDB - filter parameter was set to false instead of being calculator from the later image Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit a82d2327e34e5da617af0b7ca78a2dba90999f0a) bug(repodb): Checking signature returns error if signed image is not found - we considere a signature image orfan when the image it signs is not found - we need this to ignore such signatures in certain cases Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit d0418505f76467accd8e1ee34fcc2b2a165efae5) feat(repodb): CVE logic to use repoDB Also update some method signatures to remove usage of: github.com/google/go-containerregistry/pkg/v1 Signed-off-by: Andrei Aaron <andaaron@cisco.com> * feat(repodb): refactor repodb update logic Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(repodb): minor fixes Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): move repodb logic inside meta directory under pkg Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): change factory class for repodb initialization with factory metrod Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): simplify repodb configuration - repodb now shares config parameters with the cache - config taken directly from storage config Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(authors): fix authors information to work properly with repodb Ideally this commit would be squshed in the repodb commit but as-is it is easier to cherry-pick on other branches Signed-off-by: Andrei Aaron <andaaron@cisco.com> * feat(repodb): dynamodb support for repodb - clean-up repodb code + coverage improvements Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(dynamo): tables used by dynamo are created automatically if they don't exists - if the table exists nothing happens Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * test(repodb): coverage tests - minor fix for CVEListForImage to fix the tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): add descriptor with media type - to represent images and multi-arch images Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): support signatures on repo level - added to follow the behavior of signing and signature verification tools that work on a manifest level for each repo - all images with different tags but the same manifest will be signed at once Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): old repodb version migration support Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): tests for coverage Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): WIP fixing tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): work on patchRepoDB tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(repodb): create dynamo tables only for linux amd Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(ci): fix a typo in ci-cd.yml Signed-off-by: Andrei Aaron <aaaron@luxoft.com> Signed-off-by: Andrei Aaron <andaaron@cisco.com> Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Signed-off-by: Andrei Aaron <aaaron@luxoft.com> Co-authored-by: Andrei Aaron <andaaron@cisco.com> Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
2023-01-09 15:37:44 -05:00
r.log.Error().Err(err).Msg("unable to retrieve repo list")
return paginatedReposResult, err
}
return paginatedReposResult, nil
Update to graphql 1.17.13 We encountered some problems with using the existing folder structure, but it looks like running the tooling with the latest versions works after we regenerated the project using 'gql init' and refactoring to separate the login previously in resolvers.go. - the autogenerated code is now under the gql_generated folder - the file resolvers.go now contains only the code which is not rewritten by the gqlgen framework - the file schema.resolvers.go is rewritten when gqlgen runs, and we'll only keep there the actual resolvers matching query names Changes we observed to schema.resolvers.go when gqlgen runs include reordering methods, and renaming function parameters to match the names used in schema.graphql - we now have a gqlgen.yaml config file which governs the behavior of gqlgen (can be tweaked to restructure the folder structure of the generated code in the future) Looks like the new graphql server has better validation 1 Returns 422 instead of 200 for missing query string - had to update tests 2 Correctly uncovered an error in a test for a bad `%` in query string. As as result of 2, a `masked` bug was found in the way we check if images are signed with Notary, the signatures were reasched for with the media type of the image manifest itself instead of the media type for notation. Fixed this bug, and improved error messages. This bug would have also been reproducible with main branch if the bad `%` in the test would have fixed. Updated the linter to ignore some issues with the code which is always rewritten when running: `go run github.com/99designs/gqlgen@v0.17.13 generate` Add a workflow to test gqlgen works and has no uncommitted changes Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-15 06:10:51 -05:00
}
// ImageList is the resolver for the ImageList field.
func (r *queryResolver) ImageList(ctx context.Context, repo string, requestedPage *gql_generated.PageInput) ([]*gql_generated.ImageSummary, error) {
r.log.Info().Msg("extension api: getting a list of all images")
imageList, err := getImageList(ctx, repo, r.repoDB, r.cveInfo, requestedPage, r.log)
if err != nil {
r.log.Error().Err(err).Msgf("unable to retrieve image list for repo: %s", repo)
return imageList, err
}
return imageList, nil
}
Update to graphql 1.17.13 We encountered some problems with using the existing folder structure, but it looks like running the tooling with the latest versions works after we regenerated the project using 'gql init' and refactoring to separate the login previously in resolvers.go. - the autogenerated code is now under the gql_generated folder - the file resolvers.go now contains only the code which is not rewritten by the gqlgen framework - the file schema.resolvers.go is rewritten when gqlgen runs, and we'll only keep there the actual resolvers matching query names Changes we observed to schema.resolvers.go when gqlgen runs include reordering methods, and renaming function parameters to match the names used in schema.graphql - we now have a gqlgen.yaml config file which governs the behavior of gqlgen (can be tweaked to restructure the folder structure of the generated code in the future) Looks like the new graphql server has better validation 1 Returns 422 instead of 200 for missing query string - had to update tests 2 Correctly uncovered an error in a test for a bad `%` in query string. As as result of 2, a `masked` bug was found in the way we check if images are signed with Notary, the signatures were reasched for with the media type of the image manifest itself instead of the media type for notation. Fixed this bug, and improved error messages. This bug would have also been reproducible with main branch if the bad `%` in the test would have fixed. Updated the linter to ignore some issues with the code which is always rewritten when running: `go run github.com/99designs/gqlgen@v0.17.13 generate` Add a workflow to test gqlgen works and has no uncommitted changes Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-15 06:10:51 -05:00
// ExpandedRepoInfo is the resolver for the ExpandedRepoInfo field.
func (r *queryResolver) ExpandedRepoInfo(ctx context.Context, repo string) (*gql_generated.RepoInfo, error) {
feat(repodb): Implement RepoDB for image specific information using boltdb/dynamodb (#979) * feat(repodb): implement a DB for image specific information using boltdb (cherry picked from commit e3cb60b8566d84158409815325083ed149175063) Some other fixes/improvements on top (Andrei) Global search: The last updated attribute on repo level is now computed correctly. Global search: Fix and enhance tests: validate more fields, and fix CVE verification logic RepoListWithNewestImage: The vendors and platforms at repo level are no longer containing duplicate entries CVE: scan OCIUncompressedLayer instead of skiping them (used in tests) bug(repodb): do no try to increment download counters for signatures Signed-off-by: Andrei Aaron <andaaron@cisco.com> Add filtering to global search API (Laurentiu) (cherry picked from commit a87976d635ea876fe8ced532e8adb7c3bb24098f) Original work by Laurentiu Niculae <niculae.laurentiu1@gmail.com> Fix pagination bug - when limit was bigger than the repo count result contained empty results - now correctly returns only maximum available number of repo results Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Add history to the fields returned from RepoDB Consolidate fields used in packages - pkg/extensions/search/common/common_test - pkg/extensions/search/common/common Refactor duplicate code in GlobalSearch verification Add vulnerability scan results to image:tag reply Signed-off-by: Andrei Aaron <andaaron@cisco.com> Refactor ExpandedRepoInfo to using RepoDB Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit fd7dc85c3a9d028fd8860d3791cad4df769ed005) Init RepoDB at startup - sync with storage - ignore images without a tag Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 359898facd6541b2aa99ee95080f7aabf28c2650) Update request to get image:tag to use repodb Signed-off-by: Andrei Aaron <andaaron@cisco.com> Sync RepoDB logging - added logging for errors Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 2e128f4d01712b34c70b5468285100b0657001bb) sync-repodb minor error checking fix Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Improve tests for syncing RepoDB with storage Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit b18408c6d64e01312849fc18b929e3a2a7931e9e) Update scoring rule for repos - now prioritize matches to the end of the repo name Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 6961346ccf02223132b3b12a2132c80bd1b6b33c) Upgrade search filters to permit multiple values - multiple values for os and arch Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 3ffb72c6fc0587ff827a03fe4f76a13b27b876a0) feature(repodb): add pagination for RepoListWithNewestImage Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro> (cherry picked from commit 32c917f2dc65363b0856345289353559a8027aee) test(fix): fix tests failing since repodb is used for listing all repos 1. One of the tests was verifying disk/oci related erros and is not applicable 2. Another test was actually broken in an older PR, the default store and the substore were using the same repo names (the substore ones were unprefixed), which should not be the case, this was causing a single entry to show in the RepoDB instead of two separate entries for each test image Root cause in: https://github.com/project-zot/zot/commit/b61aff62cd5088314cee39fae42cec0ad855bd7c#diff-b86e11fa5a3102b336caebec3b30a9d35e26af554dd8658f124dba2404b7d24aR88 Signed-off-by: Andrei Aaron <andaaron@cisco.com> chore: move code reponsible for transforming objects to gql_generated types to separate package Signed-off-by: Andrei Aaron <andaaron@cisco.com> Process input for global search - Clean input: query, filter strings - Add validation for global search input Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit f1ca8670fbe4a4a327ea25cf459237dbf23bb78a) fix: only call cve scanning for data shown to the user Signed-off-by: Andrei Aaron <andaaron@cisco.com> GQL omit scanning for CVE if field is not required Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 5479ce45d6cb2abcf5fbccadeaf6f3393c3f6bf1) Fix filtering logic in RepoDB - filter parameter was set to false instead of being calculator from the later image Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit a82d2327e34e5da617af0b7ca78a2dba90999f0a) bug(repodb): Checking signature returns error if signed image is not found - we considere a signature image orfan when the image it signs is not found - we need this to ignore such signatures in certain cases Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit d0418505f76467accd8e1ee34fcc2b2a165efae5) feat(repodb): CVE logic to use repoDB Also update some method signatures to remove usage of: github.com/google/go-containerregistry/pkg/v1 Signed-off-by: Andrei Aaron <andaaron@cisco.com> * feat(repodb): refactor repodb update logic Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(repodb): minor fixes Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): move repodb logic inside meta directory under pkg Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): change factory class for repodb initialization with factory metrod Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): simplify repodb configuration - repodb now shares config parameters with the cache - config taken directly from storage config Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(authors): fix authors information to work properly with repodb Ideally this commit would be squshed in the repodb commit but as-is it is easier to cherry-pick on other branches Signed-off-by: Andrei Aaron <andaaron@cisco.com> * feat(repodb): dynamodb support for repodb - clean-up repodb code + coverage improvements Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(dynamo): tables used by dynamo are created automatically if they don't exists - if the table exists nothing happens Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * test(repodb): coverage tests - minor fix for CVEListForImage to fix the tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): add descriptor with media type - to represent images and multi-arch images Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): support signatures on repo level - added to follow the behavior of signing and signature verification tools that work on a manifest level for each repo - all images with different tags but the same manifest will be signed at once Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): old repodb version migration support Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): tests for coverage Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): WIP fixing tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): work on patchRepoDB tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(repodb): create dynamo tables only for linux amd Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(ci): fix a typo in ci-cd.yml Signed-off-by: Andrei Aaron <aaaron@luxoft.com> Signed-off-by: Andrei Aaron <andaaron@cisco.com> Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Signed-off-by: Andrei Aaron <aaaron@luxoft.com> Co-authored-by: Andrei Aaron <andaaron@cisco.com> Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
2023-01-09 15:37:44 -05:00
repoInfo, err := expandedRepoInfo(ctx, repo, r.repoDB, r.cveInfo, r.log)
feat(repodb): Implement RepoDB for image specific information using boltdb/dynamodb (#979) * feat(repodb): implement a DB for image specific information using boltdb (cherry picked from commit e3cb60b8566d84158409815325083ed149175063) Some other fixes/improvements on top (Andrei) Global search: The last updated attribute on repo level is now computed correctly. Global search: Fix and enhance tests: validate more fields, and fix CVE verification logic RepoListWithNewestImage: The vendors and platforms at repo level are no longer containing duplicate entries CVE: scan OCIUncompressedLayer instead of skiping them (used in tests) bug(repodb): do no try to increment download counters for signatures Signed-off-by: Andrei Aaron <andaaron@cisco.com> Add filtering to global search API (Laurentiu) (cherry picked from commit a87976d635ea876fe8ced532e8adb7c3bb24098f) Original work by Laurentiu Niculae <niculae.laurentiu1@gmail.com> Fix pagination bug - when limit was bigger than the repo count result contained empty results - now correctly returns only maximum available number of repo results Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Add history to the fields returned from RepoDB Consolidate fields used in packages - pkg/extensions/search/common/common_test - pkg/extensions/search/common/common Refactor duplicate code in GlobalSearch verification Add vulnerability scan results to image:tag reply Signed-off-by: Andrei Aaron <andaaron@cisco.com> Refactor ExpandedRepoInfo to using RepoDB Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit fd7dc85c3a9d028fd8860d3791cad4df769ed005) Init RepoDB at startup - sync with storage - ignore images without a tag Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 359898facd6541b2aa99ee95080f7aabf28c2650) Update request to get image:tag to use repodb Signed-off-by: Andrei Aaron <andaaron@cisco.com> Sync RepoDB logging - added logging for errors Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 2e128f4d01712b34c70b5468285100b0657001bb) sync-repodb minor error checking fix Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Improve tests for syncing RepoDB with storage Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit b18408c6d64e01312849fc18b929e3a2a7931e9e) Update scoring rule for repos - now prioritize matches to the end of the repo name Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 6961346ccf02223132b3b12a2132c80bd1b6b33c) Upgrade search filters to permit multiple values - multiple values for os and arch Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 3ffb72c6fc0587ff827a03fe4f76a13b27b876a0) feature(repodb): add pagination for RepoListWithNewestImage Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro> (cherry picked from commit 32c917f2dc65363b0856345289353559a8027aee) test(fix): fix tests failing since repodb is used for listing all repos 1. One of the tests was verifying disk/oci related erros and is not applicable 2. Another test was actually broken in an older PR, the default store and the substore were using the same repo names (the substore ones were unprefixed), which should not be the case, this was causing a single entry to show in the RepoDB instead of two separate entries for each test image Root cause in: https://github.com/project-zot/zot/commit/b61aff62cd5088314cee39fae42cec0ad855bd7c#diff-b86e11fa5a3102b336caebec3b30a9d35e26af554dd8658f124dba2404b7d24aR88 Signed-off-by: Andrei Aaron <andaaron@cisco.com> chore: move code reponsible for transforming objects to gql_generated types to separate package Signed-off-by: Andrei Aaron <andaaron@cisco.com> Process input for global search - Clean input: query, filter strings - Add validation for global search input Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit f1ca8670fbe4a4a327ea25cf459237dbf23bb78a) fix: only call cve scanning for data shown to the user Signed-off-by: Andrei Aaron <andaaron@cisco.com> GQL omit scanning for CVE if field is not required Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 5479ce45d6cb2abcf5fbccadeaf6f3393c3f6bf1) Fix filtering logic in RepoDB - filter parameter was set to false instead of being calculator from the later image Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit a82d2327e34e5da617af0b7ca78a2dba90999f0a) bug(repodb): Checking signature returns error if signed image is not found - we considere a signature image orfan when the image it signs is not found - we need this to ignore such signatures in certain cases Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit d0418505f76467accd8e1ee34fcc2b2a165efae5) feat(repodb): CVE logic to use repoDB Also update some method signatures to remove usage of: github.com/google/go-containerregistry/pkg/v1 Signed-off-by: Andrei Aaron <andaaron@cisco.com> * feat(repodb): refactor repodb update logic Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(repodb): minor fixes Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): move repodb logic inside meta directory under pkg Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): change factory class for repodb initialization with factory metrod Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): simplify repodb configuration - repodb now shares config parameters with the cache - config taken directly from storage config Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(authors): fix authors information to work properly with repodb Ideally this commit would be squshed in the repodb commit but as-is it is easier to cherry-pick on other branches Signed-off-by: Andrei Aaron <andaaron@cisco.com> * feat(repodb): dynamodb support for repodb - clean-up repodb code + coverage improvements Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(dynamo): tables used by dynamo are created automatically if they don't exists - if the table exists nothing happens Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * test(repodb): coverage tests - minor fix for CVEListForImage to fix the tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): add descriptor with media type - to represent images and multi-arch images Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): support signatures on repo level - added to follow the behavior of signing and signature verification tools that work on a manifest level for each repo - all images with different tags but the same manifest will be signed at once Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): old repodb version migration support Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): tests for coverage Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): WIP fixing tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): work on patchRepoDB tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(repodb): create dynamo tables only for linux amd Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(ci): fix a typo in ci-cd.yml Signed-off-by: Andrei Aaron <aaaron@luxoft.com> Signed-off-by: Andrei Aaron <andaaron@cisco.com> Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Signed-off-by: Andrei Aaron <aaaron@luxoft.com> Co-authored-by: Andrei Aaron <andaaron@cisco.com> Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
2023-01-09 15:37:44 -05:00
return repoInfo, err
Update to graphql 1.17.13 We encountered some problems with using the existing folder structure, but it looks like running the tooling with the latest versions works after we regenerated the project using 'gql init' and refactoring to separate the login previously in resolvers.go. - the autogenerated code is now under the gql_generated folder - the file resolvers.go now contains only the code which is not rewritten by the gqlgen framework - the file schema.resolvers.go is rewritten when gqlgen runs, and we'll only keep there the actual resolvers matching query names Changes we observed to schema.resolvers.go when gqlgen runs include reordering methods, and renaming function parameters to match the names used in schema.graphql - we now have a gqlgen.yaml config file which governs the behavior of gqlgen (can be tweaked to restructure the folder structure of the generated code in the future) Looks like the new graphql server has better validation 1 Returns 422 instead of 200 for missing query string - had to update tests 2 Correctly uncovered an error in a test for a bad `%` in query string. As as result of 2, a `masked` bug was found in the way we check if images are signed with Notary, the signatures were reasched for with the media type of the image manifest itself instead of the media type for notation. Fixed this bug, and improved error messages. This bug would have also been reproducible with main branch if the bad `%` in the test would have fixed. Updated the linter to ignore some issues with the code which is always rewritten when running: `go run github.com/99designs/gqlgen@v0.17.13 generate` Add a workflow to test gqlgen works and has no uncommitted changes Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-15 06:10:51 -05:00
}
// GlobalSearch is the resolver for the GlobalSearch field.
feat(repodb): Implement RepoDB for image specific information using boltdb/dynamodb (#979) * feat(repodb): implement a DB for image specific information using boltdb (cherry picked from commit e3cb60b8566d84158409815325083ed149175063) Some other fixes/improvements on top (Andrei) Global search: The last updated attribute on repo level is now computed correctly. Global search: Fix and enhance tests: validate more fields, and fix CVE verification logic RepoListWithNewestImage: The vendors and platforms at repo level are no longer containing duplicate entries CVE: scan OCIUncompressedLayer instead of skiping them (used in tests) bug(repodb): do no try to increment download counters for signatures Signed-off-by: Andrei Aaron <andaaron@cisco.com> Add filtering to global search API (Laurentiu) (cherry picked from commit a87976d635ea876fe8ced532e8adb7c3bb24098f) Original work by Laurentiu Niculae <niculae.laurentiu1@gmail.com> Fix pagination bug - when limit was bigger than the repo count result contained empty results - now correctly returns only maximum available number of repo results Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Add history to the fields returned from RepoDB Consolidate fields used in packages - pkg/extensions/search/common/common_test - pkg/extensions/search/common/common Refactor duplicate code in GlobalSearch verification Add vulnerability scan results to image:tag reply Signed-off-by: Andrei Aaron <andaaron@cisco.com> Refactor ExpandedRepoInfo to using RepoDB Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit fd7dc85c3a9d028fd8860d3791cad4df769ed005) Init RepoDB at startup - sync with storage - ignore images without a tag Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 359898facd6541b2aa99ee95080f7aabf28c2650) Update request to get image:tag to use repodb Signed-off-by: Andrei Aaron <andaaron@cisco.com> Sync RepoDB logging - added logging for errors Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 2e128f4d01712b34c70b5468285100b0657001bb) sync-repodb minor error checking fix Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Improve tests for syncing RepoDB with storage Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit b18408c6d64e01312849fc18b929e3a2a7931e9e) Update scoring rule for repos - now prioritize matches to the end of the repo name Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 6961346ccf02223132b3b12a2132c80bd1b6b33c) Upgrade search filters to permit multiple values - multiple values for os and arch Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 3ffb72c6fc0587ff827a03fe4f76a13b27b876a0) feature(repodb): add pagination for RepoListWithNewestImage Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro> (cherry picked from commit 32c917f2dc65363b0856345289353559a8027aee) test(fix): fix tests failing since repodb is used for listing all repos 1. One of the tests was verifying disk/oci related erros and is not applicable 2. Another test was actually broken in an older PR, the default store and the substore were using the same repo names (the substore ones were unprefixed), which should not be the case, this was causing a single entry to show in the RepoDB instead of two separate entries for each test image Root cause in: https://github.com/project-zot/zot/commit/b61aff62cd5088314cee39fae42cec0ad855bd7c#diff-b86e11fa5a3102b336caebec3b30a9d35e26af554dd8658f124dba2404b7d24aR88 Signed-off-by: Andrei Aaron <andaaron@cisco.com> chore: move code reponsible for transforming objects to gql_generated types to separate package Signed-off-by: Andrei Aaron <andaaron@cisco.com> Process input for global search - Clean input: query, filter strings - Add validation for global search input Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit f1ca8670fbe4a4a327ea25cf459237dbf23bb78a) fix: only call cve scanning for data shown to the user Signed-off-by: Andrei Aaron <andaaron@cisco.com> GQL omit scanning for CVE if field is not required Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 5479ce45d6cb2abcf5fbccadeaf6f3393c3f6bf1) Fix filtering logic in RepoDB - filter parameter was set to false instead of being calculator from the later image Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit a82d2327e34e5da617af0b7ca78a2dba90999f0a) bug(repodb): Checking signature returns error if signed image is not found - we considere a signature image orfan when the image it signs is not found - we need this to ignore such signatures in certain cases Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit d0418505f76467accd8e1ee34fcc2b2a165efae5) feat(repodb): CVE logic to use repoDB Also update some method signatures to remove usage of: github.com/google/go-containerregistry/pkg/v1 Signed-off-by: Andrei Aaron <andaaron@cisco.com> * feat(repodb): refactor repodb update logic Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(repodb): minor fixes Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): move repodb logic inside meta directory under pkg Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): change factory class for repodb initialization with factory metrod Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): simplify repodb configuration - repodb now shares config parameters with the cache - config taken directly from storage config Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(authors): fix authors information to work properly with repodb Ideally this commit would be squshed in the repodb commit but as-is it is easier to cherry-pick on other branches Signed-off-by: Andrei Aaron <andaaron@cisco.com> * feat(repodb): dynamodb support for repodb - clean-up repodb code + coverage improvements Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(dynamo): tables used by dynamo are created automatically if they don't exists - if the table exists nothing happens Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * test(repodb): coverage tests - minor fix for CVEListForImage to fix the tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): add descriptor with media type - to represent images and multi-arch images Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): support signatures on repo level - added to follow the behavior of signing and signature verification tools that work on a manifest level for each repo - all images with different tags but the same manifest will be signed at once Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): old repodb version migration support Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): tests for coverage Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): WIP fixing tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): work on patchRepoDB tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(repodb): create dynamo tables only for linux amd Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(ci): fix a typo in ci-cd.yml Signed-off-by: Andrei Aaron <aaaron@luxoft.com> Signed-off-by: Andrei Aaron <andaaron@cisco.com> Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Signed-off-by: Andrei Aaron <aaaron@luxoft.com> Co-authored-by: Andrei Aaron <andaaron@cisco.com> Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
2023-01-09 15:37:44 -05:00
func (r *queryResolver) GlobalSearch(ctx context.Context, query string, filter *gql_generated.Filter, requestedPage *gql_generated.PageInput) (*gql_generated.GlobalSearchResult, error) {
if err := validateGlobalSearchInput(query, filter, requestedPage); err != nil {
return &gql_generated.GlobalSearchResult{}, err
}
feat(repodb): Implement RepoDB for image specific information using boltdb/dynamodb (#979) * feat(repodb): implement a DB for image specific information using boltdb (cherry picked from commit e3cb60b8566d84158409815325083ed149175063) Some other fixes/improvements on top (Andrei) Global search: The last updated attribute on repo level is now computed correctly. Global search: Fix and enhance tests: validate more fields, and fix CVE verification logic RepoListWithNewestImage: The vendors and platforms at repo level are no longer containing duplicate entries CVE: scan OCIUncompressedLayer instead of skiping them (used in tests) bug(repodb): do no try to increment download counters for signatures Signed-off-by: Andrei Aaron <andaaron@cisco.com> Add filtering to global search API (Laurentiu) (cherry picked from commit a87976d635ea876fe8ced532e8adb7c3bb24098f) Original work by Laurentiu Niculae <niculae.laurentiu1@gmail.com> Fix pagination bug - when limit was bigger than the repo count result contained empty results - now correctly returns only maximum available number of repo results Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Add history to the fields returned from RepoDB Consolidate fields used in packages - pkg/extensions/search/common/common_test - pkg/extensions/search/common/common Refactor duplicate code in GlobalSearch verification Add vulnerability scan results to image:tag reply Signed-off-by: Andrei Aaron <andaaron@cisco.com> Refactor ExpandedRepoInfo to using RepoDB Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit fd7dc85c3a9d028fd8860d3791cad4df769ed005) Init RepoDB at startup - sync with storage - ignore images without a tag Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 359898facd6541b2aa99ee95080f7aabf28c2650) Update request to get image:tag to use repodb Signed-off-by: Andrei Aaron <andaaron@cisco.com> Sync RepoDB logging - added logging for errors Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 2e128f4d01712b34c70b5468285100b0657001bb) sync-repodb minor error checking fix Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Improve tests for syncing RepoDB with storage Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit b18408c6d64e01312849fc18b929e3a2a7931e9e) Update scoring rule for repos - now prioritize matches to the end of the repo name Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 6961346ccf02223132b3b12a2132c80bd1b6b33c) Upgrade search filters to permit multiple values - multiple values for os and arch Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 3ffb72c6fc0587ff827a03fe4f76a13b27b876a0) feature(repodb): add pagination for RepoListWithNewestImage Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro> (cherry picked from commit 32c917f2dc65363b0856345289353559a8027aee) test(fix): fix tests failing since repodb is used for listing all repos 1. One of the tests was verifying disk/oci related erros and is not applicable 2. Another test was actually broken in an older PR, the default store and the substore were using the same repo names (the substore ones were unprefixed), which should not be the case, this was causing a single entry to show in the RepoDB instead of two separate entries for each test image Root cause in: https://github.com/project-zot/zot/commit/b61aff62cd5088314cee39fae42cec0ad855bd7c#diff-b86e11fa5a3102b336caebec3b30a9d35e26af554dd8658f124dba2404b7d24aR88 Signed-off-by: Andrei Aaron <andaaron@cisco.com> chore: move code reponsible for transforming objects to gql_generated types to separate package Signed-off-by: Andrei Aaron <andaaron@cisco.com> Process input for global search - Clean input: query, filter strings - Add validation for global search input Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit f1ca8670fbe4a4a327ea25cf459237dbf23bb78a) fix: only call cve scanning for data shown to the user Signed-off-by: Andrei Aaron <andaaron@cisco.com> GQL omit scanning for CVE if field is not required Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 5479ce45d6cb2abcf5fbccadeaf6f3393c3f6bf1) Fix filtering logic in RepoDB - filter parameter was set to false instead of being calculator from the later image Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit a82d2327e34e5da617af0b7ca78a2dba90999f0a) bug(repodb): Checking signature returns error if signed image is not found - we considere a signature image orfan when the image it signs is not found - we need this to ignore such signatures in certain cases Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit d0418505f76467accd8e1ee34fcc2b2a165efae5) feat(repodb): CVE logic to use repoDB Also update some method signatures to remove usage of: github.com/google/go-containerregistry/pkg/v1 Signed-off-by: Andrei Aaron <andaaron@cisco.com> * feat(repodb): refactor repodb update logic Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(repodb): minor fixes Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): move repodb logic inside meta directory under pkg Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): change factory class for repodb initialization with factory metrod Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): simplify repodb configuration - repodb now shares config parameters with the cache - config taken directly from storage config Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(authors): fix authors information to work properly with repodb Ideally this commit would be squshed in the repodb commit but as-is it is easier to cherry-pick on other branches Signed-off-by: Andrei Aaron <andaaron@cisco.com> * feat(repodb): dynamodb support for repodb - clean-up repodb code + coverage improvements Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(dynamo): tables used by dynamo are created automatically if they don't exists - if the table exists nothing happens Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * test(repodb): coverage tests - minor fix for CVEListForImage to fix the tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): add descriptor with media type - to represent images and multi-arch images Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): support signatures on repo level - added to follow the behavior of signing and signature verification tools that work on a manifest level for each repo - all images with different tags but the same manifest will be signed at once Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): old repodb version migration support Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): tests for coverage Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): WIP fixing tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): work on patchRepoDB tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(repodb): create dynamo tables only for linux amd Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(ci): fix a typo in ci-cd.yml Signed-off-by: Andrei Aaron <aaaron@luxoft.com> Signed-off-by: Andrei Aaron <andaaron@cisco.com> Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Signed-off-by: Andrei Aaron <aaaron@luxoft.com> Co-authored-by: Andrei Aaron <andaaron@cisco.com> Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
2023-01-09 15:37:44 -05:00
query = cleanQuery(query)
filter = cleanFilter(filter)
paginatedReposResult, images, layers, err := globalSearch(ctx, query, r.repoDB, filter, requestedPage, r.cveInfo, r.log)
return &gql_generated.GlobalSearchResult{
Page: paginatedReposResult.Page,
Images: images,
Repos: paginatedReposResult.Results,
Layers: layers,
feat(repodb): Implement RepoDB for image specific information using boltdb/dynamodb (#979) * feat(repodb): implement a DB for image specific information using boltdb (cherry picked from commit e3cb60b8566d84158409815325083ed149175063) Some other fixes/improvements on top (Andrei) Global search: The last updated attribute on repo level is now computed correctly. Global search: Fix and enhance tests: validate more fields, and fix CVE verification logic RepoListWithNewestImage: The vendors and platforms at repo level are no longer containing duplicate entries CVE: scan OCIUncompressedLayer instead of skiping them (used in tests) bug(repodb): do no try to increment download counters for signatures Signed-off-by: Andrei Aaron <andaaron@cisco.com> Add filtering to global search API (Laurentiu) (cherry picked from commit a87976d635ea876fe8ced532e8adb7c3bb24098f) Original work by Laurentiu Niculae <niculae.laurentiu1@gmail.com> Fix pagination bug - when limit was bigger than the repo count result contained empty results - now correctly returns only maximum available number of repo results Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Add history to the fields returned from RepoDB Consolidate fields used in packages - pkg/extensions/search/common/common_test - pkg/extensions/search/common/common Refactor duplicate code in GlobalSearch verification Add vulnerability scan results to image:tag reply Signed-off-by: Andrei Aaron <andaaron@cisco.com> Refactor ExpandedRepoInfo to using RepoDB Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit fd7dc85c3a9d028fd8860d3791cad4df769ed005) Init RepoDB at startup - sync with storage - ignore images without a tag Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 359898facd6541b2aa99ee95080f7aabf28c2650) Update request to get image:tag to use repodb Signed-off-by: Andrei Aaron <andaaron@cisco.com> Sync RepoDB logging - added logging for errors Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 2e128f4d01712b34c70b5468285100b0657001bb) sync-repodb minor error checking fix Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Improve tests for syncing RepoDB with storage Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit b18408c6d64e01312849fc18b929e3a2a7931e9e) Update scoring rule for repos - now prioritize matches to the end of the repo name Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 6961346ccf02223132b3b12a2132c80bd1b6b33c) Upgrade search filters to permit multiple values - multiple values for os and arch Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 3ffb72c6fc0587ff827a03fe4f76a13b27b876a0) feature(repodb): add pagination for RepoListWithNewestImage Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro> (cherry picked from commit 32c917f2dc65363b0856345289353559a8027aee) test(fix): fix tests failing since repodb is used for listing all repos 1. One of the tests was verifying disk/oci related erros and is not applicable 2. Another test was actually broken in an older PR, the default store and the substore were using the same repo names (the substore ones were unprefixed), which should not be the case, this was causing a single entry to show in the RepoDB instead of two separate entries for each test image Root cause in: https://github.com/project-zot/zot/commit/b61aff62cd5088314cee39fae42cec0ad855bd7c#diff-b86e11fa5a3102b336caebec3b30a9d35e26af554dd8658f124dba2404b7d24aR88 Signed-off-by: Andrei Aaron <andaaron@cisco.com> chore: move code reponsible for transforming objects to gql_generated types to separate package Signed-off-by: Andrei Aaron <andaaron@cisco.com> Process input for global search - Clean input: query, filter strings - Add validation for global search input Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit f1ca8670fbe4a4a327ea25cf459237dbf23bb78a) fix: only call cve scanning for data shown to the user Signed-off-by: Andrei Aaron <andaaron@cisco.com> GQL omit scanning for CVE if field is not required Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 5479ce45d6cb2abcf5fbccadeaf6f3393c3f6bf1) Fix filtering logic in RepoDB - filter parameter was set to false instead of being calculator from the later image Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit a82d2327e34e5da617af0b7ca78a2dba90999f0a) bug(repodb): Checking signature returns error if signed image is not found - we considere a signature image orfan when the image it signs is not found - we need this to ignore such signatures in certain cases Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit d0418505f76467accd8e1ee34fcc2b2a165efae5) feat(repodb): CVE logic to use repoDB Also update some method signatures to remove usage of: github.com/google/go-containerregistry/pkg/v1 Signed-off-by: Andrei Aaron <andaaron@cisco.com> * feat(repodb): refactor repodb update logic Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(repodb): minor fixes Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): move repodb logic inside meta directory under pkg Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): change factory class for repodb initialization with factory metrod Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): simplify repodb configuration - repodb now shares config parameters with the cache - config taken directly from storage config Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(authors): fix authors information to work properly with repodb Ideally this commit would be squshed in the repodb commit but as-is it is easier to cherry-pick on other branches Signed-off-by: Andrei Aaron <andaaron@cisco.com> * feat(repodb): dynamodb support for repodb - clean-up repodb code + coverage improvements Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(dynamo): tables used by dynamo are created automatically if they don't exists - if the table exists nothing happens Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * test(repodb): coverage tests - minor fix for CVEListForImage to fix the tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): add descriptor with media type - to represent images and multi-arch images Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): support signatures on repo level - added to follow the behavior of signing and signature verification tools that work on a manifest level for each repo - all images with different tags but the same manifest will be signed at once Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): old repodb version migration support Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): tests for coverage Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): WIP fixing tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): work on patchRepoDB tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(repodb): create dynamo tables only for linux amd Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(ci): fix a typo in ci-cd.yml Signed-off-by: Andrei Aaron <aaaron@luxoft.com> Signed-off-by: Andrei Aaron <andaaron@cisco.com> Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Signed-off-by: Andrei Aaron <aaaron@luxoft.com> Co-authored-by: Andrei Aaron <andaaron@cisco.com> Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
2023-01-09 15:37:44 -05:00
}, err
}
// DependencyListForImage is the resolver for the DependencyListForImage field.
func (r *queryResolver) DerivedImageList(ctx context.Context, image string) ([]*gql_generated.ImageSummary, error) {
layoutUtils := common.NewBaseOciLayoutUtils(r.storeController, r.log)
imageList := make([]*gql_generated.ImageSummary, 0)
repoList, err := layoutUtils.GetRepositories()
if err != nil {
r.log.Error().Err(err).Msg("unable to get repositories list")
return nil, err
}
if len(repoList) == 0 {
r.log.Info().Msg("no repositories found")
return imageList, nil
}
imageDir, imageTag := common.GetImageDirAndTag(image)
if imageTag == "" {
return []*gql_generated.ImageSummary{}, gqlerror.Errorf("no reference provided")
}
imageManifest, _, err := layoutUtils.GetImageManifest(imageDir, imageTag)
if err != nil {
r.log.Info().Str("image", image).Msg("image not found")
return imageList, err
}
imageLayers := imageManifest.Layers
for _, repo := range repoList {
repoInfo, err := r.ExpandedRepoInfo(ctx, repo)
if err != nil {
r.log.Error().Err(err).Msg("unable to get image list")
return nil, err
}
imageSummaries := repoInfo.Images
// verify every image
for _, imageSummary := range imageSummaries {
if imageTag == *imageSummary.Tag && imageDir == repo {
continue
}
layers := imageSummary.Layers
sameLayer := 0
for _, l := range imageLayers {
for _, k := range layers {
if *k.Digest == l.Digest.String() {
sameLayer++
}
}
}
// if all layers are the same
if sameLayer == len(imageLayers) {
// add to returned list
imageList = append(imageList, imageSummary)
}
}
}
return imageList, nil
}
// BaseImageList is the resolver for the BaseImageList field.
func (r *queryResolver) BaseImageList(ctx context.Context, image string) ([]*gql_generated.ImageSummary, error) {
layoutUtils := common.NewBaseOciLayoutUtils(r.storeController, r.log)
imageList := make([]*gql_generated.ImageSummary, 0)
repoList, err := layoutUtils.GetRepositories()
if err != nil {
r.log.Error().Err(err).Msg("unable to get repositories list")
return nil, err
}
if len(repoList) == 0 {
r.log.Info().Msg("no repositories found")
return imageList, nil
}
imageDir, imageTag := common.GetImageDirAndTag(image)
if imageTag == "" {
return []*gql_generated.ImageSummary{}, gqlerror.Errorf("no reference provided")
}
imageManifest, _, err := layoutUtils.GetImageManifest(imageDir, imageTag)
if err != nil {
r.log.Info().Str("image", image).Msg("image not found")
return imageList, err
}
imageLayers := imageManifest.Layers
// This logic may not scale well in the future as we need to read all the
// manifest files from the disk when the call is made, we should improve in a future PR
for _, repo := range repoList {
repoInfo, err := r.ExpandedRepoInfo(ctx, repo)
if err != nil {
r.log.Error().Err(err).Msg("unable to get image list")
return nil, err
}
imageSummaries := repoInfo.Images
var addImageToList bool
// verify every image
for _, imageSummary := range imageSummaries {
if imageTag == *imageSummary.Tag && imageDir == repo {
continue
}
addImageToList = true
layers := imageSummary.Layers
for _, l := range layers {
foundLayer := false
for _, k := range imageLayers {
if *l.Digest == k.Digest.String() {
foundLayer = true
break
}
}
if !foundLayer {
addImageToList = false
break
}
}
if addImageToList {
imageList = append(imageList, imageSummary)
}
}
}
return imageList, nil
}
// Image is the resolver for the Image field.
func (r *queryResolver) Image(ctx context.Context, image string) (*gql_generated.ImageSummary, error) {
repo, tag := common.GetImageDirAndTag(image)
if tag == "" {
return &gql_generated.ImageSummary{}, gqlerror.Errorf("no reference provided")
}
feat(repodb): Implement RepoDB for image specific information using boltdb/dynamodb (#979) * feat(repodb): implement a DB for image specific information using boltdb (cherry picked from commit e3cb60b8566d84158409815325083ed149175063) Some other fixes/improvements on top (Andrei) Global search: The last updated attribute on repo level is now computed correctly. Global search: Fix and enhance tests: validate more fields, and fix CVE verification logic RepoListWithNewestImage: The vendors and platforms at repo level are no longer containing duplicate entries CVE: scan OCIUncompressedLayer instead of skiping them (used in tests) bug(repodb): do no try to increment download counters for signatures Signed-off-by: Andrei Aaron <andaaron@cisco.com> Add filtering to global search API (Laurentiu) (cherry picked from commit a87976d635ea876fe8ced532e8adb7c3bb24098f) Original work by Laurentiu Niculae <niculae.laurentiu1@gmail.com> Fix pagination bug - when limit was bigger than the repo count result contained empty results - now correctly returns only maximum available number of repo results Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Add history to the fields returned from RepoDB Consolidate fields used in packages - pkg/extensions/search/common/common_test - pkg/extensions/search/common/common Refactor duplicate code in GlobalSearch verification Add vulnerability scan results to image:tag reply Signed-off-by: Andrei Aaron <andaaron@cisco.com> Refactor ExpandedRepoInfo to using RepoDB Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit fd7dc85c3a9d028fd8860d3791cad4df769ed005) Init RepoDB at startup - sync with storage - ignore images without a tag Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 359898facd6541b2aa99ee95080f7aabf28c2650) Update request to get image:tag to use repodb Signed-off-by: Andrei Aaron <andaaron@cisco.com> Sync RepoDB logging - added logging for errors Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 2e128f4d01712b34c70b5468285100b0657001bb) sync-repodb minor error checking fix Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Improve tests for syncing RepoDB with storage Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit b18408c6d64e01312849fc18b929e3a2a7931e9e) Update scoring rule for repos - now prioritize matches to the end of the repo name Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 6961346ccf02223132b3b12a2132c80bd1b6b33c) Upgrade search filters to permit multiple values - multiple values for os and arch Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 3ffb72c6fc0587ff827a03fe4f76a13b27b876a0) feature(repodb): add pagination for RepoListWithNewestImage Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro> (cherry picked from commit 32c917f2dc65363b0856345289353559a8027aee) test(fix): fix tests failing since repodb is used for listing all repos 1. One of the tests was verifying disk/oci related erros and is not applicable 2. Another test was actually broken in an older PR, the default store and the substore were using the same repo names (the substore ones were unprefixed), which should not be the case, this was causing a single entry to show in the RepoDB instead of two separate entries for each test image Root cause in: https://github.com/project-zot/zot/commit/b61aff62cd5088314cee39fae42cec0ad855bd7c#diff-b86e11fa5a3102b336caebec3b30a9d35e26af554dd8658f124dba2404b7d24aR88 Signed-off-by: Andrei Aaron <andaaron@cisco.com> chore: move code reponsible for transforming objects to gql_generated types to separate package Signed-off-by: Andrei Aaron <andaaron@cisco.com> Process input for global search - Clean input: query, filter strings - Add validation for global search input Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit f1ca8670fbe4a4a327ea25cf459237dbf23bb78a) fix: only call cve scanning for data shown to the user Signed-off-by: Andrei Aaron <andaaron@cisco.com> GQL omit scanning for CVE if field is not required Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit 5479ce45d6cb2abcf5fbccadeaf6f3393c3f6bf1) Fix filtering logic in RepoDB - filter parameter was set to false instead of being calculator from the later image Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit a82d2327e34e5da617af0b7ca78a2dba90999f0a) bug(repodb): Checking signature returns error if signed image is not found - we considere a signature image orfan when the image it signs is not found - we need this to ignore such signatures in certain cases Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> (cherry picked from commit d0418505f76467accd8e1ee34fcc2b2a165efae5) feat(repodb): CVE logic to use repoDB Also update some method signatures to remove usage of: github.com/google/go-containerregistry/pkg/v1 Signed-off-by: Andrei Aaron <andaaron@cisco.com> * feat(repodb): refactor repodb update logic Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(repodb): minor fixes Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): move repodb logic inside meta directory under pkg Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): change factory class for repodb initialization with factory metrod Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): simplify repodb configuration - repodb now shares config parameters with the cache - config taken directly from storage config Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(authors): fix authors information to work properly with repodb Ideally this commit would be squshed in the repodb commit but as-is it is easier to cherry-pick on other branches Signed-off-by: Andrei Aaron <andaaron@cisco.com> * feat(repodb): dynamodb support for repodb - clean-up repodb code + coverage improvements Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(dynamo): tables used by dynamo are created automatically if they don't exists - if the table exists nothing happens Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * test(repodb): coverage tests - minor fix for CVEListForImage to fix the tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): add descriptor with media type - to represent images and multi-arch images Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): support signatures on repo level - added to follow the behavior of signing and signature verification tools that work on a manifest level for each repo - all images with different tags but the same manifest will be signed at once Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): old repodb version migration support Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): tests for coverage Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): WIP fixing tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * feat(repodb): work on patchRepoDB tests Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(repodb): create dynamo tables only for linux amd Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> * fix(ci): fix a typo in ci-cd.yml Signed-off-by: Andrei Aaron <aaaron@luxoft.com> Signed-off-by: Andrei Aaron <andaaron@cisco.com> Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com> Signed-off-by: Andrei Aaron <aaaron@luxoft.com> Co-authored-by: Andrei Aaron <andaaron@cisco.com> Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
2023-01-09 15:37:44 -05:00
return getImageSummary(ctx, repo, tag, r.repoDB, r.cveInfo, r.log)
}
// Referrers is the resolver for the Referrers field.
func (r *queryResolver) Referrers(ctx context.Context, repo string, digest string, typeArg string) ([]*gql_generated.Referrer, error) {
store := r.storeController.GetImageStore(repo)
referrers, err := getReferrers(store, repo, digest, typeArg, r.log)
if err != nil {
r.log.Error().Err(err).Msg("unable to get referrers from default store")
return []*gql_generated.Referrer{}, err
}
return referrers, nil
}
Update to graphql 1.17.13 We encountered some problems with using the existing folder structure, but it looks like running the tooling with the latest versions works after we regenerated the project using 'gql init' and refactoring to separate the login previously in resolvers.go. - the autogenerated code is now under the gql_generated folder - the file resolvers.go now contains only the code which is not rewritten by the gqlgen framework - the file schema.resolvers.go is rewritten when gqlgen runs, and we'll only keep there the actual resolvers matching query names Changes we observed to schema.resolvers.go when gqlgen runs include reordering methods, and renaming function parameters to match the names used in schema.graphql - we now have a gqlgen.yaml config file which governs the behavior of gqlgen (can be tweaked to restructure the folder structure of the generated code in the future) Looks like the new graphql server has better validation 1 Returns 422 instead of 200 for missing query string - had to update tests 2 Correctly uncovered an error in a test for a bad `%` in query string. As as result of 2, a `masked` bug was found in the way we check if images are signed with Notary, the signatures were reasched for with the media type of the image manifest itself instead of the media type for notation. Fixed this bug, and improved error messages. This bug would have also been reproducible with main branch if the bad `%` in the test would have fixed. Updated the linter to ignore some issues with the code which is always rewritten when running: `go run github.com/99designs/gqlgen@v0.17.13 generate` Add a workflow to test gqlgen works and has no uncommitted changes Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-07-15 06:10:51 -05:00
// Query returns gql_generated.QueryResolver implementation.
func (r *Resolver) Query() gql_generated.QueryResolver { return &queryResolver{r} }
type queryResolver struct{ *Resolver }