Infrastructure/zot
0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2024-12-23 22:27:35 -05:00
Code Issues Activity
zot/pkg/storage/cache_test.go

57 lines
1.3 KiB
Go
Raw Normal View History

dedupe: use hard links to dedupe blobs As the number of repos and layers increases, the greater the probability that layers are duplicated. We dedupe using hard links when content is the same. This is intended to be purely a storage layer optimization. Access control when available is orthogonal this optimization. Add a durable cache to help speed up layer lookups. Update README. Add more unit tests.
2020-02-17 16:57:15 -05:00
package storage_test
import (
dedupe: record relative path for cache entries In a production use case we found that the actual rootdir can be moved. Currently, cache entries for dedupe record the full blob path which doesn't work in the move use case. Only for dedupe cache entries, record relative blob paths.
2020-05-27 16:27:35 -05:00
"path"
dedupe: use hard links to dedupe blobs As the number of repos and layers increases, the greater the probability that layers are duplicated. We dedupe using hard links when content is the same. This is intended to be purely a storage layer optimization. Access control when available is orthogonal this optimization. Add a durable cache to help speed up layer lookups. Update README. Add more unit tests.
2020-02-17 16:57:15 -05:00
"testing"
. "github.com/smartystreets/goconvey/convey"
move references to zotregistry.io and project-zot Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-03 22:50:58 -05:00
"zotregistry.io/zot/errors"
"zotregistry.io/zot/pkg/log"
"zotregistry.io/zot/pkg/storage"
dedupe: use hard links to dedupe blobs As the number of repos and layers increases, the greater the probability that layers are duplicated. We dedupe using hard links when content is the same. This is intended to be purely a storage layer optimization. Access control when available is orthogonal this optimization. Add a durable cache to help speed up layer lookups. Update README. Add more unit tests.
2020-02-17 16:57:15 -05:00
)
func TestCache(t *testing.T) {
Convey("Make a new cache", t, func() {
test: use `T.TempDir` to create temporary test directory The directory created by `T.TempDir` is automatically removed when the test and all its subtests complete. Reference: https://pkg.go.dev/testing#T.TempDir Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-03-07 03:55:12 -05:00
dir := t.TempDir()
dedupe: use hard links to dedupe blobs As the number of repos and layers increases, the greater the probability that layers are duplicated. We dedupe using hard links when content is the same. This is intended to be purely a storage layer optimization. Access control when available is orthogonal this optimization. Add a durable cache to help speed up layer lookups. Update README. Add more unit tests.
2020-02-17 16:57:15 -05:00
log := log.NewLogger("debug", "")
So(log, ShouldNotBeNil)
s3: added logic for deduping blobs Because s3 doesn't support hard links we store duplicated blobs as empty files. When the original blob is deleted its content is moved to the the next duplicated blob and so on. Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-04-12 05:01:04 -05:00
So(storage.NewCache("/deadBEEF", "cache_test", true, log), ShouldBeNil)
dedupe: use hard links to dedupe blobs As the number of repos and layers increases, the greater the probability that layers are duplicated. We dedupe using hard links when content is the same. This is intended to be purely a storage layer optimization. Access control when available is orthogonal this optimization. Add a durable cache to help speed up layer lookups. Update README. Add more unit tests.
2020-02-17 16:57:15 -05:00
s3: added logic for deduping blobs Because s3 doesn't support hard links we store duplicated blobs as empty files. When the original blob is deleted its content is moved to the the next duplicated blob and so on. Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-04-12 05:01:04 -05:00
cache := storage.NewCache(dir, "cache_test", true, log)
lint: upgrade golangci-lint Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-13 14:23:31 -05:00
So(cache, ShouldNotBeNil)
dedupe: use hard links to dedupe blobs As the number of repos and layers increases, the greater the probability that layers are duplicated. We dedupe using hard links when content is the same. This is intended to be purely a storage layer optimization. Access control when available is orthogonal this optimization. Add a durable cache to help speed up layer lookups. Update README. Add more unit tests.
2020-02-17 16:57:15 -05:00
lint: upgrade golangci-lint Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-13 14:23:31 -05:00
val, err := cache.GetBlob("key")
dedupe: use hard links to dedupe blobs As the number of repos and layers increases, the greater the probability that layers are duplicated. We dedupe using hard links when content is the same. This is intended to be purely a storage layer optimization. Access control when available is orthogonal this optimization. Add a durable cache to help speed up layer lookups. Update README. Add more unit tests.
2020-02-17 16:57:15 -05:00
So(err, ShouldEqual, errors.ErrCacheMiss)
lint: upgrade golangci-lint Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-13 14:23:31 -05:00
So(val, ShouldBeEmpty)
dedupe: use hard links to dedupe blobs As the number of repos and layers increases, the greater the probability that layers are duplicated. We dedupe using hard links when content is the same. This is intended to be purely a storage layer optimization. Access control when available is orthogonal this optimization. Add a durable cache to help speed up layer lookups. Update README. Add more unit tests.
2020-02-17 16:57:15 -05:00
lint: upgrade golangci-lint Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-13 14:23:31 -05:00
exists := cache.HasBlob("key", "value")
So(exists, ShouldBeFalse)
dedupe: use hard links to dedupe blobs As the number of repos and layers increases, the greater the probability that layers are duplicated. We dedupe using hard links when content is the same. This is intended to be purely a storage layer optimization. Access control when available is orthogonal this optimization. Add a durable cache to help speed up layer lookups. Update README. Add more unit tests.
2020-02-17 16:57:15 -05:00
lint: upgrade golangci-lint Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-13 14:23:31 -05:00
err = cache.PutBlob("key", path.Join(dir, "value"))
dedupe: use hard links to dedupe blobs As the number of repos and layers increases, the greater the probability that layers are duplicated. We dedupe using hard links when content is the same. This is intended to be purely a storage layer optimization. Access control when available is orthogonal this optimization. Add a durable cache to help speed up layer lookups. Update README. Add more unit tests.
2020-02-17 16:57:15 -05:00
So(err, ShouldBeNil)
sync: Added support for syncing notary/cosign signatures, closes #261 Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-12-07 13:26:26 -05:00
err = cache.PutBlob("key", "value")
So(err, ShouldNotBeNil)
lint: upgrade golangci-lint Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-13 14:23:31 -05:00
exists = cache.HasBlob("key", "value")
So(exists, ShouldBeTrue)
dedupe: use hard links to dedupe blobs As the number of repos and layers increases, the greater the probability that layers are duplicated. We dedupe using hard links when content is the same. This is intended to be purely a storage layer optimization. Access control when available is orthogonal this optimization. Add a durable cache to help speed up layer lookups. Update README. Add more unit tests.
2020-02-17 16:57:15 -05:00
lint: upgrade golangci-lint Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-13 14:23:31 -05:00
val, err = cache.GetBlob("key")
dedupe: use hard links to dedupe blobs As the number of repos and layers increases, the greater the probability that layers are duplicated. We dedupe using hard links when content is the same. This is intended to be purely a storage layer optimization. Access control when available is orthogonal this optimization. Add a durable cache to help speed up layer lookups. Update README. Add more unit tests.
2020-02-17 16:57:15 -05:00
So(err, ShouldBeNil)
lint: upgrade golangci-lint Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-13 14:23:31 -05:00
So(val, ShouldNotBeEmpty)
dedupe: use hard links to dedupe blobs As the number of repos and layers increases, the greater the probability that layers are duplicated. We dedupe using hard links when content is the same. This is intended to be purely a storage layer optimization. Access control when available is orthogonal this optimization. Add a durable cache to help speed up layer lookups. Update README. Add more unit tests.
2020-02-17 16:57:15 -05:00
lint: upgrade golangci-lint Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-13 14:23:31 -05:00
err = cache.DeleteBlob("bogusKey", "bogusValue")
dedupe: use hard links to dedupe blobs As the number of repos and layers increases, the greater the probability that layers are duplicated. We dedupe using hard links when content is the same. This is intended to be purely a storage layer optimization. Access control when available is orthogonal this optimization. Add a durable cache to help speed up layer lookups. Update README. Add more unit tests.
2020-02-17 16:57:15 -05:00
So(err, ShouldEqual, errors.ErrCacheMiss)
lint: upgrade golangci-lint Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-13 14:23:31 -05:00
err = cache.DeleteBlob("key", "bogusValue")
dedupe: use hard links to dedupe blobs As the number of repos and layers increases, the greater the probability that layers are duplicated. We dedupe using hard links when content is the same. This is intended to be purely a storage layer optimization. Access control when available is orthogonal this optimization. Add a durable cache to help speed up layer lookups. Update README. Add more unit tests.
2020-02-17 16:57:15 -05:00
So(err, ShouldBeNil)
dedupe: stat blob path before creating link
2021-07-20 16:04:10 -05:00
// try to insert empty path
lint: upgrade golangci-lint Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-13 14:23:31 -05:00
err = cache.PutBlob("key", "")
dedupe: stat blob path before creating link
2021-07-20 16:04:10 -05:00
So(err, ShouldNotBeNil)
So(err, ShouldEqual, errors.ErrEmptyValue)
dedupe: use hard links to dedupe blobs As the number of repos and layers increases, the greater the probability that layers are duplicated. We dedupe using hard links when content is the same. This is intended to be purely a storage layer optimization. Access control when available is orthogonal this optimization. Add a durable cache to help speed up layer lookups. Update README. Add more unit tests.
2020-02-17 16:57:15 -05:00
})
}
Reference in a new issue Copy permalink