0
Fork 0
mirror of https://github.com/penpot/penpot.git synced 2025-03-10 23:01:39 -05:00
Commit graph

48 commits

Author SHA1 Message Date
Andrey Antukh
addb392ecc Add safety mechanism for direct object deletion
The main objective is prevent deletion of objects that can leave
unreachable orphan objects which we are unable to correctly track.

Additionally, this commit includes:

1. Properly implement safe cascade deletion of all participating
   tables on soft deletion in the objects-gc task;

2. Make the file thumbnail related tables also participate in the
   touch/refcount mechanism applyign to the same safety checks;

3. Add helper for db query lazy iteration using PostgreSQL support
   for server side cursors;

4. Fix efficiency issues on gc related task using server side
   cursors instead of custom chunked iteration for processing data.

   The problem resided when a large chunk of rows that has identical
   value on the deleted_at column and the chunk size is small (the
   default); when the custom chunked iteration only reads a first N
   items and skip the rest of the set to the next run.

   This has caused many objects to remain pending to be eliminated,
   taking up space for longer than expected. The server side cursor
   based iteration does not has this problem and iterates correctly
   over all objects.

5. Fix refcount issues on font variant deletion RPC methods
2024-01-03 10:56:57 +01:00
Andrey Antukh
87615ce221 💄 Fix format issues on backend module 2023-11-29 12:55:58 +01:00
Alejandro
7404933e99
Merge pull request #3854 from penpot/niwinz-develop-yetti-update
 Update yetti and simplify internal worker module
2023-11-29 12:01:33 +01:00
Andrey Antukh
da7f88c7ca 🔥 Remove executor internal dependency on storage module 2023-11-27 14:25:12 +01:00
Andrey Antukh
d82ebdc034 Add deduplication for file object thumbnails 2023-11-24 11:06:47 +01:00
Andrey Antukh
a0973b9ddf Improve storage-gc-deleted task reliability 2023-08-31 14:36:31 +02:00
Andrey Antukh
dfdc9c9fa5 ♻️ Refactor storage internal concurrency model 2023-03-14 12:30:27 +01:00
Andrey Antukh
ab3b9cba45 ♻️ Refactor storage and assets related modules
- improve internal error handling
- add more specs and more asserts
2023-02-07 18:16:55 +01:00
Andrey Antukh
db689d151e ♻️ Refactor profile and session handling
- makes the profile access more efficient (replace in-app joins to a
  simple select query on profile table
- add partial support for access-tokens (still missing some RPC methods)
- move router definitions to specific modules and simplify the main http
  module definitions to simple includes
- simplifiy authentication code related to access-tokens and sessions
- normalize db parameters with proper namespaced props
- more work on convert all modules initialization to use proper specs
  with fully-qualified keyword config props
2023-01-18 10:51:58 +01:00
luz paz
e30bea0b6f 🔧 Fix typos in source code
Found via `codespell -q 3 -S *.po,./frontend/yarn.lock -L childs,clen,fpr,inflight,ody,ot,ro,te,trys,ue`
2022-10-04 10:40:34 +02:00
Andrey Antukh
886ab0e152 Improve iteration and add concat-all and fully lazy mapcat helper 2022-09-29 10:32:18 +02:00
Andrey Antukh
41134f22e9 📎 Update license header 2022-09-20 23:23:22 +02:00
Andrey Antukh
172a39c2e2 Improve storage sharding replacing uuid/next with uuid/random 2022-09-12 15:31:14 +02:00
Andrey Antukh
ec3651d85b 🎉 Add optional rate limit support for RPC calls 2022-08-31 12:55:48 +02:00
Andrey Antukh
d533e37ae0 Improve logging on gc-deleted storage task 2022-08-11 17:31:32 +02:00
Andrey Antukh
6ee6e5e23e Improve logging on gc-touched storage task 2022-08-11 17:31:32 +02:00
Andrey Antukh
ebcb385593 ♻️ Minor refactor on storages
Fix many issues on FS & S3 backend; removes the unused and broken
DB backend. Normalize operations on bytes and byte streams on a
separated namespace: app.util.bytes
2022-06-22 11:37:45 +02:00
Andrey Antukh
3a55f07f45 🐛 Remove duplicate work on storing already existing files in storage 2022-06-20 14:17:31 +02:00
Andrey Antukh
1943877b21 Simplify d/group-by impl 2022-03-28 17:24:52 +02:00
Andrey Antukh
f7bb08382c Fix issues from previous refactor peer review 2022-03-10 13:32:06 +01:00
Andrey Antukh
1b444a42f2 ♻️ Refactor http server layer
Make it fully asynchronous.
2022-03-09 17:18:06 +01:00
Andrey Antukh
9e4a50fb15 ♻️ Refactor backend to be more async friendly 2022-03-03 16:05:52 +01:00
Andrey Antukh
84a36624a6 🎉 Add specific namespace for data macros
And additionally add optimized macros for get-in,
select-keys and str.
2022-02-25 11:07:40 +01:00
Andrey Antukh
aeed535f1b Minor improvement on reference handling on touched-gc task 2022-02-23 09:13:48 +01:00
Andrey Antukh
f1db0fea03 ♻️ Refactor storage transaction management 2022-02-16 13:58:15 +01:00
Andrey Antukh
116fafd0e1 📎 Minor log param naming change 2022-02-15 13:25:46 +01:00
Andrey Antukh
3dd94bd362 🐛 Log correct deleted number value on recheck task 2022-02-10 15:12:35 +01:00
Josh Soref
cd2d3d5fa3 🐛 Fix typos in backend 2021-11-15 09:53:10 -05:00
Andrey Antukh
b64d5ef357 🎉 Add unified logging api. 2021-09-30 11:48:14 +02:00
Andrey Antukh
efa5120fac Fix inconsistencies on storage backend usage. 2021-07-29 12:59:24 +02:00
Andrey Antukh
0c97a44a2a 🎉 Add file offloading to external storage mechanism. 2021-06-14 15:41:27 +02:00
Andrey Antukh
1a81631886 📎 Decrease default bulk buffers on storage tasks. 2021-06-04 09:41:42 +02:00
Andrey Antukh
f545e41d10 📎 Fix license header. 2021-04-12 16:49:43 +02:00
Andrey Antukh
e12a6e65a6 ♻️ Refactor logging. 2021-04-09 15:28:18 +02:00
Andrey Antukh
784a4f8ecd Add some type hints and remove legacy code. 2021-02-18 09:35:37 +01:00
Andrey Antukh
b80295a21c Fix all linter issues on backend code. 2021-02-01 09:48:28 +01:00
Andrey Antukh
26948fb68b Make storage tasks more testable and traceable. 2021-02-01 09:48:28 +01:00
Andrey Antukh
233cd8c3d6 Add expiration checking on storage functions. 2021-02-01 09:48:28 +01:00
Andrey Antukh
8253ef90d0 Improve handling of temporal files.
Store temporal files outside of main fs backend.
2021-02-01 09:48:28 +01:00
Andrey Antukh
e54b443247 🎉 Add refcount-like functionality to storages.
This allows reuse of storage objects among different files.
2021-02-01 09:48:28 +01:00
Andrey Antukh
a69a35a0b6 Improve storage recheck task and add more specs. 2021-01-26 11:13:21 +01:00
Andrey Antukh
d68286821b Add the notion of temporal files on the storage. 2021-01-26 11:13:21 +01:00
alonso.torres
66fe0048a5 Adds system to load initial project data 2021-01-25 11:27:29 +01:00
Andrey Antukh
ef25f8a721 Avoid reflection on s3 storage backend. 2021-01-19 15:37:26 +01:00
Andrey Antukh
d494e44df3 🎉 Add builtin copy fast path operation for storage. 2021-01-19 15:01:33 +01:00
Andrey Antukh
0c30d53d95 🐛 Fix wrong conn handling on some function on storage. 2021-01-08 12:37:32 +01:00
Andrey Antukh
ab944fb9ae ♻️ Integrate new storage subsystem. 2021-01-07 11:29:57 +01:00
Andrey Antukh
760eb926bf 🎉 Add plugable storages abstraction layer (with support for fs, s3 and db). 2021-01-07 11:29:57 +01:00