Andrey Antukh
88fb5e7ab5
♻️ Update integrant to latest version
...
This upgrade also includes complete elimination of use spec
from the backend codebase, completing the long running migration
to fully use malli for validation and decoding.
2024-11-13 19:09:19 +01:00
Andrey Antukh
a9d3dfab1a
✨ Use schema instead of spec for validate worker submit options
2024-10-30 13:39:38 +01:00
Andrey Antukh
ec4260830c
♻️ Add API consistency fixes for task calling
...
Also adds a helper for calling tasks synchronously
2024-06-25 13:24:17 +02:00
Andrey Antukh
25001e5b80
📎 Add minor logging improvements on worker module
2024-04-16 16:42:44 +02:00
Andrey Antukh
3a67e51f2f
✨ Move worker runner to a separated namespace
2024-04-03 17:03:06 +02:00
Andrey Antukh
4ccea6b2cf
✨ Move worker dispatcher code to a separated ns
2024-04-03 17:03:06 +02:00
Andrey Antukh
d2998e1767
✨ Move executor service initialization to a separared ns
2024-04-03 17:03:06 +02:00
Andrey Antukh
e2ddb3e31e
✨ Move worker cron related code to a separated namespace
2024-04-03 17:03:06 +02:00
Andrey Antukh
9c9d09a816
✨ Add better logging of elapsed time for cron tasks
2024-04-03 17:03:06 +02:00
Andrey Antukh
7e803eeca8
✨ Add minor improvements for fdata logging
2024-03-22 12:58:50 +01:00
Andrey Antukh
0e380a97cc
💄 Add minor cosmetic improvement to worker ns
2024-02-05 20:11:20 +01:00
Andrey Antukh
41287d8fc5
⚡ Improve migration script performance and api usability
2024-01-04 14:40:22 +01:00
Andrey Antukh
87615ce221
💄 Fix format issues on backend module
2023-11-29 12:55:58 +01:00
Andrey Antukh
c64e14859c
✨ Simplify internal executor module
2023-11-27 14:25:12 +01:00
Andrey Antukh
a23d1908e9
✨ Improve worker logging
2023-09-04 11:35:31 +02:00
Andrey Antukh
a3495800b5
✨ Add minor logging improvements to worker namespace
2023-08-31 21:09:18 +02:00
Andrey Antukh
d2d9aeff25
📎 Reduce log level of worker submit operation
...
Start logging to as TRACE instead of DEBUG
2023-08-31 20:59:58 +02:00
Andrey Antukh
5275c35002
🐛 Prevent rollback for idle-in-transaction errors on cron tasks
2023-08-24 11:18:56 +02:00
Andrey Antukh
259b05db51
✨ Add more improvements to error reporting
2023-08-04 13:10:36 +02:00
Andrey Antukh
aafbf6bc15
♻️ Refactor cocurrency model on backend
...
Mainly the followin changes:
- Pass majority of code to the old and plain synchronous style
and start using virtual threads for the RPC (and partially some
HTTP server middlewares).
- Make some improvements on how CLIMIT is handled, simplifying code
- Improve considerably performance reducing the reflection and
unnecesary funcion calls on the whole stack-trace of an RPC call.
- Improve efficiency reducing considerably the total threads number.
2023-03-14 12:30:27 +01:00
Andrey Antukh
04b321caae
✨ Add several improvements to internal worker impl
...
Mainly for make the cron jobs do not block the scheduled executor
and offload all work to a separate threads
2023-03-14 12:30:27 +01:00
Andrey Antukh
cad1851e95
🔥 Replace own scheduled executor with the one defined in promesa lib
2023-03-14 12:30:27 +01:00
Andrey Antukh
ab3b9cba45
♻️ Refactor storage and assets related modules
...
- improve internal error handling
- add more specs and more asserts
2023-02-07 18:16:55 +01:00
Andrey Antukh
071ecca875
🐛 Fix internal executor naming issue
2023-02-02 13:38:04 +01:00
Andrey Antukh
db689d151e
♻️ Refactor profile and session handling
...
- makes the profile access more efficient (replace in-app joins to a
simple select query on profile table
- add partial support for access-tokens (still missing some RPC methods)
- move router definitions to specific modules and simplify the main http
module definitions to simple includes
- simplifiy authentication code related to access-tokens and sessions
- normalize db parameters with proper namespaced props
- more work on convert all modules initialization to use proper specs
with fully-qualified keyword config props
2023-01-18 10:51:58 +01:00
Andrey Antukh
226afe98e0
💄 Change worker logging
2022-12-15 11:33:10 +01:00
Andrey Antukh
d7459db292
🎉 Add task deduplication by label
2022-12-13 23:13:11 +01:00
Andrey Antukh
c0a4b7dc76
✨ Improve worker queue management
...
and add specific worker instance for webhooks
2022-12-13 16:17:31 +01:00
Andrey Antukh
8bad9d8340
♻️ Refactor loggers/audit, auth/oidc, and http/clent modules
2022-12-05 08:53:00 +01:00
Andrey Antukh
7f7efc5760
✨ Improve exception formating on backend
2022-12-05 08:51:54 +01:00
Andrey Antukh
c451c7bb9d
🐛 Fix regression on worker task handling
2022-11-30 13:13:11 +01:00
Andrey Antukh
0600b2abe4
♻️ Make the worker abstraction more scalable
...
Start using redis for dispatcher to worker communication
and add the ability to start multiple threads to worker
for increase the concurrency.
2022-11-28 12:21:44 +01:00
Andrey Antukh
13a092b192
♻️ Normalize internal naming on the worker module
2022-11-28 12:20:58 +01:00
luz paz
e30bea0b6f
🔧 Fix typos in source code
...
Found via `codespell -q 3 -S *.po,./frontend/yarn.lock -L childs,clen,fpr,inflight,ody,ot,ro,te,trys,ue`
2022-10-04 10:40:34 +02:00
Andrey Antukh
41134f22e9
📎 Update license header
2022-09-20 23:23:22 +02:00
Andrey Antukh
9bfdcc6277
✨ Make the task retry algorithm use better backoff values
2022-09-20 23:04:37 +02:00
Andrey Antukh
6f42f4ec45
♻️ Refactor semaphore and executors
2022-09-20 11:32:45 +02:00
Andrey Antukh
7fa609d5f4
✨ Allow disable worker
2022-08-12 08:52:36 +02:00
Andrey Antukh
f2140a1421
🐛 Fix cron scheduler locking mechanism
...
And add improved logging to the worker/cron code
2022-07-04 11:32:36 +02:00
Andrey Antukh
9e4a50fb15
♻️ Refactor backend to be more async friendly
2022-03-03 16:05:52 +01:00
Andrey Antukh
d1c834e647
🐛 Fix minor issue on executors monitor
2022-03-01 14:34:13 +01:00
Andrey Antukh
e60b8a7aef
🐛 Minor fix on worker executors monitor
2022-02-28 17:21:36 +01:00
Andrey Antukh
5d2715dd32
✨ Improve monitors monitor
2022-02-28 15:29:30 +01:00
Andrey Antukh
7cf27ac86d
♻️ Refactor general resource and concurrency model on backend
2022-02-22 13:05:41 +01:00
Andrey Antukh
7eed8c5ee5
Merge remote-tracking branch 'origin/main' into develop
2022-02-16 11:23:26 +01:00
Andrey Antukh
9b78b2a432
✨ Improve error reporting on background tasks
2022-02-15 12:26:36 +01:00
Andrey Antukh
321b2c7c23
🐛 Fix error handling on s3 delete-in-bulk operation
2022-02-15 12:26:36 +01:00
Andrey Antukh
741d2b3f3c
Merge remote-tracking branch 'origin/main' into develop
2022-02-12 17:33:28 +01:00
Andrey Antukh
827c2140b7
♻️ Refactor error reporting and logging context formatting
...
The prev approach uses clojure.pprint without any limit extensivelly
for format error context data and all this is done on the calling
thread. The clojure.pprint seems very inneficient in cpu and memory
usage on pprinting large data structures.
This is improved in the following way:
- All formatting and pretty printing is moved to logging thread,
reducing unnecesary blocking and load on jetty http threads.
- Replace the clojure.pprint with fipp.edn that looks considerably
faster than the first one.
- Add some safe limits on pretty printer for strip printing some
data when the data structure is very large, very deep or both.
2022-02-10 15:12:35 +01:00
Andrey Antukh
bc2a0432b9
✨ Allow connect to read-only databases.
2022-01-27 16:11:32 +01:00
