Infrastructure/ghost
0
Fork 0
mirror of https://github.com/TryGhost/Ghost.git synced 2025-03-11 02:12:21 -05:00
Code Issues Activity

Updated max events to better control throughput

Browse source 
If we conservatively (somewhat) figure ~2500 events/min, and that the job is on a 5min cron, we should try to keep the loop cycle to ~5 mins or less in order to repeat the fetch logic, which now prioritizes opens.

Open data is the only data to update any stats, so we want to make sure if people are pressing refresh that we do our best to keep it up to date.

Meanwhile, if we don't find opens, we can move on to filling in the other event data, collecting missed events, and scheduled backfills, in that order.
This commit is contained in:
Steve Larson 2024-09-03 15:05:53 -05:00
parent 99aa2e6dbe
commit 5449d70143
1 changed files with 25 additions and 16 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

41
ghost/core/core/server/services/email-analytics/EmailAnalyticsServiceWrapper.js
View file

@ -111,30 +111,33 @@ class EmailAnalyticsServiceWrapper {
}
this.fetching = true;
// NOTE: Data shows we can process ~7500 events per minute on Pro; this can vary locally
// NOTE: Data shows we can process ~2500 events per minute on Pro for a large-ish db (150k members).
// This can vary locally, but we should be conservative with the number of events we fetch.
try {
// Prioritize opens since they are the most important (only data directly displayed to users)
await this.fetchLatestOpenedEvents({maxEvents: Infinity});
const c1 = await this.fetchLatestOpenedEvents({maxEvents: 10000});
if (c1 >= 10000) {
this._restartFetch('high opened event count');
return;
}
// Set limits on how much we fetch without checkings for opened events. During surge events (following newsletter send)
// we want to make sure we don't spend too much time collecting delivery data.
const c1 = await this.fetchLatestNonOpenedEvents({maxEvents: 20000});
if (c1 > 15000) {
this.fetching = false;
logging.info('[EmailAnalytics] Restarting fetch due to high event count');
this.startFetch();
return;
}
const c2 = await this.fetchMissing({maxEvents: 20000});
if ((c1 + c2) > 15000) {
this.fetching = false;
logging.info('[EmailAnalytics] Restarting fetch due to high event count');
this.startFetch();
const c2 = await this.fetchLatestNonOpenedEvents({maxEvents: 10000 - c1});
const c3 = await this.fetchMissing({maxEvents: 10000 - c1 - c2});
// Always restart immediately instead of waiting for the next scheduled job if we're fetching a lot of events
if ((c1 + c2 + c3) > 10000) {
this._restartFetch('high event count');
return;
}
// Only fetch scheduled if we didn't fetch a lot of normal events
await this.fetchScheduled({maxEvents: 20000 - c1 - c2});
// Only backfill if we're not currently fetching a lot of events
const c4 = await this.fetchScheduled({maxEvents: 10000});
if (c4 > 0) {
this._restartFetch('scheduled backfill');
return;
}
this.fetching = false;
} catch (e) {
@ -145,6 +148,12 @@ class EmailAnalyticsServiceWrapper {
}
this.fetching = false;
}
_restartFetch(reason) {
this.fetching = false;
logging.info(`[EmailAnalytics] Restarting fetch due to ${reason}`);
this.startFetch();
}
}
module.exports = EmailAnalyticsServiceWrapper;