Commit graph

28 commits

Author SHA1 Message Date
Adriano Caloiaro
2ce00761c7 maint: clean up testutils/TestLogger 2023-09-07 12:36:02 +02:00
Adriano Caloiaro
bc8df98894 feat: Hold connections and transactions for less time
Previously pg backend would hold connections/transactions during the
execution of job handlers. So, e.g. if a job ran for 30 seconds, a
transaction and its connection would be held for the entirety of those
30 seconds.

The reason it was originally implemented this way is because originally
the vision was to pass every job a `tx` that could be used throughout
the job. If the job failed, its `tx` would be rolled back in neoq
so that user would not have to handle rollbacks and connection handling
themselves, were they to perform database operations in their jobs.
But ultimately, that is a lot of hand-holding at the cost of a lot of
resources, for a use case that is both unlikely and does not work as
soon as the user's application and neoq tables are in different
databases.

This commit reverses that poor decision and opts to improve performance
over giving users an ergonomic transaction handling API.
2023-09-02 14:59:32 +02:00
Adriano Caloiaro
6ea1bb692f feat: Improve postgres duplicate job detection performance
Previously, neoq would query to see if new jobs' (fingerprint, status)
tuple were unique. This change adds a unique index on (fingerprint,
status) and simply allows unique key constraint violations to be
returned from `Enqueue` instead. This prevents one additional query
for every new queue item.
2023-09-02 14:59:32 +02:00
Adriano Caloiaro
342aad6fe2 fix: Incorrectly passing pgx connections params to pq
Packages `pq` and `pgx` allow different connection string parameters.
Since `pgx` is neoq's primary SQL library, but migrations use `pq`,
we had to ensure that pgx connection parameters do not get passed to
the pq library when running migrations.

This issue was introduced in version `0.20.0`.
2023-08-29 11:23:34 +02:00
Adriano Caloiaro
72594867e5 Support job deadlines
Fixes #52
2023-08-28 20:01:21 +02:00
Adriano Caloiaro
a4c54e0f49 Support postgres backend schema migrations 2023-08-28 08:59:40 +02:00
Adriano Caloiaro
631c508813 Fix memory backend support for multiple queues
Fixes #56
2023-08-20 10:16:45 -06:00
57c75deefb Set database name on connection string 2023-07-29 07:52:51 -06:00
c237f31b4e Fix slog API change issue 2023-07-17 07:54:22 -06:00
49317f4840 Make [rR]edisBackend public 2023-07-12 13:34:51 -06:00
a70e9eb188
Fix optional pool params on connection string 2023-05-02 21:24:12 -06:00
d68db7b442
Add log level to redis and memory backends (#50) 2023-05-02 20:46:04 -06:00
43a0a133f2
Add log level support (#49) 2023-05-02 20:17:53 -06:00
106ab0f18a
Fix retries in the event of handler panic (#48) 2023-04-29 12:47:52 -06:00
71dd54b180
Fix postgres Handler error handling (#47) 2023-04-26 10:54:02 -06:00
1c92db565b
Update slog (#46) 2023-04-19 17:05:43 -06:00
aaed37eeb5
Add redis support (#44) 2023-04-14 19:11:22 -06:00
219ea84d76
Move config.WithConnectionString -> postgres.WithConnectionString (#43) 2023-04-01 18:03:55 -07:00
81845a2303
Move handlers.FromJobContext() -> jobs.FromContext() (#41) 2023-04-01 17:54:05 -07:00
d8c637852c
Remove add multiple handlers per queue (#42) 2023-04-01 17:36:07 -07:00
d47845d573
Make JobWithContext private 2023-03-21 09:54:08 -07:00
900893d14c
Mem: Add multiple handlers per queue 2023-03-18 16:13:23 -07:00
4639e53980
Update docs regarding lost connections while updating jobs (#39) 2023-03-18 12:22:27 -07:00
eb430e9444
Acquire a new connection for every queue listener (#38) 2023-03-18 09:52:48 -07:00
07a336a397
Make handler context vars backend agnostic (#37) 2023-03-17 08:53:16 -07:00
784ea57578
increase test coverage over memory backend 2023-03-09 11:03:37 -07:00
621b105213
Refactor ReplaceNonAlphanum() to use regex.ReplaceAllString() 2023-03-07 09:24:46 -07:00
5b6620927e
Move backends to separate packages 2023-03-01 10:34:27 -07:00