There could be a condition where jobs were added to a queue that is not longer being watched,
yet the jobs in those queues would continue appearing in the pending job list because
the query to get pending jobs did not filter by queues being watched.
Instead of passing a single connection into the pending job processor,
use the connection pool to pull a new connection on every loop, releasing
it after every use.
The previous release attempted to fix an issue where, when LISTEN connections
drop an reconnect, some job announcements might be missed.
This fix reduces the number of connections required to do so.
When jobs are inserted into the 'neoq_jobs', the `announce_job` trigger announces the job to
listeners of the job's queue. However, if no workers are available to receive the announcement,
or the announcement is somehow lost, it meant those jobs were marooned on the queue.
This fix adds a periodic check for post-due pending jobs to ensure that a neoq restart is
not necessary to catch marooned, overdue jobs.
This makes it so that channel names for NOTIFY/LISTEN/UNLISTEN are
consistent. That is that they are double quoted in all instances except
for when they are referenced by `pg_notify`. This way the channel names
are no longer case-insensitive.
This happens because of how PostgreSQL parses the SQL statements for
NOTIFY/LISTEN/UNLISTEN and treats the arguments as an identifier. Where
as it treats the arguments to `pg_notify` as only a string. This causes
arguments that are not double quoted to be adjusted to lower case when
passed to the SQL statement.
By making all usages consistent, we no longer have to worry about
channel names being odd.
When the queue name is uppercase, the listener will never receive the
notification when the job is enqueued. This is done specifically with
the enqueuer and consumer being separate neoq instances (server A kicks
off a job, server B is listening to perform the work).
I want to be able to know what job a log message is in regards to, so
this adds the queue to all of the log messages where its available. It
also makes some of the `error` logs more consistent with each other.
Since these values are being written in a separate go routine, it is
possible for a datarace to happen since we are reading them
unatomically. This just makes it so that both the reads ands writes are
done atomically.
Resolves#76
Previously, new, non-future jobs were announced by executing `NOTIFY` in
Go code. Triggers are much better suited for this, and reduces neoq
complexity by allowing PG to perform notification work for most jobs.
"Future" jobs continue to use the `announceJob` method on a timer.
This test is to provide a minimal proof that jobs can be consumed by
multiple workers and in a way can only be consumed once. If the
execCount does not match the expected count then this test will fail
because either too many jobs were executed (like one executing twice) or
a job was dropped when it should not have been.
This reverts commit bc8df98894.
This commit was totally wrong about the motivations for holding transactions throught the duration of jobs.
Ths is an embarrassing mistake because it is the crux of the entire Postgres backend. Transactions must be held while jobs are being processed because `FOR UPDATE` requires the transaction that issued the `SELECT * FOR UPDATE ...` query to remain open to hold the lock (https://www.postgresql.org/docs/current/explicit-locking.html#LOCKING-ROWS)
The reverted commit allowed jobs to be picked up by multiple workers.
`cancelFuncs` in the pg backend was intended for gracefully shutting down the backend with Shutdown(). Adding a
new cancel function every time a job is added to a queue isn't of any use, because there is nothing to "cancel" by
cancelling the context unless a job is in progress.
This was effectively a memory leak.
This goal of this change is to simplify the API. Handlers and queues
have a 1:1 mapping, so functions that receive both a `queue` and a
`handler` take two parameters when they could take only one.
When neoq is embedded in apps already use golang-migrate, they would
both attempt to use the `schema_migrations` table for migrations.
This commit moves neoq to its own `neoq_schema_migrations` table so as
not to clash with other golang-migrate users.
Previously pg backend would hold connections/transactions during the
execution of job handlers. So, e.g. if a job ran for 30 seconds, a
transaction and its connection would be held for the entirety of those
30 seconds.
The reason it was originally implemented this way is because originally
the vision was to pass every job a `tx` that could be used throughout
the job. If the job failed, its `tx` would be rolled back in neoq
so that user would not have to handle rollbacks and connection handling
themselves, were they to perform database operations in their jobs.
But ultimately, that is a lot of hand-holding at the cost of a lot of
resources, for a use case that is both unlikely and does not work as
soon as the user's application and neoq tables are in different
databases.
This commit reverses that poor decision and opts to improve performance
over giving users an ergonomic transaction handling API.
