Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add PostgreSQLQueue and DelayedPostgreSQL Pub/Sub #34

Merged
merged 40 commits into from
Oct 25, 2024
Merged

Conversation

m110
Copy link
Member

@m110 m110 commented Aug 22, 2024

  • Added PostgreSQLQueueSchema and PostgreSQLQueueOffsetsAdapter.
    • This schema is similar to the original one, except it allows filtering messages with a custom WHERE clause.
    • It doesn't support customer groups.
    • It allows deleting messages from the table after acking (optional).
  • Added NewDelayedPostgreSQLPublisher and NewDelayedPostgresSQLSubscriber
    • They work on top of the conditional schema and use the delay component implemented in Requeuer and Message Delay watermill#469
    • The idea is to receive messages with a delay or at a given time in a simple way.
  • Reworked the SchemaAdapter and OffsetsAdapter interfaces (Breaking change -> will bump the major to v4).
    • We had a few instances where some details have been missing from one of the interface methods. It makes it difficult to extend this library with new features, as each will require a major version bump, and we want to avoid it if possible.
    • To mitigate this, we're moving to a params struct passed to the methods instead of raw arguments. It adds some verbosity but allows extending the params if needed with no breaking changes to the library.
    • The methods now also return errors.
    • Migration: if you don't use a custom schema/offsets adapter, you don't need to do anything. If you do, change the methods to implement the new interfaces. Use values from the params argument and return errors.

@m110 m110 marked this pull request as ready for review October 15, 2024 16:08
pkg/sql/conditional_offsets_adapter_postgresql.go Outdated Show resolved Hide resolved
pkg/sql/conditional_offsets_adapter_postgresql.go Outdated Show resolved Hide resolved
pkg/sql/conditional_schema_adapter_postgresql.go Outdated Show resolved Hide resolved
pkg/sql/delayed_postgresql_test.go Show resolved Hide resolved
@m110 m110 changed the title Add ConditionalPostgreSQLSchema Add PostgreSQLQueue and DelayedPostgreSQL Pub/Sub Oct 23, 2024
m110 added a commit to ThreeDotsLabs/watermill that referenced this pull request Oct 25, 2024
* Added the `Requeuer` component.
    * It works as a simpler version of the Forwarder, routing messages from one topic to another (a dynamic one).
    * Can be used to move messages that failed to process back to the original topic, so they don't block other messages.
    * Can be used together with the poison middleware and the `delay` component to delay the forwarding. 
* Added the `delay` package. It contains helpers for setting delay metadata on messages.
    * **Does not do anything by itself.** A Pub/Sub needs to support it explicitly. For now, that's the delayed postgres Pub/Sub implemented in  ThreeDotsLabs/watermill-sql#34
    * Use case 1: publishing a message after a given delay or at given time (see the example).
    * Use case 2: automatically moving messages out of the poison queue to the original topic after a delay (used together with the `Requeuer` component).
* Added the `pq` CLI tool for working with poison queues.
@m110 m110 merged commit 730d8ef into master Oct 25, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants