Is Database-as-IPC always an anti-pattern?

No universal verdict follows from the name. The criticism concerns unsuitable use, especially improvised messaging without adequate performance or failure semantics.

Does a database transaction include a worker's external action?

Usually not. Coordinating the task row with an external service requires additional design and may need duplicate-safe handling.

Can notifications eliminate all polling?

They can reduce routine checking in some designs, but reconnection and missed-signal recovery still require a way to establish authoritative work state.

AI Glossary / Data engineering and architecture / Intermediate

Database-as-IPC

Definition

Database-as-IPC means using a database as a channel for inter-process communication: one process writes information that another reads to discover work or receive a message. The label criticizes using tables as improvised message queues. Suitability depends on workload, reliability needs and implementation.

What it means

IPC is communication between running processes. A web service might insert a row describing a task, and a background worker might repeatedly query the table for pending rows. Here the database is doing more than keeping business records: it is coordinating activity between programs.

This can be convenient when the programs already share a database, but a table does not automatically provide a complete messaging system. The design must establish who claims work, what completion means and what happens when a worker stops. Conversely, database-supported queues and notifications show why every use of a database for communication should not be dismissed automatically.

How it works

A producer records a message or task with an identifier and relevant state. A consumer discovers it through polling or a supported notification mechanism, claims it under suitable concurrency controls, performs the work and records the outcome. Cleanup or retention policies manage completed records.

The difficult parts appear around failures. Two consumers must not accidentally perform conflicting claims. A consumer can complete an external action and then fail before recording completion. Retrying may then repeat the action. Transactional table updates do not automatically make external effects exactly once.

Some engines provide useful building blocks. PostgreSQL documents notifications between listening sessions and locking options useful for queue-like tables. These mechanisms have defined limits and are not interchangeable with a durable message history. Select them according to required delivery, ordering and recovery behavior rather than treating the database name as a guarantee.

Three practical examples

1. A small internal application records report-generation requests in a database table. Workers claim requests and update their status, with retries designed to avoid overwriting a completed output incorrectly.

2. A service repeatedly polls a table for short-lived signals. As the number of workers grows, the team measures empty queries and contention to decide whether the communication design remains suitable.

3. An AI processing workflow stores durable task records and uses notifications as a wake-up hint. Workers still inspect the authoritative task state after reconnecting instead of assuming a missed signal means no work exists.

Think of it this way

Think of coworkers leaving job cards on a shared noticeboard. It works only if they know who takes a card, how completion is marked and what happens when someone leaves midway. The analogy is limited: database transactions can coordinate claims precisely, while external side effects and disconnected listeners introduce failure cases that a simple noticeboard picture does not solve.

Worked example

Suppose 100 workers each poll a task table every 0.5 seconds. Ignoring processing delays, this produces 200 polling queries per second. Over one hour, that is 720,000 queries, even if only 60 tasks arrive during the hour.

The calculation does not prove that the database cannot handle the load; query cost, indexing and existing capacity matter. It does reveal a workload worth measuring, especially if most polls find nothing. Increasing the interval reduces polling but can increase discovery delay.

Now one worker claims task T9, sends an external notification and crashes before marking T9 complete. Another worker may retry T9 and send it again. A durable row has preserved the task but has not solved duplicate effects. The design needs an explicit recovery and deduplication approach appropriate to that external operation, or it must accept and handle repeated execution.

Illustrative case study

Fictional case: QueueMeadow uses a table to coordinate thumbnail generation. The initial implementation selects the first pending row and only later marks it claimed. Under concurrency, two workers select the same row and both begin processing.

The team changes claiming to use an appropriate atomic database operation and tests simultaneous consumers. It separately tests a worker dying after output creation, choosing an output identity that lets a retry recognize or safely replace the same result. Monitoring tracks waiting tasks, failed attempts and processing age.

Discovery, ownership and completion need an explicit protocol, even with database coordination tools.

Common misconceptions and mistakes

Do not assume that storing messages durably guarantees exactly-once effects. Persistence, delivery and idempotency solve different problems. Also avoid treating a notification as a substitute for checking durable state after disconnects.

Do not condemn every database-backed queue on the basis of the anti-pattern label. A modest workload with clear semantics may justify using an existing database. Equally, do not ignore missing capabilities such as delayed delivery, dead-letter handling or operational isolation merely because adding a table is easy. Compare requirements with the actual design.

Business use

Businesses should evaluate this pattern when background work shares infrastructure with important transactional services. Consider polling cost, lock contention, backlog behavior, recovery, monitoring and the impact of failures on both workloads.

A dedicated messaging system may offer useful capabilities, but it also introduces another service to operate. A database-backed design may be simpler in one environment and inappropriate in another. For AI workloads, long-running or externally acting tasks make clear ownership, retry policy and outcome tracking particularly important.

Check your understanding

A worker completes an external action but crashes before updating its task row. Why can retrying cause a duplicate?

Answer: The durable row still appears unfinished even though the external effect happened. The system needs an explicit way to recognize, prevent or tolerate the repeated effect.

Related terms

Concurrency control
Inter-process communication
message queue
polling
notification
task queue
idempotency
transaction
retry
dead-letter queue
worker

Learn next

Learn message-delivery semantics and idempotency next. Draw a task's states and test a crash before claim, after claim and after its external effect. Then measure an idle workload's polling cost. Use those observations to compare a database-backed protocol with a dedicated queue's documented behavior.

Sources

PostgreSQL, NOTIFY, describes database-supported inter-process signaling and transactional timing: https://www.postgresql.org/docs/current/sql-notify.html

PostgreSQL, SELECT locking clause, documents SKIP LOCKED's queue-like use and limitations: https://www.postgresql.org/docs/current/sql-select.html

Junade Ali, Mastering PHP Design Patterns, has a Database as IPC section located through the publisher index; full text retrieval failed: https://subscription.packtpub.com/book/programming/9781785887130/2/ch02lvl1sec13/database-as-ipc

The PostgreSQL documentation was read. The anti-pattern label is treated contextually, without attributing unread arguments to the book. Calculations and the fictional case are original illustrations.

Previous: Database Next: Database Client