Postgres is not simply taking a row from one machine and placing it on another. The publisher first writes a physical change to WAL. A separate process later decodes that storage-level change back into a logical row, reconstructs its transaction, filters it, serialises it, and sends it over the replication protocol. The subscriber then finds the corresponding local relation, applies a new write, maintains its indexes, checks constraints and produces another set of WAL records.

Every replicated change is therefore paid for several times.

This is easy to miss because logical replication is normally asynchronous. The application can continue committing on the publisher whilst the expensive part quietly moves into a walsender, an apply worker, or an ever-growing replication lag graph.

Let’s dig into where that CPU goes, and then measure how much difference a few seemingly harmless schema choices can make.

Glossary

• WAL - the write-ahead log; the physical record of every change made to heap and index pages
• logical decoding - the process of reading physical WAL records and reconstructing them as row-level changes grouped into transactions
• walsender - the publisher-side process which streams decoded changes to a subscriber
• pgoutput - the built-in output plugin which serialises decoded changes into logical replication protocol messages
• ReorderBuffer - publisher-side structure holding decoded changes until their transaction commits, so they can be emitted in commit order
• apply worker - the subscriber-side process which maps incoming messages to local tables and executes the writes
• replica identity - the columns a publisher sends so the subscriber can locate the row to UPDATE or DELETE
• replication slot - publisher-side bookmark which retains WAL until a consumer confirms it has been received
• HOT update - a heap-only tuple update; a new row version written to the same page without touching any index

A Row Becomes Physical Before It Becomes Logical Again

The strange part of logical replication is that the source database does not start with a logical stream.

Postgres executes the original INSERT, UPDATE or DELETE as normal. It modifies heap and index pages and writes WAL describing those storage-level changes. Logical decoding then has to reverse some of that process: it reads the WAL and turns it back into a coherent stream of transactions and tuples.

The full path looks roughly like this:

The built-in implementation uses a walsender on the publisher and an apply worker on the subscriber. The standard pgoutput plugin transforms decoded changes into logical replication protocol messages, and the apply worker maps those messages to local tables in transactional order. This is described in the PostgreSQL logical replication architecture.

Each arrow represents work which can become the bottleneck independently.

Where the Publisher Spends CPU

Writing enough WAL to decode later

Logical replication requires wal_level = logical, which contains everything in replica WAL plus the information needed to extract logical change sets, and Postgres explicitly warns that this can increase your WAL volume, particularly for UPDATE and DELETE operations on tables using REPLICA IDENTITY FULL.

This cost can exist even before a consumer starts reading the stream, because the publisher has to preserve enough information for a future decoder to understand what happened. See the wal_level documentation for the exact distinction.

Decoding and rebuilding transactions

WAL describes changes at the storage layer, so logical decoding has to turn those records back into an application-specific stream of tuples and transactions, which means identifying the relation, reconstructing tuple data, preserving transaction order and keeping changes around until it knows whether the transaction commits.

Postgres uses a ReorderBuffer for this. Large transactions can exceed logical_decoding_work_mem and spill decoded changes to disk, exchanging a memory problem for CPU and I/O. The spill is visible in pg_stat_replication_slots through spill_txns, spill_count and spill_bytes.

SELECT
    slot_name,
    spill_txns,
    pg_size_pretty(spill_bytes) AS spilled,
    stream_txns,
    pg_size_pretty(stream_bytes) AS streamed
FROM pg_stat_replication_slots;

This is why one transaction containing a million changes behaves differently from a million one-row transactions, even when both modify the same number of rows.

Filtering and serialising rows

pgoutput still has to turn decoded tuples into protocol messages, and by default a subscription requests those values in text format, which invokes type output on the publisher and type input on the subscriber. binary = true can avoid some of that conversion, but it is less portable between PostgreSQL versions, machine architectures and differing source/target types.

Binary mode can be faster, but it is not a free switch. PostgreSQL documents the trade-off directly under CREATE SUBSCRIPTION.

Publication filtering also executes here, so whilst column lists can reduce the data sent, row filters will consume publisher CPU to do it. UPDATE is the awkward case: Postgres evaluates the filter against both the old and new row and may transform the change into an INSERT or DELETE when the row moves across the filter boundary. The behaviour is covered in the row-filter documentation.

Where the Subscriber Spends CPU

The subscriber is not replaying the publisher’s page changes, it is applying new logical writes against its own physical layout, which is where most of the surprise comes from.

For every incoming change it may need to:

• parse the replication message and map remote columns to a local relation
• locate the target row for an UPDATE or DELETE
• create a new MVCC tuple version
• maintain every affected local index
• enforce applicable local constraints such as NOT NULL, CHECK, unique and exclusion constraints
• write heap and index changes into local WAL

Normal triggers and rules do not fire because the apply worker runs with session_replication_role = replica. This includes the triggers used to enforce foreign keys. Triggers can be explicitly enabled for replica execution, so they should not be assumed to be free in every schema. Constraints enforced by the executor and indexes still matter: an incoming unique or exclusion conflict can stop replication. PostgreSQL’s conflict documentation describes the failure modes.

Index maintenance is often the dominant subscriber cost, because a subscriber with one primary key and a subscriber with six secondary indexes receive exactly the same logical row whilst doing nowhere near the same amount of work. The second schema produces more page changes, consumes more CPU and writes more local WAL.

This is also why HOT eligibility matters, given that an update can only remain HOT when no indexed value needs changing and there is room on the heap page. Adding an index to a frequently changed subscriber column can turn one heap update into a heap update plus several B-tree writes.

Subscriptions default their apply workers to synchronous_commit = off. This avoids waiting for every subscriber flush and lets Postgres resend changes after a subscriber crash, but it does not eliminate local WAL generation.

Logical Versus Physical Replication

Physical replication gets summarised as “copy bytes” and logical replication as “execute queries”, which is not quite right, because a physical standby still spends CPU replaying WAL and modifying heap and index pages. It is not free.

The difference is that physical replication sends the existing physical WAL records and replays them against an identical physical structure. Logical replication decodes those records into row-level meaning and then generates new local changes on a potentially different schema.

Logical replication buys per-table selection, row and column filtering, cross-version replication and independently designed subscribers. The additional CPU is the price of that flexibility.

Measuring the Cost

I ran two isolated PostgreSQL 18.1 clusters on the same 10-core Apple M1 Pro with 32 GB of memory. Keeping both clusters on one machine is not representative of a production topology, but it makes the CPU competition visible and keeps the test reproducible.

The harness is run-benchmark.sh. It builds two disposable clusters, runs every scenario and tears them down again, so the numbers below can be reproduced directly. The full per-run figures are on the benchmark results page.

The main table was:

CREATE TABLE events (
    id          bigint PRIMARY KEY,
    account_id  integer NOT NULL,
    status      smallint NOT NULL,
    payload     text NOT NULL,
    updated_at  timestamptz NOT NULL
);

Eight clients committed 20,000 transactions in total, each of them inserting 25 rows with a 128-character payload, which comes to 500,000 rows per scenario.

To focus the test on CPU rather than storage durability, both clusters used fsync = off, synchronous_commit = off, full_page_writes = off and autovacuum = off. Every publisher scenario used wal_level = logical, including the no-consumer baseline, and there was no initial table copy.

I sampled CPU time for the PostgreSQL server processes, excluding pgbench and the pg_recvlogical client. Each logical scenario waited for an explicit marker row to become queryable on the subscriber; “caught up” therefore means applied, not merely received. The values below are medians from three clean runs; the raw per-run figures are on the results page.

Each column is calculated independently. Total server CPU is the median of the publisher-plus-subscriber total for each run, so it does not always equal the sum of the two independently calculated CPU medians shown beside it.

Decoding the stream without applying it increased publisher-side server CPU from 4.71 to 7.92 seconds. Applying the text stream end-to-end more than doubled total server CPU compared with the no-consumer case.

Binary transfer helped, but not dramatically for these data types. It reduced subscriber CPU by roughly 10%, whilst total measured server CPU fell by about 1%. Measure it against your real column types and version topology before turning it on.

The four additional indexes increased subscriber CPU by 59% and extended the drain from 2.49 to 3.60 seconds. The lower publisher CPU in that row should not be interpreted as an optimisation: both clusters were competing for the same cores, which is one reason publisher TPS alone is a poor logical-replication health signal.

The subscriber WAL counter makes the index cost even clearer. These values are from one representative run, with pg_stat_wal reset immediately before the workload:

Binary encoding changed the wire representation, not the local tuple or index layout, so subscriber WAL remained almost identical. Four additional indexes almost exactly doubled it.

These numbers are not a universal logical-replication multiplier, they are a CPU-focused local experiment with durability disabled and no network latency. The receiver client’s CPU is also excluded, so this undercounts total system cost. The shape is what matters: decoding costs something measurable, applying creates a second write workload, and subscriber schema design changes how expensive that is.

The REPLICA IDENTITY FULL Trap

INSERT is straightforward because the subscriber only needs to create a row. UPDATE and DELETE first need to find one.

Postgres normally uses the primary key as the replica identity, though you can explicitly select another suitable unique index instead. When neither exists, REPLICA IDENTITY FULL sends the old row as its identity.

On PostgreSQL 15 and earlier, applying this meant sequentially scanning the subscriber table for a match, and whilst PostgreSQL 16 added the ability to use a suitable B-tree index for a FULL identity lookup, without one the sequential path still exists.

The apply worker makes the choice explicitly: FindReplTupleInLocalRel calls RelationFindReplTupleByIndex when it has a usable local index, and otherwise falls back to RelationFindReplTupleSeq. You can see both branches in the PostgreSQL apply-worker source.

I created 50,000 identical rows on publisher and subscriber, configured the publisher table with REPLICA IDENTITY FULL, paused the subscription and queued updates to the final 2,000 rows. Updating rows near the end is deliberately adversarial: a sequential lookup has to walk most of the table for each change.

The only difference between the two subscriber runs was:

CREATE INDEX full_events_lookup_idx ON full_events (id);

The index made this deliberately bad case approximately 31 times faster.

This is not an argument that every REPLICA IDENTITY FULL workload will be 31x slower. Row order, table width, cache state, update distribution and candidate indexes all matter. What it shows is narrower: the wrong replica identity changes the lookup algorithm for every replicated update.

PostgreSQL’s publication documentation describes which indexes can be used with FULL. PostgreSQL 16’s release notes are also worth reading if you operate a mixture of pre-16 and newer subscribers.

What to Monitor

Logical replication lag is the final symptom, not the diagnosis. I normally want to answer four separate questions.

Is the publisher retaining WAL?

SELECT
    slot_name,
    active,
    pg_size_pretty(
        pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)
    ) AS unconfirmed_wal,
    restart_lsn,
    confirmed_flush_lsn
FROM pg_replication_slots
WHERE slot_type = 'logical';

An inactive or stalled slot can retain WAL indefinitely, and it will also hold back catalog cleanup through catalog_xmin, so disk usage is not the only thing you are risking.

Is decoding spilling?

Use pg_stat_replication_slots and watch spill_bytes as shown earlier. A growing value says the publisher is no longer keeping the decoded transaction in memory.

Is the subscriber receiving or actually applying?

pg_stat_subscription shows the worker type, received LSN and message times, whilst pg_replication_origin_status.remote_lsn records how far changes have actually been replayed for an origin. That distinction matters, because a subscriber can happily receive data faster than it can update its tables.

What is the subscriber writing?

Use pg_stat_wal for local WAL generation, pg_stat_user_tables for tuple and HOT-update behaviour, and normal OS process metrics for the logical replication workers. pg_stat_subscription_stats exposes apply errors and conflicts which can otherwise look like a performance stall.

Practical Ways to Reduce the Cost

Use a narrow, indexed replica identity

This is the single biggest win for update-heavy workloads, so you want either a primary key or a unique index you have deliberately chosen for the job. Treat REPLICA IDENTITY FULL as a fallback, and remember that pre-16 subscribers cannot use the newer indexed FULL lookup path.

Design the subscriber as a write-heavy database

A read replica tends to accumulate indexes because they make analytical queries faster, but under logical replication every one of those indexes joins the write path. Remove the redundant ones, avoid indexing frequently changed columns where you can, and check whether your subscriber updates are still HOT-eligible.

Publish less data

Column lists reduce serialisation, transfer and apply work. Row filters can reduce downstream work too, but they execute on the publisher and an update may evaluate them twice, so you are moving the cost rather than making it disappear. See the PostgreSQL column-list documentation for the replica-identity requirements.

Test binary transfer against the actual topology

binary = true can reduce type conversion, but it is more restrictive across versions and types, and it did not reduce subscriber WAL in this test because the wire encoding does not alter the local write. Use it when your measured CPU or bandwidth justifies the compatibility trade-off.

Watch transaction shape

logical_decoding_work_mem and streaming settings matter for large in-progress transactions, and PostgreSQL 16 introduced parallel application for streamed large ones, controlled by max_parallel_apply_workers_per_subscription. That does not turn every ordinary transaction in your subscription into a freely parallel workload, since transactional ordering still constrains apply.

Capacity-plan both ends

An asynchronous publisher can report perfectly healthy commit latency whilst the subscriber quietly burns CPU and falls behind, which is why your capacity planning needs publisher decode CPU, subscriber apply CPU, subscriber WAL and retained publisher WAL - not only application TPS.

Conclusion

Logical replication is not a cheap copy of data. It is a pipeline which turns a physical change back into a logical one and then turns it into a new physical change somewhere else.

That extra interpretation is what makes it useful. It allows different versions, different schemas, filtered datasets and writable subscribers. It also means that a wide row, an unnecessary index or the wrong replica identity is multiplied across every change in the stream.

Treat a logical subscriber as another write-heavy database, not as passive storage. Measure decoding and apply separately, give updates an efficient identity lookup, and include the subscriber’s indexes and WAL in the cost of every design decision.

References

• Logical replication architecture
• Logical decoding concepts
• Logical streaming replication protocol
• CREATE SUBSCRIPTION options
• Replica identity and publications
• PostgreSQL 16 logical replication changes
• Logical replication monitoring statistics
• Logical apply-worker source

This post walks through exactly how a composite index like the one below will end up getting stored on disk.

CREATE TABLE test_tbl(
  id   bigserial primary key,
  col1 int,
  col2 text
);

CREATE INDEX t_idx1 ON test_tbl(col1, col2);

Composite Indexes Are One Key, Not Two

t_idx1 is a single B-tree index whose key is the pair (col1, col2).

Postgres stores that pair as one logical key and orders it lexicographically:

1. Sort by col1
2. For rows with the same col1, sort by col2

Everything about how queries use this index flows from this rule.

On-Disk Storage: A Separate File, B-Tree Structured

t_idx1 is its own table-like structure on disk:

• It lives in pgdata/base/<db_oid>/<relfilenode>
• It uses the B-tree access method
• It is split into 8kB pages

• Pages are arranged as:

  • Meta page (block 0)
  • Internal pages (routing nodes)
  • Leaf pages (actual key + TID entries)

This file has no connection to the heap except via the TIDs stored inside each tuple.

graph TD
    subgraph file["Index File (relfilenode)"]
        M["Block 0 · Meta Page\nroot=1, level=2, fast-root=1"]
        I1["Block 1 · Internal Page\nsep: (col1=2)"]
        L1["Block 2 · Leaf Page\n(1,'aaa', TID)\n(1,'bbb', TID)"]
        L2["Block 3 · Leaf Page\n(2,'abc', TID)\n(2,'xyz', TID)"]
    end

    M --> I1
    I1 -->|"left child\ncol1 < 2"| L1
    I1 -->|"right child\ncol1 ≥ 2"| L2
    L1 <-->|"prev / next ptrs"| L2

For large indexes Postgres splits the file into 1 GB segments (.1, .2, …) but the page structure and B-tree layout remain identical across segments.

Each entry in a leaf page stores:

• col1 - 4-byte integer
• col2 - text datum
• TID - 6-byte heap pointer (block number + offset)

This structure is wrapped in an IndexTupleData header that contains flags, tuple size, and a null bitmap if needed.

Text values follow varlena rules:

• Short text: stored inline ([length][bytes])
• Large text: replaced by an 18-byte TOAST pointer rather than embedding huge strings in the index

A leaf entry is physically:

[IndexTupleHeader]
[col1 value]
[col2 varlena or toast pointer]
[TID]

How Composite Keys Are Ordered

The B-tree compares keys in order:

(col1=1, col2='aaa')
(col1=1, col2='bbb')
(col1=2, col2='abc')

If a query filters only on col2, the index cannot be traversed meaningfully because the tree is structured around col1. Column order in the index will define physical locality.

A Concrete Example: 12 Rows Across Three Levels

Consider inserting these 12 rows (insertion order is irrelevant — the index always maintains sorted order):

INSERT INTO test_tbl (col1, col2) VALUES
  (3, 'ivy'), (1, 'bee'), (4, 'koi'),
  (2, 'fox'), (1, 'ant'), (3, 'gnu'),
  (4, 'jay'), (2, 'dog'), (1, 'cat'),
  (3, 'hen'), (4, 'lynx'),(2, 'elk');

The index sorts them by (col1, col2). TIDs reflect insertion order, not sort order:

With 3 entries per leaf page the index grows to three levels: a root internal page, two level-2 internal pages, and four leaf pages.

graph TD
    META["Meta Page · block 0\nroot → block 1"]
    ROOT["Root Internal · block 1\nsep = (3)"]
    LI["Left Internal · block 2\nsep = (2)"]
    RI["Right Internal · block 3\nsep = (4)"]
    L4["Leaf · block 4\n(1,'ant') TID(0,5)\n(1,'bee') TID(0,2)\n(1,'cat') TID(0,9)"]
    L5["Leaf · block 5\n(2,'dog') TID(0,8)\n(2,'elk') TID(0,12)\n(2,'fox') TID(0,4)"]
    L6["Leaf · block 6\n(3,'gnu') TID(0,6)\n(3,'hen') TID(0,10)\n(3,'ivy') TID(0,1)"]
    L7["Leaf · block 7\n(4,'jay') TID(0,7)\n(4,'koi') TID(0,3)\n(4,'lynx') TID(0,11)"]

    META --> ROOT
    ROOT -->|"key < (3)"| LI
    ROOT -->|"key ≥ (3)"| RI
    LI -->|"key < (2)"| L4
    LI -->|"key ≥ (2)"| L5
    RI -->|"key < (4)"| L6
    RI -->|"key ≥ (4)"| L7
    L4 <-.->|"prev/next"| L5
    L5 <-.->|"prev/next"| L6
    L6 <-.->|"prev/next"| L7

Every split in this tree falls between rows with different col1 values, so suffix truncation reduces all three separator keys to a single integer:

• Root sep = (3) — split between (2,'fox') and (3,'gnu') → col1 differs → stored as (3)
• Left internal sep = (2) — split between (1,'cat') and (2,'dog') → stored as (2)
• Right internal sep = (4) — split between (3,'ivy') and (4,'jay') → stored as (4)

Query Traversal

Select a query below to step through how the B-tree is traversed:

Internal Pages Store Limited Information

Internal nodes route lookups downward but do not need the full key. They store separator keys. Since Postgres 13, the B-tree code applies suffix truncation when writing a separator: it strips trailing attributes that are not needed to distinguish the two subtrees being separated.

• If col1 alone is different at the split boundary → the separator is (col1).
• If the split falls between two rows that share the same col1 → the separator must include col2 to disambiguate, so it becomes (col1, col2).

In practice, most separators end up as just col1, keeping internal pages small and the tree shallower. But the full key can appear whenever a page boundary falls inside a run of identical col1 values.

Leaf pages store all actual index entries; structure:

[PageHeader]
[line pointers]
...
[free space]
...
[packed index tuples at the end]

Leaf pages also form a doubly-linked list, allowing efficient range scans.

Because items are sorted by (col1, col2), values with similar keys land on adjacent leaf pages.

Skip Scans: Using the Index Without the Leading Column

A query filtering only on col2 can’t traverse the B-tree meaningfully because the tree is ordered by col1 first. Before Postgres 17, the planner would ignore t_idx1 entirely for WHERE col2 = 'foo'.

Postgres 17 introduced native skip scans. The planner enumerates each distinct value of col1, then does a separate descend into the tree for each one, effectively running:

col1 = <v1> AND col2 = 'foo'
col1 = <v2> AND col2 = 'foo'
...

This is only efficient when col1 has low cardinality — the number of descents equals the number of distinct col1 values. An ENUM type is the ideal fit:

CREATE TYPE status AS ENUM ('active', 'pending', 'closed');
ALTER TABLE test_tbl ADD COLUMN status status;
CREATE INDEX t_idx2 ON test_tbl(status, col2);

With three enum values the skip scan does three tree descents instead of a full sequential scan. A high-cardinality integer col1 would make the same approach impractical (thousands of descents).

On older Postgres versions the equivalent workaround is an explicit IN list:

WHERE col1 IN (SELECT DISTINCT col1 FROM test_tbl) AND col2 = 'foo'

Practical Implications

• Column order is structural — put the most selective or most-filtered column first.
• Prefix lookups (col1 = ?) are ideal; col2-only queries require a skip scan (PG 17+) or a separate index.
• Low-cardinality leading columns (enums, status codes) make skip scans practical.
• Large text values don’t bloat the index due to TOAST pointers.
• Locality on leaf pages drives range scan speed — similar keys land on adjacent pages.

Closing

Composite indexes in Postgres are literally concatenated keys stored in B-tree leaf pages. Once you understand that physical structure, most planner decisions become predictable rather than surprising.

This post explores something deceptively simple: how UUIDs are stored across various systems, and how simple assumptions about common storage can end up surprisingly wasteful, and how much can be saved both in bytes and in real money by being more deliberate in how we serialize them. A seemingly trivial choice can translate to terabytes of wasted storage and significant financial cost.

UUIDs in Modern Systems

UUIDs are easy to generate, globally unique, and require no coordination between services. They are the obvious choice for any distributed architecture.

Many systems, use UUID v7 which is UUID that can be time-ordered, making them an easy replacement for numerical sequential IDs, as they can fit nicely into the traditional btree database indexes with no performance loss.

A UUID v7 is composed of:

• a 48-bit timestamp
• version bits
• variant bits
• 60 bits of randomness

This structure makes collisions effectively impossible and ensures that identifiers cluster by time.

Underneath all of that, a UUID is simply 32 hexadecimal characters representing 16 raw bytes. When displayed as a string, the addition of four decorative dashes increases it to 36 characters. Readable for humans, yes. Efficient for machines, no.

Protobuf’s Role

Protobuf remains a common choice for modern service ecosystems: it is compact, strongly typed, and designed for high-throughput communication.

Protobuf serializes data using:

• field numbers
• wire types
• encoded values
• optional length prefixes

This results in efficient, structured binary messages. The downside is that strings are not treated the same as fixed-length bytes. Strings are a “length-delimited” wire type, which means they incur both:

• a UTF-8 encoding cost
• a length-prefix header

When dealing with a field that contains UUID, this overhead starts to matter. And unfortunately, UUIDs do not have a native Protobuf type and as a result, most often they end up being stored as strings rather than binary byte sequences.

How UUIDs Should Be Stored

Let’s break this down.

A UUID can be stored as:

• 16 bytes - raw binary; no variable length - requires custom type
• 18 bytes - bytes type with 2-byte variable length header
• 38 bytes - string: 36 chars (incl 4 dashes) + 2-byte variable length header

The most common but least efficient approach is to store UUIDs as strings. That carries more than double the required space (a 111% increase), with no benefit to the system. Using a byte array or a dedicated fixed-length UUID type eliminates the UTF-8 overhead entirely.

This can look like a micro-optimisation, but at scale - these bytes end up, not just in storage - but also transfer, backups as well as the additional memory required to process large amounts of these messages concurrently.

Why This Matters in Modern Architectures

In high-volume event systems, data growth rarely happens slowly. It accelerates. A single table can grow into terabytes quickly, and storing data inefficiently compounds the problem.

String-encoded UUIDs double the footprint of one of the most frequently used fields. They also increase:

• network bandwidth
• Kafka topic sizes
• I/O on storage engines
• VACUUM costs
• CPU usage during encoding/decoding

All for no benefit.

Optimizing UUID storage won’t fix everything, but it is one of the cleanest, lowest-risk improvements available.

Final Thoughts

Performance gains in distributed systems often come from tightening foundational pieces the parts executed millions or billions of times.
Storing UUIDs as strings is familiar and convenient, but it is also unnecessarily expensive.

Small decisions when done at scale end up having incredibly large impact, so in a distributed system world its paramount to pay attention to the details!

Introduction

Once you move from a single server DB, schema design is more important as your data is now split between many machines. It’s no longer enough to define your columns and data types; you have to consider how your data will be physically spread across machines: both for performance benefits and also to ensure optimal load across the cluster. The two biggest impacts on these metrics are the choices you make for primary keys and indexes.

How data is stored and partitioned

In YugabyteDB a table’s primary key is also the partitioning key. This means the primary key dictates how data is divided into tablets and which nodes those tablets live on. The database constantly monitors the size of these tablets, automatically splitting them when they grow too large to ensure that they remain balanced.

Every table begins with a set number of tablets, typically equal to the number of nodes in the cluster.

Partitioning Strategies: Hash vs. Range

There are two strategies for partitioning your data:

Hash Partitioning

With hash partitioning, the database applies a hashing function to the primary key and distributes rows based on the resulting hash value. This is the default in YugabyteDB and is excellent for ensuring an even spread of data across all nodes, which helps avoid “hotspots” where one node becomes overloaded with writes. For example, a table of events with PRIMARY KEY (identifier HASH) will have its data spread evenly across the tablets and thus the cluster.

Attaching an example of how a hash primary key would be distributed on a 3 node cluster:

Range Partitioning

Range partitioning groups data based on a continuous range of primary key values. For instance, a logs table with PRIMARY KEY (created_at ASC) would place records from January on one node, February on another, and so on. This will be efficient for queries that scan over a specific range, like fetching all logs from last week. However, it carries a significant risk: if your primary key is monotonically increasing (like a timestamp or a traditional sequence), all incoming writes will target a single node, creating a hotspot. There are a few tricks that we can use to mitigate this and help distribute the writes more evenly, which I will dig into a follow-up post!

As a range-partitioned tablet grows, it will eventually split. For example, a tablet holding data for the entire year of 2023 might split into two, one for the first six months and another for the second.

Example of how a range primary key would be distributed on a 3 node cluster:

Example of how a range primary key would be split:

Indexes Are Also Distributed Tables

In Yugabyte, an index is stored as its own table with its own primary key and partitioning strategy. When you run CREATE INDEX, you’re creating another data structure that consumes storage and requires its own maintenance.

Index Creation Syntax in YugabyteDB

It’s important to understand how YugabyteDB interprets the CREATE INDEX statement. Often, what you write is shorthand for a more explicit command that defines the partitioning strategy.

For example, a simple index creation:

CREATE INDEX idx_tbl ON tbl (id);

is translated to use hash partitioning on the index’s primary key:

-- This is what YugabyteDB actually runs
CREATE INDEX idx_tbl ON tbl (id HASH);

Similarly, for a composite index, the first column is hashed by default, while subsequent columns are stored in ascending order:

CREATE INDEX idx_tbl2 ON tbl (id, created_at);

This becomes:

-- This is what YugabyteDB actually runs
CREATE INDEX idx_tbl2 ON tbl (id HASH, created_at ASC);

You can also create more complex indexes, for example, having a hash of a two columns to help with data distribution:

CREATE INDEX idx_tbl_distrb_status ON table((id, status) HASH);

Understanding this translation is key to predicting how the table that gets created will perform both for writes and also for reads.

Common Indexing Pitfalls

1. Indexing Low-Cardinality Columns: Creating an index on a column with few unique values, like an enum for status, is a classic mistake. This leads to poor data distribution, as all rows with the same status will be clumped together in a few tablets, creating hotspots. This will lead to both slower inserts - as all writes will be directed to the same tablet, and slower reads - as queries filtering on that status will hit only one tablet, negating the benefits of distribution.

2. Using Hash Indexes for Range Queries: A HASH index on a timestamp column is often not useful. While it distributes the data well, it is inefficient for range queries (e.g., WHERE created_at BETWEEN ? AND ?) because the data is not stored in chronological order.

The Need for Housekeeping

Because data is constantly being written, updated and removed there is a need for continuous housekeeping. Processes like compaction: merging small data files into larger ones are essential for maintaining read performance. While the database handles this automatically, a well-designed schema makes these operations far more efficient.

Not only do we have to maintain our table, but also the tables used for the indexes. It’s important to remember that for every index you add to your table there is an additional write cost as the index has to be kept up to date with every insert, update and delete operation.

Final Thoughts

Designing schemas for distributed SQL is about balancing trade-offs.

• Your primary key is your partitioning strategy. Choose it wisely.
• In Yugabyte, your indexes are tables. They add overhead, so use them carefully and design them for good distribution - both for write and read operations.
• The goal is to leverage the distributed nature of the system. A schema that spreads the load evenly across all available nodes will lead to a healthier, more performant, and more scalable cluster.

PostgreSQL offers an alternative - CREATE INDEX CONCURRENTLY. This command allows you to build an index without blocking write operations on the table, making it the only option for systems where writes are constant throughout the day.

Understanding CREATE INDEX CONCURRENTLY

The magic behind CREATE INDEX CONCURRENTLY is in its multi-phase approach:

1. Initial Scan and Build: It performs an initial scan of the table and starts building the index structure. During this phase, it only requires a ShareUpdateExclusiveLock on the table. This lock mode blocks schema changes (like ALTER TABLE) and VACUUM FULL, but it allows INSERT, UPDATE and DELETE operations to continue normally.
2. Waiting for Transactions: After the initial build, it waits for all transactions that started before this phase began to complete. This ensures that the index includes changes made by those transactions.
3. Second Scan: It performs a second scan of the table to incorporate changes made by transactions that occurred during the initial build phase.
4. Finalization: Briefly locks the table to ensure it’s using the latest updates, after which the index is marked as ready for use.

Lock Requirements

While CREATE INDEX CONCURRENTLY avoids the heavy AccessExclusiveLock used by the standard CREATE INDEX, however, it still needs an exclusive lock in the finalization stage:

• ShareUpdateExclusiveLock (Mode 4): This is held on the table being indexed for most of the operation. It blocks schema changes and VACUUM FULL but allows reads and writes (SELECT, INSERT, UPDATE, DELETE).
• Brief AccessExclusiveLock (Mode 8): Towards the very end, when finalizing the index and making it visible in the system catalogs, it needs to acquire a brief AccessExclusiveLock. This lock does block all other operations - but this lock is typically held for a very short duration, without noticeable effect on the performance of the system.

Why Use It?

The primary benefit is minimizing downtime. By allowing writes during the lengthy index build process, CREATE INDEX CONCURRENTLY is essential for adding indexes to busy tables in production environments without disrupting application availability or performance.

Trade-offs

It also requires two transactions. If either transaction fails, the index might be left in an “invalid” state, requiring cleanup (DROP INDEX) before retrying.

• Slower: Building an index concurrently takes significantly longer than a standard CREATE INDEX due to the extra scans and waiting phases. For hot tables, there are likely going to be multiple passes before the index is close enough to the latest transaction and being able to enter its finalization stage.
• Higher CPU/IO Load: The process consumes more system resources over a longer period.
• Cannot Run Inside a Transaction Block: CREATE INDEX CONCURRENTLY must be run outside of an explicit transaction (BEGIN/COMMIT).
• Potential Failure: If something goes wrong (e.g., unique constraint violation during the second scan, or the final locking phase times out), it can leave behind an invalid index that needs manual cleanup.
• Blocked by long running transactions: If you have longer running transactions, those will slow down the creation of the index significantly, as the index has to wait for all previously open transactions to wrap. This is problematic if the table targetted by the index creation is particularly hot and requires multiple scans to catch up.

Conclusion

CREATE INDEX CONCURRENTLY is a powerful tool in PostgreSQL for maintaining performance without sacrificing availability. While it requires careful consideration due to its longer duration, higher resource usage, and inability to run within a transaction, its ability to avoid blocking writes makes it the preferred method for adding indexes to live production tables. Understanding its locking behaviour, as well as what transactions are blocking the index creation means you can make an informed decision between CREATE INDEX and CREATE INDEX CONCURRENTLY.

fetch use-cases

Using fetch is a common pattern in Ruby, especially when dealing with hashes. It allows you to retrieve a value for a given key, and if the key is not found, you can provide a default value. This is useful for avoiding nil values and for providing a fallback value.

hash = { a: 1, b: 2 }
hash.fetch(:a) # => 1
hash.fetch(:c, 3) # => 3

fetch fallback values and fallback blocks

When using fetch, the fallback value is evaluated even if the key is present. This can lead to unexpected behavior, especially if the fallback value is a complex expression or a method call.

hash = { a: 1, b: 2 }
hash.fetch(:a, "not found" ) # => 1
hash.fetch(:c, "not found" ) # => "Key not found"

Now with a defined function which throws an exception:

def complex_fallback
  raise "This is an exception"
end

hash = { a: 1, b: 2 }
hash.fetch(:a, complex_fallback) # => This is an exception!

This is definitely not what we wanted, as the value a is present in the hash, and we’d expect for it to be returned. To avoid this we have to pass a block to fetch:

hash = { a: 1, b: 2 }
hash.fetch(:a) { complex_fallback } # => 1
hash.fetch(:c) { complex_fallback } # => This is an exception

Conclusion

If you want to use fetch with a fallback value, be careful about the evaluation of the fallback value. If the fallback value is a complex expression or a method call, it is better to use a block to avoid unexpected evaluation of the function.

Introduction

The best interview questions are more than just a test. They’re a way to ensure that both parties get a clear idea of what it would be like to work together. They should be:

• grounded in real-world problems
• encourage discussion
• avoid gimmicks
• expandable across levels

Grounded in Real Work

The strongest questions come directly from the kinds of challenges your team faces every day. These questions create context, helping the candidate visualize what it’s like to work on your codebase, with your data, or within your constraints. It’s not about mimicking production exactly;it’s about reflecting the shape of the problems. You want questions that allow candidates to showcase their problem-solving in a way that actually maps to the role.

This also helps reduce bias: instead of filtering for textbook knowledge, you’re evaluating the kinds of thinking you actually need on your team.

Enables Two-Way Alignment

A well-structured question encourages discussion. It gives candidates the opportunity to bring in their perspective, ask questions, and see whether your environment matches their strengths and interests. When done right, the interview becomes a collaborative exchange, not a one-sided interrogation. It’s important to remember that the interview is not a confrontation, and you want the candidate to succeed.

You learn not just whether they can do the work, but how they communicate, how they think through ambiguity, and whether their approach aligns with your team’s engineering culture. And just as importantly, it gives them insight into how the team work and what the collaboration looks like.

Avoids Gimmicks

Puzzles and trick questions usually produce noise. They reward prior exposure over solid reasoning which are valuable if the environment in which you operate you require a lot of specific knowledge: you can either try to find that in candidates or validate that they can pick up on tricks. They can also create unnecessary stress and exclude strong candidates who simply haven’t seen the specific pattern before.

A good question is well-scoped, has some constraints and can be reasoned through with just simple concepts. Even better if it encourages the candidate to ask clarifying questions; that’s often a good sign they’re thinking through the problem and engaging with the question.

Expandable Across Levels

A great question can cater to all levels of candidates, it can scale up or down in complexity such that everyone can enjoy the question. It also gives you flexibility during the interview: if someone finishes the core problem quickly, you can dig deeper. If they struggle initially, you can adjust without abandoning the question altogether and offers the possibility of making the interview a positive experience for the candidate even if they will not be moving forward. One has to remember that tech world can be quite small and you do not want to have negative experiences due to a bad interview question.

Example: Real-World System Design

Skip the textbook stuff it is usually very low signal. Try to ask something practical, based on a real-problem you are solving:

If you are working on a system that has to ingest a lot of data from articles, you can ask the candidate to design that, or in a coding exercise - give them an array of articles, and have the write a “processing” pipeline.

A more worked through example: Example: “We handle millions of log events per second. How would you design a system to make recent logs quickly queryable?”

Why I think this leads to better engagement:

It’s real: Based on the kind of infrastructure problems many teams tackle.

It’s relatable: Most engineers have some experience with logs or data pipelines.

It scales: You can start with a naive solution, then explore tradeoffs: consistency vs. availability, local vs. distributed stores, etc.

You can also layer in follow-ups: What happens when query volumes spikes? How would you handle hot partitions? What if logs need to be retained for compliance? How to deal with data deletion/rotation?

Takeaway

A good interview question should feel like a working session. It should be rooted in real problems, flexible for different experience levels, yet have enough ambiguity to allow for a discussion. If it results in a productive, thoughtful conversation, you’re probably on the right track.

It’s worth investing in this. A few well-designed questions can raise the bar on your hiring process and improve the experience for everyone involved.

Got an interview question you’ve seen that works well;or one that totally misses the mark? Let’s talk.

LIKE, ILIKE and why you might need to use them

LIKE and ILIKE operators allow you to use wildcards in your queries to provide a level of fuzzy-matching should you require it. The wildcard in SQL syntax is represented by the percent symbol: %. A very common use-case is to find all names or emails matching a prefix, suffix or both. Generally this kind of query can be optimised such that we avoid the LIKE or ILIKE syntax which can have pretty impactful performance considerations.

The difference between LIKE and ILIKE is that the ILIKE operator will ignore the case of the strings which it compares, ie: it will match AbC with a record containing abc or aBc - this is particularly useful anytime you are dealing with user input - as case consistency is not a given.

Alternatives to LIKE and ILIKE

Firstly, if you need to check for only a prefix - you can actually trim both strings to the desired length and perform an equality check, which can use an existing index and yield really good performance. The same can be applied if you also need to ignore casing, you can create an index which transforms the column - resulting in slightly more work at INSERT time but better performance for reads.

CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_lower_name ON
"users" (lower(name));

Allowing us to write a query like this:

SELECT *
FROM users
WHERE lower(name) = ?

The above query will be able to use the index defined above, and as long as we ensure that we pass a string which has been lower-cased from the calling point, we are ensuring great performance by using the index.

Another alternative is using the citext extension - which stands for case insensitive text, which can be used as a drop-in replacement for text columns. However, if you are expecing to have letters that have accents, those will not be normalized to the non-accented letter. There is also a small storage overhead to be paid when using citext - but it allows you to avoid the LOWER(...) calls on your queries.

Optimising LIKE and ILIKE queries

LIKE queries which check only for prefixes, eg: LIKE 'alex%' - can make use of existing BTree indexes to speed up their performance and do an IndexScan which can plug some gaps. ILIKE cannot use BTree indexes and as such cannot be optimised from. How the query plan will look like for hitting an IndexScan for a LIKE query:

EXPLAIN ANALYZE 
SELECT * 
FROM emails 
WHERE LOWER(recipient_email) LIKE 'alex%' 
LIMIT 5;
                                                                            QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.43..129.79 rows=5 width=1370) (actual time=61.852..1837.272 rows=5 loops=1)
   ->  Index Scan using idx_lower_rec_email on emails  (cost=0.43..1472852.26 rows=56930 width=1370) (actual time=61.851..1837.266 rows=5 loops=1)
         Filter: (lower((recipient_email)::text) ~~ 'alex%'::text)
         Rows Removed by Filter: 2079
 Planning Time: 0.165 ms
 Execution Time: 1842.558 ms
(6 rows)

However, sometimes you cannot get away with just a single non-leading wildcard, and you require more power.

Trigrams & pg_trgm for multiple wildcards

Enter trigrams - via the pg_trgrm extension, which splits up the words into 3 letter segments, allowing much more granularity when it comes to querying. You will also need to use a GIN or GiST index to be able to leverage the trigrams.

[Trigrams for "Charlie"]
+-------+
| cha   |
| har   |
| arl   |
| rli   |
| lie   |
+-------+

[Trigram GIN Index]
+---------+------------+
| Trigram | Row IDs    |
+---------+------------+
| cha     | {3}        |
| har     | {3}        |
| arl     | {3}        |
| rli     | {3}        |
| lie     | {3}        |
+----------------------+

This will allow queries like SELECT * FROM users WHERE name LIKE "%arl%" to have very very fast results. However, it is obviously going to come at a pretty large cost in terms of storage, as now instead of storing a single reference to the name, you have split it up in threes:

This is a classic exponential situation which:

Storage considerations aside, you have to consider that the trigrams have to be computed at INSERT time and saved in the index. This will increase the latency of the data ingestion, however trading that off for the faster data lookups.

Recommendations

• Try to use LOWER(...) = functions if possible to do prefix matching, rather than a LIKE "alex%". Create an index that computes the LOWER(...) and can be used to speed up these queries.
• Use citext if you require case-insensitive, prefix-lookups and want to avoid the LOWER(...) changes.
• Use pg_trgm for short strings which require multiple wildcards. Be mindful of the insert time overhead.

What is SELECT ... FOR UPDATE SKIP LOCKED

Introduced in Postgres 9.5 - the SKIP LOCKED syntax can be tacked at the end of a SELECT ... FOR UPDATE statement and will return only rows which are not locked by any other ongoing transaction. The normal behaviour of SELECT ... FOR UPDATE operation is to wait for the locked rows to be returned, thus making it quite difficult for the user to parallelize such SELECT statements on a single table without having appropriate mechanisms to deal with either updating rows locked, or targeting different rows by an additional attribute (eg: partitioning by modulo).

The introduction of SKIP LOCKED, enabled a very powerful way in which users can parallelize their processing without having to worry about writing locking mechanisms in their application, and can instead rely on the robust locking mechanism provided by Postgres.

Table Setup

Given a simple table structure like:

CREATE TABLE t_random AS
    SELECT
        uuid_generate_v4(),
        ROUND(RANDOM()*100) AS c1,
        ROUND(RANDOM()*1000) AS c2,
        ROUND(RANDOM()*100) AS c3
    FROM GENERATE_SERIES(1,100000) s;

On this table, lets apply a status column - and assume that each row in the table is a job that would need to be executed.

ALTER TABLE your_table_name ADD COLUMN status VARCHAR(255);

UPDATE your_table_name
SET status =
    CASE
        WHEN (c1 + c2 + c3) < 300 THEN 'PENDING'
        WHEN (c1 + c2 + c3) BETWEEN 300 AND 500 THEN 'INPROGRESS'
        ELSE 'COMPLETED'
    END;

Running in 2 side by side transactions you can now experiment with the SELECT ... FOR UPDATE and SKIP LOCKED.

-- session 1

begin;
SELECT id FROM t_random WHERE status='PENDING' ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED;
-- the above will return 100 rows.

-- session 2

begin;
SELECT id FROM t_random WHERE status='PENDING' ORDER BY ID LIMIT 100 FOR UPDATE SKIP LOCKED;
-- this will return a different set of 100 rows, allowing you to processes these
-- in parallel.

end; -- the 100 rows are unlocked and are now VISIBLE to other sesions.

Row locking under the hood

So how does row locking actually work under the hood? When running a SELECT ... FOR UPDATE query Postgres will find the targeted rows and then use a LockTupleExclusive row lock, which in turn will update the header of the tuple with xmin and xmax values. These values affect which transactions can view the row. This requires both a write to disk, which will persist the tuple change, and also it writes the lock in shared memory and the lock is now viewable via the pg_locks view.

This immediately highlights that row locking is an operation which has I/O, it also shows that to check for row locks the database engine needs to read the tuple information. This means we can no longer have Index Only Scan queries, which will negatively affect performance.

You can put an EXPLAIN ANALYZE when running the above queries to have a peek under the hood:

EXPLAIN ANALYZE SELECT id FROM t_random WHERE status = 'PENDING' ORDER BY ID LIMIT 100;
----------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.42..22.91 rows=100 width=16) (actual time=4.240..4.642 rows=100 loops=1)
   ->  Index Only Scan using trand_status on t_random  (cost=0.42..80932.71 rows=359853 width=16) (actual time=4.237..4.630 rows=100 loops=1)
         Heap Fetches: 91
 Planning Time: 0.399 ms
 Execution Time: 4.687 ms
(5 rows)

EXPLAIN ANALYZE SELECT id FROM t_random WHERE status = 'PENDING' ORDER BY ID LIMIT 100 FOR UPDATE SKIP LOCKED;
                                                                   QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.42..41.94 rows=100 width=22) (actual time=0.552..0.780 rows=100 loops=1)
   ->  LockRows  (cost=0.42..149404.06 rows=359853 width=22) (actual time=0.551..0.769 rows=100 loops=1)
         ->  Index Scan using trand_status on t_random  (cost=0.42..145805.53 rows=359853 width=22) (actual time=0.536..0.676 rows=100 loops=1)
               Filter: ((status)::text = 'PENDING'::text)
 Planning Time: 0.212 ms
 Execution Time: 0.813 ms
(6 rows)

The difference in the query plan between a query without FOR UPDATE and one with FOR UPDATE is the presence of the LockRows statement which forces PG to check the row tuple headers - meaning we now doing 2 reads - one from the index, and one from the table.

Recommendation

It is recommended to only lock a small number of records in one transaction. This is because the row locking has both a disk I/O cost as well as a memory overhead, as the locks get stored in shared memory.

If a large number of rows are locked in a table, the query performance will start to noticeably drop and trend towards more of a SEQSCAN performance level due to the sheer number of rows it has to skip.

TL;DR - be mindful that sometimes although you have an index, the query plan can be changed underneath you and it can result in a seq scan.

Introduction

As part of an upgrade to Rails 7.1 we started seeing various CI failures which all related to either Postgres becoming unavailable, or the test run timing out or completely random rspec errors. This was only affecting 1 particular spec file, but we could not immediately isolate the culprit.

Investigation

I have started by disabling all initializers for rspec, reducing DB pool size, and kept continuing until I ended up having to disable activejob. It turns out that ActiveJob by default in test mode uses :async as its running method, which will require an active DB connection so it can cache DB schema.

OK, but what was calling activejob, as we only make use of sidekiq in the application? Well, it turns out that the root cause is that activestorage depends on activejob and in turn this lead to the following backtrace:

/ruby/3.1.4/lib/ruby/gems/3.1.0/gems/activestorage-7.1.2/app/jobs/active_storage/analyze_job.rb:5

The solution

The solution was swapping the ActiveJob running mode to :test in the environment config file (config/environments/test.rb). By adding the below configuration it has immediately resolved all of our CI issues:

config.active_job.queue_adapter = :test

This was hinted at in this Rails issue.