A read stopped being a block in 18

The assumption nobody wrote down

For most of PostgreSQL’s history, one entry in the read counter meant one block, and one block meant eight kilobytes. That was never documented as a contract, and it did not need to be, because it was simply how the server worked: a backend wanted a page, it asked the operating system for that page, the counter went up by one. Every I/O dashboard in existence has that fact baked into it, usually as a literal * 8192 somewhere in a query, and it went on being true long enough that nobody thinks of it as an assumption.

Version 18 makes the size of a read a tuning parameter. Adjacent pages that a scan is about to want anyway are fetched in one operation, and how many is decided by a setting that a user can change for the duration of a session. The volume of data moved is unchanged. The number of operations it took is now a function of configuration rather than of the data.

Nothing errors. The counter still exists, still increments, still means something. It just stopped meaning what the dashboard thinks.

The same scan, three times

The fixture is a table of three hundred thousand rows, a little over a hundred megabytes, on a cluster with a buffer pool far too small to hold it. The scan at the end matches nothing, so the only work is reading.

CREATE TABLE scan_target (row_id bigint, filler text);
INSERT INTO scan_target SELECT g, repeat('s', 400) FROM generate_series(1, 300000) AS g;
CHECKPOINT;

CREATE TABLE
INSERT 0 300000
CHECKPOINT

Start on 17, and ask the question the way a dashboard asks it: count the reads, multiply by the block size, call the result the volume.

SELECT pg_stat_reset_shared('io');
SELECT count(*) AS matched FROM scan_target WHERE filler LIKE '%q%';
SELECT sum(reads) AS reads, sum(reads) * 8192 / 1024 / 1024 AS mb_by_the_old_arithmetic
FROM pg_stat_io WHERE context = 'bulkread';

 pg_stat_reset_shared 
----------------------
 
(1 row)

 matched 
---------
       0
(1 row)

 reads | mb_by_the_old_arithmetic 
-------+--------------------------
 13042 |     101.8906250000000000
(1 row)

Hold on to that answer, because it is already wrong and 17 gives you no way to know. The table is about a hundred and thirty megabytes and the scan read all of it; the arithmetic says a hundred and two. Read combining arrived a release earlier than the accounting for it did, so on 17 some of those operations already moved more than one block and the multiplication quietly under-reported by a quarter. This is the oldest kind of monitoring bug: the query still runs, the number still moves in the right direction, and it is off by an amount nothing in the system will ever tell you.

On 18 the same scan, with combining turned down to a single block so that each read is one page, which is what the dashboard has always believed.

SET io_combine_limit = 1;
SELECT pg_stat_reset_shared('io');
SELECT count(*) AS matched FROM scan_target WHERE filler LIKE '%q%';
SELECT sum(reads) AS reads,
       pg_size_pretty(sum(read_bytes)) AS volume,
       round((sum(read_bytes) / nullif(sum(reads), 0) / 1024)::numeric, 1) AS kb_per_read
FROM pg_stat_io WHERE context = 'bulkread';

SET
 pg_stat_reset_shared 
----------------------
 
(1 row)

 matched 
---------
       0
(1 row)

 reads | volume | kb_per_read 
-------+--------+-------------
 16667 | 130 MB |         8.0
(1 row)

Sixteen thousand six hundred and sixty-seven reads, a hundred and thirty megabytes, exactly eight kilobytes each. That is the world the dashboard was written for, and 18 will give it to you if you ask.

Now the same scan again with the default.

SET io_combine_limit = 16;
SELECT pg_stat_reset_shared('io');
SELECT count(*) AS matched FROM scan_target WHERE filler LIKE '%q%';
SELECT sum(reads) AS reads,
       pg_size_pretty(sum(read_bytes)) AS volume,
       round((sum(read_bytes) / nullif(sum(reads), 0) / 1024)::numeric, 1) AS kb_per_read
FROM pg_stat_io WHERE context = 'bulkread';

SET
 pg_stat_reset_shared 
----------------------
 
(1 row)

 matched 
---------
       0
(1 row)

 reads | volume | kb_per_read 
-------+--------+-------------
  1530 | 102 MB |        68.4
(1 row)

Same table, same query, same server, one session-level setting different: sixteen thousand reads became fifteen hundred. The average read is sixty-eight kilobytes rather than eight. The volume is lower than the previous run because part of the table was still resident from it, which is worth noticing for its own reasons and does not touch the point.

A panel that graphs reads per second across that boundary shows a ninety percent drop in I/O on the day of the upgrade. A capacity model fed by the same number concludes the storage has an order of magnitude more headroom than it did. An alert on read rate stops firing. Every one of those is a confident statement about a workload that did not change at all.

Why sixty-eight and not a hundred and twenty-eight

Sixteen blocks at eight kilobytes each is a hundred and twenty-eight kilobytes, and the measured average is about half that. That gap is the useful part of this measurement rather than a flaw in it.

Combining only happens where the pages are adjacent and the server knows in advance that it wants them. A sequential scan is the best case and still does not get the full width every time: the scan works through a ring of buffers rather than the whole pool, the run of pages it can request ahead is bounded by what is free in that ring, and every boundary in the file is a place where a batch has to stop short. Different scan shapes do much worse. A bitmap heap scan fetches runs of varying length and averages far less. An index scan chasing individual rows combines nothing at all, and its reads stay at one block each on every version.

So the practical shape of this is that the read count now depends on the access pattern as much as on the amount of data, and two clusters that look identical can report read counts that differ by a factor of ten because one of them is running sequential scans and the other is running index lookups. Comparing read counts between clusters was always a slightly dubious exercise. On 18 it is meaningless unless you compare bytes.

The setting comes in two parts, which is the other thing to know before touching it.

SELECT name, setting, unit, context FROM pg_settings
WHERE name IN ('io_combine_limit', 'io_max_combine_limit') ORDER BY name;

         name         | setting | unit |  context   
----------------------+---------+------+------------
 io_combine_limit     | 16      | 8kB  | user
 io_max_combine_limit | 16      | 8kB  | postmaster
(2 rows)

The first can be changed by anybody, in any session, at any time, and applies immediately. The second is a server-wide ceiling that clamps it and needs a restart to move. Raising the per-session value above the ceiling silently gets you the ceiling, which is the right behaviour and is also a good way to spend an afternoon wondering why a change had no effect. Both are expressed in blocks, which the unit column spells out.

What to change in the monitoring

Three things, and the first of them is the whole job.

• Delete every multiplication of an operation count by a block size and read the byte columns instead. They are reported directly on 18, and the page on the column that used to be the multiplier covers what replaced what.
• Re-baseline anything expressed as reads per second at the upgrade boundary rather than carrying the series across it. The old numbers are not wrong, they are answering a different question, and splicing them produces a cliff that people will spend a week explaining.
• Where an operation count is genuinely what you want, keep it and say so. Operations per second against a storage device with a published queue depth is a real signal; it is simply a different one from throughput, and the two have been conflated for years because they used to be a constant apart.

Average bytes per read, which the third query above computes, is worth adding as a panel in its own right. It is the fastest way to see whether a workload is doing bulk work or point lookups, it needs no knowledge of the schema, and on any version before 18 it was not available at all.

Explaining the change to people who do not read release notes

Two conversations follow this change and both go badly if the numbers are presented without the reason.

The first is with whoever owns the storage. An operations count that falls by an order of magnitude overnight looks, from the storage side, like the database stopped using the array. If that team is sizing or billing on operations, which many do, a cluster that upgraded on Tuesday appears to have freed up an enormous amount of capacity. It has not. It is moving the same bytes in fewer, larger requests, and larger requests are generally cheaper per byte and not free. The number to bring to that conversation is throughput, which did not change, with the average request size beside it as the explanation.

The second is with whoever set the alert thresholds. Every threshold expressed in operations per second is now calibrated against a unit that changed size, and they will not fire. A quiet alert is worse than a wrong one, because nobody investigates silence. The fix is mechanical, and the thing to resist is the temptation to scale the old threshold by the observed ratio: the ratio depends on the access pattern, so a threshold divided by ten is correct for the sequential work that produced the ten and wrong for everything else.

Underneath both is a fact worth stating plainly, because it outlasts this release. An operation count was never a measure of I/O volume; it was a measure of volume divided by a constant that happened to hold. PostgreSQL held that constant for decades, which is long enough for an assumption to become an article of faith. The versions where it stopped holding are the ones where it becomes obvious that the byte columns were always the right thing to collect and that the operation counts were a convenient substitute.

A mixed fleet makes this concrete and slightly worse. A cluster on 17 and a cluster on 18 running the same workload will report read counts that differ substantially, and a dashboard that sums or averages across the fleet is now adding two different units together. Splitting the panel by major version for the duration of the migration is ugly and is the only honest option.

What it costs

Nothing here needs an extension or a restart unless you intend to raise the server-wide ceiling. Reading the byte columns costs the same as reading the operation counts did, because the server was tracking the volume regardless and was simply not reporting it.

The one caution is about turning combining down to reproduce old numbers. It works, as the second measurement above shows, and it is a reasonable thing to do for half an hour to confirm a theory. It is not a migration strategy: the setting exists because larger reads are faster on the storage most fleets run on, and configuring a cluster to be slower so that a dashboard keeps reading the same is a trade nobody would make if it were written down that way.