MySQL - UUID/created_at cursor based pagination?

Question

For a large dataset, paginating with an OFFSET is known to be slow and not the best way to paginate. A much better way to paginate is with a cursor, which is just a unique identifier on the row so we know where to continue paginating from where we last left off from the last cursor position.

When it comes to a cursor where it is an auto incrementing id value, it's fairly easily to implement:

SELECT * FROM users
WHERE id <= %cursor // cursor is the auto incrementing id, ex. 100000
ORDER BY id DESC
LIMIT %limit

What we're not certain about, is if instead of an auto incrementing id cursor, the only unique sequential identifiers for the cursor are uuid and created_at on the table rows.

We can certainly query based on the uuid to get the created_at, and then select all users that are <= created_at but the issue is what if there are multiple instances of the same created_at timestamp in the users table? Any idea how to query the userstable based on uuid/created_at cursor combination to ensure we get the correct datasets (just as if we were using auto incrementing id)? Again, the only unique field is uuid since created_at may be duplicate, but their combination would be unique per row.

Rick James · Answer 1 · 2024-02-14T22:00:04.160

WHERE   created_at <= x
  AND ( created_at < x OR uuid < y )
ORDER BY created_at DESC,
         uuid       DESC

or this equivalent:

WHERE (     created_at < x
       OR ( created_at = x AND uuid < y )
      )
ORDER BY created_at DESC,
         uuid       DESC

This technique works for any pair of columns where the first one (created_at) could have duplicates and the second is unique (uuid or id).

And this is required:

INDEX(created_at, uuid)

Note that both parts of the WHERE are DESC. Mixing ASC and DESC will defeat the usability of the INDEX. (MySQL 8.0 can work around that.)

Note also, that this assumes you don't care what order the rows are in when created_at is duplicated, but you do want a consistent ordering. Note that uuid will be seemingly random, but still consistent. With that said, id (with or without Galera) and uuid work equally well.

(UUIDs suck, but that is a different discussion.)

Remember where you left

More on pagination without using OFFSET .

In your case, that is the combination of created_at and whatever the PRIMARY KEY is. The former gives the desired ordering; the latter handles what to do in the case of dup dates.

This technique further optimizes things:

PRIMARY KEY(create_at, id), -- or (created_at, UUID)
INDEX(id)                   -- or (UUID)

Federico Razzoli · Accepted Answer · 2022-11-16T19:07:41.887

I will answer what you asked, but first let me tell you that I don't understand why you want to do that. An autoincremental id is very good for this task. But it is correct to also use a timestamp column, because it is a bad practice to rely on an id for sorting. Why? Because there are cases when its order might not be chronological - for example, if you use Galera cluster and you have failovers.

To do what you asked, first create this index:

ALTER TABLE users
    ADD INDEX idx_created_at_uuid (created_at, uuid);

The order columns is important. If you reverse it, the index will not be useful.

Now you just need to run a query like this:

SELECT some_columns
    FROM users
    WHERE (created_at, uuid) < (x, y)
    ORDER BY created_at DESC;

uuid is only needed because created_at is not unique. The expression above uses a syntax known as row constructors. It is intuitive: it's the same as created_at < x, unless these values are equal, in which case it compares uuidandy`.

If created_at is not the first column in the index, MySQL will have to read all rows and copy them to a temporary table (which could be in-memory or on-disk) to sort them.

If you decide to use the id, just keep the above snippets, but replace uuid with id.

EDIT 16 Nov 22: The WHERE clause was incorrect. I fixed it and I explained the syntax.

MySQL - UUID/created_at cursor based pagination?

2 Answers2

Linked