feat(page): add typed keyset pagination for Bob
Add versioned typed cursors, composable native query mods, bidirectional page assembly and opt-in authorized counts. Cover SQLite and PostgreSQL outcomes, strict cursor boundaries and a runnable example; require PostgreSQL service tests in CI. Refs: #6
This commit is contained in:
1 parent
da1ff94fe2
commit
8615d8ca66
11 files changed
+1142
-11
No files matched your search
+11
-11
@@ -1,8 +1,8 @@
|
||||
---
|
||||
title: Preserve supported value types in versioned page cursors
|
||||
date: 2026-10-27
|
||||
status: proposed
|
||||
implementation: not-started
|
||||
status: accepted
|
||||
implementation: in-progress
|
||||
summary: Pair a plain sort with a typed row key function and preserve supported concrete values with tagged cursors.
|
||||
tags:
|
||||
- pagination
|
||||
@@ -33,9 +33,9 @@ Type preservation here means retaining each supported concrete value across enco
|
||||
|
||||
## Decision Outcome
|
||||
|
||||
Propose plain Sort/key inputs with tagged JSON, wrapped in unpadded base64url (`base64.RawURLEncoding`). This record remains proposed pending independent review and owner acceptance; implementation has not started.
|
||||
Use plain Sort/key inputs with tagged JSON, wrapped in unpadded base64url (`base64.RawURLEncoding`). The owner accepted this reviewed design on 2026-10-27, including the two-input API and continuation flags. Implementation is present in the working branch; delivery remains in progress pending independent implementation review and a real PR. No PR delivery evidence is recorded yet.
|
||||
|
||||
Sort contains a stable caller-owned ID and an ordered list of columns, each with a stable ID, Bob expression, direction and unique marker. Reject empty sorts, missing/duplicate IDs, nil expressions, invalid directions and a missing unique marker on the final column. The marker asserts that column alone is unique in the selected relation under database collation/comparison semantics, including joins. The library cannot prove schema truth. Composite-only uniqueness is outside the issue's final unique-column contract. Expressions and IDs come from trusted application declarations, never cursor-controlled SQL. The sort ID identifies the full immutable interpretation: ordered expressions, directions, key types, uniqueness and comparison semantics. Changing any of these requires a new ID. The backend selects the expected trusted Sort for the query; a decoded token ID must equal its ID. There is no global registry, descriptor fallback or hash subsystem. Local column metadata remains available for validation and query generation; it is not repeated on the wire.
|
||||
Sort contains a stable caller-owned ID and an ordered list of columns, each with a Bob expression, direction and unique marker. Reject empty sorts, a missing or invalid UTF-8 Sort ID, nil expressions, invalid directions and a missing unique marker on the final column. Columns have no redundant ID field. The marker asserts that column alone is unique in the selected relation under database collation/comparison semantics, including joins. The library cannot prove schema truth. Composite-only uniqueness is outside the issue's final unique-column contract. Expressions and IDs come from trusted application declarations, never cursor-controlled SQL. The sort ID identifies the full immutable interpretation: ordered expressions, directions, key types, uniqueness and comparison semantics. Changing any of these requires a new ID. The backend selects the expected trusted Sort for the query; a decoded token ID must equal its ID. There is no global registry, descriptor fallback or hash subsystem. Local column metadata remains available for validation and query generation; it is not repeated on the wire.
|
||||
|
||||
A typed key function returns values in exactly that order. Check `len(key(row)) == len(sort.Columns)` for every returned row used to assemble a page, including the fetched extra row. A mismatch returns a clear programmer-error `KeyError` with expected/actual lengths; never truncate, zip a prefix or emit a bad cursor. Reject nil key functions. Column/value correspondence remains an explicit caller contract; no field discovery or hidden mapping is introduced.
|
||||
|
||||
@@ -49,22 +49,22 @@ For this exact example, compact JSON is 105 bytes and its unpadded base64url tok
|
||||
|
||||
UUID support specifically means Go 1.27.1's standard `uuid.UUID`: its Parse/String methods support this representation and database/sql's default converter converts it to a string for drivers. Third-party UUID types require explicit caller conversion to this type. Named user-defined scalar types, unsigned integers, floats, booleans, pointers, nullable wrappers, byte slices and arbitrary custom types return a typed `ValueError`; no reflection, implicit coercion, custom binder or registry is proposed. A true Go type alias retains its underlying supported identity. Additional types require an explicit format/support decision. V0 sort columns must be non-null and ordered consistently by WHERE and ORDER BY; callers extract actual stored/scanned values, including timestamp precision.
|
||||
|
||||
Apply the same proposed maximum of 16 KiB (16,384 bytes) to both emitted and received tokens, measured after final unpadded base64url encoding, including the full envelope. Check received token bytes before decoding, and decode before constructing a mod. Encoding a key or envelope that cannot fit returns a typed `CursorError` and no partial Page. Reject bad base64/JSON, unknown/duplicate/missing envelope fields (only `v`, `sort` and `values` are allowed, so `columns` is rejected), trailing data, version mismatch, a sort ID unequal to the expected Sort ID, wrong value count, nulls, non-array value entries, pairs of any arity other than two, non-string tags/payloads, unknown tags and invalid/range-overflowing payloads. The outer `values` array must contain exactly `len(sort.Columns)` pairs; the former type/value objects are rejected. An exported `CursorError`, inspectable with `errors.As`, distinguishes malformed, version, sort and unsupported-value failures. Encoding unsupported key values returns `ValueError` identifying the position and type, without leaking values. Keys and token shape are validated before producing a Page; errors return no partial page. Cursors are not authenticated: a well-formed edit to a supported value or tag remains a seek request, not a tamper-detection failure. Without expected type metadata in Sort, the library cannot prove an edited supported tag matches a SQL column; driver/database errors propagate normally. Authorization stays in the query.
|
||||
Apply the same maximum of 16 KiB (16,384 bytes) to both emitted and received tokens, measured after final unpadded base64url encoding, including the full envelope. Check received token bytes before decoding, and decode before constructing a mod. Encoding a key or envelope that cannot fit returns a typed `CursorError` and no partial Page. Reject bad base64/JSON, unknown/duplicate/missing envelope fields (only `v`, `sort` and `values` are allowed, so `columns` is rejected), trailing data, version mismatch, a sort ID unequal to the expected Sort ID, wrong value count, nulls, non-array value entries, pairs of any arity other than two, non-string tags/payloads, unknown tags and invalid/range-overflowing payloads. The outer `values` array must contain exactly `len(sort.Columns)` pairs; the former type/value objects are rejected. An exported `CursorError`, inspectable with `errors.As`, distinguishes malformed, version, sort and unsupported-value failures. Encoding unsupported key values returns `ValueError` identifying the position and type, without leaking values. Keys and token shape are validated before producing a Page; errors return no partial page. Cursors are not authenticated: a well-formed edit to a supported value or tag remains a seek request, not a tamper-detection failure. Without expected type metadata in Sort, the library cannot prove an edited supported tag matches a SQL column; driver/database errors propagate normally. Authorization stays in the query.
|
||||
|
||||
Prepared pagination exposes one generic Bob `Mod[Q]` targeting native `AppendWhere(...any)`, `AppendOrder(bob.Expression)` and `SetLimit(any)` methods shared by PostgreSQL, SQLite and MySQL SelectQuery. Use Bob expressions and `clause.OrderDef`, fixed operators/directions, and an integer limit. Prepare validates requests/cursors first because ordinary Bob Apply has no error return. Cursor values only enter `expr.Arg(value)`; Bob's existing `any` boundary retains their concrete dynamic type. No custom SQL renderer is needed. Apply once to a query without ORDER BY, LIMIT, OFFSET or FETCH; callers may append their dynamic/authorization filters but must not override pagination/order.
|
||||
|
||||
For `(created DESC, id ASC)`, after means `created < c OR (created = c AND id > i)`, grouped as one WHERE condition. Before reverses each comparison. Backward also reverses each ORDER BY direction, fetches limit+1, trims overflow, then reverses retained rows into canonical order. Page[T] holds Items, StartCursor, EndCursor, HasNext and HasPrevious; cursors describe the canonical first/last returned items, or are empty when no items exist. Deleted anchors remain value boundaries. Writes or changed filters between requests may skip/repeat items; no snapshot is promised.
|
||||
|
||||
Forward maps first/after and Backward maps last/before to a transport-neutral Direction, Size and Cursor request. Proposed defaults: zero direction is Forward, zero/absent size uses configured default, negative sizes and invalid directions fail, oversized sizes clamp to maximum; require `1 <= default <= maximum` and overflow-safe limit+1. Empty cursor means no anchor. The owner selected continuation semantics: forward HasNext=overflow and HasPrevious=nonempty after; backward HasPrevious=overflow and HasNext=nonempty before. This applies even to empty/stale results. An anchor-derived opposite flag does not prove rows exist; no extra existence probes are required.
|
||||
Forward maps first/after and Backward maps last/before to a transport-neutral Direction, Size and Cursor request. Request rules: zero direction is Forward, zero/absent size uses configured default, negative sizes and invalid directions fail, oversized sizes clamp to maximum; require `1 <= default <= maximum` and overflow-safe limit+1. Empty cursor means no anchor. The owner selected continuation semantics: forward HasNext=overflow and HasPrevious=nonempty after; backward HasPrevious=overflow and HasNext=nonempty before. This applies even to empty/stale results. An anchor-derived opposite flag does not prove rows exist; no extra existence probes are required.
|
||||
|
||||
Bob v0.50.0 distinguishes generated column references from generated typed WHERE helpers. Generated references such as `models.Orders.Columns.CreatedAt` (schematic; this repository has no generated models) satisfy `bob.Expression` and can replace handwritten `expr.Quote` identifiers in Sort. Their generated per-table column type embeds the dialect Expression and has no Go value-type parameter: neither reference nor `expr.Quote` statically links that expression to a `time.Time` key. In contrast, generated `models.SelectWhere.Orders.CreatedAt` uses `WhereMod[Q, time.Time]`; its `EQ`/`LT`/`GT` arguments must be assignable to `time.Time`, and Q must satisfy Bob's Filterable constraint. These helpers produce WHERE mods, not sortable column expressions, and expose no public expression accessor. They cannot be substituted directly for Sort expressions or consume a heterogeneous `[]any` element without recovering the expected concrete type. The chosen API preserves the concrete dynamic values through the codec/binder and checks lengths at runtime; it promises no compile-time column/key type or order agreement. The owner's question about stronger static type safety remains open; any alternative API needs a separate owner decision. No adapter or stronger typed abstraction is proposed here.
|
||||
Bob v0.50.0 distinguishes generated column references from generated typed WHERE helpers. Generated references such as `models.Orders.Columns.CreatedAt` (schematic; this repository has no generated models) satisfy `bob.Expression` and can replace handwritten `expr.Quote` identifiers in Sort. Their generated per-table column type embeds the dialect Expression and has no Go value-type parameter: neither reference nor `expr.Quote` statically links that expression to a `time.Time` key. In contrast, generated `models.SelectWhere.Orders.CreatedAt` uses `WhereMod[Q, time.Time]`; its `EQ`/`LT`/`GT` arguments must be assignable to `time.Time`, and Q must satisfy Bob's Filterable constraint. These helpers produce WHERE mods, not sortable column expressions, and expose no public expression accessor. They cannot be substituted directly for Sort expressions or consume a heterogeneous `[]any` element without recovering the expected concrete type. The chosen API preserves the concrete dynamic values through the codec/binder and checks lengths at runtime; it promises no compile-time column/key type or order agreement. The owner deferred stronger compile-time field correspondence to [code generation issue 11](https://gitea.auvem.com/corestack/go/issues/11); no code generation or stronger typed abstraction is implemented here.
|
||||
|
||||
A concrete proposed call shape is:
|
||||
A concrete call shape using schematic Bob-generated references is (this repository has no generated models):
|
||||
|
||||
```go
|
||||
sort := page.Sort{ID: "orders/recent/v1", Columns: []page.Column{
|
||||
{ID: "created", Expression: expr.Quote("orders", "created_at"), Direction: page.Desc},
|
||||
{ID: "id", Expression: expr.Quote("orders", "id"), Direction: page.Asc, Unique: true},
|
||||
{Expression: models.Orders.Columns.CreatedAt, Direction: page.Desc},
|
||||
{Expression: models.Orders.Columns.ID, Direction: page.Asc, Unique: true},
|
||||
}}
|
||||
key := func(o Order) []any { return []any{o.CreatedAt, o.ID} } // time.Time, int64
|
||||
prepared, err := page.Prepare(sort, key, page.Request{
|
||||
@@ -79,7 +79,7 @@ result, err := prepared.Page(rows)
|
||||
|
||||
Backward changes Direction and supplies the before cursor. `pgdialect`/`sm` are Bob's PostgreSQL dialect/select-mod packages; corresponding SQLite/MySQL select types use the same mod. `Page` retains the typed `Order` key function; `o.ID` is restored as int64, including above 2^53. A separate opt-in `Count(ctx, executor, callerBuiltCountQuery)` executes an explicitly counting, authorized unpaginated Bob query. It never guesses how to strip pagination or count grouped/distinct caller relations.
|
||||
|
||||
Bob v0.50.0 is source-feasible for all three dialects, with two reported limitations: generic `mods.OrderBy` expects a different AppendOrder signature from actual selects (native ModFunc matches); MySQL's select writer discards LIMIT expression arguments (use the normal native integer LIMIT API, while cursor WHERE values remain bound). Runtime dialect execution is unverified. Bob belongs in the root module under the accepted module-boundary ADR; no UUID dependency is needed. No persistence, writers, migrations, secrets, workers or timers are introduced.
|
||||
Bob v0.50.0 is source-feasible for all three dialects, with two reported limitations: generic `mods.OrderBy` expects a different AppendOrder signature from actual selects (native ModFunc matches); MySQL's select writer discards LIMIT expression arguments (use the normal native integer LIMIT API, while cursor WHERE values remain bound). SQLite runs in-process in the normal test suite. PostgreSQL uses the same pagination outcome suite and a required CI service container. MySQL has compile-time native mod composition coverage; runtime behavior remains unverified. Bob belongs in the root module under the accepted module-boundary ADR; no UUID dependency is needed. No persistence, writers, migrations, secrets, workers or timers are introduced.
|
||||
|
||||
### Consequences
|
||||
|
||||
|
||||
Reference in new issue
Block a user