GeoParquet provider¶
A read-only OGC API Features provider that serves GeoParquet as a collection without converting it first: the file stays on a local path or in an object-storage bucket, and the query goes to it. CQL2 filters — spatial predicates included — are evaluated inside DuckDB, so the network carries the answer rather than the dataset.
Install the extra:
Configuration¶
Declare it by dotted path in the pygeoapi configuration:
resources:
lakes:
type: collection
title:
en: Lakes
description:
en: Lakes served from GeoParquet
keywords:
en: [lakes]
extents:
spatial:
bbox: [-180, -90, 180, 90]
crs: http://www.opengis.net/def/crs/OGC/1.3/CRS84
providers:
- type: feature
name: app.provider.geoparquet.GeoParquetProvider
data: s3://my-bucket/tenants/acme/lakes
id_field: id
geometry_column: geom
time_field: observed_at
| Key | Required | Meaning |
|---|---|---|
data | yes | Dataset source: a file, a glob, or a directory root |
id_field | yes | Column carrying the feature identifier |
geometry_column | no | Geometry column name (default geom) |
time_field | no | Column the datetime parameter filters on |
properties | no | Restrict which columns are returned by default |
bbox_column | no | Covering column to pre-filter on (auto-detected) |
union_by_name | no | false takes the schema from the first file only |
file_index | no | stac-geoparquet file or STAC collection of the files |
file_index_asset | no | Asset of the index with each file's href (data) |
count | no | false drops numberMatched from item responses |
store_options | no | Store settings: region, skip_signature, endpoint |
engine_options | no | DuckDB settings, e.g. memory_limit, threads |
Source shapes¶
| Shape | Example |
|---|---|
| Single file | s3://bucket/lakes.parquet |
| Glob | s3://bucket/lakes/*.parquet |
| Hive-partitioned root | s3://bucket/lakes → scanned as lakes/**/*.parquet |
A directory root is read with hive partitioning enabled, so a query that filters on a partition column reads only the matching partitions.
On a bucket DuckDB expands the wildcard itself, in one request. That also keeps the dataset isolated from the process environment: the object-store layer reads the standard variables in every constructor, so a deployment whose AWS_ENDPOINT_URL_S3 names its own store would send the listing there and be refused for a dataset that lives elsewhere. Where the object-store layer is the reader (the obstore fallback below, the PMTiles provider), an explicit endpoint in store_options wins over that variable: name the dataset's own endpoint (for AWS, s3.<region>.amazonaws.com) and the reads go there.
On the obstore fallback below the objects are listed instead: DuckDB cannot expand a glob through that bridge — an explicit file works while *.parquet raises "No files found" — so the provider enumerates the prefix once, when it is built, and hands the engine explicit paths.
Sources and credentials¶
data accepts the same sources as the configuration itself — local paths, s3://, gs://, az://, and any S3-compatible endpoint — and credentials come exclusively from each provider's standard environment variables. See Config from cloud object storage; the same rules apply here.
A single .parquet file can also sit at an https:// URL, such as a public bucket served by its own domain. DuckDB reads it by range, with no credentials and no store_options. HTTP has no listing, so a glob or a folder there is refused when the provider starts: read those from the bucket URL instead.
Bucket data is read by DuckDB itself — httpfs for S3-compatible stores and GCS, the azure extension for Azure — with the per-dataset settings carried in a DuckDB secret. Without explicit keys the secret uses DuckDB's credential chain, which reads the same standard environment variables listed above, so nothing changes for an operator.
That choice is measured, not stylistic: reading through a Python filesystem bridge prevented DuckDB from caching the blocks it had already fetched, so every request repaid its bytes — the same query cost 21 s on each repetition against 0.7 s natively, and a bbox window over a 4.5 GB dataset went from 44 s to 0.9 s once warm.
The object store still loads the configuration and writes the artifact. If a deployment needs the previous path back:
Local paths are read by DuckDB directly, with no configuration at all.
Per-dataset store settings travel in store_options. A public dataset must say so and name its region. Without skip_signature the request is signed with whatever credentials the process happens to carry — and any deployment that keeps its own data on an S3-compatible store carries some — which a public bucket answers with 403 Forbidden. On a machine with no credentials at all it fails differently but no better, after roughly twenty seconds of EC2-metadata retries:
data: s3://overturemaps-us-west-2/release/2026-09-23.1/theme=divisions/type=division/
store_options:
region: us-west-2
skip_signature: true
The same key carries endpoint for an S3-compatible service.
Query capabilities¶
| Capability | Notes |
|---|---|
limit / offset | Pushed down as LIMIT/OFFSET |
bbox | ST_Intersects against the request envelope, in-engine |
datetime | Instants and intervals, .. for open ends; needs time_field |
| Property equality | e.g. ?country=it |
properties (select) | Column projection |
skipGeometry | Skips the geometry entirely |
sortby | Ascending and descending |
resulttype=hits | COUNT only, no features materialised |
filter (CQL2) | cql2-text and cql2-json |
CQL2 support covers comparisons, AND/OR/NOT, LIKE, IN, BETWEEN, IS NULL, arithmetic, the spatial predicates (S_INTERSECTS, S_WITHIN, S_CONTAINS, S_DISJOINT, S_TOUCHES, S_CROSSES, S_OVERLAPS, S_EQUALS) and the instant temporal operators (T_AFTER, T_BEFORE, T_EQUALS). Interval temporal operators are refused with a clear error rather than approximated, since honouring them would need a second time column.
Example:
curl "https://example.org/geoapi/collections/lakes/items?\
filter-lang=cql2-text&\
filter=S_INTERSECTS(geom, POLYGON((11 41, 14 41, 14 43, 11 43, 11 41))) AND depth > 10"
Performance: the file and the reader both decide¶
Measured against the public Overture Maps release (theme=divisions, one 578 MB file in us-west-2, read from Europe):
| Same file, remote | Same file, local disk | 7 MB regional extract | |
|---|---|---|---|
| First page | ~15–20 s | 31 ms | 5 ms |
| Warm page | ~15 s | 30 ms | 3 ms |
bbox window | ~20 s | 36 ms | 5 ms |
Two separate effects hide in those columns. Locality is worth roughly 500× (the middle column is the very same file, only nearer), while dataset size is worth about 10×. Shrinking the data is the smaller lever; moving it is the larger one.
Give the file the shape the format expects¶
Row-group pruning only works when the file carries usable statistics. Following the GeoParquet spec's own distribution guidance:
- zstd compression, level 15 or above;
- spatially sorted (Hilbert) before writing — without it, every row group's bounding box spans most of the dataset and nothing can be skipped;
- row groups of 50k–150k rows;
- a covering
bboxcolumn (GeoParquet 1.1) or native geometry types (2.0); the provider detects the covering automatically, or takesbbox_column; - partition beyond a couple of gigabytes.
geoparquet-io applies these as defaults, and a plain DuckDB COPY … TO … (FORMAT parquet, ROW_GROUP_SIZE 100000) gets you most of the way.
Turn off union_by_name on a large glob of one schema¶
By default the provider unites the schemas of all the files of a dataset, which means reading the footer of each of them when it starts. On a glob over many large files this dominates the start: the 512 files of the Overture buildings (258 GiB) took 117 s and 3.8 GB of memory to describe with ten threads, and did not finish in 15 minutes with one. When the files share one schema, as the files of an Overture type do, union_by_name: false takes the schema from the first file instead, and the same glob is described in about 4 s with 63 MB. The other files are then read against that schema: a column the first file lacks is left out without a warning, and a file that lacks a column of the first fails the query with schema mismatch in glob. Leave the default on a dataset whose files were written at different times.
Read only the files a request needs¶
A request with a bbox on a glob still opens the footer of every file, because the bounds of each row group are in the footers. When the provider has a file_index, it hands DuckDB only the files whose bbox meets the request, and answers with no feature, without reading anything, when none does. A request without a bbox but with another filter (a property, a datetime, CQL2) reads every file, and so does /items/{id} unless the provider has an id_index (see the next section).
When every row of the index has a table:row_count, a request with no filter and no sortby opens no footer to count: numberMatched is the sum of the counts, and a page reads only the files its rows fall in, however far it is from the first. If one row has no count, DuckDB counts and pages the whole dataset over every file. The counts are taken from the index as they are; a wrong count gives a wrong numberMatched and shifts the pages.
The index is a stac-geoparquet file, one row per file with its bbox, the href of its data asset and, optionally, its table:row_count, or the URL of a STAC collection whose items name the files; the provider reads the items once and keeps the result in its cache until the collection changes. A relative file_index is read from beside the configuration. If the index cannot be read, the provider logs a warning and reads the glob.
With an index the provider takes the schema from the first file it lists, whatever union_by_name says, so the files of an indexed dataset should share one schema. With union_by_name: false the schema of each request comes from the first file it selects, and a request whose first file lacks a column of the others fails.
providers:
- type: feature
name: app.provider.geoparquet.GeoParquetProvider
data: s3://overturemaps-us-west-2/release/2026-10-21.0/theme=buildings/type=building/*.parquet
union_by_name: false
file_index: indexes/portolan-building.parquet
Of the 512 files of the Overture buildings one meets a window over Rome. Read from a laptop with sixteen DuckDB threads, the first request on that window took 125 s on the glob and 9 s on the one file, and a repeated request 6 s and 2 s. On a server with one dedicated vCPU in the bucket's region, the same request took 44 s and then 28 s on the glob, and 2.8 s and then 0.5 s with the index, while the process used 625 MB at most instead of 1.6 GB.
Find a feature by id¶
/items/{id} on a glob reads the id column of every file, because there's nothing that says which file holds the id. For example, on the 512 files of the Overture buildings a lookup takes several minutes and a lot of memory. Some publishers ship an index of their ids, with one row per feature naming the file that holds it, sorted by id across a few Parquet files. The GERS registry of Overture Maps is one example. With id_index the provider looks the id up and reads only that file:
providers:
- type: feature
name: app.provider.geoparquet.GeoParquetProvider
data: s3://overturemaps-us-west-2/release/2026-10-21.0/theme=buildings/type=building/*.parquet
store_options:
region: us-west-2
skip_signature: true
id_index:
type: retrieved
location: s3://overturemaps-us-west-2/registry/*.parquet
columns:
id: id
path: path
bbox: bbox
href_base: s3://overturemaps-us-west-2/release/2026-10-21.0/
columns names the id column, the column with the file (href for a full href, or path for a path that href_base is joined to) and, optionally, a bbox struct of xmin, ymin, xmax and ymax. The index is read with the store_options of the provider. A relative location is read from beside the configuration.
The provider learns the range of ids of each index file from its footer, keeps the ranges in its cache with the ETag of each file, and checks them with one listing of location per lookup. A lookup opens only the index file whose range holds the id, and DuckDB reads only the row groups whose bounds hold it. Ids are compared as text, so an integer id matches an index that stores it as a string.
The answer follows the row:
- No row for the id:
404, without reading any data file. - A row naming a file of the dataset: only that file is read. With a
bboxin the row and a covering column in the data, DuckDB also skips the row groups whose bounds do not meet it. - A row with no file, or naming a file that is not part of the dataset or that the store cannot read: every file is read, as without an index. A file is part of the dataset when the
file_indexlists it or, without one, when thedataglob matches it (*stays within one folder, as in DuckDB) or it lies under the folderdatanames. - An index that cannot be read: a warning, and every file is read.
The third rule covers a registry that spans other themes, or that has moved to a newer release than the one served. The Overture registry follows the latest release and its paths name the files of that release, so joined to the href_base of an older one they name no file of the dataset, and the provider reads every file instead.
An id_index of type: generated names an index built from the data. Nothing builds one yet, so the provider logs a warning and ignores it.
Keep the data near the server, and read it with a caching reader¶
Cross-region latency is a floor no predicate can lower: reading the Overture file from Europe cost seconds per request no matter how well the file was laid out — its row groups are in fact reasonably sorted (each covers about 5% of world longitude, and a window over Rome touches 16 of 256).
Repeated reads are the other half of the story. DuckDB caches the blocks it has already fetched, but only on its own I/O path: the same query through the obstore fsspec bridge stayed at ~12 s on every repetition, while DuckDB's native httpfs answered the second and third in milliseconds. This mirrors what GDAL sees on the desktop side (GDAL #8225: 542 MB downloaded for a 507 MB file), and it is why "cloud-native" can end up costing more than a plain download when the reader cannot cache.
Practical order of effect: put the data in the same region as the server, shape it as above, and only then reach for materialising a local copy.
The schema is read once per version of the files¶
The provider needs the dataset's schema when the server starts and on every configuration reload. For a dataset on a bucket it lists the objects with the store's client, reads their footers, and keeps the schema in schemas under the cache root (FASTGEOAPI_CACHE_DIR, or .cache in the working directory), keyed by the objects' ETags. A server started again over unchanged files lists them and reads nothing else: on a slow route to us-west-2, such a restart took 7 seconds, where reading the schema through DuckDB had taken 11 minutes. When the environment sets AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL, DuckDB reads the schema itself, because the store's client would send the listing to that endpoint, and it does so too when the faster path fails.
What that costs on a live deployment¶
The public demo runs both arrangements side by side on the same 1 GB / 1 vCPU machine in Paris, over the same query — a bbox over the centre of Rome, five features. lazio-roads is an Overture extract staged in a bucket in the same region; overture-places is read where Overture publishes it, in us-west-2, with no copy.
| Collection | Placement | Cold | Warm |
|---|---|---|---|
lazio-roads | same region | 5–6 s | 1.0 s |
overture-places | cross-region | 19–29 s | 2.2–2.5 s |
Warm, locality is worth about 2×; cold, four to five times that. The cold figures are ranges because they are: the cross-region first request was measured at 18.8 s and 29.0 s on two different cold starts, and a number that swings by ten seconds is a number to design around rather than quote. A machine that suspends, redeploys or scales from zero pays it again on the next request, and half a minute is not a response a user waits for. Staging an extract is the difference between a demo that always answers and one that sometimes does.
Cheap wins on the request itself¶
count: falsedropsnumberMatchedfrom item responses (aresulttype=hitsrequest still answers) — worth seconds per request on a remote dataset.- Ask for the columns you need with
properties=— Parquet is columnar, so unrequested columns are never read.
Preparing datasets¶
Any GeoParquet writer works. For partitioning, spatial sorting and bbox covering columns, geoparquet-io is a convenient offline tool — it is not needed at runtime.
DuckDB can also produce a partitioned dataset directly:
Staging an Overture extract¶
scripts/stage_overture.py does the whole job for an Overture release: it reads a window, flattens the nested fields worth exposing, writes the covering bbox column and the Hilbert ordering described above, and uploads the result.
uv run python scripts/stage_overture.py \
--theme transportation --type segment \
--bbox 11.4,41.2,14.1,42.9 \
--local /tmp/lazio.parquet \
--dest s3://my-bucket/overture/transportation-lazio.parquet
Destination credentials come from the standard environment variables; the public Overture bucket is read anonymously regardless of them. --local keeps the extract on disk, so a failed upload does not discard the read — re-running with the same path uploads what is already there. The Lazio extract above is 743k segments and 133 MB, read in about 50 seconds.
This is an offline tool. Nothing at runtime depends on it, and it is not installed with the package.
Geometry: native or WKB¶
DuckDB writes its own GEOMETRY type; GeoPandas, GDAL and Sedona write the geometry as a WKB blob. The provider detects which it is holding and converts only when it has to, so both kinds of file are served without configuration — including a dataset written by one tool and extended by another.
Running on a read-only runtime (AWS Lambda and friends)¶
fastgeoapi can be deployed as a Lambda function (AWS_LAMBDA_DEPLOY, served through Mangum). There the filesystem is read-only except /tmp, and two DuckDB defaults do not fit:
- the spatial extension must already be there. A missing extension would otherwise be installed into
~/.duckdb, which fails; the provider now says so explicitly instead of leaking a path error. Vendor it in the image (theDockerfiledoes), or pointDUCKDB_EXTENSION_DIRECTORYat a writable path. - the spill directory defaults to
.tmpnext to the working directory. The provider now sets it fromTMPDIR(which such runtimes set to/tmp), so a query that needs to spill has somewhere to go.
Size the engine against the function rather than the host it lands on:
Two more things worth knowing about that deployment shape. Execution environments are reused across invocations, so the plugin cache means the engine is opened once per container rather than per request — but a cold start still pays it. And /tmp survives warm invocations, which makes it a reasonable place to materialise a dataset that would otherwise be read across a region boundary on every cold container.
The reload webhook does not fan out. Each execution environment holds its own configuration, so POST /admin/config/reload reaches exactly one of them; the others keep serving the previous configuration until they are recycled. On a single long-lived instance this is a non-issue; on a function runtime, treat configuration as versioned and plan for eventual convergence.
Limitations¶
- Read-only: no create, update or delete.
- A query whose reads time out at the store answers
504 Gateway Timeout. A filter on a column the files are not sorted by can read a whole file, which across regions takes minutes. - The
spatialextension is vendored in the container image. Outside it, the first connection installs the extension, which needs network access once. - Geometries are served in the dataset's own CRS; declare it with
storage_crswhen it is not CRS84.