libviprs

Takes large images and multi-page documents of many kinds — high-resolution scans, PDFs, aerial and satellite imagery, gigapixel photographs, historical maps, digitised archives — extracts raster data, optionally geo-references it, and generates tile pyramids (DeepZoom, XYZ, Google Maps) ready to serve to web-based viewers such as MapLibre GL, the open-source WebGL map engine, for interactive pan and zoom. Inspired by libvips, built from scratch.

Features

• PDF extraction — extract embedded raster images from scanned document PDFs via lopdf (pure Rust, no C dependencies)
• PDF rendering — render vector PDFs (text, paths, and other vector exports) via PDFium, with optional memory-budgeted rendering
• Image decoding — JPEG, PNG, TIFF via the image crate
• Tile pyramid generation — multi-threaded engine with backpressure, configurable tile size and overlap; optional streaming mode processes images in horizontal strips for bounded peak memory
• True source-side PDF streaming — PdfiumStripSource renders one strip of a vector PDF at a time via PDFium's clipped matrix path, including correct /Rotate handling, so multi-gigapixel pages stay inside the configured memory budget
• libvips-exact PDF page sizing — a page at a DPI comes out at the size libvips gives (Letter at 300 dpi is 2550×3300), with PageSizing::LegacyTruncated to keep the 0.5.x sizes
• Memory-budget policy — StreamingConfig.budget_policy selects BudgetPolicy::Error (fail loudly when a strip cannot fit) or BudgetPolicy::AutoAdjust (raise the budget to the smallest workable strip); the engine also pre-flights the budget against canvas geometry before any work starts and returns a typed BudgetExceeded error
• PMTiles v3 storage (default) puts the whole pyramid in one .pmtiles archive via PmTilesSink, indexed by z/x/y so a viewer fetches one tile with a ranged read; pmtiles::Reader reads it back over any byte-range source, and --storage directory still writes the classic tree
• Layout formats — DeepZoom (.dzi + directory tree), XYZ (z/x/y), and Google Maps (z/y/x, power-of-2 grids)
• Centre support — centre image within the tile grid with even background padding on all sides
• Tile encoding — PNG, lossless WebP, JPEG (configurable quality), or raw pixel output
• Blank tile optimization — emit full tiles or write 1-byte placeholders for uniform-color regions
• Edge tile background — configurable background color for padding partial tiles at image edges
• Geo-referencing — affine transform mapping pixel coordinates to geographic coordinates, GCP support
• Observability — progress events, per-level callbacks, peak memory tracking
• Object-storage sinks — write tiles directly to S3-compatible buckets via s3:// URIs; enabled with --features s3
• Manifest v1 + versioning — ManifestV1 / ManifestBuilder emit a versioned JSON manifest alongside the pyramid; unknown fields are ignored for forward compatibility
• Blank-tile tolerance — BlankTileStrategy::PlaceholderWithTolerance collapses near-uniform tiles within a configurable per-channel delta, reducing output for slightly noisy scans
• Tracing spans — structured spans (libviprs::pipeline, libviprs::level, libviprs::tile) for OpenTelemetry-compatible collectors; enabled with --features tracing
• Resumable jobs — EngineBuilder::with_resume(ResumePolicy) writes a checkpoint file and supports Overwrite, Resume, and Verify modes; interrupted runs restart from the last completed tile. Directory storage only: a PMTiles archive keeps no checkpoint, so under the default --storage pmtiles Verify checks the finished archive against the plan instead of walking a tree, and Resume has no checkpoint to skip from, so it regenerates
• Retry + failure policies — per-tile retry with configurable back-off; FailurePolicy::RetryThenSkip marks a tile as missing rather than aborting the entire job
• Checksums — tile-level Blake3 or SHA-256 digests written into or verified from the manifest; controlled by ChecksumMode and ChecksumAlgo
• Deduplication — identical tiles are stored once and referenced via symlink, hardlink, or manifest-only pointer; DedupeStrategy::Blanks targets only placeholder tiles; DedupeStrategy::All dedupes every tile
• Packfile archive sinks — write the whole pyramid into a .tar, .tar.gz, or .zip archive; enabled with --features packfile
• Extended EngineResult metrics — bytes_read, bytes_written, retry_count, queue_pressure_peak, duration, stage_durations for observability pipelines

Quick Start

Try it interactively: the CLI & API generator lets you tick flags and copy a tailored version of this snippet — see the pyramid base setup and toggle features in the generator panel.

use libviprs::{
    extract_page_image,
    BlankTileStrategy,
    EngineBuilder,
    EngineKind,
    FsSink,
    Layout,
    PyramidPlanner,
    TileFormat,
};
use std::path::Path;

// ──────────────────────────────────────────────────────────────────────
// 1. Decode the source.
//    extract_page_image pulls the embedded raster out of a scanned PDF;
//    use libviprs::decode_file for plain image inputs (PNG/JPEG/TIFF).
// ──────────────────────────────────────────────────────────────────────
let raster = extract_page_image(
    Path::new("document.pdf"),  // input path (PDF here; any image works too)
    1,                           // 1-based PDF page number
).unwrap();

// ──────────────────────────────────────────────────────────────────────
// 2. Plan the pyramid.
//    PyramidPlanner computes per-level dimensions, tile counts, and
//    canvas size — no pixels are touched yet.
// ──────────────────────────────────────────────────────────────────────
let planner = PyramidPlanner::new(
    raster.width(),    // source width in pixels
    raster.height(),   // source height in pixels
    256,               // tile size (DeepZoom default; 512 for HiDPI)
    0,                 // pixel overlap between adjacent tiles
    Layout::DeepZoom,  // tile naming convention (also: Xyz, Google)
).unwrap();

let plan = planner.plan();

// ──────────────────────────────────────────────────────────────────────
// 3. Configure where the tiles get written.
//    FsSink writes to a local directory; libviprs also ships
//    ObjectStoreSink (S3) and PackfileSink (.tar/.tar.gz/.zip).
// ──────────────────────────────────────────────────────────────────────
let sink = FsSink::new(
    "output_tiles",  // output directory (created if missing)
    plan.clone(),    // pyramid plan tile paths are derived from
)
.with_format(TileFormat::Png);  // also: TileFormat::Webp | TileFormat::Jpeg { quality: u8 } | Raw

// ──────────────────────────────────────────────────────────────────────
// 4. Run the engine.
//    EngineKind::Auto picks monolithic / streaming / mapreduce based on
//    the source kind and any with_memory_budget value supplied.
// ──────────────────────────────────────────────────────────────────────
let result = EngineBuilder::new(
    &raster,  // source raster from step 1
    plan,     // pyramid plan from step 2
    sink,     // tile sink from step 3
)
.with_engine(EngineKind::Auto)                        // auto-select engine
.with_concurrency(4)                                   // worker threads for tile extraction
.with_blank_strategy(BlankTileStrategy::Placeholder)   // collapse uniform tiles into 1-byte placeholders
.run()
.unwrap();

println!(
    "{} tiles across {} levels ({} blank tiles skipped)",
    result.tiles_produced,    // total tile files written
    result.levels_processed,  // number of pyramid levels
    result.tiles_skipped,     // tiles emitted as blank placeholders
);

Modules

Streaming Engine

Large raster images — scanned documents at 300 DPI, aerial surveys, gigapixel photographs — can easily exceed available RAM when the monolithic engine materialises the full canvas. The streaming engine solves this by processing the pyramid in horizontal strips, reducing peak memory by orders of magnitude while producing pixel-exact output.

Unlike libvips, which implements a fully lazy demand-driven pipeline where each pixel is computed on demand through a complex DAG of operations, libviprs takes a pragmatic middle ground: strips are processed eagerly within each band, but the full canvas is never materialised. This keeps the pipeline architecture simple and auditable while delivering the memory savings that matter for gigapixel-scale images.

The caller sets a memory budget; the engine maximises strip height within that budget to balance memory and throughput. When the budget is large enough for the full canvas, the monolithic engine is used automatically — no code changes needed.

For vector PDFs, PdfiumStripSource renders one strip at a time directly from the PDF using PDFium's clipped matrix path (with full /Rotate support), so the full page bitmap is never materialised. BudgetPolicy::Error fails loudly if the chosen budget cannot fit the smallest workable strip; BudgetPolicy::AutoAdjust raises the budget and continues. The engine also pre-flights the budget against canvas geometry before allocating, surfacing a typed BudgetExceeded error rather than running until OOM.

PDF Page Sizing

A PDF page is measured in points (1/72 inch), so rendering it at a DPI means turning points into whole pixels. libviprs now does that the way libvips pdfload does: per axis, rint(points * (dpi / 72.0)), computed in f64 and rounded to nearest with ties to even. pdfium then stretches the page into a bitmap of exactly that size, which is the same call libvips makes, so nothing is padded or cropped. US Letter at 300 dpi is 2550×3300, and a pyramid built from a PDF now lines up pixel for pixel with the libvips one.

The old rule (0.5.x) multiplied the page by dpi as f32 / 72.0 and cut off the decimals. When the product landed a hair under a whole number it lost a pixel, and then pdfium's aspect-fit rounded a second time and pulled the other axis down with it. Rasters came out 0 to 2 px smaller than libvips, so Letter at 300 dpi was 2549×3299. The planner, the cached render and the streaming source all use the one rule now, so they agree with each other as well.

PageSizing

PageSizing names the rule. It is #[non_exhaustive], so a match on it needs a wildcard arm.

PageSizing::pixel_dims(width_pts, height_pts, dpi) is the one function every site calls, so you can ask for a page's raster size without rendering it. The points are the page size pdfium reports after the page box and /Rotate are applied. Results saturate at u32::MAX.

use libviprs::PageSizing;

// US Letter at 300 dpi.
assert_eq!(PageSizing::Exact.pixel_dims(612.0, 792.0, 300), (2550, 3300));
assert_eq!(PageSizing::default(), PageSizing::Exact);

Functions and the strip source

The existing names keep working and run at Exact. Each one has a _with twin that takes the policy:

For streaming, PdfiumStripSource::builder(..) takes .mode(), .sizing() and .budget(), then .build(). PdfiumStripSource::sizing() reads the choice back, and width() and height() report the new sizes. Strips are scaled per axis, so a streamed strip is the same pixels as the matching rows of a cached render.

use libviprs::{PageSizing, PdfiumStripSource, render_page_pdfium_with};

// Same pixels as 0.5.x, for pyramids already on disk.
let raster = render_page_pdfium_with(path, 1, 300, PageSizing::LegacyTruncated)?;

let src = PdfiumStripSource::builder(path, 1, 300)
    .mode(libviprs::PdfiumRenderMode::Streaming)
    .sizing(PageSizing::LegacyTruncated)
    .build()?;

The pixel budget is checked on the new size, so a page that only fit because of the truncation can now come back as RenderBudgetExceeded, and render_page_pdfium_budgeted steps the DPI down until the rounded size fits. UserUnit is still ignored, as in pdfium, libvips and pdftoppm, and a fractional DPI still rounds to a whole one. The migration guide has the recipe for each old constructor.

From the CLI

viprs pyramid and viprs plan take --page-sizing <exact|legacy-truncated>, defaulting to exact. It also applies to --match-page-size.

viprs pyramid document.pdf out/ --dpi 300                                  # 2550x3300 for Letter
viprs pyramid document.pdf out/ --dpi 300 --page-sizing legacy-truncated   # 2549x3299, the 0.5.x size

Migrating from 0.5.x

Nothing fails to compile, but the numbers change: Letter at 300 dpi goes from 2549×3299 to 2550×3300, Tabloid from 3299×5098 to 3300×5100, A3 from 3507×4959 to 3508×4961, A1 from 7015×9932 to 7016×9933, and ARCH D from 7199×10799 to 7200×10800. You only need to act if you have output on disk at the old sizes and will add to it or re-render part of it; anything that reads a finished tree or archive keeps working, because those describe their own size. A plan you computed yourself in f64 already matches the new size, so use PageSizing::Exact.pixel_dims to plan with. To keep the old sizes, pass PageSizing::LegacyTruncated (or --page-sizing legacy-truncated). Those will not match libvips, so use them only for continuity with existing output.

Sizes at 72, 150 and 300 dpi

What PageSizing::Exact gives for the standard sheets, in pixels. I computed these with the rule above (page points widened from f32 as pdfium reports them) and checked them against the libvips pdfload reference set the tests use (135 renders, all matching). Points are shown rounded to three decimals.

ANSI A is Letter and ANSI B is Tabloid (11×17 in). ARCH A to E are 9×12 up to 36×48 in, and ARCH E1 is 30×42 in.

PMTiles v3 Storage (default)

A pyramid is one file now. viprs pyramid scan.tif writes scan.pmtiles, a PMTiles v3 archive holding every tile, its index and its metadata, instead of scattering a quarter of a million small files across a directory tree. PmTilesSink is the default sink. FsSink and the directory tree it writes are still there, behind --storage directory.

The reason is operational, not aesthetic. A deep pyramid is mostly tiny files, and tiny files are what every layer underneath is worst at: an inode each on disk, a request each over the wire, a PUT each into object storage, and a long tail of latency in anything that has to walk them. One archive copies, uploads, versions and deletes as one object, and a viewer still fetches a single tile, because PMTiles indexes tiles by z/x/y and answers with a ranged read.

use libviprs::{EngineBuilder, PmTilesSink, TileFormat};

let sink = PmTilesSink::try_new("atlas.pmtiles", plan.clone(), TileFormat::Png)?;
EngineBuilder::new(&raster, plan, &sink).run()?;

Reading needs no server and no unpacking step. pmtiles::Reader runs against any caller-provided RangeReader backend, and a tile lookup touches the header, a directory and the tile rather than the whole file. Two backends ship: FileRangeReader over a local file, and ObjectStoreRangeReader over any ObjectStore you inject (behind the object-store-sink feature). Neither carries an HTTP or S3 client, and that is the point: the transport is two methods on the trait, only read_range is required, so a consumer that already has the stack writes it against its own client.

use libviprs::pmtiles::Reader;

let reader = Reader::try_open("atlas.pmtiles")?;
let tile = reader.get_tile(3, 4, 5)?;  // Option<Vec<u8>>, None when the archive does not address it

The archive is a published format, not a private one, so a libviprs pyramid opens in any PMTiles-aware viewer or server, and an archive written elsewhere opens here. The writer streams, so peak memory tracks the tile being written rather than the size of the pyramid. The CLI side lives in the viprs pmtiles command group: read a header, pull one tile out, check that an archive holds together, or unpack it back into a tree.

Rasters in a PVec

A scanned underlay, an aerial or the images of a PDF can go into the same .pvec as the linework. libviprs encodes the tiles and PVec stores them as they are: it never decodes a pixel, nothing is resampled on the way, and a 16-bit or palette source is converted before tiling and the report says so. A 2x3 affine places the raster in the drawing's source space, and the georeference behind it is stored once, out of band, in the PMTiles metadata (vnd.libviprs.geo_transform). The format and the reasons are in ADR 0018 in libviprs/pvec.

The placement is composed from the raster's GeoTransform and the drawing's calibration only when the raster is in the calibration's native CRS and the calibration is affine. Anything else is a typed refusal (CrsMismatch, ProjectiveCalibration and the rest), never a silent resample. A PDF's images are baked first: /SMask into alpha, /ImageMask in the fill colour, /Indexed, ICC, Cal and Lab colour into sRGB, /Decode applied and clips into alpha plus a visible box. A DWG IMAGE names a path no reader here opens, so it is a frame in the report and no placement is written.

These are the rules libviprs follows. The table is copied from the page generated in the repository, docs/pvec-raster.md, which a test keeps equal to the code, so go there if the two ever disagree.

Three commands in libviprs-cli put this on the command line: viprs pyramid --storage pvec, viprs pmtiles to-pvec and viprs pvec rasters. They are in the default build (the pvec-raster feature is on by default), so a plain cargo install has them and so do the release binaries from the next release on.

Phase 3 Hardening

Phase 3 adds production-hardening features: durable storage sinks, resumable jobs, retry policies, checksums, deduplication, structured tracing, and versioned manifests. All features are opt-in via Cargo feature flags.

Object-storage sinks (--features s3)

Write tiles directly to an S3-compatible bucket via ObjectStoreSink. The sink runs against any caller-provided ObjectStore backend.

use libviprs::{EngineBuilder, ObjectStoreSink, ObjectStoreConfig, TileFormat};

let sink = ObjectStoreSink::new(store, ObjectStoreConfig::default(), plan.clone())
    .with_format(TileFormat::Png);
EngineBuilder::new(&raster, plan, sink).run().unwrap();

Manifest v1 + versioning

ManifestBuilder produces a ManifestV1 JSON sidecar next to the pyramid root. The manifest carries a schema_version field; readers ignore unknown keys, so older consumers continue working as new fields are added. Attach the builder to an FsSink via with_manifest.

use libviprs::{FsSink, ManifestBuilder, ChecksumAlgo, TileFormat};

let manifest = ManifestBuilder::new(&plan)
    .checksum_algo(ChecksumAlgo::Blake3);
let sink = FsSink::new("output_tiles", plan.clone())
    .with_format(TileFormat::Png)
    .with_manifest(manifest);
// serialises to: {"schema_version":1, "tiles": [...], ...}

Blank-tile tolerance

Use BlankTileStrategy::PlaceholderWithTolerance to treat tiles whose every channel varies by at most max_channel_delta as blank. Useful for slightly noisy scans where pure-uniform detection misses near-white regions.

use libviprs::{BlankTileStrategy, EngineBuilder};

let result = EngineBuilder::new(&raster, plan, sink)
    .with_blank_strategy(BlankTileStrategy::PlaceholderWithTolerance {
        max_channel_delta: 2,
    })
    .run()
    .unwrap();

CLI equivalent: --blank-tolerance 2

Tracing spans (--features tracing)

When the tracing feature is enabled, the engine emits structured spans compatible with any tracing-subscriber, including OpenTelemetry exporters. Span names: libviprs::pipeline, libviprs::level, libviprs::tile.

# Cargo.toml
libviprs = { version = "0.4", features = ["tracing"] }

CLI: --trace-level info (values: error, warn, info, debug, trace).

Resumable jobs

EngineBuilder::with_resume(ResumePolicy) writes a checkpoint file (.libviprs-job.json) at the output root after each tile. A re-run with ResumePolicy::resume() skips already-written tiles; ResumePolicy::verify() re-reads them and asserts their checksums. Both of them want the tree. PmTilesSink keeps no checkpoint, so ResumePolicy::verify() works on an archive too: it validates the container, then confirms the archive holds every coordinate the plan asked for and that none of them is stored empty, which is viprs pmtiles verify plus the plan. ResumePolicy::resume() is the one an archive cannot do, because it keeps no checkpoint, so it finds nothing to skip and regenerates the lot, and the sink builder refuses it by name if you set that mode on the sink. Reach for FsSink (--storage directory) when you want a resumable run, and for viprs pmtiles verify when you want an archive checked.

use libviprs::{EngineBuilder, ResumePolicy};

EngineBuilder::new(&raster, plan, sink)
    .with_resume(ResumePolicy::resume())
    .run()
    .unwrap();

CLI: --resume / --overwrite / --verify

Retry + failure policies

Tile writes that fail transiently (network blip, lock contention) can be retried automatically. FailurePolicy::RetryThenSkip records the missing tile in the manifest instead of aborting the job.

use libviprs::{EngineBuilder, FailurePolicy, RetryPolicy};

let result = EngineBuilder::new(&raster, plan, sink)
    .with_failure_policy(FailurePolicy::RetryThenFail)
    .with_retry(RetryPolicy { max_attempts: 3, backoff_ms: 200 })
    .run()
    .unwrap();

CLI: --retry-max 3 --retry-backoff 200 --failure-policy retry-then-fail

Checksums

The engine can compute a digest for every tile and embed it in the manifest (EmitOnly) or verify existing digests on re-run (Verify).

use libviprs::{ChecksumAlgo, ChecksumMode, FsSink, TileFormat};

let sink = FsSink::new("output_tiles", plan.clone())
    .with_format(TileFormat::Png)
    .with_checksums(ChecksumAlgo::Blake3)
    .with_checksum_mode(ChecksumMode::EmitOnly);

CLI: --manifest-emit-checksums --checksum-algo blake3

Deduplication

Identical tile content is stored once. The engine tries symlink first, then hardlink, then falls back to a manifest-only reference (for sinks that do not support links, such as S3).

use libviprs::{ChecksumAlgo, DedupeStrategy, EngineBuilder};

// Dedupe only blank placeholder tiles (fast, low overhead)
EngineBuilder::new(&raster, plan.clone(), sink)
    .with_dedupe(DedupeStrategy::Blanks)
    .run()
    .unwrap();

// Dedupe every tile using Blake3 content hashes
EngineBuilder::new(&raster, plan, sink)
    .with_dedupe(DedupeStrategy::All { algo: ChecksumAlgo::Blake3 })
    .run()
    .unwrap();

CLI: --dedupe-blanks / --dedupe-all

Packfile archive sinks (--features packfile)

Archive the entire pyramid into a single file instead of a directory tree. Useful for cold storage, artifact upload, or reproducible builds.

# .tar, .tar.gz, and .zip are all supported
viprs pyramid document.pdf --sink packfile://output.tar.gz

Extended EngineResult metrics

The EngineResult struct now carries detailed I/O and timing counters for integration with monitoring systems.

let result = EngineBuilder::new(&raster, plan, sink).run().unwrap();
println!(
    "read {} bytes, wrote {} bytes, {} retries, peak queue {}, {:?}",
    result.bytes_read, result.bytes_written,
    result.retry_count, result.queue_pressure_peak,
    result.duration,
);

Feature Flags

Enable multiple features in Cargo.toml:

libviprs = { version = "0.4", features = ["s3", "tracing", "packfile"] }

Requirements

• Rust 1.99+ (edition 2024)
• libpdfium shared library or static archive (only if using the pdfium / pdfium-static feature) — prebuilt in libviprs-dep releases

Native Dependencies

PDFium is built from source and published as GitHub Releases under libviprs/libviprs-dep. Every release tag ships four archives covering the full matrix of {linux, musl} × {amd64, arm64}, and each archive contains both the shared library (libpdfium.so) and the static archive (libpdfium.a):

Loading a glibc .so from a musl process — or vice versa — fails at dlopen time, so match the archive libc to the runtime you deploy against. Fully-static musl binaries that cannot dlopen should link libpdfium.a via pdfium-render/static.

The build tooling is documented in full at libviprs-dep/MANUAL.md (man-page-style reference: CLI, options, artifact layout, environment, exit statuses, troubleshooting). A pipeline-level overview lives at libviprs-dep/pdfium/README.md.

Related Crates