Skip to contents

Design Rationale

Source: vignettes/design-rationale.qmd

Overview

charport is trying to fill in a gap between ALTREP string producers and the packages that consume their vectors. ALTREP gives the producer control over storage, but it does not provide a general way for another package to read that data in bulk. The consumer either uses R’s string API, which may materialize the vector, or writes a separate integration for each ALTREP class.

charport supplies a small shared contract instead. A producer can expose byte views and state how long they remain valid and whether they may be accessed concurrently. A consumer uses the same Reader interface for registered ALTREP classes and ordinary R character vectors, with R’s standard access path as the fallback.

Range access first

Range access fits R’s vectorized style. charport uses range and indexed callbacks instead of scalar callbacks, with separate entry points for string views, byte views, lengths, and encodings. The producer fills caller-owned arrays for the shape the consumer requested. This gives consumers a bulk path without forcing either side to allocate one wrapper object per element. It also lets producers optimize the common contiguous case with memcpy when their storage is already arranged that way.

No pointer-only access

A const char * returned by charport is a byte pointer, not necessarily a null-terminated C string. Without the byte length, a consumer cannot know where the string ends, and calling strlen() may read past the view. Requiring every provider to add a terminator would rule out useful storage layouts or force a copy solely to satisfy the access API.

For that reason, byteviews() returns pointers and lengths together. A length is useful by itself for tasks such as sizing an output buffer, so lengths() also has its own entry point. A pointer by itself is not a complete view, so there is no corresponding pointer-only entry point.

Reader fallback

Reader first uses ordinary R storage when it is already available: DATAPTR_OR_NULL(x) gives a direct pointer array without forcing. That path has persistent views, but it is still R-managed storage, so charport does not advertise concurrent access.

If the vector is ALTREP and its class is registered, Reader asks the class to start a borrow. A registered class may still decline a particular vector by returning NULL from its init callback. In that case, or when the ALTREP class is not registered, Reader falls back to STRING_PTR_RO(x).

That fallback is intentionally plain R behavior. It may materialize the ALTREP, but it preserves correctness for every character vector. charport is an escape hatch from materialization when a producer allows it.

Construction and access use different errors

Reader construction is an R-facing operation. A registered provider may call R during initialization, and the fallback may allocate while materializing an ALTREP vector. Reader construction therefore allows R errors, and consumers must handle them. The Reader itself does not control any state before resolution succeeds, so it does not itself need cleanup after an R error.

Rcpp and cpp11 consumers can choose named construction adapters when they need an R error converted to the exception type used by their wrapper. Effectively, this allows the reader to be an extension of Rcpp and cpp11.

Access has different constraints. It may run on worker threads, where calling R is forbidden, and a realistic provider may need to allocate. Access callbacks return a plain integer status and Reader converts a failure status to a standard C++ exception.

Capability bits

The two capabilities record the promises a consumer actually needs after a bulk read.

Reader::persistent_views() answers a practical question: can the returned byte pointers survive the next reader call? If false, use a pointer before calling the Reader again. If true, views can be collected and processed later while the borrow is active.

Reader::concurrent_access() answers a slightly different question: whether reader callbacks may run on multiple threads, including worker threads.

Reader::reentrant() is therefore derived, not stored: it is the case where both promises are true.

Why cetype_ext_t

charport carries encoding marks because bytes alone are not enough to reconstruct R strings correctly. Rather than use R’s cetype_t directly, it uses an equivalent and efficient one-byte type for string data. It is a one-byte struct rather than an enum so that both C and C++ can use the same type.

cetype_ext_t provides a small, fixed representation. It takes up less space using one byte (instead of R’s plain int enum), carries comprehensive encoding information, and includes CETYPE_EXT_NA so an encoding view can carry missingness without consulting separate parts.

It also distinguishes ASCII with CETYPE_EXT_ASCII. R stores an ASCII bit for every string. When it is set, it always co-occurs with CETYPE_EXT_NATIVE, so CETYPE_EXT_ASCII represents a mutually exclusive state.

CETYPE_EXT_ASCII_OR_UTF8 lets a producer report if a string is UTF-8-compatible without scanning bytes to decide whether the string is also ASCII. R makes that check when it creates a CHARSXP, so doing it in the producer can duplicate work. Base R always checks if a string is ASCII during materialization or CHARSXP creation. So forcing ASCII vs. UTF-8 resolution would frequently become a redundant calculation if the string data source doesn’t distinguish between ASCII and UTF-8. Data sources outside of R usually don’t, and R will determine the resolution later anyway.

cetype_ext_t is complete for string data: it represents every encoding state R uses for CHARSXPs, along with CETYPE_EXT_ASCII, CETYPE_EXT_ASCII_OR_UTF8, and CETYPE_EXT_NA. The two omitted cetype_t values, CE_SYMBOL and CE_ANY, are unused as encoding states for CHARSXPs.

charvec: efficient and flexible ALTREP storage

charvec is both a reference implementation and a useful output class. Its storage is shaped for the reader contract: three contiguous metadata arrays for pointers, byte lengths, and encoding marks, plus owned byte slices that hold the payload.

The separate metadata arrays make contiguous reads cheap. For charvec, a range request can copy pointers, lengths, and encodings directly into the consumer’s output arrays.

The string data is stored in slices instead of one giant buffer or one allocation per string. One giant buffer requires knowing the final byte size up front and is awkward for streaming input. On the other hand, one allocation per string is expensive and fragmented.

Slices give a middle ground: append-friendly construction, stable pointers during a read-only borrow and fewer allocation calls.

The three different builder classes correspond to common vector building patterns:

  • Builder handles ordinary serial output when strings arrive one at a time.
  • ParallelBuilder gives each worker its own slice chain while sharing the final metadata arrays. That avoids concurrent writes to the same byte buffer while still producing one charvec.
  • GrowableBuilder handles serial output whose final vector length is not yet known at the start.

Builder conversion creates an external pointer with a null address, registers its finalizer, and creates the ALTREP object. It moves the Store and installs the address only after those R allocations succeed. On failure, the caller’s Store remains intact, so to_sexp_with_rcpp() and to_sexp_with_cpp11() can unwind without losing it.

When a charvec is fully materialized, it caches an ordinary STRSXP in ALTREP data2 and releases the charvec::Store held in data1. The reader then naturally uses the ordinary fallback path over the cached STRSXP.