Charport Package Developer Guide

Overview

At its core, charport is a small contract between ALTREP producer and consumer packages. ALTREP producers expose byte views to other packages using one safe read-only interface, and consumers read those byte views from the other side of the interface. A package can register ALTREP strings, consume ALTREP strings, or use charvec, a built-in ALTREP class, directly. The diagram below shows how those roles are organized in this package.

The guide goes through three main use cases:

• Using the Universal Reader: read ALTREP strings through one C++ interface.
• Registering an ALTREP Class: make an ALTREP string class readable by other packages through charport.
• Using charvec Directly: use the built-in charvec ALTREP class directly with serial or multithreaded builders.

charport can be used from C or from C++11 and later. Its error model was designed and tested with particular care so that R errors and C++ exceptions are propagated properly from both languages. This guide explains the relevant rules alongside the APIs they govern; the Error handling vignette gives a more complete design and rationale.

Linking to charport

In DESCRIPTION, declare both the runtime dependency and the header dependency:

Imports:
    charport
LinkingTo:
    charport

In NAMESPACE, import charport so its runtime entry points are available:

Namespace Setup

• roxygen2
• NAMESPACE

#' @useDynLib ExamplePackage, .registration = TRUE
#' @import charport
"_PACKAGE"

useDynLib(ExamplePackage, .registration = TRUE)
import(charport)

There is one public header: charport.h. If you use Rcpp or cpp11, you should include that framework’s header first.

The same header can be included from C. Its C interface exposes ABI checks, reader borrows, vector inspection, and complete-vector charvec construction through R’s R_GetCCallable(). C++ packages also get the Reader class, charvec builders, and ALTREP registration. The C++ interface requires C++11 or higher.

The main entry points have direct C and C++ counterparts:

Operation	C	C++
Check the loaded ABI	charport_abi_version() == CHARPORT_ABI_VERSION	charport::check_abi()
Acquire a reader borrow	charport_resolve(x)	charport::Reader(x)
Release a raw reader borrow	charport_reader_release(&reader)	RAII destruction in charport::Reader
Inspect a string vector	charport_get_sexp_info(x)	charport::sexp_info(x)
Build a charvec	charport_charvec_from_views(...)	charvec::Builder

Using the Universal Reader

The universal reader provides the same read-only interface for both ordinary (non-ALTREP) strings and ALTREP strings. For ordinary strings, there is no per-element overhead. The C++ class charport::Reader is a class wrapper around the C struct charport_reader. The C++ class can be constructed as follows:

#include "charport.h"

charport::Reader reader(robj); // use if you do not use Rcpp or cpp11
charport::Reader rcpp_reader = charport::Reader::with_rcpp(robj); // use if your package uses Rcpp; include Rcpp.h first
charport::Reader cpp11_reader = charport::Reader::with_cpp11(robj); // use if your package uses cpp11; include cpp11.hpp first

Since the reader constructor calls into R, it can generate an R error (for example, if you pass an INTSXP instead of a STRSXP). Therefore, the three C++ constructors differ in error handling. The first generic constructor allows the R error to propagate, and the caller should handle it manually. The with_rcpp constructor wraps the R error in Rcpp::unwindProtect and uses Rcpp’s unwind mechanics. Similarly, the with_cpp11 constructor wraps the R error in cpp11::unwind_protect. Other than error handling they behave identically.

A Reader can also be created without a borrow and initialized later with reset():

charport::Reader reader;
reader.reset(robj);

This form lets a caller place the Reader in an outer C++ scope before resolving the input. reset() follows the same R error policy as Reader(robj). A successful reset replaces the current borrow. If resolution raises an R error, the existing borrow remains unchanged. Do not use the access methods before the first successful reset.

The corresponding C structure is charport_reader initialized with charport_resolve().

#include "charport.h"

charport_reader c_reader = charport_resolve(robj);

Unlike charport::Reader, this is an ordinary struct with no class methods. charport_resolve() can generate an R error.

Every successful charport_resolve() should be paired with charport_reader_release():

charport_reader_release(&c_reader);

Range Access

Range access is the main path for processing a full vector or a contiguous chunk. The pattern below demonstrates two access patterns for reading pointer and length for each string.

• C++ example
• C example

#include "charport.h"

static void process_strings_impl(SEXP x) {
  charport::Reader input(x);
  const R_xlen_t n = input.size();

  charport::ByteViews values(n);
  input.byteviews(0, n, values);

  const char ** ptrs = values.ptrs();
  const int * lengths = values.lengths();
  for(R_xlen_t i = 0; i < n; ++i) {
    if(lengths[i] != NA_INTEGER) {
      // use ptrs and lengths arrays
    }

    charport::ByteView el = values[i];
    // Process el.ptr and el.len
  }
}

#include <stdlib.h>
#include "charport.h"

static void process_strings_impl(SEXP x) {
  charport_reader input = charport_resolve(x);
  const R_xlen_t n = input.n;
  const char ** ptrs = (const char **) malloc((size_t) n * sizeof *ptrs);
  int * lengths = (int *) malloc((size_t) n * sizeof *lengths);

  int status = input.range.byteviews(input.state, 0, n, ptrs, lengths);
  if(status != CHARPORT_STATUS_OK) {
    free(lengths);
    free(ptrs);
    charport_reader_release(&input);
    Rf_error("charport reader access failed with status %d", status);
  }

  for(R_xlen_t i = 0; i < n; ++i) {
    if(lengths[i] != NA_INTEGER) {
      /* Process ptrs[i] and lengths[i]. */
    }
  }

  free(lengths);
  free(ptrs);
  charport_reader_release(&input);
}

Missing strings are reported as ptr = nullptr and len = NA_INTEGER. String pointers are not guaranteed to be null-terminated, so always read pointer and length together.

Besides byteviews(), there are several additional access methods:

• views() reads pointers, byte lengths, and encoding marks.
• lengths() reads byte lengths only.
• encodings() reads encoding marks only.

Each method has range, indexed, and scalar forms.

Scalar Reader Access

Scalar methods are convenience functions for single-element reads. For example, to read the first element:

charport::Reader r(robj);
charport::StrView s = r.view(0);
if(!s.is_na()) {
  // s.ptr, s.len, s.enc
}

r.byteview(i), r.length(i), and r.encoding(i) return the narrower scalar forms. operator[] is an alias for r.view(i).

Indexed Reader Access

Indexed access gathers arbitrary 0-based positions into output arrays. This matches R’s index selection (but is zero-based):

charport::Reader r(robj);
std::vector<R_xlen_t> idx = {0, 10, 20};

charport::ByteViews byteviews(idx.size());
r.byteviews(idx.data(), idx.size(), byteviews); // outputs elements 0, 10 and 20

Encoding

charport passes through the same string encoding information available from base R strings. It doesn’t do any string normalization or reinterpretation; it only makes the encoding marks available as defined by the producer.

Value	Meaning
CETYPE_EXT_NATIVE	Native encoding based on machine locale.
CETYPE_EXT_UTF8	UTF-8.
CETYPE_EXT_LATIN1	Latin-1.
CETYPE_EXT_BYTES	Bytes encoding.
CETYPE_EXT_ASCII_OR_UTF8	ASCII/UTF-8-compatible bytes.
CETYPE_EXT_ASCII	ASCII bytes, equivalent to Rf_charIsASCII(x).
CETYPE_EXT_NA	Missing string. ptr is NULL and len is NA_INTEGER.

The names are the same in both languages. In C++ they are typed cetype_ext_t constants comparable with ==; in C they are plain integers comparable with .value.

R’s encoding scheme is aggregated into this single type. CETYPE_EXT_ASCII, CETYPE_EXT_ASCII_OR_UTF8, and CETYPE_EXT_NA are simplifications: CETYPE_EXT_ASCII corresponds to the ASCII mark available from Rf_charIsASCII(). In R, the ASCII mark always occurs with CETYPE_EXT_NATIVE, so it represents a mutually exclusive state.

CETYPE_EXT_ASCII_OR_UTF8 exists because base R materialization always checks whether a string is ASCII, so differentiating ASCII and UTF-8 early is often redundant work.

CETYPE_EXT_NA carries the missing-string case alongside the pointer and length.

Ordinary cetype_t values correspond to cetype_ext_t values, so you can wrap R’s encoding mark when you already have a cetype_t:

cetype_ext_t enc = charport_cetype_ext(static_cast<uint8_t>(Rf_getCharCE(x)));

That cast preserves the encoding marks for native, UTF-8, Latin-1 and byte strings.

Reader Lifetime and Multithreading

In Rust terms, initializing the Reader is a strict borrow. The Reader is a temporary view of the vector’s current string storage, valid only while that storage remains protected and unmodified. Access to and modification of the original R object (STRING_ELT, STRING_PTR_RO, DATAPTR_RO, etc.) can invalidate the Reader.

An exception is attributes; you can handle attributes on the original R object while Reader exists.

Reader makes available several class-specific capability flags related to multithreading and memory lifetime:

• persistent_views() means returned pointers remain valid after another Reader call.
• concurrent_access() means Reader access may run concurrently on worker threads.
• reentrant() is true when both capabilities are available.

The fallback Reader path (on ordinary R strings) has persistent_views() but not concurrent_access().

Registering an ALTREP Class

A package can register its ALTREP class to make non-materializing views available to charport::Reader. Registration can happen directly in package initialization, or from .onLoad().

ALTREP registration is a C++ API:

void charport::register_altrep(
  R_altrep_class_t cls,
  charport_reader_state_fns state_fns,
  charport_reader_range_fns range_fns,
  charport_reader_index_fns index_fns,
  charport_reader_capabilities capabilities
);

The inputs are grouped structs of related parts:

struct charport_reader_state_fns {
  charport_reader_init_fn init;
  charport_reader_release_fn release;
};

struct charport_reader_range_fns {
  charport_reader_strviews_range_fn strviews;
  charport_reader_byteviews_range_fn byteviews;
  charport_reader_lengths_range_fn lengths;
  charport_reader_encodings_range_fn encodings;
};

struct charport_reader_index_fns {
  charport_reader_strviews_index_fn strviews;
  charport_reader_byteviews_index_fn byteviews;
  charport_reader_lengths_index_fn lengths;
  charport_reader_encodings_index_fn encodings;
};

struct charport_reader_capabilities {
  bool persistent_views;
  bool concurrent_access;
};

init(SEXP x) starts the reader and returns the state used by the access callbacks. release cleans up that state when the Reader is destroyed and may be nullptr when no cleanup is needed. init should not allow C++ exceptions to fall through, but instead convert them to R errors.

Returning NULL from init means this particular vector cannot satisfy the reader contract, so charport falls back to default access.

While charport asks for many callback functions during registration, they are all pretty similar and can be built from the same boilerplate. Range callbacks (charport_reader_range_fns) fill output arrays for the 0-based interval [start, start + size):

int range_strviews(
    void * state, R_xlen_t start, R_xlen_t size,
    const char ** out_ptrs, int * out_lens, cetype_ext_t * out_encs);

int range_byteviews(
    void * state, R_xlen_t start, R_xlen_t size,
    const char ** out_ptrs, int * out_lens);

int range_lengths(
    void * state, R_xlen_t start, R_xlen_t size,
    int * out_lens);

int range_encodings(
    void * state, R_xlen_t start, R_xlen_t size,
    cetype_ext_t * out_encs);

Similarly, indexed callbacks (charport_reader_index_fns) take const R_xlen_t * indices in place of start; everything else matches the range forms.

Bounds are a caller precondition. For a reader of length n, a range request must satisfy 0 <= start <= n and 0 <= size <= n - start. This admits an empty range at start == n and avoids an overflowing start + size calculation. An indexed request must have a nonnegative size, and every requested index must satisfy 0 <= indices[j] < n. Scalar C++ accessors have the same index requirement.

Reader forwards requests without checking these bounds. The providers shipped with charport do not check them either, so an invalid request to a built-in provider has undefined behavior. A provider may validate requests and return CHARPORT_STATUS_OUT_OF_RANGE, but callers cannot rely on that validation.

Missing strings should be reported as ptr = nullptr, len = NA_INTEGER, and enc = CETYPE_EXT_NA.

Access callbacks cross a C ABI, so they report errors through the return value. Return CHARPORT_STATUS_OK after filling the outputs. On failure, return a nonzero status. The output arrays are unspecified after failure.

The other defined status values are:

Status	Meaning
CHARPORT_STATUS_ERROR	The access failed for another reason.
CHARPORT_STATUS_NO_MEMORY	Native allocation failed.
CHARPORT_STATUS_OUT_OF_RANGE	The provider rejected an index or range.

A C++ provider can translate exceptions using the convert_current_exception_to_status() helper function, which maps C++ exceptions to their corresponding error statuses:

int range_lengths(/* callback arguments */) {
  try {
    // Fill the output array.
    return CHARPORT_STATUS_OK;
  } catch(...) {
    return charport::convert_current_exception_to_status();
  }
}

convert_current_exception_to_status() must be called from inside a catch handler. It maps std::bad_alloc to CHARPORT_STATUS_NO_MEMORY, std::out_of_range to CHARPORT_STATUS_OUT_OF_RANGE, and every other exception to CHARPORT_STATUS_ERROR.

Registration during package initialization

Register after creating the ALTREP class in the package initialization hook:

• Rcpp
• cpp11

#include <Rcpp.h>
#include "charport.h"

static R_altrep_class_t example_altrep;

// [[Rcpp::init]]
void init_ExamplePackage(DllInfo * dll) {
  if(!charport::check_abi()) {
    (Rf_error)("ExamplePackage was built against an incompatible charport ABI");
  }

  example_altrep = R_make_altstring_class(
    "example_altstring", "ExamplePackage", dll
  );

  charport::register_altrep(
    example_altrep,
    charport_reader_state_fns{reader_init, reader_release},
    charport_reader_range_fns{
      range_strviews,
      range_byteviews,
      range_lengths,
      range_encodings
    },
    charport_reader_index_fns{
      index_strviews,
      index_byteviews,
      index_lengths,
      index_encodings
    },
    charport_reader_capabilities{true, true}
  );
}

extern "C" void R_unload_ExamplePackage(DllInfo * dll) {
  charport::unregister_altrep(example_altrep);
  (void)dll;
}

#include <cpp11.hpp>
#include "charport.h"

static R_altrep_class_t example_altrep;

[[cpp11::init]]
void init_ExamplePackage(DllInfo * dll) {
  if(!charport::check_abi()) {
    (Rf_error)("ExamplePackage was built against an incompatible charport ABI");
  }

  example_altrep = R_make_altstring_class(
    "example_altstring", "ExamplePackage", dll
  );

  charport::register_altrep(
    example_altrep,
    charport_reader_state_fns{reader_init, reader_release},
    charport_reader_range_fns{
      range_strviews,
      range_byteviews,
      range_lengths,
      range_encodings
    },
    charport_reader_index_fns{
      index_strviews,
      index_byteviews,
      index_lengths,
      index_encodings
    },
    charport_reader_capabilities{true, true}
  );
}

extern "C" void R_unload_ExamplePackage(DllInfo * dll) {
  charport::unregister_altrep(example_altrep);
  (void)dll;
}

Using charvec Directly

charvec is charport’s built-in general-purpose ALTREP string class. It is an efficient drop-in representation that you can use for many applications, especially for large string output.

Each charvec is backed by a charvec::Store behind its R external pointer. The store owns two things:

1. SliceChain, a linked list of large contiguous byte slices holding the string data.
2. RecordTable, the per-string metadata: a pointer into the SliceChain, a byte length, and an encoding mark.

Conceptually:

class Store {
public:
  SliceChain slices;
  RecordTable records;
};

class RecordTable {
  std::unique_ptr<const char *[]> ptrs;  // one pointer per string
  std::unique_ptr<int[]> lens;           // length of each string
  std::unique_ptr<cetype_ext_t[]> encs;  // encoding of each string
  size_t vector_length;
  size_t capacity;
};

For non-empty strings, ptrs[i] points into one of the byte slices. Empty strings point at shared empty storage. Missing strings are stored as ptrs[i] = nullptr, lens[i] = NA_INTEGER, and encs[i] = CETYPE_EXT_NA.

Builder: basic fixed-length construction

charport has three utility classes for building a charvec. charvec::Builder is for the simple case: the output length is known at construction time and one serial producer fills elements by index. The builder packs strings into shared slices; its first slice scales with the vector length up to 16 KiB, and later slices grow up to 256 KiB. A string larger than that cap gets an exact-fit slice.

A string can be inserted into a builder with either set() or reserve().

Use set() when you have a buffer that only needs to be copied over.

charvec::Builder b(n);
b.set(i, ptr, len, CETYPE_EXT_UTF8);

Use reserve() when the output bytes can be written directly into the store:

char * dst = b.reserve(i, len, CETYPE_EXT_UTF8);
// write to dst directly

Call to_sexp() on the main R thread when the vector is done building: the Builder gives up its storage to the returned ALTREP SEXP. to_sexp() cannot throw a C++ exception, but can theoretically raise an out-of-memory R allocation error during the wrapping of Store. Packages can instead use the conversion methods Builder::to_sexp_with_rcpp() and Builder::to_sexp_with_cpp11() to adapt an R error to the framework’s C++ exception type.

Builder::reset(n) reuses the same builder object for a subsequent vector. This is useful when one operation returns several string vectors of known length.

If construction finishes on a worker thread, you can instead use release_store() to release a charvec::Store by value. This object can then later be wrapped to an ALTREP SEXP:

// worker thread
charvec::Store store = b.release_store();

// On the main R thread
SEXP out = charvec::wrap(std::move(store));

charvec::wrap() has the same error behavior as Builder::to_sexp(): it cannot throw a C++ exception, but may raise an R error while creating the ALTREP object. Similarly, you can use charvec::wrap_with_rcpp() or charvec::wrap_with_cpp11() to adapt an R error.

ParallelBuilder: sharded construction

You can use charvec::ParallelBuilder when several workers fill disjoint ranges of one fixed-length vector, via RcppParallel, RcppThread, OpenMP, etc. It keeps separate slice state for each shard while sharing the final record table, and then combines the slices on the main thread.

charvec::ParallelBuilder b(n, n_workers);

// worker_id is 0 .. n_workers-1
parallel_for_each_worker(worker_id, n_workers) {
  R_xlen_t begin = n * worker_id / n_workers;
  R_xlen_t end = n * (worker_id + 1) / n_workers;

  for(R_xlen_t i = begin; i < end; ++i) {
    char * dst = b.reserve(worker_id, i, len[i], CETYPE_EXT_UTF8);
    // write this worker's bytes into dst
  }
}

// main thread
SEXP out = b.to_sexp();

GrowableBuilder: discovered-length construction

Use charvec::GrowableBuilder when the output length is not known ahead of time. This is like building a std::vector by sequential push_back().

charvec::GrowableBuilder b;

b.append(view);
b.append(ptr, len, CETYPE_EXT_UTF8);

char * dst = b.append_reserve(len, CETYPE_EXT_UTF8);
// write to dst directly

size_t n = b.size();
SEXP out = b.to_sexp();

append() copies bytes. append_reserve() adds one record and returns its writable storage. The record table grows geometrically as elements are appended.

Using Store directly

charvec::Store is a public struct with records and slices. The constructor allocates a fixed record table and, when initial_slice_bytes is nonzero, one slice of storage. The caller must keep every record pointer, length, encoding, and slice consistent.

Example:

charvec::Store store(3, total_bytes);

char * data = store.slices.front_data();
std::memcpy(data, src, total_bytes);

store.records.set(0, data + offset0, len0, CETYPE_EXT_UTF8);
store.records.set(1, data + offset1, len1, CETYPE_EXT_LATIN1);
store.records.set_na(2);

SEXP out = charvec::wrap(std::move(store));

Allocate another slice with store.slices.push_front(bytes). It returns the new slice’s data pointer.

For a one-element result, Store::scalar() copies the bytes and fills record zero:

charvec::Store store = charvec::Store::scalar(ptr, len, CETYPE_EXT_UTF8);
SEXP out = charvec::wrap(std::move(store));

Building charvec from C

C callers can construct a charvec from complete pointer, length, and encoding arrays with charport_charvec_from_views().

const char * ptrs[] = {"alpha", NULL, ""};
const int lengths[] = {5, NA_INTEGER, 0};
const cetype_ext_t encodings[] = {
  CETYPE_EXT_ASCII,
  CETYPE_EXT_NA,
  CETYPE_EXT_ASCII
};

SEXP out = charport_charvec_from_views(3, ptrs, lengths, encodings);

ABI Compatibility

Every package compiled against charport should call check_abi() during package loading, before registering a class or using Reader and Builder entry points. The registration examples above include the check. A consumer-only package should make the same call from its initialization hook.

In C, compare charport_abi_version() with CHARPORT_ABI_VERSION instead.