axelera.runtime API

The low-level Python API for loading compiled models and running them directly on the AIPU. This gives you fine-grained control over device selection, buffer management, and multi-instance execution.

For most applications, the InferenceStream API is simpler and handles all of this automatically. Use axelera.runtime when you need direct tensor-level access to the hardware.

import axelera.runtime as axr

Typical usage​

import numpy as np
import axelera.runtime as axr

with axr.Context() as ctx:
    devices = ctx.list_devices()
    conn = ctx.device_connect(devices[0])

    model = ctx.load_model("build/yolov8n-coco/yolov8n-coco/1/model.json")
    instance = conn.load_model_instance(model, aipu_cores=4, double_buffer=True)

    # Prepare inputs as numpy arrays
    inputs = [np.zeros(t.unpadded_shape, dtype=np.float32) for t in model.inputs()]
    outputs = [np.zeros(t.shape, dtype=np.int8) for t in model.outputs()]

    instance.run(inputs, outputs)

Context​

The root object. Manages all devices and resources. Use as a context manager or call release() explicitly.

ctx = axr.Context()
# or:
with axr.Context() as ctx:
    ...

Methods​

configure_device properties​

Connection​

A reserved connection to a device (or sub-device group). Created by ctx.device_connect().

Methods​

load_model_instance kwargs​

Model​

Represents a loaded compiled model. Created by ctx.load_model().

Methods​

Properties​

ModelInstance​

A model loaded onto a specific device connection. Created by conn.load_model_instance().

Methods​

inputs and outputs are lists of numpy arrays (or DMA buffer file descriptors if dmabuf mode is enabled). Shapes must match model.inputs() / model.outputs(). Raises an exception on failure.

TensorInfo​

Describes one input or output tensor, including quantization and padding metadata.

Quantize an input:

t = model.inputs()[0]
src = np.zeros(t.unpadded_shape, dtype=np.float32)
quant = np.round((src / t.scale) + t.zero_point).clip(-128, 127).astype(np.int8)
padded = np.pad(quant, t.padding, constant_values=t.zero_point)

Dequantise an output:

t = model.outputs()[0]
out = np.zeros(t.shape, dtype=np.int8)
# (populate out from instance.run())
depadded = out[tuple(slice(b, -e if e else None) for b, e in t.padding)]
dequant = (depadded.astype(np.float32) - t.zero_point) * t.scale

DeviceInfo​

Returned by ctx.list_devices().

Exceptions​

See also​

• InferenceStream API — higher-level Python API (recommended for most applications)
• axrunmodel — CLI tool built on the same runtime
• Model Formats — the model.json file this API loads