Driver API

The API provides functions to query the system for available hardware accelerators, identify their capabilities (e.g., core count, supported data types), and establish a context for execution. This involves:

• Enumerating devices on the PCIe or system bus.
• Opening a handle to a specific device.
• Querying device properties via get_info-style calls.
• Initializing necessary runtime contexts and command queues.

A critical function is managing the movement of data between host (CPU) and device (NPU) memory spaces. The API allocates and deallocates device memory buffers and provides mechanisms for efficient data transfer.

• Device Memory Allocation: Functions like malloc_device() allocate buffers in NPU High-Bandwidth Memory (HBM).
• Unified Memory: Some APIs support managed memory, where a single pointer is accessible from both host and device, with data migration handled automatically.
• Explicit Transfers: Commands like memcpy_host_to_device() and memcpy_device_to_host() perform direct, synchronous, or asynchronous data copies over the system bus (e.g., PCIe).

The core function is submitting computational tasks (kernels) to the hardware. The API packages kernel code, arguments, and execution configuration into a command that is placed on a queue for the device to process.

• Kernel Arguments: Setting pointers and scalar values for the kernel's parameters.
• Launch Configuration: Defining the grid and block dimensions (for GPU-like architectures) or task graph dependencies (for dataflow NPUs).
• Command Queues: Work is submitted to asynchronous queues (compute, copy). The API provides synchronization primitives like events and barriers to manage dependencies between queued operations.

Because execution is asynchronous, the API provides mechanisms to synchronize host application logic with device operations. This ensures data is ready before being read and resources are not prematurely reused.

• Events: Create, record on a queue, and wait for events to signal operation completion.
• Stream Barriers: Insert dependencies within a command queue.
• Device Synchronization: Functions like device_synchronize() block the host thread until all previously submitted commands on a device have completed.
• Callbacks: Some APIs allow registering host functions to be called asynchronously upon event completion.

Robust error codes and status query functions are essential for debugging and building resilient applications. The API returns specific error codes for invalid arguments, out-of-memory conditions, or device faults.

• Function Return Codes: Every API call typically returns a status code (e.g., SUCCESS, ERROR_INVALID_VALUE).
• Asynchronous Error Checking: Errors from kernel execution may be reported asynchronously; the API provides functions to retrieve these errors.
• Device Status: Queries to check if a device is lost or in an error state, allowing for graceful application recovery or restart.

The Driver API must integrate with broader system software. Key functions include:

• Graph Compiler Integration: Accepting pre-compiled kernel binaries or graphs from a vendor's graph compiler.
• Runtime Library Handoff: Interfacing with a higher-level Vendor Runtime for graph scheduling and memory pooling.
• Foreign Handle Interop: Exchanging memory handles (like DMA-BUF file descriptors on Linux) with other APIs (e.g., OpenCL, Vulkan) for zero-copy data sharing between different accelerators or with display drivers.

A vendor-specific Software Development Kit that provides the foundational libraries, headers, compilers, and debugging tools required to build applications for a hardware accelerator like an NPU. It typically includes the Driver API as its core runtime component, along with higher-level frameworks and utilities for model optimization and deployment.

• Purpose: Abstracts hardware complexity for application developers.
• Components: Includes the driver, compiler toolchain, profilers, and sample code.
• Example: NVIDIA's CUDA Toolkit or Intel's oneAPI Base Toolkit for their respective accelerators.

The operating system kernel module that provides direct, privileged control over a physical hardware device. The Driver API is the user-space interface exposed by this kernel driver.

• Role: Manages physical resources (memory, interrupts, DMA), enforces security/isolation, and handles low-level command submission.
• Location: Executes in kernel space (Ring 0 on x86).
• Interaction: User-space applications call the Driver API, which makes system calls (e.g., ioctl) to communicate requests to the kernel driver.

A vendor-provided library that manages the execution environment and lifecycle for workloads on the accelerator. It sits above the Driver API, offering convenience functions for common tasks.

• Functions: Handles device discovery, context management, memory allocation/pinning, and asynchronous kernel execution queues.
• Abstraction: Provides a more object-oriented or task-graph based interface than the raw Driver API.
• Example: The CUDA Runtime API is a higher-level wrapper over the lower-level CUDA Driver API.

A thin software layer that provides a uniform interface to hardware-specific functionalities, insulating higher-level software (like a framework) from the details of the underlying Driver API and hardware.

• Purpose: Enables portability of frameworks across different vendor hardware.
• Implementation: Translates framework operations (e.g., 'launch a convolution') into the appropriate sequence of Driver API calls for the specific NPU in use.
• Example: A machine learning framework's 'device backend' for different NPUs is a form of HAL.

The low-level contract between separately compiled binary modules, defining how functions are called and data is structured in memory. The Driver API conforms to a specific ABI.

• Scope: Includes calling conventions (register usage, stack management), data type sizes/alignment, and name mangling.
• Importance: Ensures binary compatibility. User-space applications compiled against the Driver API's headers must match the ABI of the driver's shared library (e.g., libcuda.so).
• Stability: A stable Driver API ABI allows applications to run with newer driver versions without recompilation.

A non-standard, vendor-specific application programming interface. A Driver API is almost always a Proprietary API, as it provides direct access to the unique architectural features of a vendor's NPU.

• Characteristics: Lock-in to a specific hardware vendor. Documentation and support are controlled by the vendor.
• Contrast with Open Standards: Unlike cross-vendor standards (e.g., OpenCL, Vulkan), a proprietary Driver API allows for deeper optimization and access to cutting-edge hardware features.
• Trade-off: Maximizes performance for a specific platform at the cost of portability.