Vendor ISA

The core set of machine-level instructions that the processor or accelerator can execute. For an NPU, this typically includes specialized instructions for tensor operations (e.g., matrix multiplication, convolution), data movement, and control flow. Each instruction is encoded as a binary opcode that the hardware decodes and executes.

Defines the architectural registers (small, fast storage locations) available to programmers and the data types they can hold. Key components include:

• General-Purpose Registers (GPRs): For scalar values and addresses.
• Tensor/Vector Registers: Wide registers designed to hold multi-dimensional data (e.g., 8x8 FP16 matrices).
• Special Registers: Status, control, and configuration registers (e.g., for setting execution modes).
• Supported Data Types: Precision formats like INT4, INT8, FP16, BF16, and FP32.

Specifies how instructions reference memory. This includes:

• Addressing Modes: The methods for calculating operand addresses (e.g., immediate, register indirect, indexed).
• Address Space: Defines separate spaces for code, data, and I/O.
• Alignment Requirements: Rules for how data must be aligned in memory for efficient access.
• Coherence & Consistency Models: Rules for how memory accesses by multiple threads or cores are ordered and made visible.

Defines the fundamental model of program execution and how parallel work is organized. This encompasses:

• Threading Model: The hierarchy of threads, warps/wavefronts, and thread blocks.
• SIMD/SIMT Execution: How Single Instruction, Multiple Data (or Thread) parallelism is exposed and managed.
• Synchronization Primitives: Instructions for barriers, atomic operations, and memory fences to coordinate parallel execution.
• Predication & Masking: Mechanisms for conditionally executing instructions across vector lanes.

A set of special registers used to configure, control, and monitor the hardware state. Software uses CSRs to:

• Enable/disable hardware features or execution modes.
• Configure memory protection units or cache behavior.
• Read performance counters (e.g., cycles, instructions retired, cache misses).
• Handle interrupts and exceptions by accessing cause and status registers.

Defines the mechanism by which the hardware responds to asynchronous events (interrupts) and synchronous error conditions (exceptions/traps). This includes:

• Exception Types: What events cause an exception (e.g., illegal instruction, memory fault, division by zero).
• Interrupt Vector Table: The location in memory of handler routine addresses.
• Context Saving: Which processor state is automatically saved upon an exception.
• Privilege Levels: Different modes of operation (e.g., user vs. supervisor) that restrict certain instructions.

The most common entry point is a Vendor SDK, which provides high-level libraries and frameworks (e.g., TensorFlow or PyTorch integrations). Developers write model code in a familiar framework, and the SDK's graph compiler translates it into a sequence of vendor-specific operations. This abstracts the underlying ISA but may limit access to unique hardware features. Key components include:

• Graph compilers that optimize and lower neural network graphs.
• Runtime libraries for memory management and kernel scheduling.
• Profiling tools to analyze performance bottlenecks.

For finer control, developers use hardware intrinsics. These are special functions or data types in a high-level language (like C/C++) that map directly to specific ISA instructions, such as a tensor multiply-accumulate (MAC) operation. Compiler intrinsics are provided by the vendor's compiler to expose these low-level operations without writing assembly. This allows for:

• Explicit use of SIMD (Single Instruction, Multiple Data) and tensor cores.
• Manual optimization of critical computational kernels.
• Control over data types (e.g., FP16, INT8) and memory access patterns.

The lowest-level method is writing inline assembly code within a C/C++ program or authoring pure assembly files. This grants the developer absolute control over the instruction stream, enabling:

• Hand-optimized loops and register allocation.
• Direct use of esoteric or proprietary instructions not exposed via intrinsics.
• Cycle-accurate tuning for latency-critical sections. This approach is complex, non-portable, and typically reserved for vendor library developers or performance architects creating foundational kernels.

A vendor toolchain is the suite of tools that transforms source code into executable binaries for the target ISA. It includes:

• A cross-compiler (e.g., clang or gcc with a vendor backend) that generates machine code from C/C++/intrinsics.
• An assembler that processes assembly source files.
• A linker that combines object files, resolves symbols, and creates the final binary using a linker script to define the NPU's memory map.
• Tools for generating fat binaries containing code for multiple architectures (e.g., host CPU and NPU).

The compiled code interacts with the hardware through a software stack defined by an Application Binary Interface (ABI). This low-level contract governs:

• The calling convention: how functions pass arguments and return values between host CPU and NPU kernels.
• Binary format (e.g., ELF sections) and symbol table structure.
• Interaction with the Vendor Runtime and Driver API. The runtime manages device execution, while the driver handles resource allocation and command submission via the OS kernel. The ABI ensures the linked binary can be correctly loaded and executed by the vendor's software stack.

Vendors often supplement their ISA with proprietary APIs and closed-source SDKs. Developers interface with these through:

• Vendor header files that declare function prototypes for proprietary libraries.
• Static libraries (.a files) or dynamic libraries (.so/.dll files) that are linked into the application. These libraries contain pre-optimized kernels (e.g., for convolution or attention) that are effectively black boxes.
• Stub libraries may be used during development to simulate hardware. This model provides high performance but creates vendor lock-in and limits transparency.

Low-level programming constructs that map directly to specific machine instructions of a processor, allowing developers to access hardware features like SIMD or tensor operations without writing assembly. They provide a high-level language interface (e.g., in C/C++) to low-level hardware capabilities, offering a balance of performance and portability.

• Example: An intrinsic function like _mm256_add_ps for an x86 AVX instruction.
• Purpose: Enables manual optimization for critical loops while maintaining compiler-managed register allocation.

A vendor-specific software development kit that provides the essential toolchain for programming an accelerator. It typically includes:

• Compiler and assembler targeting the Vendor ISA.
• Libraries of optimized kernels (e.g., for matrix multiplication).
• Profiling and debugging tools.
• Runtime libraries for device management.

An SDK abstracts the raw ISA but is built directly upon it, serving as the primary interface for most application developers.

The low-level contract between separately compiled software modules, defining how they interoperate at the machine level. For a Vendor ISA, the ABI specifies:

• Calling conventions: How functions pass arguments and return values using the ISA's registers and stack.
• Data structure layout: Alignment and padding rules for data types.
• Binary file format (e.g., ELF sections) for object code.

The ABI ensures that code compiled by one tool can correctly call functions from a library compiled by another, even if they both target the same ISA.

The complete suite of vendor-specific build tools required to transform source code into an executable binary for the target hardware. Its core components are:

• Compiler: Translates high-level code (C/C++) to Vendor ISA assembly.
• Assembler: Converts assembly mnemonics to machine code (object files).
• Linker: Combines multiple object files and libraries into a final executable, resolving symbols based on the ABI.
• Debugger (e.g., GDB port) and Profiler for performance analysis.

This toolchain is often built around a cross-compiler that runs on a standard x86 host system.

The software layers that manage the accelerator's execution environment. While the ISA defines the instructions, these layers handle their execution.

• Driver API/Kernel Driver: The lowest-level software in the OS kernel that directly controls the hardware, managing memory, interrupts, and command queues.
• Vendor Runtime: A user-space library that provides a higher-level interface to the driver. It handles tasks like:
  • Device initialization and context management.
  • Memory allocation and transfer between host and device.
  • Scheduling and launching kernels (blocks of ISA code) on the accelerator.
  • Synchronization (e.g., waiting for kernel completion).

A software layer that provides a uniform, standardized interface to hardware-specific functionalities. Its role relative to a Vendor ISA is critical:

• Insulates higher-level frameworks (like TensorFlow or PyTorch) from the details of any single Vendor ISA.
• Defines a common set of operations (e.g., GEMM, Convolution) that each vendor implements by mapping them to their own ISA-optimized kernels.
• Enables portability: The same model can, in theory, run on different NPUs because the framework calls the HAL, which dispatches to the vendor's ISA-specific implementation.

It sits above the Vendor SDK, translating generic compute requests into vendor-specific API calls.