Week 11 · AI on FPGAs · Lesson 23 of 24

AI inference on FPGAs: INT8 quantisation and systolic arrays

Run a quantised network layer in FPGA fabric: INT8 scales and zero points, requantisation with a multiplier and shift, systolic MAC arrays, hls4ml and FINN.

advanced1 hUpdated Fact-checked Free

What you will learn

  • Explain when an FPGA is a good inference target in terms of latency, determinism and custom precision
  • Quantise a fully connected layer to INT8 with per-channel weight scales and an INT32 bias
  • Implement requantisation with a fixed-point multiplier and a rounding shift, bit-exact with a Python model
  • Describe weight-stationary and output-stationary dataflows and map a layer onto an N x N systolic array
  • Estimate cycles and on-chip memory bandwidth for a layer and choose between hand-written RTL, hls4ml, FINN and vendor AI tools

Before you start

  • FPGA memory and DSP: block RAM, FIFOs and MACs (lesson 14)
  • Verification with cocotb (lesson 10)
  • INT8 quantisation basics, covered in the Robotics and AI course lesson on deployment

After this lesson you can take a small trained network layer, quantise it to 8-bit integers, and run it in FPGA fabric with results that match a Python model bit for bit. You will build the two blocks every INT8 accelerator needs: a systolic array of multiply-accumulate (MAC) units and a requantisation stage that turns 32-bit sums back into 8-bit activations.

The arithmetic builds on the MAC and fixed-point material in memory and DSP. The quantisation formulas themselves are introduced in the Robotics and AI lesson on deployment and safety. Read its INT8 section first if scale and zero point are new to you. This lesson starts where that one stops: what the integer arithmetic looks like in hardware.

Why run inference on an FPGA

A CPU or GPU runs a network as a sequence of instructions over memory. An FPGA can instead lay the network out as a pipeline of dedicated multipliers, adders and RAMs. Three properties follow.

  • Latency. With every layer in fabric and the weights in on-chip memory, there is no instruction fetch, no operating system and no off-chip memory round trip. The hls4ml case study by Duarte et al. fitted a jet-classification network for a particle-physics trigger on an FPGA with a latency of about 100 ns.
  • Determinism. A pipeline built from registers has a fixed latency in clock cycles. The systolic array in this lesson always produces a tile 2N - 1 cycles after its last input beat. A control loop or a trigger system can budget for that number exactly.
  • Custom precision. Fabric multipliers can be any width. hls4ml lets you choose the fixed-point width. FINN goes to the extreme: Umuroglu et al. map binarised networks, with 1-bit weights and activations, onto XNOR and popcount logic.

The costs are longer design time and a less flexible deployment: a model that changes every week is easier to run on a processor. FPGAs pay off when latency, power or determinism matter most, or when the inference sits next to other FPGA logic such as a camera pipeline.

INT8 arithmetic for hardware

The affine mapping

Jacob et al. define the quantisation scheme adopted in TensorFlow Lite. A real value rr is represented by an integer qq with a scale SS and a zero point ZZ:

r=S (q−Z)r = S\,(q - Z)

The current LiteRT (TensorFlow Lite) 8-bit specification fixes the details that matter for hardware:

  • Activations: INT8 in [−128,127][-128, 127], asymmetric, so the zero point can be anywhere in that range.
  • Weights: INT8 in [−127,127][-127, 127], symmetric with Z=0Z = 0, and one scale per output channel (“per-axis”) for convolutions.
  • Bias: INT32 with scale SxSwS_x S_w and zero point 0, so it adds straight into the accumulator.

Per-channel weight scales cost almost nothing in hardware. The scale only appears at requantisation, once per output, and the requantisation unit takes its multiplier as an input.

The integer layer

A fully connected layer computes yn=∑kwnkxk+bny_n = \sum_k w_{nk} x_k + b_n. Substituting the quantised forms (with Zw=0Z_w = 0):

yn=SxSw,n(∑kqw,nk (qx,k−Zx)+qb,n)y_n = S_x S_{w,n} \Big( \sum_k q_{w,nk}\,(q_{x,k} - Z_x) + q_{b,n} \Big)

Expand the bracket. The term Zx∑kqw,nkZ_x \sum_k q_{w,nk} depends only on the weights, so it is a constant per output channel. Jacob et al. use the same expansion to keep zero points out of the inner loop. Fold it into the bias offline:

accn=∑kqw,nk qx,k⏟INT8×INT8 in the array+qb,n−Zx∑kqw,nk⏟biasn, INT32\text{acc}_n = \underbrace{\sum_k q_{w,nk}\, q_{x,k}}_{\text{INT8} \times \text{INT8 in the array}} + \underbrace{q_{b,n} - Z_x \sum_k q_{w,nk}}_{\text{bias}_n\text{, INT32}}

The array now multiplies plain INT8 numbers. Each product fits in 16 bits: the largest magnitude is (−128)×(−128)=16384(-128) \times (-128) = 16384. An INT32 accumulator therefore holds ⌊(231−1)/214⌋=131071\lfloor (2^{31} - 1) / 2^{14} \rfloor = 131071 worst-case products without overflow, far more than the 64 inputs of the example.

Requantisation: multiplier and shift

The output is INT8 again, with its own scale SyS_y and zero point ZyZ_y:

qy,n=Zy+Mn⋅accn,Mn=SxSw,nSyq_{y,n} = Z_y + M_n \cdot \text{acc}_n, \qquad M_n = \frac{S_x S_{w,n}}{S_y}

MnM_n is a real number. Jacob et al. report that it is always in (0,1)(0, 1) in practice and write it as M=2−nM0M = 2^{-n} M_0 with M0∈[0.5,1)M_0 \in [0.5, 1). Stored as a 32-bit integer, M0⋅231M_0 \cdot 2^{31} is at least 2302^{30}, so it keeps at least 30 bits of relative precision. The multiplication by 2−n2^{-n} becomes a shift that must round to nearest.

In this lesson m0 is the integer round(M0⋅231)\text{round}(M_0 \cdot 2^{31}) and shift is nn. The whole requantisation is:

q=clamp⁡(⌊(acc+bias)⋅m0 / 231+n+12⌋+Zy, lo, 127)q = \operatorname{clamp}\Big( \big\lfloor (\text{acc} + \text{bias}) \cdot m_0 \,/\, 2^{31+n} + \tfrac{1}{2} \big\rfloor + Z_y,\ \text{lo},\ 127 \Big)

with lo=−128\text{lo} = -128, or lo=Zy\text{lo} = Z_y to apply a ReLU (real value 0 maps to ZyZ_y). The rounding here is round half up. Frameworks differ in how they break exact ties. If you need bit-exact agreement with a specific framework, copy its rounding function, and test it the way the cocotb tests below do.

def quantize_multiplier(m: float) -> tuple[int, int]:
    frac, exp = math.frexp(m)          # m = frac * 2^exp, 0.5 <= frac < 1
    m0 = round(frac * (1 << 31))
    if m0 == 1 << 31:                  # rounding reached 1.0: renormalise
        m0 //= 2
        exp += 1
    return m0, -exp                    # shift = -exp

def requantize(acc, bias, m0, shift, zp, relu=False) -> int:
    p = (acc + bias) * m0
    r = (p + (1 << (30 + shift))) >> (31 + shift)   # round half up
    return max(zp if relu else -128, min(127, r + zp))

requant.sv

The RTL has three pipeline stages. Stage 1 adds the bias in 33 bits, so acc + bias cannot overflow. Stage 2 forms the full 65-bit product. Stage 3 adds the rounding constant, shifts, adds the zero point and saturates:

always_comb begin
    half    = 65'sd1 <<< (30 + int'(sh2));
    shifted = (prod2 + half) >>> (31 + int'(sh2));
    y       = shifted + 65'(zp2);
    lo      = relu2 ? 65'(zp2) : -65'sd128;
end

always_ff @(posedge clk) begin
    if (y > 65'sd127)  q <= 8'sd127;
    else if (y < lo)   q <= lo[7:0];
    else               q <= y[7:0];
end

Because m0, shift and zp are inputs, one instance serves all output channels: feed each accumulator together with its channel’s parameters. The 33 x 32-bit multiply is larger than one DSP slice (25 x 18 bits in AMD 7-series devices, up to 27 x 27 in Intel variable-precision blocks). Synthesis splits it across several slices, and you may want an extra pipeline register for timing at high clock rates.

Download requant.sv

Systolic arrays

H. T. Kung’s 1982 paper explains systolic architectures, a concept first proposed for VLSI implementation of matrix operations, around one observation: if a data item is fetched from memory once and then used by many processing elements as it is pumped from cell to cell, throughput can rise without more memory bandwidth. His Figure 1 makes the point with numbers: a memory with a 100 ns cycle feeding one processing element limits it to 5 million operations per second; the same memory feeding a chain of six cells allows 30 million.

Dataflows

Matrix multiplication is the natural fit. Each output Cij=∑kAikBkjC_{ij} = \sum_k A_{ik} B_{kj} needs a row of A and a column of B, and every A value is used by every column. The question is what stays still in each processing element (PE). Sze et al. name the common choices:

  • Weight stationary: each weight is loaded into a PE once and stays there. Inputs stream through, and partial sums move from PE to PE. This minimises the energy spent reading weights. Kung’s design B1 (“broadcast inputs, move results, weights stay”) is an early example.
  • Output stationary: each PE keeps the partial sum of one output. Inputs and weights stream past. This minimises the energy of reading and writing partial sums. Kung’s design B2 (“broadcast inputs, move weights, results stay”) is of this kind.

The reason the choice matters is energy and bandwidth. In the memory hierarchy figure of Sze et al., a DRAM access is shown at about 200 times the energy of a register-file access next to the arithmetic unit. Every byte that stays in a PE register is a byte not fetched from further away.

The array in this lesson

systolic_array.sv is an output-stationary N x N array. It computes one N x N tile of C=A×BC = A \times B, with INT8 operands and INT32 accumulators.

A 3 by 3 grid of processing elements PE(0,0) to PE(2,2), each holding an accumulator c_ij. Elements of A enter each row from the left, with row 1 delayed by one idle slot and row 2 by two. Elements of B enter each column from the top, with column 1 delayed by one slot and column 2 by two. Arrows pass a values to the right and b values downwards. A side box lists what each PE does per clock: if valid, accumulate a times b, or load a times b on the first beat, pass a and the flags right, pass b down, and keep the result on the last beat.
Output-stationary systolic array for C = A x B. Skewing row i of A and column j of B by i and j cycles makes A[i][k] and B[k][j] meet in PE(i, j) on the same clock edge.

Each clock, column kk of A enters on the left and row kk of B on the top. Row ii is delayed by ii cycles and column jj by jj cycles. A value then needs jj more cycles to reach column jj, and a B value ii more cycles to reach row ii, so both reach PE(i,j)(i, j) exactly i+ji + j cycles after they entered. Flags for valid, first and last travel with the A values. The core of each PE:

assign prod = a_h[i][j] * b_v[i][j];                         // 16-bit signed
assign sum  = (f_h[i][j][1] ? 32'sd0 : acc) + 32'(prod);   // first beat restarts

always_ff @(posedge clk) begin
    if (f_h[i][j][0]) begin                 // valid
        acc <= sum;
        if (f_h[i][j][2]) res[i][j] <= sum; // last: keep the result
    end
end

The data registers have no reset, as in the MAC of lesson 14; only the flag pipeline is reset. PE(N−1,N−1)(N-1, N-1) finishes last, 2N−12N - 1 cycles after the last beat, and its done flag drives out_valid. The interface rule follows: a new tile’s last beat must come at least 2N−12N - 1 cycles after the previous one, or PE(0,0)(0,0) overwrites its result before out_valid. For K≥2N−1K \ge 2N - 1 tiles can run back to back with no idle cycles.

Download systolic_array.sv

Bandwidth

Per cycle the array reads 2N2N INT8 operands and performs N2N^2 MACs. Each operand is used NN times. For N=16N = 16 that is 32 bytes per cycle for 256 MACs. Those bytes come from block RAM, so the design question becomes how many block RAM ports you need and how wide they are. Kung’s point applies directly: a larger array needs more compute but its memory traffic grows only with NN.

The other memory question is capacity. A fully parallel design such as the hls4ml case study keeps every weight on chip, in the programmable logic. If the weights do not fit, they stream from external memory, and the external bandwidth sets the speed. Count the weight bytes before choosing an architecture.

Worked example: an INT8 digit classifier

int8_layer.py builds a small, reproducible problem. It draws ten 8 x 8 seven-segment style digit glyphs, then makes noisy copies with random one-pixel shifts, random stroke intensity and Gaussian noise (σ=0.2\sigma = 0.2), all from a fixed seed. There are 1500 training images and 500 test images. No external dataset is downloaded.

Two rows of ten 8 by 8 pixel images. The top row shows clean seven-segment style glyphs for the digits 0 to 9. The bottom row shows one noisy, shifted test image for each digit.
Synthetic 8 x 8 digit dataset: the ten clean glyphs (top) and one noisy test image per class (bottom).

A linear classifier (64 inputs, 10 outputs) is trained in floating point with NumPy. The script then quantises it as described above:

  • inputs: pixels in [0,1][0, 1] with Sx=1/255S_x = 1/255 and Zx=−128Z_x = -128,
  • weights: per-channel symmetric INT8,
  • bias: INT32 with the input zero point folded in,
  • outputs: INT8 with a scale and zero point calibrated on the training logits.
$ python int8_layer.py
test samples: 500
float accuracy: 0.932   int8 accuracy: 0.928   same class: 0.996
input  scale 1/255, zero point -128
output scale 0.08733, zero point 8
weight scales: [0.02519 0.01719 0.02143 0.01809 0.02133 0.02594 0.02525 0.01962 0.01412
 0.01767]
channel 0: M = 1.131267e-03 -> m0 = 1243841297, shift = 9
largest |accumulator| = 241526

The INT8 layer picks the same class as the float layer for 99.6 % of the test images. Accuracy drops by 0.4 percentage points. The largest accumulator, 241526, uses 19 bits of the 32 available.

Scatter plot of INT8 logit error against float logit for 5000 values. The errors spread roughly evenly between about -0.14 and +0.12 across the whole logit range from -10 to +10. A shaded band of plus or minus 0.044 marks half an output step.
Error of the dequantised INT8 logits against the float logits on the test set. Output rounding alone gives at most half a step (shaded band); rounding the inputs and weights adds the rest.

Mapping the layer onto the array

The layer is a matrix product: a batch of images XqX_q (M x 64) times the transposed weights Wq⊤W_q^\top (64 x 10). On a 4 x 4 array, 8 images and 10 outputs (padded to 12 with zero weights) make 2 x 3 = 6 tiles. Each tile streams K=64K = 64 beats, so the 6 tiles take 384 cycles plus 7 cycles of latency, and the array does 16 MACs per cycle. The accumulators then go through requant.sv, one per cycle, with each channel’s bias, m0 and shift.

Verification

The cocotb tests compare the RTL with the Python model bit for bit.

  • cocotb_systolic.py sends 60 random tiles with KK from 1 to 40, about 30 % extreme values (−128-128, 127127 and others) and random idle cycles. It also tests tiles at the minimum spacing K=2N−1K = 2N - 1, the largest-magnitude cases, and the classifier layer itself, checking every accumulator against int_matmul.
  • cocotb_requant.py checks 4360 values, including the INT32 limits for acc and bias, the extreme m0 and shift values and both ReLU settings, against requantize. It then checks that the classifier accumulators of 40 images produce exactly the INT8 logits of the Python model.
cd public/code/fpga-design/ai-inference-on-fpgas
/Volumes/M2-Dev/conda/envs/toptech/bin/python -m pytest -q
# 11 passed

The pytest wrapper builds each module with Icarus Verilog in a temporary directory and fails if any cocotb test fails. Removing the rounding constant or the first restart from the RTL makes the suite fail.

Download int8_layer.py Download test_int8_layer.py Download cocotb_systolic.py Download cocotb_requant.py Download test_cocotb_ai.py

Tools that generate the hardware

Hand-written RTL gives full control and shows what the hardware does. For whole networks, generators save a lot of work.

  • hls4ml translates trained models into high-level synthesis (HLS) code. Duarte et al. describe the flow for Keras and PyTorch models, fixed-point arithmetic with a configurable width, and a “reuse factor” RR: each multiplier is used RR times per layer, so R=1R = 1 is fully parallel and larger RR trades latency for fewer multipliers. The current documentation lists Keras, PyTorch and ONNX front ends (with quantised variants such as Brevitas and QONNX) and several back ends, including AMD Vivado/Vitis, Altera and Catapult.
  • FINN, from AMD’s research labs, generates a dataflow architecture tailored to each quantised network. The original paper used a heterogeneous streaming architecture, with compute resources per layer sized to a throughput target and parameters kept on chip. On a ZC706 board it reported up to 12.3 million MNIST classifications per second at 0.31 µs latency for a binarised network.
  • Vendor AI tools usually take a different route: a fixed, configurable neural-network processor plus a compiler, rather than a custom circuit per network. AMD describes Vitis AI as an inference development platform for its adaptive SoCs, with a compiler, a neural processing unit (NPU) IP, runtime software and the Quark quantiser; the current release lists general access support for the Versal AI Edge Series and its Gen 2. Microchip’s VectorBlox Accelerator SDK compiles quantised INT8 TFLite models for AI/ML inference on PolarFire FPGAs and PolarFire SoC FPGAs. Supported device lists change between releases, so check the vendor page.

Whichever tool you use, the checks in this lesson still apply: confirm the accumulator width, the rounding of the requantisation, and the accuracy of the quantised model on held-out data.

Exercise

Requantise by hand

An output channel has M=0.0123M = 0.0123 and output zero point Zy=8Z_y = 8. Find m0 and shift. Then requantise an accumulator of 10000 with bias −500-500. Finally requantise an accumulator of −20000-20000 (bias 0) without and with ReLU.

Show solution

math.frexp(0.0123) gives 0.7872×2−60.7872 \times 2^{-6}, so shift = 6 and m0 = round(0.7872×231)(0.7872 \times 2^{31}) = 1690499128. Check: 1690499128/237=0.01230001690499128 / 2^{37} = 0.0123000.

acc+bias=9500\text{acc} + \text{bias} = 9500. The real product is 9500×0.0123=116.859500 \times 0.0123 = 116.85, which rounds to 117. Adding ZyZ_y gives q=125q = 125. The integer path: 9500×1690499128=16 059 741 716 0009500 \times 1690499128 = 16\,059\,741\,716\,000; adding 2362^{36} and shifting right by 37 gives 117. requantize(10000, -500, 1690499128, 6, 8) returns 125.

For −20000-20000: −20000×0.0123=−246-20000 \times 0.0123 = -246, plus 8 is −238-238, which saturates to −128-128. With ReLU the lower clamp is Zy=8Z_y = 8, so q=8q = 8, which dequantises to exactly 0.

Exercise

Size an array for a larger layer

A layer has 784 inputs and 128 outputs. You process batches of 16 images on a 16 x 16 output-stationary array with the interface of systolic_array.sv. How many tiles are there, how many cycles does a batch take, and how many weight bytes must be on chip? Assume a 200 MHz clock and give the batch time.

Show solution

Tiles: ⌈16/16⌉×⌈128/16⌉=8\lceil 16/16 \rceil \times \lceil 128/16 \rceil = 8, each with K=784K = 784 beats. Since K≥2N−1=31K \ge 2N - 1 = 31, the tiles run back to back: 8×784=62728 \times 784 = 6272 cycles, plus 31 cycles until the last out_valid, so 6303 cycles. At the assumed 200 MHz that is about 31.5 µs per batch of 16 images.

Check with MAC counts: 16×784×128=1 605 63216 \times 784 \times 128 = 1\,605\,632 MACs, divided by 256 MACs per cycle, is 6272 cycles. The array is fully used while streaming.

Weights: 784×128=100 352784 \times 128 = 100\,352 bytes (about 98 KiB) of INT8, plus 128 INT32 biases and the per-channel m0 and shift. Compare this with the block RAM of your device. Each cycle reads 16 weight bytes and 16 input bytes, so the weight store must deliver 128 bits per cycle.

Summary

  • FPGAs suit inference when latency, determinism, power or custom precision matter more than peak throughput.
  • INT8 inference keeps INT8 x INT8 products in INT32 accumulators. Symmetric per-channel weights and an asymmetric input zero point folded into the INT32 bias keep the inner loop a plain integer MAC.
  • Requantisation multiplies by M=SxSw/SyM = S_x S_w / S_y as an integer m0 in Q0.31 and a rounding right shift, then adds the output zero point and saturates.
  • Systolic arrays reuse each fetched operand N times. Weight-stationary arrays keep weights in the PEs; output-stationary arrays, like the one here, keep partial sums.
  • Verify against an integer Python model bit for bit, including extreme values, idle cycles and the tile spacing rule.
  • hls4ml and FINN generate per-network hardware; vendor AI suites compile networks onto a configurable processor IP.

Check your understanding

Pick an answer for each question, then check. Answers stay hidden until you submit.

1. Why does the requantisation step multiply by an integer M0 and then shift, instead of multiplying by the real scale ratio M?
2. In the output-stationary array of this lesson, what stays inside each processing element for the whole tile?
3. The input zero point is z_x = -128. Where does the term -z_x times the sum of a weight row go in this lesson's design?
4. An 8 x 8 output-stationary array streams column k of A and row k of B each cycle. How many INT8 operands does it read per cycle, and how many MACs does it do?
5. Your design uses the systolic array with N = 4. Two tiles with K = 3 must run back to back. What does the interface rule of systolic_array.sv require?

Further reading

Found a mistake or something unclear?Report a correction.