6.1. Execution Throughput#

A typical microarchitecture decomposes instructions into microoperations. The microoperations are issued to pipelines that execute them. Depending on the microoperation, it may be executed by one of several pipelines. Typically, an execution pipeline can accept one new microoperation per cycle.

Execution throughput is the maximum number of instructions in a given instruction group that a microarchitecture can execute per cycle. This assumes ideal conditions where there are no resource conflicts, data dependencies, or memory delays. For example, assume that a microarchitecture has an execution throughput of 4 for the Neon FMLA (vector) instruction. This means that under ideal conditions, we obtain the results of four FMLA (vector) instructions per cycle.

Table 6.1.1 shows the theoretical floating-point throughput of an example core. The 3.3 GHz is the nominal frequency of the NVIDIA Grace Superchip system microbenchmarked below. It is slightly higher than the 3.1 GHz listed in the corresponding datasheet. The execution throughput of the three example instructions is expressed in instructions per cycle and meets the microarchitecture specifications outlined in Table 3-16 of the Neoverse V2 Software Optimization Guide.

The number of operations (ops) for a given instruction that a microarchitecture can execute per cycle is given as:

In the case of FMLA (vector), we may be interested in FP32 arithmetic instructions with arrangement specifier 4S. In this case, each instruction performs a fused-multiply add on four-element vectors, resulting in 8 floating-point operations per instruction. This gives us the throughput $32=4\cdot 8$ for Neoverse V2 in terms of operations per cycle.

We get the operations per second by also considering the clock frequency:

For example, in the case of a Grace core and FMLA (vector) with arrangement specifier 4S, we get $105.6\text{GOPS}=3.3\text{GHz}\cdot 32\text{ops/cycle}$. This means that a core can perform 105.6 FP32 giga (${10}^9$) operations per second using FMLA (vector) instructions under ideal conditions. Of all the instructions available, this is also the highest theoretical throughput for FP32 arithmetic. We refer to this number as the core’s theoretical FP32 peak performance.

So far we have discussed instruction throughput from a theoretical point of view. We will now discuss how to write microbenchmarks that allow us to measure execution throughput. To maximize instruction throughput, we must 1) utilize all available pipelines and 2) keep the pipelines full. In summary, we design the throughput microbenchmarks to execute many independent instructions.

 1loop_bench:
 2    sub x0, x0, #1
 3
 4    .rept 100
 5    fmla  v0.4s,  v8.4s, v16.4s
 6    fmla  v1.4s,  v9.4s, v17.4s
 7    fmla  v2.4s, v10.4s, v18.4s
 8    fmla  v3.4s, v11.4s, v19.4s
 9
10    fmla  v4.4s, v12.4s, v20.4s
11    fmla  v5.4s, v13.4s, v21.4s
12    fmla  v6.4s, v14.4s, v22.4s
13    fmla  v7.4s, v15.4s, v23.4s
14
15    fmla  v8.4s, v16.4s, v24.4s
16    fmla  v9.4s, v17.4s, v25.4s
17    fmla v10.4s, v18.4s, v26.4s
18    fmla v11.4s, v19.4s, v27.4s
19
20    fmla v12.4s, v20.4s, v28.4s
21    fmla v13.4s, v21.4s, v29.4s
22    fmla v14.4s, v22.4s, v30.4s
23    fmla v15.4s, v23.4s, v31.4s
24
25    fmla v16.4s, v24.4s,  v0.4s
26    fmla v17.4s, v25.4s,  v1.4s
27    fmla v18.4s, v26.4s,  v2.4s
28    fmla v19.4s, v27.4s,  v3.4s
29
30    fmla v20.4s, v28.4s,  v4.4s
31    fmla v21.4s, v29.4s,  v5.4s
32    fmla v22.4s, v30.4s,  v6.4s
33    fmla v23.4s, v31.4s,  v7.4s
34
35    fmla v24.4s,  v0.4s,  v8.4s
36    fmla v25.4s,  v1.4s,  v9.4s
37    fmla v26.4s,  v2.4s, v10.4s
38    fmla v27.4s,  v3.4s, v11.4s
39
40    fmla v28.4s,  v4.4s, v12.4s
41    fmla v29.4s,  v5.4s, v13.4s
42    fmla v30.4s,  v6.4s, v14.4s
43    fmla v31.4s,  v7.4s, v15.4s
44    .endr
45
46    cbnz x0, loop_bench

Listing 6.1.1 shows the loop of a microbenchmark that measures the execution throughput of FMLA (vector) using the 4S arrangement specifier. The benchmark consists of two parts: a loop and a block of code that is executed at each iteration of the loop. The loop decrements the value of X0 at each iteration (line 2) and continues iterating only if the value in X0 is not zero (line 46). The value in X0 is the only function parameter of the microbenchmark, allowing the surrounding scope to determine how many times the code block is executed.

The code block itself consists of 3200 instructions, realized by a subblock of 32 instructions (lines 5-43), repeated 100 times by the .rept and .endr directives in lines 4 and 44. Looking at the subblock, we see that the first eight instructions in lines 5-13 are completely independent. The following eight instructions in lines 15-23 have write-after-read dependencies with respect to the previous eight instructions in lines 5-13. For example, the ninth instruction fmla v8.4s, v16.4s, v24.4s in line 15 writes to register V8.4s, which is read by the instruction fmla v0.4s, v8.4s, v16.4s in line 5. Write-after-read dependencies are typically resolved by the renaming capabilities of the microarchitecture.

The first true dependencies (read-after-write) occur in the second half of the 32-instruction subblock. For example, the fmla v16.4s, v24.4s, v0.4s instruction in line 25 reads from source register V0.4s, which is written by the fmla v0.4s, v8.4s, v16.4s instruction in line 5. Read-after-write dependencies cannot be resolved by the microarchitecture, meaning that the corresponding result must be available before the instruction with the dependent read can be executed. Thus, this situation can become a bottleneck if we run out of independent instructions to keep the pipelines busy. The number of cycles required before the result of an instruction is available to other instructions is called latency and is discussed in Section 6.2.

Using the microbenchmark structure in Listing 6.1.1 to measure the execution throughput of FMLA (vector) and FMUL (scalar) on Grace (Neoverse V2), we obtain the performance shown in Table 6.1.2. In summary, our microbenchmarks are able to achieve the execution throughput we expected from theoretical considerations.

fmla  v0.4s, v30.4s, v31.4s
fmla  v1.4s, v30.4s, v31.4s
// [...]
fmla v29.4s, v30.4s, v31.4s

6.2. Execution Latency#

An instruction’s execution latency is the minimum number of processor cycles between the time an instruction’s microoperations are issued into the execution pipeline and the time its result is available – via write-back or forwarding – to a dependent instruction. Again, as with execution throughput, we assume ideal conditions.

As discussed in Section 6.1, we need independent instructions to fully utilize all available execution pipelines. On the other hand, if the instructions are interdependent, our workload can become latency-bound. Suppose we have $n$ instructions $I_1$ to $I_n$ and each instruction depends on the previous one. This means that we have the dependency chain $I_1\to I_2\to \dots \to I_{n-1}\to I_n$. In this case, we lose core-level parallelism in two ways. First, we can only use one execution pipeline at a time. Second, pipeline parallelism is either completely inhibited or significantly reduced. More specifically, if the instruction $I_k$ requires $I_{k-1}$ to write the result back to the register file, we cannot use pipeline parallelism. However, if $I_{k-1}$ can share the required results with $I_k$ via late forwarding, then at least a limited amount of pipeline parallelism is possible. Late forwarding lets an earlier microoperation forward its result directly to a dependent microoperation without waiting for register-file write-back. We can now write microbenchmarks for both cases.

 1loop_bench:
 2    sub x0, x0, #1
 3
 4    .rept 100
 5    fmla v0.4s, v0.4s,  v1.4s
 6    fmla v0.4s, v0.4s,  v2.4s
 7    fmla v0.4s, v0.4s,  v3.4s
 8    fmla v0.4s, v0.4s,  v4.4s
 9
10    fmla v0.4s, v0.4s,  v5.4s
11    fmla v0.4s, v0.4s,  v6.4s
12    fmla v0.4s, v0.4s,  v7.4s
13    fmla v0.4s, v0.4s,  v8.4s
14
15    fmla v0.4s, v0.4s,  v9.4s
16    fmla v0.4s, v0.4s, v10.4s
17    fmla v0.4s, v0.4s, v11.4s
18    fmla v0.4s, v0.4s, v12.4s
19
20    fmla v0.4s, v0.4s, v13.4s
21    fmla v0.4s, v0.4s, v14.4s
22    fmla v0.4s, v0.4s, v15.4s
23    fmla v0.4s, v0.4s, v16.4s
24
25    fmla v0.4s, v0.4s, v17.4s
26    fmla v0.4s, v0.4s, v18.4s
27    fmla v0.4s, v0.4s, v19.4s
28    fmla v0.4s, v0.4s, v20.4s
29
30    fmla v0.4s, v0.4s, v21.4s
31    fmla v0.4s, v0.4s, v22.4s
32    fmla v0.4s, v0.4s, v23.4s
33    fmla v0.4s, v0.4s, v24.4s
34
35    fmla v0.4s, v0.4s, v25.4s
36    fmla v0.4s, v0.4s, v26.4s
37    fmla v0.4s, v0.4s, v27.4s
38    fmla v0.4s, v0.4s, v28.4s
39
40    fmla v0.4s, v0.4s, v29.4s
41    fmla v0.4s, v0.4s, v30.4s
42    fmla v0.4s, v0.4s, v31.4s
43    .endr
44
45    cbnz x0, loop_bench

Listing 6.2.1 shows a microbenchmark loop for the write-back case using FMLA (vector) with arrangement specifier 4S as an example. The dependency chain is given by means of Neon register V0. All instructions use V0 as their first source and destination register. In the case of Neoverse V2, Table 3-16 of the Software Optimization Guide specifies an execution latency of 4 cycles for this case. This means that we can expect to execute 0.25 instructions per cycle for the microbenchmark.

 1loop_bench:
 2    sub x0, x0, #1
 3
 4    .rept 100
 5    fmla v0.4s,  v1.4s,  v9.4s
 6    fmla v0.4s,  v2.4s, v10.4s
 7    fmla v0.4s,  v3.4s, v11.4s
 8    fmla v0.4s,  v4.4s, v12.4s
 9
10    fmla v0.4s,  v5.4s, v13.4s
11    fmla v0.4s,  v6.4s, v14.4s
12    fmla v0.4s,  v7.4s, v15.4s
13    fmla v0.4s,  v8.4s, v16.4s
14
15    fmla v0.4s,  v9.4s, v17.4s
16    fmla v0.4s, v10.4s, v18.4s
17    fmla v0.4s, v11.4s, v19.4s
18    fmla v0.4s, v12.4s, v20.4s
19
20    fmla v0.4s, v13.4s, v21.4s
21    fmla v0.4s, v14.4s, v22.4s
22    fmla v0.4s, v15.4s, v23.4s
23    fmla v0.4s, v16.4s, v24.4s
24
25    fmla v0.4s, v17.4s, v25.4s
26    fmla v0.4s, v18.4s, v26.4s
27    fmla v0.4s, v19.4s, v27.4s
28    fmla v0.4s, v20.4s, v28.4s
29
30    fmla v0.4s, v21.4s, v29.4s
31    fmla v0.4s, v22.4s, v30.4s
32    fmla v0.4s, v23.4s, v31.4s
33    fmla v0.4s, v24.4s,  v1.4s
34
35    fmla v0.4s, v25.4s,  v2.4s
36    fmla v0.4s, v26.4s,  v3.4s
37    fmla v0.4s, v27.4s,  v4.4s
38    fmla v0.4s, v28.4s,  v5.4s
39
40    fmla v0.4s, v29.4s,  v6.4s
41    fmla v0.4s, v30.4s,  v7.4s
42    fmla v0.4s, v31.4s,  v8.4s
43    .endr
44
45    cbnz x0, loop_bench

Listing 6.2.2 shows a microbenchmark loop for the forwarding case. The dependency chain now exists only on the destination register, not on any of the source registers. In the case of Neoverse V2, the microarchitecture supports late-forwarding of accumulate operands, and an execution latency of 2 cycles is specified in this case. This means that we can expect to execute 0.5 instructions per cycle for the microbenchmark.

We can confirm the theoretical Neoverse V2 execution latencies by running the microbenchmarks. Table 6.2.1 shows the measured latencies for the two cases. We see that we can reproduce the theoretical execution latencies given in the Software Optimization Guide.

In particular, the 6.6 GFLOPS measured for source register dependencies represents a 16-fold slowdown compared to the 105.6 GFLOPS throughput achieved when all pipelines and pipeline parallelism are fully utilized. In summary, we need to design our compute kernels very carefully to fully exploit the available parallelism of the target microarchitecture.