4
\$\begingroup\$

I have found an ARM Branch instruction appears to take 6 cycles to run on an ARM7TDMI processor. It seems it shouldn't be happening because in all references I've found, an ARM7TDMI branch instruction should take only 3 cycles. But:

The C function:

start_time = TC;
for (int i=0; i<120; i++) {
  __asm("NOP");
}
end_time = TC;

The disassembly shows the loop as: (Update: instruction addresses added):

0x120             MOV R1, 0
0x124             B LOC0
            start:    
0x128             NOP
0x12C             ADD R1, R1, 1
            LOC0:     
0x130             CMP R1, 120
0x134             BLT start

Now the result shows the loop takes 1080 cycles (converted from a timer counter put in TC), i.e., 9 cycles per loop kernel. Since NOP, ADD, CMP are all single cycle instructions, BLT has to be 6 cycles.

I once suspect if my timing method has flaws. But if I add 1 NOP in the loop kernel, the time increase would amount to exactly 1 cycle.

What's wrong here?

(Update: fix: the original disassembly code miswrote ADD R1, R1, 1 as ADD R1, R1)

Update: Answer Accepted: Flash access stall causes the 3 extra cycles

Thanks all for the helpful answers and comments, especially @supercat, @Dzarda, @DaveTweed, @IgorSkochinsky, @WoutervanOoijen. I am running code from flash. The CPU is a LPC23xx. According to the User Manual, it does include a Memory Acceleration Module (MAM) for bufferred flash access. And the suggested flash fetch cycles under my CPU speed is exactly 3 cycles.

The start in the above penalized loop kernel is aligning to a 8-byte boundary. If I change the alignment of start to 16-byte boundary, then the 3 extra cycles penalty disappeares. This can be explained by the 128 bits (16 bytes) flash prefetch buffer size of my CPU.

(@WoutervanOoijen) Note the 3-cycle MAM flash fetch time is not done by ARM CPU, but by the MAM module which prefetches the flash data in parallel with CPU. So in my code with start aligning at 8-byte boundary, CMP is the first instruction in the 128-bit (4-instruction) MAM prefetch buffer. When the ARM CPU executes BLT, it takes the 1st cycle to "understand" the instruction. Then it tries to fetch NOP instruction which is not in the MAM prefetch buffer. That should be the moment when the extra 3 cycles happens when the MAM accesses the flash. When the NOP instruction is in the buffer (along with 3 other instructions in the 32-byte flash line), the ARM CPU can actually re-fill the pipeling by fetching NOP (5th cycle) and decoding NOP (6th cycle). That's where the total 6 cycles come from.

So the answer to my question is Yes, a 6-cycle branch instruction is possible if there's a flash access stall.

Final Unresolved Question

As @WoutervanOoijen points out, the above reasoning has a flaw. LPC23xx's Memory Acceleration Module has an additional Branch Trail buffer that is supposed to avoid this kind of repeated re-fetch loop branches. The LPC23XX User Manual states:

The Branch Trail buffer captures the line to which such a non-sequential break occurs. If the same branch is taken again, the next instruction is taken from the Branch Trail buffer

This statement doesn't seem to be very clear about what's exactly being put into the Branch Trail buffer. It could be the last prefetched flash line, or the last branch destination flash line. In either case, the flash access penalty shouldn't have happened because the flash line (0x120 ~ 0x12F) including the branch destination instruction (NOP) should already be in the Branch Trail buffer when BLT is being executed (at least from the second time on).

(BTW, I verified the MAM is put in Fully Enabled mode, i.e. MAM_mode_control is 2.)

I will update this question after I find more information about this. And I'll appreciate it if you have any comments on what might be happening here, or what test can be done to look for clues.

\$\endgroup\$
  • 3
    \$\begingroup\$ My bet would be, that it has something to do with cache misses or something in that ballpark. Anyway, ARM is a pretty complex architecture and you should not expect cycle-precise execution. There are many things that regular engineers don't know (don't have to know) about the thing. \$\endgroup\$ – Dzarda May 5 '14 at 16:47
  • 1
    \$\begingroup\$ First of all, are you sure that disassembly is correct? It looks like it should double R1 (not increment it) on every iteration. Second, there may be a pipeline stall induced by the data dependency between the add and the cmp instructions. Finally, 3 cycles for blt might be just a minimum value; additonal clocks might be required if portions of the decode pipeline need to be flushed when the branch is taken. \$\endgroup\$ – Dave Tweed May 5 '14 at 16:53
  • 3
    \$\begingroup\$ Which exact chip is it? (NXP?) Does it have a flash accelerator/cache? Do you account for flash access time? Do you get a different result if you run from RAM? \$\endgroup\$ – Igor Skochinsky May 5 '14 at 17:03
  • 1
    \$\begingroup\$ For an ARM7 that does not have to wait (for operand fetches, for instance) that loop should indeed be 1+1+1+3 cycles. 1) Check whether it is really this code that is executed (the nop seems to be at the wrong place, and as David said the loop counting seems to be wrong). 2) Check whether the CPU has to wait for instruction fetches (Flash often has an extra wait cycle). \$\endgroup\$ – Wouter van Ooijen May 5 '14 at 17:20
  • 1
    \$\begingroup\$ @xiaobai the ARM7TDMI is a CPU, the chip manufacturer adds the memory (RAM, FLash), which can include some form of caching or buffering (the LPC2148 for instance has a primitive Flash buffer). So why don't you tell us exactly which chip you are using and with which settings? \$\endgroup\$ – Wouter van Ooijen May 5 '14 at 20:34
7
\$\begingroup\$

Are you running code from RAM or from flash? ARM processors that run code from flash often require wait states in at least some circumstances; such processors often include hardware which can eliminate most of the wait states in common code, but such hardware may be as simple as a single-line buffer which allows an access to the same line of flash as the previous access to avoid the wait state. If the branch target is the last word of a flash line, then the flash would require two or three cycles to fetch that word, and two or three cycles to fetch the following word. If one of the cycles is performed concurrently with some other CPU operation, that would leave a three-cycle penalty.

\$\endgroup\$
  • \$\begingroup\$ Thanks a lot for the answer. I am running code from flash. The CPU is a LPC23xx. According to the User Manual, it does include a Memory Acceleration Module (MAM) for bufferred flash access. And the suggested flash fetch cycles under my CPU speed is exactly 3 cycles. \$\endgroup\$ – Penghe Geng May 5 '14 at 21:00
  • \$\begingroup\$ @xiaobai: Did I correctly guess the alignment relative to the cache line, or was I off by a byte? Would shifting the code to start at a multiple of 8 words (32 bytes) make it run faster? \$\endgroup\$ – supercat May 6 '14 at 1:50
  • \$\begingroup\$ Yes you are correct that the loop kernel needs to align with the flash buffer line size. For my case, the flash prefetch buffer is 128 bits (16 bytes). I verified that the extra cycles doesn't happen if start aligns to 16-byte boundary. Previously it's in the 8-byte boundary. \$\endgroup\$ – Penghe Geng May 6 '14 at 14:30
1
\$\begingroup\$

Take a peek at the ARM info center, figure 2, keeping in mind that you're working with the ARM7 pipeline and not the M3 3-stage pipeline. The point remains valid.

There can be cycles in between fetch and execution. It's very difficult to count clock ticks for instruction cycles on a modern pipelined core. I'm not sure it's even deterministic

I'm wondering if the pipleline needs to start over at each branch. You might consider stacking a bunch of these NOPs instead of branching to see if your resulting behavior is more deterministic as a debugging step.

Indeed, I've been warned about using NOPs for precise delays on ARM platforms for this reason.

enter image description here

\$\endgroup\$
  • \$\begingroup\$ -1: Your answer is based on timing diagrams for M3 and M4 (16 bit) CPU's, but the question is about the (32-bit) ARM7. \$\endgroup\$ – Wouter van Ooijen May 5 '14 at 17:46
  • 1
    \$\begingroup\$ @WoutervanOoijen I was just assuming the 8-stage A7 pipeline would be an even more difficult situation than the 3-stage M pipeline, and couldn't find a good illustration for the A7, even after some looking. Glad to replace the image and link if you can point me to a better one as illustrative of the point I'm trying to make. \$\endgroup\$ – Scott Seidman May 5 '14 at 17:57
  • 2
    \$\begingroup\$ @WoutervanOoijen, M3 and M4 are 32 bit CPU's. Are you referring to the Instruction Set? \$\endgroup\$ – Scott Seidman May 5 '14 at 19:17
  • 2
    \$\begingroup\$ sorry for the wording, what I meant is that the Mx are (mostly) 16-bit instruction CPUs, while the question is about ARM7TDMI in ARM mode, which is a totally different CPU, which even has a different instruction with. (But the same width is not enough, M0 and M0+ for instance have the same instruction set but different pipelines). I am not sure what you mean by A7, ARM7 is not a Cortex A7! Without stalls the ARM7 pipeline is Fetch-Decode-Excute, with a 2 cycle penalty for changing the Pc. \$\endgroup\$ – Wouter van Ooijen May 5 '14 at 20:39
  • \$\begingroup\$ @ScottSeidman Thanks for the suggestion of using NOP instead of loops. Using NOP the result would be correct. \$\endgroup\$ – Penghe Geng May 5 '14 at 21:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.