SPI flash bit-banging access over memory-mapped IO

Question

I'm faced with a problem where we have an SPI NOR flash attached to an Intel I210 PCIe-ethernet adapter on a custom Cortex A9 based board. The SPI flash is normally used by the ethernet adapter to hold configuration data and firmware binary, but at the moment it's empty, in which case the ARM tools that Intel provided cannot touch the external SPI flash, and our ethernet chip now sits there unable to do anything helpful.

After digging around in the Intel chip's datasheet, I realised that there is a register with address offset 0x1201C called "FLA", and some bits in this register correspond with the sclk, miso, mosi and slave-select pins of the SPI flash. The datasheet recommends that, with an empty flash chip, the host should bit-bang commands to the SPI flash to enable writing to it through some other channels. I thought this would be a more promising approach than trying to port Intel's x86 drivers and tools to our board.

My problem is that, since the SPI flash operates at a much slower clock rate than the PCIe bus, how do I make sure that the slave-select and mosi signals arrive with sufficient setup and hold times relative to sclk?

Say, if I drive slave-select low in one memory-write operation and pulse the clock high in the next, does the PCIe infrastructure give any guarantee about the time between the packets arriving? Will it guarantee that the packet containing slave-select will arrive before the one containing the rising edge of sclk at all? Is using nanosleep() to try and add a delay between them a reasonable approach, for example?

I feel as if I've misunderstood some things and am going about it in a wrong way. How is this sort of bit-banging access usually done to ensure that the timing constraints are met?

Answer 1

I just recently did something very similar - bit banging both a 16 bit parallel interface and two QSPI interfaces from userspace over PCIe. In this case, an FPGA was on the other end, not an ASIC, but the interface is the same: a few bits in a couple of MMIO registers. In this case, the device advertised the BAR as 32 bit non-prefetchable, though I'm not sure if prefetchable vs. non-prefetchable makes any difference for writes. At any rate, writes are posted, and as such the PCIe spec requires them to traverse the bus in-order, so there were no special considerations required as far as ordering is concerned. However, timing was a slightly different story, and in a few places I had to perform dummy reads of the register to slow things down a little bit. Reading is non-posted, so presuming things are not being cached, performing a read can be used as a barrier, ensuring that all writes initiated before the read will complete before the read itself completes.

The general method is exactly what you might think: For writing, set CS and MOSI in one write, then set SCK high in the next write, then set SCK low and update MOSI in the next write, then set SCK high in the next write, etc. For reading, the idea is similar, read MISO, set SCK high, set SCK low, read MISO, etc. For the QSPI flash I was interfacing with over a PCIe gen 3 x16 link, I did not have to do anything more than that - no sleeps, busy loops, or dummy reads required. For the parallel flash, I did have to add a couple of dummy reads, namely one dummy read to extend the high period of the signals during a write, and one dummy read before the actual read to give things time to settle.

I would recommend figuring out how to read the flash ID registers reliably first before moving on to writing the flash.