0
\$\begingroup\$

I'm faced with a problem where we have an SPI NOR flash attached to an Intel I210 PCIe-ethernet adapter on a custom Cortex A9 based board. The SPI flash is normally used by the ethernet adapter to hold configuration data and firmware binary, but at the moment it's empty, in which case the ARM tools that Intel provided cannot touch the external SPI flash, and our ethernet chip now sits there unable to do anything helpful.

After digging around in the Intel chip's datasheet, I realised that there is a register with address offset 0x1201C called "FLA", and some bits in this register correspond with the sclk, miso, mosi and slave-select pins of the SPI flash. The datasheet recommends that, with an empty flash chip, the host should bit-bang commands to the SPI flash to enable writing to it through some other channels. I thought this would be a more promising approach than trying to port Intel's x86 drivers and tools to our board.

My problem is that, since the SPI flash operates at a much slower clock rate than the PCIe bus, how do I make sure that the slave-select and mosi signals arrive with sufficient setup and hold times relative to sclk?

Say, if I drive slave-select low in one memory-write operation and pulse the clock high in the next, does the PCIe infrastructure give any guarantee about the time between the packets arriving? Will it guarantee that the packet containing slave-select will arrive before the one containing the rising edge of sclk at all? Is using nanosleep() to try and add a delay between them a reasonable approach, for example?

I feel as if I've misunderstood some things and am going about it in a wrong way. How is this sort of bit-banging access usually done to ensure that the timing constraints are met?

\$\endgroup\$
  • \$\begingroup\$ Bit-banging is usually plenty slow that you'll have ample, even vast, amounts of setup time. \$\endgroup\$ – Ignacio Vazquez-Abrams Nov 23 '17 at 23:50
  • 1
    \$\begingroup\$ @IgnacioVazquez-Abrams I'm worried that maybe the processor or the PCIe circuitry will do some clever things and, say, keep the transactions in the L1 cache and fire them in bursts, or maybe re-ordering them, making the signals that go out to the flash chip rubbish. \$\endgroup\$ – user3109672 Nov 23 '17 at 23:57
  • \$\begingroup\$ That is indeed a concern worthy of at least thinking about. You can generally mark memory as unchacheable, if the amount of data to be transferred is small you may also utilize delays longer than the maximum possible "bunching" behavior. If you don't need to make a lot of these (or reprogram them in the field), you might consider holding the chip in reset and using a spring clip to do the programming. If you do try to implement this, picking up a cheap USB-fifo logic analyzer that streams to disk via sigrok will let you run a lot of trials and programmatically analyze their timing. \$\endgroup\$ – Chris Stratton Nov 24 '17 at 0:46
  • \$\begingroup\$ If you actually have the source for an x86 based driver, porting this part of it may not be that bad - or at the least, you could probably learn a lot by studying it. \$\endgroup\$ – Chris Stratton Nov 24 '17 at 0:49
  • \$\begingroup\$ DMB night be another way to enforce the ordering if it's not appropriate to define the memory region as DEVICE. \$\endgroup\$ – Sean Houlihane Nov 26 '17 at 11:01
1
\$\begingroup\$

I just recently did something very similar - bit banging both a 16 bit parallel interface and two QSPI interfaces from userspace over PCIe. In this case, an FPGA was on the other end, not an ASIC, but the interface is the same: a few bits in a couple of MMIO registers. In this case, the device advertised the BAR as 32 bit non-prefetchable, though I'm not sure if prefetchable vs. non-prefetchable makes any difference for writes. At any rate, writes are posted, and as such the PCIe spec requires them to traverse the bus in-order, so there were no special considerations required as far as ordering is concerned. However, timing was a slightly different story, and in a few places I had to perform dummy reads of the register to slow things down a little bit. Reading is non-posted, so presuming things are not being cached, performing a read can be used as a barrier, ensuring that all writes initiated before the read will complete before the read itself completes.

The general method is exactly what you might think: For writing, set CS and MOSI in one write, then set SCK high in the next write, then set SCK low and update MOSI in the next write, then set SCK high in the next write, etc. For reading, the idea is similar, read MISO, set SCK high, set SCK low, read MISO, etc. For the QSPI flash I was interfacing with over a PCIe gen 3 x16 link, I did not have to do anything more than that - no sleeps, busy loops, or dummy reads required. For the parallel flash, I did have to add a couple of dummy reads, namely one dummy read to extend the high period of the signals during a write, and one dummy read before the actual read to give things time to settle.

I would recommend figuring out how to read the flash ID registers reliably first before moving on to writing the flash.

|improve this answer|||||
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.