Faster SoC and MCU communication
Hi I figured I should open a topic here first. In brief I'm looking to implement changes in the Tessel 2 firmware and Openwrt image to speed up communication between the SoC and MCU. Before getting myself too deeply into that I want to write out here what I'm looking to do and get any thoughts and advice on it.
I'm currently planning two phases of work (and one optional phase):
- (optional) Replace
usleep calls with busy waits.
kmod-tessel-spi that writes spi messages to and from the MCU for the SoC in place of
spid and the
spidev kernel module. Rewrite
spid to be a convenience layer for existing Tessel apps that read and write directly to the current unix sockets.
- Write a
mt7620-spi kernel module based on the
rt2880 currently in use and either use hardware fifo or dma to send spi messages actually asynchronously opening the cpu to do real work instead of the current busy waiting.
From best I can tell, and I'll write more on it with numbers and such soon,
spidev are the primary uses of time. One of the simple things to track down is
spid's use of
usleep. Replacing that with any form of busy wait speeds up this work. And doing so can better use the CPU as the time to change to another process and back seems to be more time than we really need to wait. This first optional step provides a small boost that could benefit users transparently.
usleep there are the
ioctl calls in
spid that hand off to
spidev's handling that lead to a
spi_async call. Like
usleep that sleeps the thread until a
kthread can perform the underlying work. Sometimes this does the spi work immediately but often only for the header message. The payload message often waits I'm guessing because the linux scheduler figures the spi thread just got to do work and something else deserves a chance (This is a wild assumption. I have benchmark timing that supports this, with the header giving times too small to context switch, and the payload regularly giving times too large to not be a context switch. I need to learn more about the scheduler and the one in use in openwrt/tessel to better confirm).
I believe writing a kernel module to do the work of
spidev is the best way to remove this regular spot for context switching to source that sends and receives the spi messages without sleeping. This also has the potential for other speed ups, removing the kernel context switching time for the other IO calls, faster IRQ response to async, possibly quicker interaction with gpio calls, extra memory copying, and removing cache thrashing due to existing sleeps.
Being a kernel module this would have greater potential for causing a tessel device to malfunction (with a kernel panic) than the userland
spid program. But I think we can offset that with plenty of community testing and review before making it a normal update through
t2. (Having not read or written a kernel module before doing this research I was really surprised how "normal" the code and APIs are. I was expecting a whole mountain of dragons to be inside.)
The last phase easily seems the biggest work to me. I read the issue relating to Tessel 2 boot time and agree with @kevin that the MT7620 programming guide and datasheets info on the SPI and DMA hardware is lacking. When I take a swing at this phase I don't know if I'll be able to finish figuring out this mystery. Though the goal is very tempting to reach. Seeing faster boot times and being able to work on data and communicate at roughly the same time would be great.
I'll come back with some of my rough benchmark data. I look forward to everyone's thoughts. Cheers!