I'm a little surprised to see 40us gap there. The driver, if you look at it, does send one byte out per interrupt, then waits for another interrupt before it does the next one. I would have anticipated more like 10us between them however.
This is usually more of an issue with SPI running at 2MHz or so, where it really makes a huge difference and you can fix that by changing the code to tight loop and keep stuffing bytes in, by the time you've written one and checked the register, it's been clocked out and you can do the next one, if you ever do catch up, then you just wait for the next interrupt and do another batch. At 2 or 4 Mhz SPI, the CPU can't keep up so the buffer is always ready.
At 400KHz however that's 20us a byte, so there will be some downtime between bytes. If full speed is really important, writing your own driver is probably the only way to go.
