Your uart communication is sending one byte at the time. this means a lot of cpu processing at priority APP_IRQ_PRIORITY_LOWEST. What priority are you using for SPI? I am guessing it has the same priority so it is first come, first serve. You could try to increase the priority of spi, but that means UART throughput might be affected.
Maybe a better way is to use bigger buffers in both cases, so you have less cpu interaction all over. Maybe consider using libuarte instead of app_uart.
