Zephyr FATFS Flash disk cache commit slow down by 200x when mass storage USB disconnected

RE: Zephyr FATFS Flash disk cache commit slow down by 200x when mass storage USB disconnected

JustinL — Thu, 22 Jun 2023 08:40:47 GMT

Hi,

The issue is resolved.

I had a old compilation unit still in the cmake which should not have been there. The file did contain some old SPI initialisation code. It was obtaining a handle to the nrfx SPIM instance. Not sure why that caused this issue but removing it from cmake resolved the issue.

Thanks for you help along the way.

RE: Zephyr FATFS Flash disk cache commit slow down by 200x when mass storage USB disconnected

JustinL — Wed, 21 Jun 2023 16:55:23 GMT

I have narrowed it down to the spi_context_wait_for_completion() function. When the USB is not connected every third SPI transaction this function it takes 200ms, which is the configured SPI transaction timeout.

With the USB plugged in every third transaction takes just 1ms. So right there is the 200x difference.
(See output below).

The spi_context_wait_for_completion() function is waiting on a semaphore

k_sem_take(&ctx->sync, timeout) where timeout is 200ms in my case.

So for some reason when the SPI transaction has finished it is not being signalled when there is no USB device.

[00:00:24.977,600] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 200ms

[00:00:24.986,877] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 0ms

[00:00:24.995,971] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 0ms

[00:00:25.205,139] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 200ms

[00:00:25.214,416] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 0ms

[00:00:25.223,510] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 0ms

[00:00:25.432,678] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 200ms

[00:00:25.441,955] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 0ms

[00:00:25.451,049] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 0ms

[00:00:25.660,186] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 200ms

[00:00:25.669,464] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 0ms

[00:00:25.678,558] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 0ms

[00:00:14.746,673] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 1ms

[00:00:14.755,798] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 0ms

[00:00:14.764,892] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 0ms

[00:00:14.774,688] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 1ms

[00:00:14.783,782] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 0ms

[00:00:14.792,877] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 0ms

[00:00:14.802,673] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 1ms

[00:00:14.811,798] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 0ms

[00:00:14.820,892] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 0ms

[00:00:14.830,688] <err> spi_nrfx_spim: spi_context_wait_for_completion elasped time: 1ms

RE: Zephyr FATFS Flash disk cache commit slow down by 200x when mass storage USB disconnected

Jørgen Holmefjord — Wed, 21 Jun 2023 14:28:14 GMT

Are you able to narrow it down to the exact location within this snippet where the code is stuck?

I will then check with out developers if they have any explanation to why this could happen.

RE: Zephyr FATFS Flash disk cache commit slow down by 200x when mass storage USB disconnected

JustinL — Tue, 20 Jun 2023 16:10:29 GMT

I think I have narrowed down the issue to to the transceive function of spi_nrfx_spim.c

Using a processor ticks timer I find that the following code within the transceive function is around 8.9 million ticks with the usb cable out and 128k ticks with the cable in which is a factor of 70X.

What can be conflicting with the SPI when the USB is not attached? I have usb_cdc and usb mass storage.

spi_context_lock(&dev_data->ctx, asynchronous, cb, userdata, spi_cfg);

error = configure(dev, spi_cfg);

if (error == 0) {

dev_data->busy = true;

spi_context_buffers_setup(&dev_data->ctx, tx_bufs, rx_bufs, 1);

spi_context_cs_control(&dev_data->ctx, true);

transfer_next_chunk(dev);

error = spi_context_wait_for_completion(&dev_data->ctx);

}

spi_context_release(&dev_data->ctx, error);

RE: Zephyr FATFS Flash disk cache commit slow down by 200x when mass storage USB disconnected

JustinL — Tue, 20 Jun 2023 14:41:26 GMT

Hi,

I have used the Thread Analyzer library starting with no usb and then plugging it in.

There doesn't seem to be much difference from the thread stats. In both cases about 80% of time is spent in idle and 18% in main.

I have also put the SPI bus on the scope and can confirm there is no change in bus frequency, but there is a massive change in the bus transactions. Without the usb plugged in the SPI bus becomes very quite with sporadic transactions.

I would imagine you are right that the SPI thread is waiting for something held by another thread. I would think the disk access api must be synchronising access to the flash disk between threads? Unless the USB peripheral and the SPIM need to be synchronised, I guess they both us DMA?

I am not sure how to find out time a thread spends on waiting for what?

Below is the thread library output if it helps. Starts with no usb, then is plugged in.

[00:00:16.696,166] <inf> wr_ecm: Writen 2000 bytes to target file. took 7283 ms
[00:00:16.708,862] <inf> wr_ecm: Writen 3000 bytes to target file. took 7295 ms
[00:00:16.721,557] <inf> wr_ecm: Writen 4000 bytes to target file. took 7308 ms
[00:00:16.817,626] <inf> flashdisk: JXL about to flashdisk_cache_commit
Thread analyze:
thread_analyzer : STACK: unused 200 usage 824 / 1024 (80 %); CPU: 1 %
: Total CPU cycles used: 7651
usb_mass : STACK: unused 3976 usage 120 / 4096 (2 %); CPU: 0 %
: Total CPU cycles used: 0
usbd_workq : STACK: unused 3960 usage 136 / 4096 (3 %); CPU: 0 %
: Total CPU cycles used: 0
sysworkq : STACK: unused 3960 usage 136 / 4096 (3 %); CPU: 0 %
: Total CPU cycles used: 0
usbworkq : STACK: unused 3960 usage 136 / 4096 (3 %); CPU: 0 %
: Total CPU cycles used: 1
idle : STACK: unused 252 usage 68 / 320 (21 %); CPU: 81 %
: Total CPU cycles used: 489880
main : STACK: unused 3032 usage 1064 / 4096 (25 %); CPU: 17 %
: Total CPU cycles used: 102531
ISR0 : STACK: unused 1664 usage 384 / 2048 (18 %)
[00:00:19.738,952] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 2921ms
[00:00:20.423,278] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:23.632,629] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 3209ms
[00:00:23.845,520] <inf> wr_ecm: Writen 5000 bytes to target file. took 14432 ms
[00:00:23.858,367] <inf> wr_ecm: Writen 6000 bytes to target file. took 14445 ms
[00:00:23.871,185] <inf> wr_ecm: Writen 7000 bytes to target file. took 14458 ms
[00:00:23.884,002] <inf> wr_ecm: Writen 8000 bytes to target file. took 14471 ms
[00:00:23.977,722] <inf> flashdisk: JXL about to flashdisk_cache_commit
Thread analyze:
thread_analyzer : STACK: unused 200 usage 824 / 1024 (80 %); CPU: 1 %
: Total CPU cycles used: 10191
usb_mass : STACK: unused 3976 usage 120 / 4096 (2 %); CPU: 0 %
: Total CPU cycles used: 0
usbd_workq : STACK: unused 3960 usage 136 / 4096 (3 %); CPU: 0 %
: Total CPU cycles used: 0
sysworkq : STACK: unused 3960 usage 136 / 4096 (3 %); CPU: 0 %
: Total CPU cycles used: 0
usbworkq : STACK: unused 3960 usage 136 / 4096 (3 %); CPU: 0 %
: Total CPU cycles used: 1
idle : STACK: unused 252 usage 68 / 320 (21 %); CPU: 84 %
: Total CPU cycles used: 647177
main : STACK: unused 3016 usage 1080 / 4096 (26 %); CPU: 14 %
: Total CPU cycles used: 109372
ISR0 : STACK: unused 1664 usage 384 / 2048 (18 %)
[00:00:27.064,666] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 3087ms
[00:00:27.749,908] <inf> flashdisk: JXL about to flashdisk_cache_commit
Thread analyze:
thread_analyzer : STACK: unused 200 usage 824 / 1024 (80 %); CPU: 1 %
: Total CPU cycles used: 12733
usb_mass : STACK: unused 3976 usage 120 / 4096 (2 %); CPU: 0 %
: Total CPU cycles used: 0
usbd_workq : STACK: unused 3960 usage 136 / 4096 (3 %); CPU: 0 %
: Total CPU cycles used: 0
sysworkq : STACK: unused 3960 usage 136 / 4096 (3 %); CPU: 0 %
: Total CPU cycles used: 0
usbworkq : STACK: unused 3960 usage 136 / 4096 (3 %); CPU: 0 %
: Total CPU cycles used: 1
idle : STACK: unused 252 usage 68 / 320 (21 %); CPU: 86 %
: Total CPU cycles used: 807905
main : STACK: unused 3016 usage 1080 / 4096 (26 %); CPU: 11 %
: Total CPU cycles used: 111922
ISR0 : STACK: unused 1664 usage 384 / 2048 (18 %)
[00:00:30.203,765] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 2454ms
[00:00:30.206,634] <inf> usb_cdc_acm: Device suspended
[00:00:30.317,138] <inf> usb_cdc_acm: Device resumed
[00:00:30.322,570] <inf> usb_cdc_acm: from suspend
[00:00:30.332,275] <inf> wr_ecm: Writen 9000 bytes to target file. took 20919 ms
[00:00:30.345,123] <inf> wr_ecm: Writen 10000 bytes to target file. took 20932 ms
[00:00:30.358,032] <inf> wr_ecm: Writen 11000 bytes to target file. took 20945 ms
[00:00:30.370,971] <inf> wr_ecm: Writen 12000 bytes to target file. took 20957 ms
[00:00:30.466,644] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:30.488,769] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 22ms
[00:00:30.538,635] <inf> usb_cdc_acm: Device configured
[00:00:30.589,294] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:30.612,823] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:30.629,730] <inf> wr_ecm: Writen 13000 bytes to target file. took 21216 ms
[00:00:30.648,132] <inf> wr_ecm: Writen 14000 bytes to target file. took 21235 ms
[00:00:30.664,825] <inf> wr_ecm: Writen 15000 bytes to target file. took 21251 ms
[00:00:30.684,326] <inf> wr_ecm: Writen 16000 bytes to target file. took 21271 ms
[00:00:30.786,651] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:30.809,783] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:30.918,945] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:30.941,772] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:30.957,977] <inf> wr_ecm: Writen 17000 bytes to target file. took 21544 ms
[00:00:30.974,456] <inf> wr_ecm: Writen 18000 bytes to target file. took 21561 ms
[00:00:30.992,828] <inf> wr_ecm: Writen 19000 bytes to target file. took 21579 ms
[00:00:31.007,965] <inf> wr_ecm: Writen 20000 bytes to target file. took 21594 ms
[00:00:31.107,299] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:31.130,767] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:31.240,905] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:31.263,763] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:31.279,510] <inf> wr_ecm: Writen 21000 bytes to target file. took 21866 ms
[00:00:31.293,609] <inf> wr_ecm: Writen 22000 bytes to target file. took 21880 ms
[00:00:31.308,044] <inf> wr_ecm: Writen 23000 bytes to target file. took 21894 ms
[00:00:31.322,479] <inf> wr_ecm: Writen 24000 bytes to target file. took 21909 ms
[00:00:31.417,755] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:31.440,765] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:31.548,461] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:31.571,777] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:31.587,036] <inf> wr_ecm: Writen 25000 bytes to target file. took 22174 ms
[00:00:31.601,928] <inf> wr_ecm: Writen 26000 bytes to target file. took 22188 ms
[00:00:31.617,187] <inf> wr_ecm: Writen 27000 bytes to target file. took 22204 ms
[00:00:31.631,317] <inf> wr_ecm: Writen 28000 bytes to target file. took 22218 ms
[00:00:31.732,421] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:31.755,767] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:31.862,365] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:31.885,772] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:31.900,543] <inf> wr_ecm: Writen 29000 bytes to target file. took 22487 ms
[00:00:31.915,374] <inf> wr_ecm: Writen 30000 bytes to target file. took 22502 ms
[00:00:31.929,931] <inf> wr_ecm: Writen 31000 bytes to target file. took 22516 ms
[00:00:31.943,695] <inf> wr_ecm: Writen 32000 bytes to target file. took 22530 ms
[00:00:32.048,889] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:32.071,777] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:32.176,635] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:32.199,768] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:32.214,050] <inf> wr_ecm: Writen 33000 bytes to target file. took 22801 ms
[00:00:32.226,989] <inf> wr_ecm: Writen 34000 bytes to target file. took 22813 ms
[00:00:32.240,112] <inf> wr_ecm: Writen 35000 bytes to target file. took 22827 ms
[00:00:32.253,967] <inf> wr_ecm: Writen 36000 bytes to target file. took 22840 ms
[00:00:32.352,478] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:32.375,793] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:32.437,316] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:32.460,784] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:32.553,436] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:32.576,782] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:32.590,576] <inf> wr_ecm: Writen 37000 bytes to target file. took 23177 ms
[00:00:32.603,546] <inf> wr_ecm: Writen 38000 bytes to target file. took 23190 ms
[00:00:32.616,485] <inf> wr_ecm: Writen 39000 bytes to target file. took 23203 ms
[00:00:32.629,455] <inf> wr_ecm: Writen 40000 bytes to target file. took 23216 ms
[00:00:32.723,876] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:32.746,795] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:32.849,456] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:32.872,772] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:32.886,108] <inf> wr_ecm: Writen 41000 bytes to target file. took 23473 ms
[00:00:32.899,047] <inf> wr_ecm: Writen 42000 bytes to target file. took 23486 ms
[00:00:32.911,987] <inf> wr_ecm: Writen 43000 bytes to target file. took 23498 ms
[00:00:32.924,957] <inf> wr_ecm: Writen 44000 bytes to target file. took 23511 ms
[00:00:32.937,896] <inf> wr_ecm: Writen 45000 bytes to target file. took 23524 ms
[00:00:33.035,552] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:33.058,776] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:33.159,393] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:33.182,769] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:33.200,561] <inf> wr_ecm: Writen 46000 bytes to target file. took 23787 ms
[00:00:33.213,500] <inf> wr_ecm: Writen 47000 bytes to target file. took 23800 ms
[00:00:33.226,470] <inf> wr_ecm: Writen 48000 bytes to target file. took 23813 ms
[00:00:33.239,410] <inf> wr_ecm: Writen 49000 bytes to target file. took 23826 ms
[00:00:33.337,493] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:33.360,778] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:33.458,099] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:33.480,773] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 22ms
[00:00:33.498,077] <inf> wr_ecm: Writen 50000 bytes to target file. took 24085 ms
[00:00:33.511,047] <inf> wr_ecm: Writen 51000 bytes to target file. took 24098 ms
[00:00:33.523,986] <inf> wr_ecm: Writen 52000 bytes to target file. took 24110 ms
[00:00:33.536,926] <inf> wr_ecm: Writen 53000 bytes to target file. took 24123 ms
[00:00:33.628,021] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:33.650,787] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 22ms
[00:00:33.717,987] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:33.740,783] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:33.828,063] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:33.850,799] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 22ms
[00:00:33.867,614] <inf> wr_ecm: Writen 54000 bytes to target file. took 24454 ms
[00:00:33.880,584] <inf> wr_ecm: Writen 55000 bytes to target file. took 24467 ms
[00:00:33.893,524] <inf> wr_ecm: Writen 56000 bytes to target file. took 24480 ms
[00:00:33.906,494] <inf> wr_ecm: Writen 57000 bytes to target file. took 24493 ms
[00:00:34.005,584] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:34.028,778] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:34.100,494] <inf> flashdisk: JXL about to flashdisk_cache_commit
[00:00:34.123,779] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:34.213,226] <inf> flashdisk: JXL about to flashdisk_cache_commit
Thread analyze:
thread_analyzer : STACK: unused 200 usage 824 / 1024 (80 %); CPU: 1 %
: Total CPU cycles used: 15274
usb_mass : STACK: unused 3632 usage 464 / 4096 (11 %); CPU: 0 %
[00:00:34.235,778] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 22ms
: Total CPU cycles used: 106
[00:00:34.252,136] <inf> wr_ecm: Writen 58000 bytes to target file. took 24839 ms
[00:00:34.265,106] <inf> wr_ecm: Writen 59000 bytes to target file. took 24852 ms
[00:00:34.278,076] <inf> wr_ecm: Writen 60000 bytes to target file. took 24865 ms
[00:00:34.291,015] <inf> wr_ecm: Writen 61000 bytes to target file. took 24877 ms
[00:00:34.388,366] <inf> flashdisk: JXL about to flashdisk_cache_commit
usbd_workq : STACK: unused 3192 usage 904 / 4096 (22 %); CPU: 0 %
: Total CPU cycles used: 4246
sysworkq : STACK: unused 3960 usage 136 / 4096 (3 %); CPU: 0 %
[00:00:34.411,804] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
: Total CPU cycles used: 0
usbworkq : STACK: unused 3960 usage 136 / 4096 (3 %); CPU: 0 %
[00:00:34.476,989] <inf> flashdisk: JXL about to flashdisk_cache_commit
: Total CPU cycles used: 1
idle : STACK: unused 252 usage 68 / 320 (21 %); CPU: 79 %
: Total CPU cycles used: 866128
[00:00:34.499,786] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
main : STACK: unused 2984 usage 1112 / 4096 (27 %); CPU: 18 %
[00:00:34.591,400] <inf> flashdisk: JXL about to flashdisk_cache_commit
: Total CPU cycles used: 207254
ISR0 : STACK: unused 1664 usage 384 / 2048 (18 %)
[00:00:34.614,807] <inf> flashdisk: flashdisk_cache_commit: 4096 write time: 23ms
[00:00:34.630,676] <inf> wr_ecm: Writen 62000 bytes to target file. took 25217 ms
[00:00:34.643,615] <inf> wr_ecm: Writen 63000 bytes to target file. took 25230 ms

RE: Zephyr FATFS Flash disk cache commit slow down by 200x when mass storage USB disconnected

JustinL — Tue, 20 Jun 2023 13:45:35 GMT

I will give that a try.

RE: Zephyr FATFS Flash disk cache commit slow down by 200x when mass storage USB disconnected

Jørgen Holmefjord — Tue, 20 Jun 2023 13:31:00 GMT

Hi,

There may be some thread that is checking the USB state or similar that is blocking the CPU from executing your flash write operations.

Can you try out the Thread analyzer library, to see if some thread is using more CPU time than expected?

Best regards,
Jørgen