nrf5340 is unable to trigger rpmsg(ipc) transfer

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Edvin — Tue, 06 Dec 2022 21:20:03 GMT

Hello,

Sorry for the late reply. I was unexpectedly out of office since Thursday.

That would be great. If I can have a look, reproduce it and run it by our SoftDevice controller, I can ask them whether they have any hints on whether it is the intended workflow, or if this is in fact a bug.

Best regards,

Edvin

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Dumam — Tue, 06 Dec 2022 17:25:45 GMT

Both nrf52840 and nrf5340 have bigger throughput with Zephyr stack.
If I replace this with SoftDevice stack, then throughput drops to 500kbps.

Edvin If you're interested I can prepare an example project which will reproduce my case only with two nrf5340 boards, without any SPI device (it can be in loopback).

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Dumam — Sat, 03 Dec 2022 09:52:30 GMT

Hi,
The issue persists. I was able to partially overcome it, by changing net core BLE Stack implementation in hci_rpmsg_nrf5340dk_nrf5340_cpuapp.conf to:

# select ll layer
CONFIG_BT_LL_SW_SPLIT=y
CONFIG_BT_LL_SOFTDEVICE=n
#CONFIG_BT_CTLR_SDC_MAX_CONN_EVENT_LEN_DEFAULT=4000000

After this change, I was able to achieve 1,1Mbps of throughput.
I still will be fighting with this code to reach 1,4Mbps, but It can be hard to do so.

According to System View, not much room left for processing.

If I find something useful, I will post it here.

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Dumam — Wed, 30 Nov 2022 21:16:09 GMT

Thanks for this reply, but:

1. I posted URL to my code where you can easily track that I'm not using work queue but separate thread.
I posted a print screen from Seeger System View, where there is only info about thread
2. I call there bt_gatt_write_without_response_cb(), which calls bt_gatt_write_without_response_cb(). This calls still blocks thread execution.
3. My issues are the same as ones mentioned here, for the same board, for the same situation:
What's different in my case is that I'm using a `write without response` not notify.
Fast notifications with nRF5340 and nRF Connect SDK

4. For this board, if you use BLE stack anywhere, then net CPU is included.
This converts the project to multicore one and adds a simple rpmsg project for net-core and installs SoftDevice by default on it.
After that in config you can also see that HCI RPMSG stack is enabled for app core.
So every command from app-core to HCI BLE is sent via RPMSG module.
This module uses RPROC protocol and MBOX module to sync over IPC.
Sadly, to process this in correct manner, MBOX module is using WORKQUEUE.

I guess this is the main reason for locks.
If this issue is true.
bt_gatt_notify() can block when running on Zephyr workqueue.

5. nrf52840 has different driver HCI RPC, but there is similar WORKQUEUE implemented to sync with SoftDevice.

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Edvin — Wed, 30 Nov 2022 11:33:06 GMT

Can you please check my colleague, Einar's replies in this ticket:
bt_gatt_notify() can block when running on Zephyr workqueue.

In general, you should not call bt_gatt_notify_cb() from a workqueue, because it can lead to a deadlock, or in some cases, blocking calls, as you see here. Try calling it from another context.

Best regards,

Edvin

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Dumam — Tue, 29 Nov 2022 23:25:25 GMT

Dear Edvin
bt_write_without_response is locking function, which is bad for me because my SPI do not have time to prepare new data for connection event.
I'm looking for something what will help me in determining if i have still space in in my TX buffers.
According to docs.zephyrproject.org/.../l2cap.html , there is nothing like that.

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Dumam — Tue, 29 Nov 2022 03:02:17 GMT

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Dumam — Tue, 29 Nov 2022 02:56:34 GMT

Ok, I added BLE call inside SPI callback.
https://github.com/DuMaM/bitly_nrf5x/pull/13
I got even worse performance ;(

BTW. I'm using bt_gett_write_without_response, which is under the hood:

static inline int bt_gatt_write_without_response(struct bt_conn *conn,
						 uint16_t handle, const void *data,
						 uint16_t length, bool sign)
{
	return bt_gatt_write_without_response_cb(conn, handle, data, length,
						 sign, NULL, NULL);
}

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Edvin — Mon, 28 Nov 2022 11:54:07 GMT

I believe it is rather the work queue holding you back than the BLE stack. You can try to either skip the work queue, and call the bt_gatt_notify_cb() directly from your spi callback handler.

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Dumam — Fri, 25 Nov 2022 03:10:28 GMT

I also increased systick speed and this didnt helped.

# clock settings
# https://github.com/zephyrproject-rtos/zephyr/issues/28469#issuecomment-704164283
# https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/zephyr/kernel/services/timing/clocks.html
# https://devzone.nordicsemi.com/f/nordic-q-a/92297/increasing-cpu-clock-frequency-for-bl653
CONFIG_NRF_RTC_TIMER=n
CONFIG_CORTEX_M_SYSTICK=y
CONFIG_SYS_CLOCK_HW_CYCLES_PER_SEC=64000000
CONFIG_SYS_CLOCK_TICKS_PER_SEC=6400000

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Dumam — Fri, 25 Nov 2022 03:09:42 GMT

[quote userid="26071" url="~/f/nordic-q-a/94053/nrf5340-is-unable-to-trigger-rpmsg-ipc-transfer/397343"]

Whether or not that is recursive, I am not sure. That is, if you have queued packet #1 and #2 when packet #1 goes on air. Then you queue packet #3 before packet #2 has started airing, I am not sure whether it will change the "more data" bit in packet #2 or not. If not, then all packets needs to be queued before the connection event starts.

[/quote]

As far as I can tell, write request is locked since there will be a free space in rpmsg buffer.
I don't understand why I see only 2 work queue execution per, connection event, even thought I have more data prepared for transmission. I think this is a one of settings from RPMSG.

I'm not sure about PPI. My guess is that SPI is taking too long, and I'm unable to provide next packet for BT_TX thread.

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Edvin — Thu, 24 Nov 2022 09:41:31 GMT

[quote user="Dumam"]Please do not close this ticket yet.[/quote]

No problem. We can go on for a while more. Just please note that I will be out of office tomorrow, but I'll be back on Monday.

I am glad you figured it out! That makes sense. The data needs to be queued up before the connection event. Obviously, all individual packets needs to be queued before they go on air. And in each packet, there is a bit that says whether or not it has more data. So the Bluetooth stack (SoftDevice Controller) needs to at least be aware of the second packet before it starts transmitting the initial packet. Whether or not that is recursive, I am not sure. That is, if you have queued packet #1 and #2 when packet #1 goes on air. Then you queue packet #3 before packet #2 has started airing, I am not sure whether it will change the "more data" bit in packet #2 or not. If not, then all packets needs to be queued before the connection event starts.

As for the SPI. I am not aware of any SPI with EasyDMA samples in Zephyr, but you can set up a ppi approach. On how to use PPI, please see the sample found in NCS\zephyr\samples\boards\nrf\nrfx. What you can do here is to chain the transfer end event with a transfer start, so that it runs in the background, if that is what you are looking for. Then you would typically use a timer in counter mode to also count up, and then set an interrupt when the timer (counter) reaches a certain value, to then trigger a "timeout" interrupt in your application.

Best regards,

Edvin

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Dumam — Thu, 24 Nov 2022 02:52:30 GMT

Hi Edvin,

You were right. There was problem with connection interval.
Zephyr needs a constant feed of data at the beginning of a connection event.
If this steam is not delivered, then the connection is instantly terminated.
I was planning to just retransmit my SPI data without any additional buffering, but this will not work in that case.

So, I increased SPI buffer, and now it provides space for data from 35ms long SPI stream.
At the end, I was able to reach 0.5Mbits/s of throughput ✈

That's so big step forward 🙏 thanks for this help.

Sadly, I still require some help.
I noticed that SPI is now a blocking factor.
Do you have nice example how to work with EasyDMA SPI in zephyr, so I can fetch data only on interrupt from it? Additional GPIO, takes too many CPU cycles in ISR, and it duplicates SPI work.

Please do not close this ticket yet.

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Dumam — Wed, 23 Nov 2022 11:55:45 GMT

[quote userid="26071" url="~/f/nordic-q-a/94053/nrf5340-is-unable-to-trigger-rpmsg-ipc-transfer/397115"]

For debugging purposes, what if you instead of calling bt_performance_test_write(), just add up the length of what you were going to send, and look at what sort of total length/throughput you would have got without actually sending the data? Is it still lower than only sending dummy data without the SPI?

[/quote]

I checked this. I'm always sending 251 bytes.

You gave me other idea how I can debug this. I will try to use dummy data with SPI in background and see if this is a problem on thread sync level or maybe some ISR/MCU components.
This way I separate SPI thread from BLE one.

Did you manage to find something on your side?

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Edvin — Wed, 23 Nov 2022 10:15:02 GMT

[quote user="Dumam"]I'm not sure what you mean by `No actual peripheral transferring data?`[/quote]

I mean that you are not doing any SPI transactions inside that function, which means that you need to wait for the transaction to finish inside this callback? If so, that will introduce a delay.

Ok, we have a more similar understanding of what your project does now, which is a good thing 🙂

My claim is that bt_performance_test_write() does not care whether the data comes from a buffer of all 0's or a buffer consisting of SPI data. So the lower throughput has to come from elsewhere.

BR,

Edvin

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Dumam — Tue, 22 Nov 2022 15:42:54 GMT

> Ok, I am trying to navigate your project. I am struggling to see exactly how it relates to the throughput sample.

I renamed it to performance_tests 😉
I wanted to extract this form nrf sdk, mostly because in feature I'm going to modify it.
This also helped me to have a stable code base.

> Where exactly are you sending the BLE packets. Can you please point to it?

Sure. I'm using writes here, as it was in example. I'm sending packages here:
https://github.com/DuMaM/bitly_nrf5x/blob/e6cf66e1655bf3da2c65d58fedb561893194ba80/src/app/cmd_run_adc.c#L61

> Where it sends, and how long are the packets that you are sending? There should be a call to bt_gatt_notify_cb() or bt_gatt_write() or something like that.

I'm trying to fill package as much as possible. Until there are data in buffer I'm sending `data_len->tx_max_len`, otherwise when there will be some leftovers I placing them as a whole.

> You say that your throughput is much higher when you disable the SPI and just use dummy data. So the MTU and connection interval and everything else is the same, just that you wait for the SPI to finish?

I think yes. 😉
SPI is driven by GPIO called DRDY. This pin is used by hooked up device to notify master that it should fetch data. Maximum speed here is 32000 samples/s, but with current BLE capabilities the maximum reasonable value is 4000k samples/s (1400kbps), for tests i'm using 2000k samples/s.

> What is it that triggers send_test_ecg_data()?

This semaphore and GPIO callback:
https://github.com/DuMaM/bitly_nrf5x/blob/e6cf66e1655bf3da2c65d58fedb561893194ba80/src/app/spi_adc.c#L442
github.com/.../spi_adc.c

> ads129x_get_data() only triggers a read from a buffer? No actual peripheral transferring data? > When will it continue; and when will it send data (which I assume it does inside bt_performance_test_write()

It triggers only on when there is a timeout on pipe k_polling or requested number of data was delivered from SPI pipe. I'm not sure what you mean by `No actual peripheral transferring data?`
Anyway when it fetch data from pipe it will transfer them to `bt_performance_test_write` which is wrapper for `bt_gatt_write_without_response`
It's a bit stupid but thats how it was designed in example. I didnt have time to change it too much.
github.com/.../performance_test.c

> And when bt_performance_test_write() is called, what is the length of the data that is being sent?

As I said before it's max len set for single connection event - mostly 251 bytes.
If SPI finish it's data transfer and it's not aligned to 251 bytes I send this remainder.

> Sorry, a lot of unstructured questions? I was reading more and more into your project.

No problem, I really appreciate it.

> So the question is, how often is send_test_ecg_data() called, and how often is bt_performance_test_write() called? Is bt_performance_test_write() called every time send_test_ecg_data() is called, or does it often skip it because of continue; ?

It will be hard to answer for this, because it's constantly changing. Depending on priories I set for threads and ISR. What I can tell for sure is that it's able to pull data from pipe without issue.
I started to suspect that bt_gatt_write_without_response waits for something. I'm not sure what, but when I interrupt it with SPI ISR, this event is lost.
This behavior would explain why synthetic dummy tests are ok.

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Edvin — Tue, 22 Nov 2022 14:55:17 GMT

Ok, I am trying to navigate your project. I am struggling to see exactly how it relates to the throughput sample. Where exactly are you sending the BLE packets. Can you please point to it? Where it sends, and how long are the packets that you are sending? There should be a call to bt_gatt_notify_cb() or bt_gatt_write() or something like that.

You say that your throughput is much higher when you disable the SPI and just use dummy data. So the MTU and connection interval and everything else is the same, just that you wait for the SPI to finish?

What is it that triggers send_test_ecg_data()?

ads129x_get_data() only triggers a read from a buffer? No actual peripheral transferring data? When will it continue; and when will it send data (which I assume it does inside bt_performance_test_write(). And when bt_performance_test_write() is called, what is the length of the data that is being sent?

Sorry, a lot of unstructured questions, but I was reading more and more into your project. So the question is, how often is send_test_ecg_data() called, and how often is bt_performance_test_write() called? Is bt_performance_test_write() called every time send_test_ecg_data() is called, or does it often skip it because of continue; ?

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Dumam — Tue, 22 Nov 2022 14:12:53 GMT

Edvin I'm soo grateful for this. I'm trying to figure it out for 2 months without any luck. :(

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Edvin — Tue, 22 Nov 2022 14:11:51 GMT

Oh, my impression was that this was someone else's repository. I understand. I will have a look at it as soon as possible.

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Dumam — Tue, 22 Nov 2022 14:08:32 GMT

My repo is public you can download or clone it without any issues.

https://github.com/DuMaM/bitly_nrf5x
I also placed this in attachment.

SPI Thread.
More here: github.com/.../spi_adc.c

void ads129x_set_data()
{
    /* add timestamp */
    tx_data.packet.timestamp = k_uptime_get() - timestamp;
    conv_u24_to_raw(tx_data.packet.timestamp, tx_data.buffer, 0);

    /* do processing */
    /* NOTE: Work directly on a ring buffer memory */
    int ret = spi_read(ads129x_spi, &ads129x_spi_cfg, &ads129x_rx);
    if (!ret)
    {
        /* add missing leads */
        lead1 = conv_u24_to_i32(conv_raw_to_u24(tx_data.packet.leads._buffer, ADS129x_LEAD1_OFFSET));
        lead2 = conv_u24_to_i32(conv_raw_to_u24(tx_data.packet.leads._buffer, ADS129x_LEAD2_OFFSET));
        conv_u24_to_raw(ads129x_get_leadIII(lead1, lead2), tx_data.packet.leads._buffer, ADS129x_LEAD3_OFFSET);
        conv_u24_to_raw(ads129x_get_aVR(lead1, lead2), tx_data.packet.leads._buffer, ADS129x_AVR_OFFSET);
        conv_u24_to_raw(ads129x_get_aVL(lead1, lead2), tx_data.packet.leads._buffer, ADS129x_AVL_OFFSET);
        conv_u24_to_raw(ads129x_get_aVF(lead1, lead2), tx_data.packet.leads._buffer, ADS129x_AVF_OFFSET);

        //ads129x_dump_data(tx_data.packet.leads._buffer);
    }

    /* send data to consumers */
    /* send data to the consumers */
    k_pipe_put(&ads129x_pipe, &tx_data.buffer, total_size, &bytes_written, sizeof(pipe_packet_u), K_NO_WAIT);
}

void ads129x_th(void)
{
    /* setup ecg */
    ads129x_setup();

    for (;;)
    {
        /*
         * Wait for semaphore from ISR; if acquired, do related work, then
         * go to next loop iteration (the semaphore might have been given
         * again); else, make the CPU idle.
         */
        if (k_sem_take(&ads129x_new_data, K_FOREVER) == 0)
        {
            ads129x_set_data();
        }
    }
}

K_THREAD_DEFINE(thread_ads129x, STACKSIZE, ads129x_th, NULL, NULL, NULL, PRIORITY, K_ESSENTIAL, 0);

BLE Thread:
More here: https://github.com/DuMaM/bitly_nrf5x/blob/e6cf66e1655bf3da2c65d58fedb561893194ba80/src/app/cmd_run_adc.c#L104

K_THREAD_STACK_DEFINE(ecg_ble_stack, 4096);
struct k_thread ecg_ble_thread;
static uint32_t bytes_to_send = 1024;

static uint32_t send_test_ecg_data(uint32_t _bytes_to_send)
{
    uint32_t prog = 0;
    uint8_t *analog_data_ptr = test_data_buffer;
    uint32_t analog_data_size = 0;
    int err = 0;

    /*
     * we always waiting for data to match whole buffer
     * so data requested should also match it
     * otherwise we should ignore it
     */
    uint16_t remainder = _bytes_to_send % ADS129x_DATA_BUFFER_SIZE;
    if (remainder) {
        _bytes_to_send = ((_bytes_to_send / ADS129x_DATA_BUFFER_SIZE) + 1) * ADS129x_DATA_BUFFER_SIZE;
    }
    LOG_INF("Sending %"PRIu32" bytes (value after rounding to max packet size)", _bytes_to_send);

    while (prog < _bytes_to_send)
    {
        analog_data_size = _bytes_to_send - prog;
        if (test_params.data_len->tx_max_len <= analog_data_size) {
            analog_data_size = test_params.data_len->tx_max_len;
        }

        /*
         * TODO: add scaling from shell
         * It would be good to send optimal number of data
         * during each transfer
         * also when we change a number o leads
         */
        if (!ads129x_get_data(analog_data_ptr, analog_data_size))
        {
            continue;
        }

        err = bt_performance_test_write(&performance_test, analog_data_ptr, analog_data_size);
        if (err)
        {
            LOG_ERR("GATT write failed (err %d)", err);
            break;
        }

        prog += analog_data_size;
    }
    return prog;
}

devzone.nordicsemi.com/.../bitly_5F00_nrf5x_2D00_main.zip

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Edvin — Tue, 22 Nov 2022 13:57:32 GMT

[quote user="Dumam"]The problematic code do the same but gets data from SPI. [/quote]

Can you please show me how you do that?

In case I have follow up questions, feel free to zip and upload the entire project here (just drag and drop into the window where you are typing).

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Dumam — Tue, 22 Nov 2022 13:48:05 GMT

Yep, this is my command i wrote for testing purposes.

Everything base on this example.
https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/samples/bluetooth/throughput/README.html

I changed few parameters, to fit my needs. cmd_sim is basically this function from sample.
github.com/.../main.c

The problematic code do the same but gets data from SPI.

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Edvin — Tue, 22 Nov 2022 13:36:04 GMT

[quote user="Dumam"]I think the issue here is that `cmd_sim` function is sending everything in one thread,[/quote]

Where is this cmd_sim? Is it part of the repository that you are linking to? I am not familiar with that application, and it is not written by anyone from Nordic.

Best regards,
Edvin

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Dumam — Tue, 22 Nov 2022 10:41:46 GMT

Edvin I think it's more a problem of how communication between threads is set up.

I checked this with Particle Xenon board and I observed exactly the same behavior.
I also switched from pipe to ring_buffers to save copy operations. This also didn't helped.

Here is BLE thread:
https://github.com/DuMaM/bitly_nrf5x/blob/e6cf66e1655bf3da2c65d58fedb561893194ba80/src/app/cmd_run_adc.c

Here is SPI thread:
github.com/.../spi_adc.c

RE: nrf5340 is unable to trigger rpmsg(ipc) transfer

Dumam — Tue, 22 Nov 2022 10:35:49 GMT

It makes sense to me but this happens also for longer data transfers. I can accept some delay while i start communication, but for me it looks like there is some interference between components.

For example If I would like to sent 3MB. Instead of stable stream of data, it looks like my app is buffering them in HCI driver and then after some time, it starts to send them in every connection event (like in picture). To set some reference, time spent on buffering is 30-100ms, and when transfer starts time between each frame is around 3-10ms.

At the end my throughput is around 60kbps.

My connection interval is set to 28units now.
https://github.com/DuMaM/bitly_nrf5x/blob/496ad8dfe21107d0d8ec1a2fd75c939aa89076fb/include/app/cmd.h#L11

What makes it more interesting is that I observe such behavior only when SPI is enabled.
For testing purposes I prepared artificial data set which I sent in exactly the same way. With this approach I'm able to get throughput around 1100kbps.
With exactly the same devices, with the same connection setups, and parameters, programs.
https://github.com/DuMaM/bitly_nrf5x/blob/496ad8dfe21107d0d8ec1a2fd75c939aa89076fb/src/app/cmd_run_sim.c

I think the issue here is that `cmd_sim` function is sending everything in one thread, by locking itself on `bt_gatt_write_without_response`. This way it dont have to switch thread context.
Where case with SPI needs to use mutexes and ISR to switch into different thread and protect shared resources.