IPC problem with a custom board nrf5340

RE: IPC problem with a custom board nrf5340

Edvin — Mon, 03 Mar 2025 08:40:35 GMT

Hello,

Did you run your design through a review process? If not, I suggest you create a ticket and upload your design (schematics), and ask for a review. I see that you have a ticket about this, but I am not sure whether this is with the faulty or the working board.

Then someone with more HW expertise can have a look. They can probably answer whether the XTAL that you are using is compatible with/without the external capacitors linked to it. Also, perhaps some suggestions on how to check if the XTAL is actually working as intended.

Best regards,

Edvin

RE: IPC problem with a custom board nrf5340

brendow — Thu, 27 Feb 2025 17:09:17 GMT

Hi Edvin,

I tried using these configurations, and I also removed the external capacitors and enabled the internals.

Unfortunately, none of these solutions worked.

I'm not entirely sure, but I’m starting to accept that the issue may be with the crystal oscillator on the board. However, it's strange to me that the same components work on one board but not on the other.

Is there any additional advice you could offer to help resolve this issue or prevent it from occurring in the future?

Thanks for your support,

Brendow

RE: IPC problem with a custom board nrf5340

Edvin — Thu, 27 Feb 2025 14:28:44 GMT

If that alone doesn't help, try adding:

CONFIG_SOC_HFXO_CAP_INTERNAL=n

I don't have your HW to test on, so not 100% sure how these work.

RE: IPC problem with a custom board nrf5340

Edvin — Thu, 27 Feb 2025 14:25:29 GMT

Hello,

I am no hardware expert, but I asked a colleague of mine, and he said:

If they are using external capacitors with the XTAL, then they need to disable the internal ones, which by default is set to 12pF. So if they have both, they will see 24pF, which may cause it not to start up [the crystal], or that it starts very slowly.

To disable the internal capacitors, add this to your prj.conf:

CONFIG_SOC_HFXO_CAP_EXTERNAL=y

And see if that helps.

RE: IPC problem with a custom board nrf5340

brendow — Wed, 26 Feb 2025 14:43:33 GMT

Hi Edvin,

You're right, there's an issue with the HFCLK because the loop never ends in the B1. However, I'm even more confused now.

[quote userid="26071" url="~/f/nordic-q-a/118189/ipc-problem-with-a-custom-board-nrf5340/524790"]Can you check the soldering around the HFXTAL? Does it look ok? Is it mounted correctly? (We have seen some times that they are mounted 90 degrees off, so make sure it is in the same orientation as the other working board.)[/quote]

I made these two boards manufactured by JLCPCB using the same Gerber file, CPL, and BOM (I only removed the LFCLK from B2). The soldering looks fine to me.

This is the connection I made in my project. Did I do something wrong?

My project

Thingy53

I followed the HFCLK placement from Thingy53, although the trace distance in my design is longer, and I used external capacitors.

Could this affect the probability of my board working properly? In my project, I used an 8pF XTAL so the external capacitors are both 12pF.

One more thing. I'm not sure if this will help identify the source of the problem. I know the correct approach would be to use an oscilloscope to measure the XTAL clock, but I don't have one here. Instead, I used a multimeter to measure the voltage on the capacitors to see if I could find anything.

On my B2 (working board), both XC1 and XC2 have 800mV, but on my B1, XC1 has 800mV while XC2 only has 200mV.

Why is there this voltage difference between the capacitors? What could be causing this difference?

Best Regards,

Brendow

RE: IPC problem with a custom board nrf5340

Edvin — Wed, 26 Feb 2025 11:30:47 GMT

Hello,

The MPSL Assert 112, 1557 suggests that the HFXTAL (32MHz crystal) is not ramping up quick enough.

Without doing any modifications, you can check whether the HFXTAL actually starts by running this piece of code in the very beginning of main():

    LOG_INF("Starting HFXO");
    NRF_CLOCK->TASKS_HFCLKSTART=1;
    while (NRF_CLOCK->EVENTS_HFCLKSTARTED == false)
    {
        // Wait...        
    }
    LOG_INF("HFXO started.");

Can you check the soldering around the HFXTAL? Does it look ok? Is it mounted correctly? (We have seen some times that they are mounted 90 degrees off, so make sure it is in the same orientation as the other working board.)

Best regards,

Edvin

RE: IPC problem with a custom board nrf5340

Edvin — Wed, 26 Feb 2025 11:13:54 GMT

Ok. This suggests that there is a deviation in the chip. Let me run this assert through the SDK team, and based on their response, we might have to forward this ticket to our QA (Quality Assurance) department. Stay tuned.

Best regards,

Edvin

RE: IPC problem with a custom board nrf5340

brendow — Tue, 25 Feb 2025 16:35:59 GMT

Yes, I just commented on the lines used by the #include <dk_buttons_and_leds.h>

	for (;;) {
		// dk_set_led(RUN_STATUS_LED, (++blink_status) % 2);
		k_sleep(K_MSEC(RUN_LED_BLINK_INTERVAL));
	}

because if I didn't do that it would show me a continuous error log:

[00:01:06.631,652] <err> dk_buttons_and_leds: LED index out of the range
[00:01:07.631,835] <err> dk_buttons_and_leds: LED index out of the range
[00:01:08.631,988] <err> dk_buttons_and_leds: LED index out of the range
[00:01:09.632,171] <err> dk_buttons_and_leds: LED index out of the range
[00:01:10.632,324] <err> dk_buttons_and_leds: LED index out of the range
[00:01:11.632,507] <err> dk_buttons_and_leds: LED index out of the range
[00:01:12.632,659] <err> dk_buttons_and_leds: LED index out of the range
[00:01:13.632,843] <err> dk_buttons_and_leds: LED index out of the range
[00:01:14.632,995] <err> dk_buttons_and_leds: LED index out of the range
[00:01:15.633,178] <err> dk_buttons_and_leds: LED index out of the range
[00:01:16.633,331] <err> dk_buttons_and_leds: LED index out of the range

but the rest of the code remains intact

RE: IPC problem with a custom board nrf5340

Edvin — Tue, 25 Feb 2025 14:42:36 GMT

Is that the unmodified peripheral_uart sample from NCS v2.9.0?

RE: IPC problem with a custom board nrf5340

brendow — Tue, 25 Feb 2025 14:05:49 GMT

Hi Edvin,

[quote userid="26071" url="~/f/nordic-q-a/118189/ipc-problem-with-a-custom-board-nrf5340/523410"]Are you able to run it?[/quote]

Yes, CPU_APP works, but in CPU_NET an MPSL assertion occurs.

[00:00:00.057,098] <inf> ipc_radio: Received hci message of 36 bytes.
[00:00:00.057,434] <inf> ipc_radio: Sent message of 7 bytes.
[00:00:00.057,739] <inf> ipc_radio: Received hci message of 36 bytes.
[00:00:00.058,074] <inf> ipc_radio: Sent message of 7 bytes.
[00:00:00.058,410] <inf> ipc_radio: Received hci message of 5 bytes.
[00:00:00.058,837] <inf> ipc_radio: Sent message of 7 bytes.

ASSERTION FAIL [0] @ WEST_TOPDIR/nrf/subsys/mpsl/init/mpsl_init.c:299
MPSL ASSERT: 112, 1557

[00:00:05.591,186] <err> os: ***** HARD FAULT *****
[00:00:05.591,186] <err> os: Fault escalation (see below)
[00:00:05.591,217] <err> os: ARCH_EXCEPT with reason 4

[00:00:05.591,247] <err> os: r0/a1: 0x00000004 r1/a2: 0x0000012b r2/a3: 0x00000002
[00:00:05.591,247] <err> os: r3/a4: 0x00000004 r12/ip: 0x00000010 r14/lr: 0x010221d3
[00:00:05.591,278] <err> os: xpsr: 0x0100001c
[00:00:05.591,278] <err> os: Faulting instruction address (r15/pc): 0x0102854a
[00:00:05.591,339] <err> os: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
[00:00:05.591,369] <err> os: Fault during interrupt handling

[00:00:05.591,400] <err> os: Current thread: 0x21006e98 (unknown)
[00:00:05.907,012] <err> os: Halting system

[quote userid="26071" url="~/f/nordic-q-a/118189/ipc-problem-with-a-custom-board-nrf5340/523410"]

Can you see any advertisements if you scan for nearby devices using your phone (nRF Connect for Android/iOS)?

Can you connect to it? Does it maintain a stable connection?

[/quote]

I can't because CPU_RADIO stopped working due to the assertion. I tested on my B2 to see if it was a Kconfig or overlay that I configured wrong, but on B2 the sample worked.

Best Regards,

Brendow

RE: IPC problem with a custom board nrf5340

Edvin — Tue, 18 Feb 2025 08:04:16 GMT

Hello Brendow,

Ok, let's check some basics on your B1.

For debugging purposes, can you please try to run a BLE sample? E.g.:

NCS\nrf\samples\bluetooth\peripheral_uart

Are you able to run it? Can you see any advertisements if you scan for nearby devices using your phone (nRF Connect for Android/iOS)?

Can you connect to it? Does it maintain a stable connection?

BR,

Edvin

RE: IPC problem with a custom board nrf5340

brendow — Wed, 12 Feb 2025 12:59:19 GMT

[quote userid="26071" url="~/f/nordic-q-a/118189/ipc-problem-with-a-custom-board-nrf5340/522572"]Was this before or after using the internal oscillator?[/quote]

Using the external and the internal is the same result. The shell only appears when I've added the delay on my B1.

[quote userid="26071" url="~/f/nordic-q-a/118189/ipc-problem-with-a-custom-board-nrf5340/522572"]So it is only one device where you are seeing these issues?[/quote]

Yes, just on a specific board (B1).

[quote userid="26071" url="~/f/nordic-q-a/118189/ipc-problem-with-a-custom-board-nrf5340/522572"]Is it in a particularly noisy environment?[/quote]

I don't think so, the board is connected to my computer USB, and I'm trying to create a Thread network using two boards with the ot-cli sample.

[quote userid="26071" url="~/f/nordic-q-a/118189/ipc-problem-with-a-custom-board-nrf5340/522572"]Have you tried it on a DK? Do you see the same issue there?[/quote]

Yes, I tried and No, I don't. Everything works the same when testing on B2 or DK. I don't think this might be a problem in your code, it's more about an issue in this specific custom board (B1) that I'm trying to figure out what's going on.

[quote userid="26071" url="~/f/nordic-q-a/118189/ipc-problem-with-a-custom-board-nrf5340/522572"]Can you please specify the chain of ot-cli commands that you call, and where it gets stuck (on what command)?[/quote]

I use the following commands:

ot scan

here is the image of my shell using the B1 and the image using the B2. I only used ot scan on both boards after connecting to the shell.

The B1 gets stuck because the IRQ on the CPU_NET never happens after the CPU_APP calls the nrf_802154_transmit_csma_ca_raw(). Then, the nrf_802154_transmitted_raw() is never called on the B1's CPU_NET.

Best regards,

Brendow

RE: IPC problem with a custom board nrf5340

Edvin — Wed, 12 Feb 2025 08:24:37 GMT

[quote user="brendow"]k_msleep(100) inside the while loop in the clock_init(), it started working[/quote]

Was this before or after using the internal oscillator?

So it is only one device where you are seeing these issues? Is it in a particularly noisy environment? Have you tried it on a DK? Do you see the same issue there?

Also, with the board that is getting stuck. Can you please specify the chain of ot-cli commands that you call, and where it gets stuck (on what command)?

Best regards,

Edvin

RE: IPC problem with a custom board nrf5340

brendow — Mon, 10 Feb 2025 20:31:20 GMT

Hi Edvin, I bring some updates.

I have two custom boards here. One of them, which I'm having trouble with, has an external LFXTAL , while the other one doesn’t. I decided to test both boards using only the internal oscillator.

The board with the external LFXTAL and the issue will be called B1, and the board without it will be called B2.

There seems to be something wrong with the nrf5340 on the B1. As I mentioned in the previous thread, when I was flashing the radio_shell, the terminal I was accessing via USB was blank. However, after I added a k_msleep(100) inside the while loop in the clock_init(), it started working.

I’m not sure why adding this 100ms delay in the loop makes a difference. The only explanation I can think of is that the while loop isn’t giving the CPU enough time to run other processes, but that shouldn’t be happening.

So, I tested the same code without the 100ms delay on B2, and it worked fine.

After this, I flashed the ot-cli sample on B2, and it also worked. So, the issue seems to be with B1.

While debugging both boards using CONFIG_NRF_802154_SER_LOG=y, I noticed that the CPU_APP on B1 gets stuck at net_generic_bool_response_await, specifically at this semaphore.

Meanwhile, on B1, CPU_NET’s last message is sending a spinel frame.

However, when I debugged B2, I saw that after CPU_NET sends a frame to CPU_APP, it calls the function nrf_802154_transmitted_raw because of an IRQ in swi_irq_handler(), which triggers nrf_802154_notification_swi() and eventually calls nrf_802154_transmitted_raw().

But when I debug B1, I see that the IRQ caused by nrf_swi never happens. Why is that? Could it be an issue with the nrf5340? Since I’m now using the RC oscillator, is there something on the board that could be causing problems?

I don't know why, but the IRQ that should be called after CPU_APP calls nrf_802154_transmit_csma_ca_raw never happens.

Best Regards,

Brendow

RE: IPC problem with a custom board nrf5340

Edvin — Thu, 30 Jan 2025 13:47:56 GMT

Hello,

I know that there have been issues with the IPC samples before (although I don't know the current state of it). The IPC communication between the application core and network core on the nRF5340, on the other hand, usually works without any issues. There aren't any external components needed for this, so it is probably not related to the issue that you are seeing.

[quote user="brendow"]Could it be a problem with my devicetree, Kconfig or maybe HFXO?[/quote]

That is more likely to be the "issue". Either that something is not configured correctly, or possibly that some components are out of specification.

Does your custom board have an LFXTAL? If not, try adding this to your prj.conf:

CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC=y

Best regards,

Edvin

RE: IPC problem with a custom board nrf5340

brendow — Mon, 27 Jan 2025 19:35:17 GMT

Hi Edvin,

[quote userid="26071" url="~/f/nordic-q-a/118189/ipc-problem-with-a-custom-board-nrf5340/519887"]Are you able to reproduce the original issue you are having on the openthread sample on a DK? [/quote]

I don't have a problem building these samples in my DK, only when using my custom board.

[quote userid="26071" url="~/f/nordic-q-a/118189/ipc-problem-with-a-custom-board-nrf5340/519887"]Are you using nRF Connect SDK v2.9.0 for everything here?[/quote]

Yes.

My question is: How can I see if IPC is working on my custom board? Because I'm unable to access the ipc shell from radio_test sample and the ipc_service had results different from your examples.

Could it be a problem with my devicetree, Kconfig or maybe HFXO?

Best regards,

Brendow

RE: IPC problem with a custom board nrf5340

Edvin — Thu, 23 Jan 2025 22:58:51 GMT

Hello,

A lot of information here. Are you able to reproduce the original issue you are having on the openthread sample on a DK? If so, is it possible to upload that application here, so that I can have a look?

Are you using nRF Connect SDK v2.9.0 for everything here?

Best regards,

Edvin