i2s peripheral not generating pointer update events when MCUBoot enabled on nrf5340 device

[responsible_duck]

Hello,

As soon as we enable CONFIG_BOOTLOADER_MCUBOOT and have an operational MCUBoot image, we noticed that our audio pipeline is no longer working for our i2s omni-microphone. Specifically we fail to allocate buffers. When we pull out the bootloader configuration it works again. This is on 2.6.1 of the SDK / toolchain. After days of debugging this is what we found:

The reason for the audio errors is that the I2S peripheral is not generating pointer update events, so the ISR is not refreshing buffers and our audio thread times out after 2 seconds. We have some PDM mics as well which do not have an issue on a similar audio processing stack; it's limited to I2S. And we don't suspect anything with the I2S mic itself, as the nRF I2S peripheral generates the clocks - even if there was no mic connected, things would still run. Looked extensively at the driver changes in 2.6.1 and nothing stood out. Copied over the 2.5.2 I2S drivers into 2.6.1 and that didn't help.

Also copied the nrfx drivers from 2.6.1 into 2.5.2, and used the pre-2.6.1 FW, and that still worked.
Messed around in the debugger on the working and non-working builds and we can reproduce the behavior by poking registers, so I'm not suspecting runtime issues (though could possibly be init). We backed out bootloader, staying with SDK 2.6.1, and things work again.

But we need to get MCUBoot and the audio pipeline working. Please we are so close. So here are some questions:

• Could this be related to enabling the internal 32 khz clock RC for mcuboot? (ie CONFIG_CLOCK_CONTROL_NRF_K32SRC_RC and CONFIG_CLOCK_CONTROL_NRF_K32SRC_500PPM)
• Are there any hardware inits in bootloader, specifically with clocks or voltage, that may affect the running image?

Here are some things we're thinking of trying next but its a bit of a rabbit hole:

• I don't think the symptoms indicate any pin conflicts, but check if there are any leftover configs in bootloader.
• Is it possible to enable bootloader in 2.5.2 with older FW, just for comparison?
• Can you hack bootloader to skip most of its operations and just transfer to the main image?
• Is NET processor being initialized, and causing some side effect?
• Leftover flash, or some memory layout issue which is causing second-order effects?

For reference of our configuration files, you can see here:  nRF5340 Custom Board App not booting when enabling CONFIG_BOOTLOADER_MCUBOOT 

[responsible_duck]

We narrowed it down to serial recovery being enabled:

`CONFIG_MCUBOOT_SERIAL=y`

With MCUBOOT enabled and `CONFIG_MCUBOOT_SERIAL=n` in the `child_image/mcuboot.conf`, our audio pipeline via i2s works fine. However, we need serial recovery mode enabled. 

Any insight as to why this would cause i2s to fail?

[AHaug]

Hi,

Glad you were able to resolve this. I were about to reach out to you regarding setting the serial config to n based on some input from my colleagues but it got lost when I were trying to find the reason for why this would be the fix.

From what I were able to uncover after discussing with my colleagues it seems to be caused by something in the cleanup after booting causing this configuration to make it so that some GPIOs still are in use. I will see if I can find more information regarding this or if it has been patched in the soon-to-be-released V2.7.0, but for now this is alteast whats causing it.

Kind regards,
Andreas

[AHaug]

Hi again,

It looks to be related to this PR: https://github.com/nrfconnect/sdk-mcuboot/pull/303 

I will ask around internally for status regarding this issue.

Kind regards,
Andreas

[responsible_duck]

Hi,

Thanks for this update AHaug ! Unfortunately this is blocking us could you instruct me as to how to point my project's MCUBoot to a branch or local version so I can use this fix in the mean time? 

[AHaug]

You can always cherrypick the suggested fix into your fork of the main ncs branch: https://docs.nordicsemi.com/bundle/ncs-latest/page/nrf/dev_model_and_contributions.html contains resources for how to manage your own fork and add cherrypick fixes to it. 

I see that the PR is not merged into main, but I suspect it will happen soon based on the comment from yesterday.

In addition, I'm a bit uncertain if this PR fixes the issue, but your symptoms (and the fix) suggests it may do. If your I2s uses the same pins as your UART1 does, then is should be reinitialized by the I2s driver when your app start. Could you check and verify if you're using a different pinout when you're in bootloader than when you're in the app?

Kind regards,
Andreas

[responsible_duck]

We use P0.19-21 for the i2s lines in our app but don't have a specified overlay for our bootloader (besides button to enter serial recovery). So I guess we would use the bootloader's default pinout? Do these correspond with the lines used for UARTE?

[AHaug]

responsible_duck said:

So I guess we would use the bootloader's default pinout?

That might resolve the issue, but it would require verification through testing

responsible_duck said:

Do these correspond with the lines used for UARTE?

Atleast for the DK it is set up as explained in https://docs.nordicsemi.com/bundle/ug_nrf5340_dk/page/UG/dk/vir_com_port.html#ariaid-title1 

But since you're using a custom board it will depend your board files

Kind regards,
Andreas

[responsible_duck]

I tried cherrypicking the fix mentioned above (https://github.com/nrfconnect/sdk-mcuboot/pull/303) in my local v2.6.1 sdk but it did not resolve the issue.

So we have a nrf5340_cpuapp_common-pinctrl.dtsi file that we use mainly to build for devkit. This contains a uart definition that seems to reflect the uart configuration you linked:

&pinctrl {
	uart0_default: uart0_default {
		group1 {
			psels = <NRF_PSEL(UART_TX, 0, 20)>,
				<NRF_PSEL(UART_RTS, 0, 19)>;
		};
		group2 {
			psels = <NRF_PSEL(UART_RX, 0, 22)>,
				<NRF_PSEL(UART_CTS, 0, 21)>;
			bias-pull-up;
		};
	};
}

In our custom board overlay we reconfigure these pins to be i2s:

/delete-node/ &{/pin-controller/uart0_default/group1/};
/delete-node/ &{/pin-controller/uart0_default/group2/};
/delete-node/ &{/pin-controller/uart0_sleep/group1/};

&uart0_default {
    group1 {
        psels = <NRF_PSEL(UART_RX, 0, 9)>, <NRF_PSEL(UART_TX, 0, 11)>;
    };
};

//  Pin configurations for PDM device
&pinctrl {
    pdm0_default_alt: pdm0_default_alt {
        group1 {
            psels = <NRF_PSEL(PDM_CLK, 0, 4)>,
                    <NRF_PSEL(PDM_DIN, 0, 5)>;
        };
    };

    i2s0_default: i2s0_default {
        group1 {
            psels = <NRF_PSEL(I2S_LRCK_M, 0, 20)>,
                    <NRF_PSEL(I2S_SDIN, 0, 21)>,
                    <NRF_PSEL(I2S_SCK_M, 0, 19)>;
        };
    };
}

This seems to work for our app but I think mcuboot only picks up the common dts configurations and not the custom board overlays? Why is that even though it is built together with the app?

Any other suggestions on why i2s is not functioning after entering app from serial recovery mode?