nrf5340 SDK 1.9.1 idle mode consumption and power management

Question

Problem: I see a constant 1.4-1.5mA power consumption that I am unable to explain, can;t reduce despite trying a few things and need help debugging further

SDK: 1.9.1

MCU: nrf5340 running at 128MHz (not the default 64MHz)

Board: custom with multiple external components (PMIC, Accel, and few other things)

Environment: SES

Build: secure build

We have BLE enabled (few 100 bytes being sent exactly every second), MCUBoot, and DFU.. everything works as expected with zero issues (functionally and timing wise).. except for this current consumption we can not explain

What I have done

1. Basic stuff
Turned off serial/uart/log in prj.conf (for application core AND network core (hci_rpmsg).. we use MCUboot and turned off UART there too
CONFIG_ASSERT=n
CONFIG_LOG=n
CONFIG_CONSOLE=n
CONFIG_SERIAL=n

// UART debug port
&uart0 {
status = "okay";
//status = "disabled";
/delete-property/ rts-pin;
/delete-property/ cts-pin;
tx-pin = < 0x19 >;
rx-pin = < 0x1a >;
current-speed = <460800>;
};

#power management
CONFIG_PM=y
CONFIG_PM_DEVICE=y
CONFIG_PM_DEVICE_RUNTIME=y

and i have added the following too

#include <pm/pm.h>
#include <device.h>

I have added
/* Prevent deep sleep (system off) from being entered */
pm_constraint_set(PM_STATE_SOFT_OFF);

Questions:

1a. FPU is ON.. does it make any difference (we need it and hence can't turn it off)..

1b: shutting UART off should reduce power by 500uA right ? or is it more ?

2. I have a few GPIOs configured as interrupt pins.. most are edge triggered GPIO_INT_EDGE_FALLING or GPIO_INT_EDGE_RISING (won;t work otherwise so, i cant go for level triggered at this time)..Only button is GPIO_INT_LEVEL_INACTIVE.. Looks like in 1.9.1 we need to add an entry in overlay to take care of masking for sense-edge and i did..

&gpio0 {
status = "okay";
sense-edge-mask = < 0xffffffff >;
};

&gpio1 {
status = "okay";
sense-edge-mask = < 0xffffffff >;
};

Will making interrupts edge triggered add 100s of uA like it is mentioned in some older tickets ? is there a way around that I am missing ?

3. do we have to add CONFIG_PM=y and CONFIG_PM_DEVICE=y to mcuboot and hci_rpmsg conf files too ? I haven't tried it yet

4. looks like idle thread runs.. (i even added a dummy idle with low priority to see if CPU is always busy doing stuff.. and i see that my dummy thread gets called ample times too..

just for testing, I enabled UART and added a print (

if (pm_system_suspend(_kernel.idle) == false) {
printk("\n suspend failed");
k_cpu_idle();
}

and I see that print suspend failed ALL the time which is worrying.. but wasn't sure if it's the print that's making it fail or is it really failing.. hence asking you guys if this is something worth worrying and investigating

5. using my PMIC , I cut off power for all external components and I still see suspend failed and still this additional power consumption. I disconnect the JTAG and reset the board after flashing .. just to avoid answering "is JTAG somehow enabled.. have you reset the board"

6. I do have I2c2 and 3 turned on configured at I2C_BITRATE_STANDARD (not doing transactions at the time of measurement).. according to datasheet, this should not add 100s of uAs but may be 10of uA.. I also have one SPI port configured (but NOT doing any transactions).. Is this ok or should I shut down everything except when i am using (makes the application design a bit more complex :-) )

7. I use QSPI (single wire) to talk to a Display controller and when display is not in use, QSPI is shutdown and pins are disconnected.

8. Eventually, I shutdown all peripherals and I still see no evidence of CPU going to idle and power consumption is close to 1 mA which really kind of shocked me.so I am questioning what else do i have to do .. base don what i have read, CONFIG_PM=y will ensure idle thread will run (and i verified that it does) and will ALSO push the cores to a lower power mode (which I don't see)..
CONFIG_PM_DEVICE=y will need explicit power management api calls and implementations from individual drivers.. i have done that for my display driver (and have verified that it shuts down)
CONFIG_PM_DEVICE_RUNTIME=y --> I am not sure if this needed or should I removed it

I am close to the point of factory production of our device and this issue has become critical given this extra current consumption has reduced the battery life to half of what we would need it to be