Issues with MCUmgr over serial with nRF52 devkit/J-Link

[FARLY7]

I am currently experiencing an issue using MCUmgr CLI over a serial connection provided by the on-board J-Link debuggers on nRF52 devkits.

When using MCUmgr CLI to send smaller simpler commands like echo and image list to the devkit, the first one succeeds but all subsequent commands result in an NMP timeout. I will need to disconnect and reconnect the nRF52 devkit for the first command to succeed again. When attempting to actually update an image, it will never pass 0%.

There are no issues when I try this same process using our production board with direct access to the UART interface through a CP2105 USB-to-UART bridge.

I suspect it has something to do with the serial comms being routed through the J-Link. Some timing/buffer limitation issues? I have tried different variations of the mtu=xxx parameter but no success unfortunately. I have also tested on Linux and macOS, but not Windows.

I appreciate this is likely more of an MCUmgr issue, but I guessed since it involved the onboard J-Link on the nRF52 devkits perhaps you have some previous experience. 

[Håkon Alseth]

Hi,

 

Are you seeing this issue when using the smp_svr example? Could you share the configuration that you're using so I can try to recreate the issue on my side?

 

Kind regards,

Håkon

[FARLY7]

Hi Hakon,

Sorry for the delay in response.

Yes this is using the smp_svr example. I think you should be able to easily recreate it just by flashing the smp_svr example to the nRF52 devkit and attempting to upload an image over the USB serial port.

Thanks,

Sean

[Håkon Alseth]

Hi,

 

If you have issues with the usb-serial bridge in the debugger, it might be beneficial to disable the mass storage device in the debugger itself.

This is done by opening J-Link Commander (JLinkExe on mac/linux) and writing:

msddisable

Remember to power cycle the board afterwards before testing again.

 

Kind regards,

Håkon

[FARLY7]

Yes, I have done this already. This is one of the first things I do with a new devkit.

I have also tried different MTU sizes in the mcumgr connection parameters, but no luck.

I saw your post on this similar issue, I will try your suggestion here and see if it works.

[Håkon Alseth]

Hi,

 

FARLY7 said:

I saw your post on this similar issue, I will try your suggestion here and see if it works.

That is specifically for the nrf9160dk_nrf9160_ns target.

I have seen similar issues in the past myself and disabling the MSD has usually been the fix.

I just tested ncs v1.8.0 and ncs v1.9.0, with board nrf52dk_nrf52832, stock smp_svr (w/overlay-serial.conf):

$ mcumgr --conntype serial --connstring /dev/ttyACM0 image upload zephyr/app_update.bin
 48.14 KiB / 48.14 KiB [====================================================================================] 100.00% 3.46 KiB/s 13s
Done

image list shows: 

Images:
 image=0 slot=0
    version: 0.0.0
    bootable: true
    flags: active confirmed
    hash: ed10f5869785d67af2fa7d416016a6836c78c6254cd330b4446ba21b316ac2e4
 image=0 slot=1
    version: 0.0.0
    bootable: true
    flags: 
    hash: 1ffb1291735140f39b1ba1f920b14e07e26136abde2d92109ddacbd047d509a7
Split status: N/A (0)

  

Do you see the same behavior on other PCs as well?

 

Kind regards,

Håkon

[FARLY7]

Hi Hakon,

I just tried the suggestion from your answer in the other issue thread, and this worked.

I will also test the smp_svr example again like you did. A little bit more investigation is required on my side to clarify where the issue is and how it is fixed.

Thanks,

Sean

[Håkon Alseth]

Hi,

 

FARLY7 said:

I just tried the suggestion from your answer in the other issue thread, and this worked.

This is interesting!

Was the CONFIG_HEAP_MEM_POOL_SIZE set to non-zero value prior to adjusting it?

 

Kind regards,

Håkon

[FARLY7]

No problem Hakon, I will check this tomorrow and get back to you.

One other question perhaps you can help with: Do you know how to increase the upload speed? It’s taking me 3.5mins to upload one ~256KB image. I am using serial recovery mode in MCUBoot if that makes any difference.

What kind of upload speed/times do you see?

I think I read somewhere else that there is a hard coded delay of 20ms between each packet in mcumgr tool, but I’ve not investigated this.

[rerickson]

FARLY7 said:

Do you know how to increase the upload speed?

In the past upping the MTU will increase the update speed:

CONFIG_MCUMGR=y

CONFIG_MCUMGR_SMP_SHELL=y

CONFIG_MCUMGR_SMP_SHELL_MTU=2048

CONFIG_MCUMGR_BUF_SIZE=2050

CONFIG_MGMT_CBORATTR_MAX_SIZE=2048

CONFIG_FS_MGMT_UL_CHUNK_SIZE=2048

[FARLY7]

Hi, late reply. But my speed related issue was actually related to this bug in Mcumgr: https://github.com/apache/mynewt-newtmgr/issues/185.

Applying the provided patch, I was able to reduce the upload times down from 3.5mins to a more respectable 44s @ 115,200 baud:

Original at 115,200 baud:
247.23 KiB / 247.23 KiB [===============================] 100.00% 1.14 KiB/s 3m37s
with patch:
247.23 KiB / 247.23 KiB [===============================] 100.00% 5.56 KiB/s 44s

For me, this issue was only occurring while using Serial Recovery mode of MCUBoot, it was not occurring when updating the image using application-level SMP server. I guess this has something to do with the flash alignment of the primary slot vs secondary slot. In Serial Recovery mode, Mcumgr was writing directly to primary slot, where as application-level updates the update is written to the secondary slot.

Another interesting point for those running macOS. The serial_posix.go file in the Tarm serial library used by Mcumgr CLI tool limits the baud rate to 115,200, which is a standard POSIX limit. We also created a patch that removed the use of constants to define the baud rate and instead use the absolute baud rate value. This allowed us to run baud rates beyond 115,200. The highest we tested was 921,600 baud, bringing the upload times further down to 11s! A massive improvement from 3.5mins.

With patch at 921,600 baud:
247.03 KiB / 247.03 KiB [===============================] 100.00% 21.89 KiB/s 11s