NRF9160 - BSD lib crash - UDP recv() - potential security issue

RE: NRF9160 - BSD lib crash - UDP recv() - potential security issue

Didrik Rokhaug — Fri, 10 Jun 2022 07:31:12 GMT

Hi,

You might want to check out NCS 2.0.0 and modem firmware 1.3.2.

We have made some changes under the hood which should increase the flow control between the modem and application cores, reducing the chances of lockups due to full buffers.

Best regards,

Didrik

RE: NRF9160 - BSD lib crash - UDP recv() - potential security issue

emh203 — Wed, 08 Jun 2022 15:51:34 GMT

Hello:

To clarify, the issue happens if you read() "slowy". For example, if my thread sleeps for 250mS in between read calls. If several packets come in quickly, send() will block forever. The condition appears to be unrecoverable other than a watchdog reset.

Having separate threads for send and recv() may help but makes the code/logic messy.

The most stable workaround is

1.) Implement a task_wdt system that will reset the system if there is a lockup.

2.) When calling recv(), do something like this in a while loop:

while((rcvd = recv(krusty_client_fd, (uint8_t *)&krusty_rx_buf, sizeof(krusty_rx_buf), MSG_DONTWAIT) ) >0)

This will help drain the queue. In my case, I have send/recv in the same thread/state machine. If I do not use the while() loop, the system can lock up in the time in between the send() and recv() call.

3.) Minimize an delay/sleep time in between a send() and recv() call. Initially my network thread had some sleep calls in the state machine and I had to make sure I call recv() frequently.

4.) Make the packet heap as large as possible.

It is pretty easy to send a bunch of packets from a server quickly to cause the lockup.

It would be nice if the modem library had its source code in the NCS/Zephyr repository. Simply having an option to never have send() block would be beneficial (at the expense of dropping some incoming packets). Applications based upon UDP expect potential packet loss anyway.

RE: NRF9160 - BSD lib crash - UDP recv() - potential security issue

wileycwj_ath — Tue, 07 Jun 2022 00:13:26 GMT

I also am experiencing this same issue on SDK 1.6.1 and modem_fw 1.3.1.

Observing same as @emh203 where TCP sockets are failing after some amount of minutes / hours depending on how many requests are sent.

[quote userid="14473" url="~/f/nordic-q-a/75598/nrf9160---bsd-lib-crash---udp-recv---potential-security-issue/313459#313459"]

- you observe that when you do read, the send() function will block eventually; this needs investigation

[/quote]

Is there a patch or solution for this that doesn't involve just shutting down the modem and application?

RE: NRF9160 - BSD lib crash - UDP recv() - potential security issue

emdi — Thu, 03 Jun 2021 14:01:31 GMT

Hi,

if you do not read incoming data, it is expected that the Heap becomes full and allocations start to fail so I don't see that as an indication of a leak.

To sum up what I gathered so far:

- you observe that when you stop reading, the send() call will block; this is a known issue.

- you observe that when you stop reading, the heap becomes full and allocation fails; this is expected behavior

- you observe that when you do read, the send() function will block eventually; this needs investigation

Are you able to observe the heap filling up and allocations failing when you keep reading from the socket? That would be an indication of a leak.

RE: NRF9160 - BSD lib crash - UDP recv() - potential security issue

emh203 — Thu, 03 Jun 2021 12:22:50 GMT

Didrik Rokhaug

Thanks. My temporary workaround is to enable a task watchdog to reset the NRF9160. Let me know if there is a new branch I can pull from to test a fix.

RE: NRF9160 - BSD lib crash - UDP recv() - potential security issue

emh203 — Thu, 03 Jun 2021 12:16:36 GMT

Hello Emdi:

If you read the data after the Heap has become full, you should see both the Heap being emptied and your send() call unblocking.

Please read the problem report more carefully. Once this issue occurs, it is unrecoverable. I initially discovered in my application that performs recv(). Disabling recv() allows us to demonstrate the error more quickly. (seconds instead of hours/days). Even with recv() being called frequently, several packets coming in quickly can lock up the modem library.

Once the allocator fails, it looks like the modem library blocks forever (verified w/ debugger). It is not possible to perform any network operations at that point. The library is closed source so the developers need to look into it.

RE: NRF9160 - BSD lib crash - UDP recv() - potential security issue

emdi — Thu, 03 Jun 2021 09:52:24 GMT

Hi,

It is expected that if you don't read incoming data, the Heap memory will become full like your logs show. If you read the data after the Heap has become full, you should see both the Heap being emptied and your send() call unblocking.

RE: NRF9160 - BSD lib crash - UDP recv() - potential security issue

visu — Thu, 03 Jun 2021 09:29:54 GMT

Hi Didrik and emh203,

It wasn't this problem. I will investigate the problem and open a new ticket if necessary.

RE: NRF9160 - BSD lib crash - UDP recv() - potential security issue

Didrik Rokhaug — Thu, 03 Jun 2021 09:26:39 GMT

Hi visu,

I believe the problems described by emh203 only happens when you receive data.

I recommend that you open a new ticket with your problem, as it can be hard to keep track of two different problems in one ticket.

emh203, thanks for the logs. I have forwarded them to our developers, and will let you know when they have looked at them.

Best regards,

Didrik

RE: NRF9160 - BSD lib crash - UDP recv() - potential security issue

emh203 — Wed, 02 Jun 2021 16:37:51 GMT

This issue became apparent in the read case for me. Note that it also affect TCP as well. I have not tested "write only". I would turn on:

CONFIG_NRF_MODEM_LIB_DEBUG_ALLOC=y

CONFIG_NRF_MODEM_LIB_DEBUG_SHM_TX_ALLOC=y

CONFIG_NRF_MODEM_LIB_HEAP_DUMP_PERIODIC=y

CONFIG_NRF_MODEM_LIB_SHM_TX_DUMP_PERIODIC=y

CONFIG_NRF_MODEM_LIB_LOG_LEVEL_DBG=y

If it can happen in the write-only case, then it should be apparent in the UDP example in nrf_connect as well (it writes data only).

The "read" side of things makes it much easier to reproduce (within seconds)

RE: NRF9160 - BSD lib crash - UDP recv() - potential security issue

visu — Wed, 02 Jun 2021 09:01:00 GMT

Hi, I am a litle bit curious regarding this issue. Does this only happen if the nrf9160 receives data?

I have an application which sends udp data every 5-7 seconds, but I never read the recv because I do not expect any data to receive (it sends udp data to a server which does not respond, similiar to your udp-sample-test). the application seems to crash after 1hour-1h30min.

I didn't debugged it yet, but I am just curious if this maybe something has to do with this issue :)

RE: NRF9160 - BSD lib crash - UDP recv() - potential security issue

emh203 — Tue, 01 Jun 2021 15:43:23 GMT

Hello Didrik:

I was able to turn on those diagnostics and gather a log. Also, I used the UDP example in

nrf/samples/nrf9160/udp

to demonstrate the issue on the NRF9160DK

Setup:

1.) Build the sample in nrf/samples/nrf9160/udp

I added the following to the prj.conf

CONFIG_NRF_MODEM_LIB_DEBUG_ALLOC=y

CONFIG_NRF_MODEM_LIB_DEBUG_SHM_TX_ALLOC=y

CONFIG_NRF_MODEM_LIB_HEAP_DUMP_PERIODIC=y

CONFIG_NRF_MODEM_LIB_SHM_TX_DUMP_PERIODIC=y

CONFIG_NRF_MODEM_LIB_LOG_LEVEL_DBG=y

CONFIG_UDP_DATA_UPLOAD_SIZE_BYTES=10

CONFIG_UDP_DATA_UPLOAD_FREQUENCY_SECONDS=10

CONFIG_UDP_SERVER_ADDRESS_STATIC="104.236.49.95"

CONFIG_UDP_SERVER_PORT=2100

CONFIG_UDP_PSM_ENABLE=n

CONFIG_UDP_EDRX_ENABLE=n

CONFIG_UDP_RAI_ENABLE=n

Note: The IP address and port is for a test server.

2.) Start the UDP test script on the test server. I have attached "udp_test.py" to this This code was started on the server terminal:

python3 udp_test.py

This code listens on port 2100 (a random test port I picked). It will print the received data and then send back the string "Hello UDP Client" six times for every message it receives. I chose six as it will trigger the issue a bit quicker.

3.) Start the UDP Sample and capture the output/log messages via putty.

Results:

See NRF9160_putty.log. This is the all the printk/log messages from boot until the program stops. You will be able to see where the memory allocation fails.

Once the failure occurs, the function to periodically send data on the work queue will stop.

Note: In this sample, incoming is never read as it will trigger the issue more quickly. I initially found that the issue will still eventually occur even if the socket is periodically read. (it might take several hours or days)

In this test setup, the fault will occur after 4 transmissions from the client. Here is the terminal output from the python script.

forge@quaint-crater:~$ python3 udp_test.py
UDP server up and listening
Message from Client:b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
Client IP Address:('12.153.231.186', 60881)
Message from Client:b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
Client IP Address:('12.153.231.186', 60881)
Message from Client:b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
Client IP Address:('12.153.231.186', 60881)
Message from Client:b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
Client IP Address:('12.153.231.186', 60881)

Here is the putty log from the NRF9160. Note that I captured data until the modem_lib process locked up.

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2021.06.01 11:22:21 =~=~=~=~=~=~=~=~=~=~=~=
I: alloc(16) -> 0x200154f4
I: alloc(12) -> 0x2001550c
*** Booting Zephyr OS build v2.4.99-ncs1-3525-g4d068de3f50f  ***
I: alloc(16) -> 0x2001551c
I: alloc(12) -> 0x20015534
UDP sample has started
I: shm_tx_alloc(15) -> 0x20010524
I: shm_tx_free(0x20010524)
I: alloc(20) -> 0x20015544
I: free(0x20015544)
I: shm_tx_alloc(10) -> 0x20010524
I: shm_tx_free(0x20010524)
I: alloc(20) -> 0x20015544
I: free(0x20015544)
I: shm_tx_alloc(10) -> 0x20010524
I: shm_tx_free(0x20010524)
I: alloc(20) -> 0x20015544
I: free(0x20015544)
I: shm_tx_alloc(9) -> 0x20010524
I: shm_tx_free(0x20010524)
I: alloc(20) -> 0x20015544
I: free(0x20015544)
I: shm_tx_alloc(11) -> 0x20010524
I: shm_tx_free(0x20010524)
I: alloc(20) -> 0x20015544
I: free(0x20015544)
I: shm_tx_alloc(22) -> 0x20010524
I: shm_tx_free(0x20010524)
I: alloc(20) -> 0x20015544
I: free(0x20015544)
I: shm_tx_alloc(10) -> 0x20010524
I: shm_tx_free(0x20010524)
I: alloc(20) -> 0x20015544
I: free(0x20015544)
I: shm_tx_alloc(10) -> 0x20010524
I: shm_tx_free(0x20010524)
I: alloc(20) -> 0x20015544
I: free(0x20015544)
I: shm_tx_alloc(9) -> 0x20010524
I: shm_tx_free(0x20010524)
I: alloc(20) -> 0x20015544
I: free(0x20015544)
nrf_modem tx dump:
Heap at 0x200104e8 contains 1023 units in 10 buckets

  bucket#    min units        total      largest      largest
             threshold       chunks      (units)      (bytes)
  -----------------------------------------------------------
        9          512            1         1016         8124

8124 free bytes, 0 allocated bytes, overhead = 64 bytes (0.8%)
Failed allocations: 0
nrf_modem heap dump:
Heap at 0x200154c0 contains 127 units in 7 buckets

  bucket#    min units        total      largest      largest
             threshold       chunks      (units)      (bytes)
  -----------------------------------------------------------
        6           64            1          111          884

884 free bytes, 64 allocated bytes, overhead = 72 bytes (7.1%)
Failed allocations: 0
nrf_modem tx dump:
Heap at 0x200104e8 contains 1023 units in 10 buckets

  bucket#    min units        total      largest      largest
             threshold       chunks      (units)      (bytes)
  -----------------------------------------------------------
        9          512            1         1016         8124

8124 free bytes, 0 allocated bytes, overhead = 64 bytes (0.8%)
Failed allocations: 0
nrf_modem heap dump:
Heap at 0x200154c0 contains 127 units in 7 buckets

  bucket#    min units        total      largest      largest
             threshold       chunks      (units)      (bytes)
  -----------------------------------------------------------
        6           64            1          111          884

884 free bytes, 64 allocated bytes, overhead = 72 bytes (7.1%)
Failed allocations: 0
nrf_modem tx dump:
Heap at 0x200104e8 contains 1023 units in 10 buckets

  bucket#    min units        total      largest      largest
             threshold       chunks      (units)      (bytes)
  -----------------------------------------------------------
        9          512            1         1016         8124

8124 free bytes, 0 allocated bytes, overhead = 64 bytes (0.8%)
Failed allocations: 0
nrf_modem heap dump:
Heap at 0x200154c0 contains 127 units in 7 buckets

  bucket#    min units        total      largest      largest
             threshold       chunks      (units)      (bytes)
  -----------------------------------------------------------
        6           64            1          111          884

884 free bytes, 64 allocated bytes, overhead = 72 bytes (7.1%)
Failed allocations: 0
I: alloc(20) -> 0x20015544
I: free(0x20015544)
LTE cell changed: Cell ID: 61774082, Tracking area: 61706
I: alloc(20) -> 0x20015544
I: free(0x20015544)
RRC mode: Connected
I: alloc(20) -> 0x20015544
I: free(0x20015544)
I: alloc(20) -> 0x20015544
I: free(0x20015544)
RRC mode: Idle

I: alloc(20) -> 0x20015544
I: free(0x20015544)
LTE cell changed: Cell ID: 122020869, Tracking area: 22018
I: alloc(20) -> 0x20015544
I: free(0x20015544)
RRC mode: Connected
I: alloc(20) -> 0x20015544
I: free(0x20015544)
RRC mode: Idle

I: alloc(20) -> 0x20015544
I: free(0x20015544)
I: alloc(20) -> 0x20015544
I: free(0x20015544)
RRC mode: Connected
I: alloc(20) -> 0x20015544
I: free(0x20015544)
Network registration status: Connected - roaming

I: alloc(16) -> 0x20015544
I: alloc(12) -> 0x2001555c
PSTransmitting UDP/IP payload of 38 bytes to the IP address 104.236.49.95, port number 2100
I: shm_tx_alloc(10) -> 0x20010524
M parameter update: TAU: -1, Active time: -1
I: shm_tx_free(0x20010524)
I: alloc(16) -> 0x2001556c
I: alloc(20) -> 0x20015584
I: alloc(16) -> 0x2001559c
I: alloc(20) -> 0x200155b4
I: alloc(16) -> 0x200155cc
I: alloc(20) -> 0x200155e4
I: alloc(16) -> 0x200155fc
I: alloc(20) -> 0x20015614
I: alloc(16) -> 0x2001562c
I: alloc(20) -> 0x20015644
I: alloc(16) -> 0x2001565c
I: alloc(20) -> 0x20015674
nrf_modem tx dump:
Heap at 0x200104e8 contains 1023 units in 10 buckets

  bucket#    min units        total      largest      largest
             threshold       chunks      (units)      (bytes)
  -----------------------------------------------------------
        9          512            1         1016         8124

8124 free bytes, 0 allocated bytes, overhead = 64 bytes (0.8%)
Failed allocations: 0
nrf_modem heap dump:
Heap at 0x200154c0 contains 127 units in 7 buckets

  bucket#    min units        total      largest      largest
             threshold       chunks      (units)      (bytes)
  -----------------------------------------------------------
        6           64            1           70          556

556 free bytes, 336 allocated bytes, overhead = 128 bytes (12.5%)
Failed allocations: 0
Transmitting UDP/IP payload of 38 bytes to the IP address 104.236.49.95, port number 2100
I: shm_tx_alloc(10) -> 0x20010524
I: shm_tx_free(0x20010524)
I: alloc(16) -> 0x2001568c
I: alloc(20) -> 0x200156a4
I: alloc(16) -> 0x200156bc
I: alloc(20) -> 0x200156d4
I: alloc(16) -> 0x200156ec
I: alloc(20) -> 0x20015704
I: alloc(16) -> 0x2001571c
I: alloc(20) -> 0x20015734
I: alloc(16) -> 0x2001574c
I: alloc(20) -> 0x20015764
I: alloc(16) -> 0x2001577c
I: alloc(20) -> 0x20015794
Transmitting UDP/IP payload of 38 bytes to the IP address 104.236.49.95, port number 2100
I: shm_tx_alloc(10) -> 0x20010524
I: shm_tx_free(0x20010524)
I: alloc(16) -> 0x200157ac
I: alloc(20) -> 0x200157c4
I: alloc(16) -> 0x200157dc
I: alloc(20) -> 0x200157f4
I: alloc(16) -> 0x2001580c
I: alloc(20) -> 0x20015824
I: alloc(16) -> 0x2001583c
I: alloc(20) -> 0x20015854
I: alloc(16) -> 0x2001586c
I: alloc(20) -> 0x20015884
I: alloc(16) -> 0x2001589c
W: alloc(20) -> (nil)
W: alloc(16) -> (nil)
nrf_modem tx dump:
Heap at 0x200104e8 contains 1023 units in 10 buckets

  bucket#    min units        total      largest      largest
             threshold       chunks      (units)      (bytes)
  -----------------------------------------------------------
        9          512            1         1016         8124

8124 free bytes, 0 allocated bytes, overhead = 64 bytes (0.8%)
Failed allocations: 0
nrf_modem heap dump:
Heap at 0x200154c0 contains 127 units in 7 buckets

  bucket#    min units        total      largest      largest
             threshold       chunks      (units)      (bytes)
  -----------------------------------------------------------
        0            1            1            1            4

4 free bytes, 796 allocated bytes, overhead = 220 bytes (21.6%)
Failed allocations: 2
Transmitting UDP/IP payload of 38 bytes to the IP address 104.236.49.95, port number 2100
I: shm_tx_alloc(10) -> 0x20010524
W: alloc(16) -> (nil)
W: alloc(16) -> (nil)
W: alloc(16) -> (nil)
W: alloc(16) -> (nil)
W: alloc(16) -> (nil)

devzone.nordicsemi.com/.../nrf9160_5F00_putty.log

devzone.nordicsemi.com/.../udp_5F00_test.py devzone.nordicsemi.com/.../ServerOutput.txt

RE: NRF9160 - BSD lib crash - UDP recv() - potential security issue

Didrik Rokhaug — Mon, 31 May 2021 10:24:33 GMT

[quote user="emh203"]While I understand the need to recv() to read data, I believe it is a serious bug for the modem library to ever block indefinitely. Data should simply be dropped.[/quote]

We can't drop the data as that would break TCP amongst other things.

But, it also looks like you have found a memory leak. Could you turn on the modem_lib diagnostics and send us the log?

CONFIG_NRF_MODEM_LIB_DEBUG_ALLOC=y
CONFIG_NRF_MODEM_LIB_DEBUG_SHM_TX_ALLOC=y
CONFIG_NRF_MODEM_LIB_HEAP_DUMP_PERIODIC=y
CONFIG_NRF_MODEM_LIB_SHM_TX_DUMP_PERIODIC=y
CONFIG_NRF_MODEM_LIB_LOG_LEVEL_DBG=y

RE: NRF9160 - BSD lib crash - UDP recv() - potential security issue

emh203 — Thu, 27 May 2021 15:58:33 GMT

I have been able to narrow down the issue even further. My goal was to figure out how to trigger the issue on demand and then make it better/worse. Also, I was able to use Segger Ozone w/ Zephyr thread aware debugging to see what is happening.

Test 1 - Simple UDP Server Response - No recv()

I have a dedicated zephyr thread that will open/connect to UDP socket on a server. I will send a small message to the server every 10 seconds and it will respond with 4 small data packets.

In my UDP thread, I disable recv().

The thread will get locked up almost immediately. I found that the thread is lockup up waiting on a semaphore. See attached image

It looks like the thread is blocked @ interface_socket_wait()

If I change this parameter:

CONFIG_NRF_MODEM_LIB_HEAP_SIZE=2048

The thread will no longer lock up immediately . It would take longer but it would eventually lock up.

Test 2 - Simple UDP Server Response - Enable recv()

Same as #1 but instead I perform a recv() every 100mS to read incoming packets.

In this test case, the thread will work for awhile but will eventual get to a locked state *even with recv()* calls. It can take several hours but it will eventually happen

The server responds to my client with 4 packets. Packet one is 40 bytes and Packet 2,3 and 4 are 96 bytes. Packets 1 and 2 are sent immediately and 2,3 and 4 are spaced by 50mS

It feels like the back to back packets might be an issue.

Test 3 - Simple TCP Server w/ Netcat

This test can be used to easily replicate the issue. In my test thread, I would connect to a remote server using TCP instead of UDP:

client_fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);

On the server side, I could use netcat to accept connections. I would then simply paste data in the console to send data.

A.) In my test thread, I disabled recv() calls. The thread would lock up almost immediately when the server response. The thread will also block at the same place as test #1

B.) Changing to CONFIG_NRF_MODEM_LIB_HEAP_SIZE=2048 helps the problem but it is still possible to lock it up.

C.) If I enable recv() with a 100mS read rate in my thread, the thread will still eventually lock up.

Conclusion:

I believe there is a memory allocation issue in the modem library. Increasing the heap size helps but does not fix the issue. Even when periodically calling recv(), it will eventually block in the interface_socket_wait semaphore.

I would appreciate a developer to looking at the the modem_lib code as it is not provided in nrf_connect sdk. The problem can be easily recreated w/ netcat on a test server.

While I understand the need to recv() to read data, I believe it is a serious bug for the modem library to ever block indefinitely. Data should simply be dropped.

My only workaround at this point is to enable the watchdog to reset the CPU. This is occurring with a trivial socket use case with small amounts of data.

RE: NRF9160 - BSD lib crash - UDP recv() - potential security issue

emh203 — Wed, 26 May 2021 16:52:56 GMT

Hello Didrik:

I understand this. In my description, I completely disabled reads to trigger the issue quicker. In my normal application, I do keep reading but there is a transient error that locks up the modem library completely.

After some more experimentation, I actually found that if

CONFIG_NRF_MODEM_LIB_HEAP_SIZE=512 (default)

I can trigger the bug.

If I change to:

CONFIG_NRF_MODEM_LIB_HEAP_SIZE=2048

Then my test case where I turn off recv() will not cause the modem_lib to get hung up.

In any case, not reading a socket fast enough should never cause the underlying modem library to lock up. Can I get source code to see why this is the case? Or can you please ask one of the developers? I noticed that there are only binary blobs in nrf connect sdk.

I think the underlying issue will trigger if several packets come in quickly. I believe the modem library should simply drop packets if there is a memory allocation issue.

I would like to at least see if there is a way I can see if there is an ASSERT, etc firing in this case. It really looks like there is a heap failure and the modem lib is locked up.

RE: NRF9160 - BSD lib crash - UDP recv() - potential security issue

Didrik Rokhaug — Wed, 26 May 2021 14:41:19 GMT

Hi,

If you do not read the data sent from the modem to the application, you will eventually fill up the shared memory buffer, which will prevent other messages from being queued.

You should therefore always make sure that you read all data sent from the modem to the application.

This behavior is explained in the documentation: https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrfxlib/nrf_modem/doc/architecture.html#rx-area

Best regards,

Didrik