serial_lte_modem app gets stuck after multiple AT#XHTTPCREQ commands

RE: serial_lte_modem app gets stuck after multiple AT#XHTTPCREQ commands

Carl Richard — Wed, 09 Dec 2020 14:06:06 GMT

Good to hear, Michal! That sounds like a good workaround.

We will of course continue on the work with fixing the actual issue, so that others can avoid this in the future.

Best regards,
Carl Richard

RE: serial_lte_modem app gets stuck after multiple AT#XHTTPCREQ commands

Michal Vacha — Wed, 09 Dec 2020 13:42:33 GMT

Hi,

I was able to work around this issue by adding a watchdog into the slm_at_host.c. I feed it in the uart_rx_handler fuction, so if nothing is received for CONFIG_WATCHDOG_TIMEOUT_MSEC I assume the device got stuck and it reboots.

I previously I had the wdt attached to the work queue, but the queue processing does not get stuck, so that was the reason it did not work for me.

Best Regards, Michal V.

RE: serial_lte_modem app gets stuck after multiple AT#XHTTPCREQ commands

Carl Richard — Thu, 03 Dec 2020 14:14:39 GMT

Hi!

That was unfortunate. Thanks for the log, even though it's not very informative. I will forward it, along with the workaround to the developer!

Best regards,
Carl Richard

RE: serial_lte_modem app gets stuck after multiple AT#XHTTPCREQ commands

Michal Vacha — Tue, 01 Dec 2020 13:58:54 GMT

Hi, unfortunately it does not resolve this issue for us. I have tried to capture a better log than the one I send you, but without any luck. The new log just contains additional http request logging.

In the meantime we have found out a possible workaround - adding sleep after each response from AT is read. It definetly improves the issue, but of course it does not solve the root cause.

Thank you for your help, BR, Michal V.

00> 41 54 23 58 48 54 54 50  43 43 4f 4e 3d 31 2c 22 |AT#XHTT[00:07:03.963,653] <inf> at_host: RX_DISABLED
00> P CCON=1,"
00> 69 6e 67 72 65 73 73 2e  67 77 78 63 6c 6f 75 64 |ingress. gwxcloud
00> 2e 63 6f 6d 22 2c 38 30                          |.com",80         
00> [00:07:03.966,461] <inf> httpc: Attempting to connect over IPv6
00> [00:07:03.968,719] <err> httpc: Unable to connect, errno 114
00> [00:07:03.970,336] <inf> httpc: Attempting to connect over IPv4
00> [00:07:04.510,040] <inf> httpc: Connected to ingress.gwxcloud.com
00> [00:07:04.510,345] <inf> at_host: TX
00> 23 58 48 54 54 50 43 43  4f 4e 3a 31 0d 0a       |#XHTTPCC ON:1..  
00> [00:07:04.526,184] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:07:04.535,919] <inf> at_host: RX_DISABLED
00> [00:07:04.537,353] <inf> at_host: RX
00> 41 54 23 58 48 54 54 50  43 52 45 51 3d 22 50 4f |AT#XHTTP CREQ="PO
00> 53 54 22 2c 22 2f 61 75  74 68 22 2c 22 43 6f 6e |ST","/au th","Con
00> 74 65 6e 74 2d 4c 65 6e  67 74 68 3a 20 31 31 38 |tent-Len gth: 118
00> 0d 0a 22 2c 31 31 38                             |..",118          
00> [00:07:04.539,459] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:07:04.540,039] <dbg> net_http.Data to send
00> 50 4f 53 54 20 2f 61 75  74 68 20 48 54 54 50 2f |POST /au th HTTP/
00> 31 2e 31 0d 0a 48 6f 73  74 3a 20 69 6e 67 72 65 |1.1..Hos t: ingre
00> 73 73 2e 67 77 78 63 6c  6f 75 64 2e 63 6f 6d 0d |ss.gwxcl oud.com.
00> 0a                                               |.                
00> [00:07:04.544,525] <dbg> net_http.Data to send
00> 0d 0a                                            |..               
00> [00:07:04.559,051] <inf> at_host: TX
00> 23 58 48 54 54 50 43 52  45 51 3a 31 0d 0a       |#XHTTPCR EQ:1..  
00> [00:07:04.573,333] <inf> a[00:07:04.573,852] <inf> at_host: RX_DISABLED
00> t_host: RX
00> 7b 60 64 61 74 61 60 3a  60 39 41 34 30 44 43 39 |{`data`: `9A40DC9
00> 44 32 31 45 39 30 42 39  32 43 30 39 36 35 39 37 |D21E90B9 2C096597
00> 45 39 35 43 37 42 39 36  41 30 30 41 30 30 30 34 |E95C7B96 A00A0004
00> 30 60 2c 60 64 65 62 75  67 60 3a 7b 60 76 45 53 |0`,`debu g`:{`vES
00> 50 60 3a 60 31 2e 36 2e  32 60 2c 60 76 4e 52 46 |P`:`1.6. 2`,`vNRF
00> 60 3a 60 31 2e 36 2e 30  60 2c 60 77 69 66 69 5f |`:`1.6.0 `,`wifi_
00> 72 73 73 69 60 3a 30 2c  60 75 70 74 69 6d 65 60 |rssi`:0, `uptime`
00> 3a 34 32 34 7d 7d                                |:424}}           
00> [00:07:04.579,162] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:07:04.580,413] <inf> at_host: TX
00> 23 58 48 54 54 50 43 52  45 51 3a 30 0d 0a       |#XHTTPCR EQ:0..  
00> [00:07:04.580,963] <dbg> net_http.http_client_req: (0x20021628): Sent 190 bytes
00> [00:07:05.343,231] <dbg> net_http.on_message_begin: (0x20021628): -- HTTP POST response (headers) --
00> [00:07:05.345,062] <dbg> net_http.on_status: (0x20021628): HTTP response status 200 OK
00> [00:07:05.345,825] <dbg> net_http.print_header_field: (0x20021628): [4] Date
00> [00:07:05.346,557] <dbg> net_http.print_header_field: (0x20021628): [29] Tue, 01 Dec 2020 13:46:08 GMT
00> [00:07:05.347,351] <dbg> net_http.print_header_field: (0x20021628): [12] Content-Type
00> [00:07:05.348,083] <dbg> net_http.print_header_field: (0x20021628): [31] application/json; charset=utf-8
00> [00:07:05.348,876] <dbg> net_http.print_header_field: (0x20021628): [17] Transfer-Encoding
00> [00:07:05.349,639] <dbg> net_http.print_header_field: (0x20021628): [7] chunked
00> [00:07:05.350,311] <dbg> net_http.print_header_field: (0x20021628): [10] Connection
00> [00:07:05.351,013] <dbg> net_http.print_header_field: (0x20021628): [10] keep-alive
00> [00:07:05.351,715] <dbg> net_http.print_header_field: (0x20021628): [10] Set-Cookie
00> [00:07:05.352,416] <dbg> net_http.print_header_field: (0x20021628): [127] __cfduid=df6d2cea734298689b4474c814bfcc0b51606830368; expires=Thu, 31-Dec-20 13:46:08 GMT; path=/; domain=.gwxcloud.com; HttpOn
00> [00:07:05.353,393] <dbg> net_http.print_header_field: (0x20021628): [4] Vary
00> [00:07:05.355,895] <dbg> net_http.print_header_field: (0x20021628): [15] Accept-Encoding
00> [00:07:05.356,872] <dbg> net_http.print_header_field: (0x20021628): [15] CF-Cache-Status
00> [00:07:05.357,513] <dbg> net_http.print_header_field: (0x20021628): [7] DYNAMIC
00> [00:07:05.358,093] <dbg> net_http.print_header_field: (0x20021628): [13] cf-request-id
00> [00:07:05.359,191] <dbg> net_http.print_header_field: (0x20021628): [32] 06c026bee1000072879cb90000000001
00> [00:07:05.360,198] <dbg> net_http.print_header_field: (0x20021628): [9] Report-To
00> [00:07:05.361,083] <dbg> net_http.print_header_field: (0x20021628): [127] {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=QLHQt3De14vfKIZEAM%2B2tQ%2BidFbctYw3mT%2Bc2oZGo6f9qe6tfp6RfGAo2z
00> [00:07:05.361,877] <dbg> net_http.print_header_field: (0x20021628): [3] NEL
00> [00:07:05.362,335] <dbg> net_http.print_header_field: (0x20021628): [39] {"report_to":"cf-nel","max_age":604800}
00> [00:07:05.362,854] <dbg> net_http.print_header_field: (0x20021628): [6] Server
00> [00:07:05.369,079] <dbg> net_http.print_header_field: (0x20021628): [4] clou
00> [00:07:05.407,226] <dbg> net_http.print_header_field: (0x20021628): [6] dflare
00> [00:07:05.407,714] <dbg> net_http.print_header_field: (0x20021628): [6] CF-RAY
00> [00:07:05.441,711] <dbg> net_http.print_header_field: (0x20021628): [20] 5fad40ab0efb7287-AMS
00> [00:07:05.442,108] <dbg> net_http.on_headers_complete: (0x20021628): No body expected
00> [00:07:05.442,749] <dbg> net_http.on_message_complete: (0x20021628): -- HTTP POST response (complete) --
00> [00:07:05.443,206] <inf> at_host: TX
00> 23 58 48 54 54 50 43 52  53 50 3a 38 34 30 2c 30 |#XHTTPCR SP:840,0
00> 0d 0a                                            |..               
00> [00:07:05.445,678] <inf> at_host: TX
00> 48 54 54 50 2f 31 2e 31  20 32 30 30 20 4f 4b 0d |HTTP/1.1  200 OK.
00> 0a 44 61 74 65 3a 20 54  75 65 2c 20 30 31 20 44 |.Date: T ue, 01 D
00> 65 63 20 32 30 32 30 20  31 33 3a 34 36 3a 30 38 |ec 2020  13:46:08
00> 20 47 4d 54 0d 0a 43 6f  6e 74 65 6e 74 2d 54 79 | GMT..Co ntent-Ty
00> 70 65 3a 20 61 70 70 6c  69 63 61 74 69 6f 6e 2f |pe: appl ication/
00> 6a 73 6f 6e 3b 20 63 68  61 72 73 65 74 3d 75 74 |json; ch arset=ut
00> 66 2d 38 0d 0a 54 72 61  6e 73 66 65 72 2d 45 6e |f-8..Tra nsfer-En
00> 63 6f 64 69 6e 67 3a 20  63 68 75 6e 6b 65 64 0d |coding:  chunked.
00> 0a 43 6f 6e 6e 65 63 74  69 6f 6e 3a 20 6b 65 65 |.Connect ion: kee
00> 70 2d 61 6c 69 76 65 0d  0a 53 65 74 2d 43 6f 6f |p-alive. .Set-Coo
00> 6b 69 65 3a 20 5f 5f 63  66 64 75 69 64 3d 64 66 |kie: __c fduid=df
00> 36 64 32 63 65 61 37 33  34 32 39 38 36 38 39 62 |6d2cea73 4298689b
00> 34 34 37 34 63 38 31 34  62 66 63 63 30 62 35 31 |4474c814 bfcc0b51
00> 36 30 36 38 33 30 33 36  38 3b 20 65 78 70 69 72 |60683036 8; expir
00> 65 73 3d 54 68 75 2c 20  33 31 2d 44 65 63 2d 32 |es=Thu,  31-Dec-2
00> 30 20 31 33 3a 34 36 3a  30 38 20 47 4d 54 3b 20 |0 13:46: 08 GMT; 
00> 70 61 74 68 3d 2f 3b 20  64 6f 6d 61 69 6e 3d 2e |path=/;  domain=.
00> 67 77 78 63 6c 6f 75 64  2e 63 6f 6d 3b 20 48 74 |gwxcloud .com; Ht
00> 74 70 4f 6e 6c 79 3b 20  53 61 6d 65 53 69 74 65 |tpOnly;  SameSite
00> 3d 4c 61 78 0d 0a 56 61  72 79 3a 20 41 63 63 65 |=Lax..Va ry: Acce
00> 70 74 2d 45 6e 63 6f 64  69 6e 67 0d 0a 43 46 2d |pt-Encod ing..CF-
00> 43 61 63 68 65 2d 53 74  61 74 75 73 3a 20 44 59 |Cache-St atus: DY
00> 4e 41 4d 49 43 0d 0a 63  66 2d 72 65 71 75 65 73 |NAMIC..c f-reques
00> 74 2d 69 64 3a 20 30 36  63 30 32 36 62 65 65 31 |t-id: 06 c026bee1
00> 30 30 30 30 37 32 38 37  39 63 62 39 30 30 30 30 |00007287 9cb90000
00> 30 30 30 30 30 31 0d 0a  52 65 70 6f 72 74 2d 54 |000001.. Report-T
00> 6f 3a 20 7b 22 65 6e 64  70 6f 69 6e 74 73 22 3a |o: {"end points":
00> 5b 7b 22 75 72 6c 22 3a  22 68 74 74 70 73 3a 5c |[{"url": "https:\
00> 2f 5c 2f 61 2e 6e 65 6c  2e 63 6c 6f 75 64 66 6c |/\/a.nel .cloudfl
00> 61 72 65 2e 63 6f 6d 5c  2f 72 65 70 6f 72 74 3f |are.com\ /report?
00> 73 3d 51 4c 48 51 74 33  44 65 31 34 76 66 4b 49 |s=QLHQt3 De14vfKI
00> 5a 45 41 4d 25 32 42 32  74 51 25 32 42 69 64 46 |ZEAM%2B2 tQ%2BidF
00> 62 63 74 59 77 33 6d 54  25 32 42 63 32 6f 5a 47 |bctYw3mT %2Bc2oZG
00> 6f 36 66 39 71 65 36 74  66 70 36 52 66 47 41 6f |o6f9qe6t fp6RfGAo
00> 32 7a 41 38 7a 25 32 46  6d 74 4e 47 78 72 44 66 |2zA8z%2F mtNGxrDf
00> 51 77 78 55 6f 5a 36 6d  4b 53 5a 6f 6e 42 56 7a |QwxUoZ6m KSZonBVz
00> 67 4a 31 48 73 30 4b 6e  69 61 50 71 4f 46 68 51 |gJ1Hs0Kn iaPqOFhQ
00> 53 52 4a 6d 79 7a 62 61  64 49 4b 77 25 33 44 25 |SRJmyzba dIKw%3D%
00> 33 44 22 7d 5d 2c 22 67  72 6f 75 70 22 3a 22 63 |3D"}],"g roup":"c
00> 66 2d 6e 65 6c 22 2c 22  6d 61 78 5f 61 67 65 22 |f-nel"," max_age"
00> 3a 36 30 34 38 30 30 7d  0d 0a 4e 45 4c 3a 20 7b |:604800} ..NEL: {
00> 22 72 65 70 6f 72 74 5f  74 6f 22 3a 22 63 66 2d |"report_ to":"cf-
00> 6e 65 6c 22 2c 22 6d 61  78 5f 61 67 65 22 3a 36 |nel","ma x_age":6
00> 30 34 38 30 30 7d 0d 0a  53 65 72 76 65 72 3a 20 |04800}.. Server: 
00> 63 6c 6f 75 64 66 6c 61  72 65 0d 0a 43 46 2d 52 |cloudfla re..CF-R
00> 41 59 3a 20 35 66 61 64  34 30 61 62 30 65 66 62 |AY: 5fad 40ab0efb
00> 37 32 38 37 2d 41 4d 53  0d 0a 0d 0a 35 36 0d 0a |7287-AMS ....56..
00> 7b 22 64 61 74 61 22 3a  22 62 39 65 63 31 66 37 |{"data": "b9ec1f7
00> 62 63 61 62 61 37 64 35  36 30 33 65 66 33 63 35 |bcaba7d5 603ef3c5
00> 38 36 65 33 63 38 61 30  30 35 36 65 31 39 61 33 |86e3c8a0 056e19a3
00> 62 62 65 39 38 61 64 63  63 33 63 62 36 34 63 61 |bbe98adc c3cb64ca
00> 30 37 63 36 61 65 32 36  34 22 2c 22 63 6d 64 22 |07c6ae26 4","cmd"
00> 3a 6e 75 6c 6c 7d 0d 0a                          |:null}..         
00> [00:07:05.487,945] <dbg> net_http.http_client_req: (0x20021628): Received 840 bytes
00> [00:07:12.658,843] <inf> at_host: RX
00> 41 54                                            |AT               
00> [00:07:12.659,362] <inf> at_host: RX_DISABLED
00> [00:07:12.659,637] <inf> at_cmd: Sending command AT
00> [00:07:12.660,034] <inf> at_cmd: Awaiting response for AT
00> [00:07:12.661,193] <inf> at_cmd: at_cmd_rx 5 bytes, OK
00> 
00> [00:07:12.661,437] <inf> at_cmd: Enqueueing response for sync call
00> [00:07:12.661,743] <inf> at_host: TX
00> [00:07:12.661,987] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:07:12.662,536] <inf> at_cmd: Writing any pending command
00> [00:07:12.662,811] <inf> at_cmd: Listening on socket
00> [00:07:22.659,790] <inf> at_host: RX
00> 41 54                                            |AT               
00> [00:07:22.660,308] <inf> at_host: RX_DISABLED
00> [00:07:22.660,583] <inf> at_cmd: Sending command AT
00> [00:07:22.660,980] <inf> at_cmd: Awaiting response for AT
00> [00:07:22.662,109] <inf> at_cmd: at_cmd_rx 5 bytes, OK
00> 
00> [00:07:22.662,384] <inf> at_cmd: Enqueueing response for sync call
00> [00:07:22.662,689] <inf> at_host: TX
00> [00:07:22.662,933] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:07:22.663,452] <inf> at_cmd: Writing any pending command
00> [00:07:22.663,726] <inf> at_cmd: Listening on socket
00> [00:07:32.660,736] <inf> at_host: RX
00> 41 54                                            |AT               
00> [00:07:32.661,254] <inf> at_host: RX_DISABLED
00> [00:07:32.661,529] <inf> at_cmd: Sending command AT
00> [00:07:32.668,029] <inf> at_cmd: Awaiting response for AT
00> [00:07:32.669,830] <inf> at_cmd: at_cmd_rx 5 bytes, OK
00> 
00> [00:07:32.670,104] <inf> at_cmd: Enqueueing response for sync call
00> [00:07:32.670,410] <inf> at_host: TX
00> [00:07:32.670,623] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:07:32.671,173] <inf> at_cmd: Writing any pending command
00> [00:07:32.671,447] <inf> at_cmd: Listening on socket
00> [00:07:42.661,407] <inf> at_host: RX
00> 41 54                                            |AT               
00> [00:07:42.661,926] <inf> at_host: RX_DISABLED
00> [00:07:42.662,200] <inf> at_cmd: Sending command AT
00> [00:07:42.668,701] <inf> at_cmd: Awaiting response for AT
00> [00:07:42.670,501] <inf> at_cmd: at_cmd_rx 5 bytes, OK
00> 
00> [00:07:42.670,776] <inf> at_cmd: Enqueueing response for sync call
00> [00:07:42.671,081] <inf> at_host: TX
00> [00:07:42.671,325] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:07:42.671,844] <inf> at_cmd: Writing any pending command
00> [00:07:42.672,119] <inf> at_cmd: Listening on socket
00> [00:07:52.663,085] <inf> at_host: RX
00> 41 54                                            |AT               
00> [00:07:52.663,604] <inf> at_host: RX_DISABLED
00> [00:07:52.663,879] <inf> at_cmd: Sending command AT
00> [00:07:52.670,379] <inf> at_cmd: Awaiting response for AT
00> [00:07:52.672,180] <inf> at_cmd: at_cmd_rx 5 bytes, OK
00> 
00> [00:07:52.672,454] <inf> at_cmd: Enqueueing response for sync call
00> [00:07:52.672,760] <inf> at_host: TX
00> [00:07:52.673,004] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:07:52.673,553] <inf> at_cmd: Writing any pending command
00> [00:07:52.673,797] <inf> at_cmd: Listening on socket
00> [00:07:56.733,245] <wrn> at_host: RX_STOPPED (4)
00> [00:07:56.733,489] <wrn> at_host: RX_STOPPED (8)
00> [00:07:56.733,734] <wrn> at_host: RX_STOPPED (4)
00> [00:07:56.733,978] <wrn> at_host: RX_STOPPED (4)
00> [00:07:56.734,222] <inf> at_host: RX_DISABLED

RE: serial_lte_modem app gets stuck after multiple AT#XHTTPCREQ commands

Carl Richard — Mon, 30 Nov 2020 08:57:13 GMT

Hi again!

The developer suggested that the updated slm_at_httpc.c from your other ticket should mitigate this issue as well. Please report back if that's not the case.

Best regards,
Carl Richard

RE: serial_lte_modem app gets stuck after multiple AT#XHTTPCREQ commands

Carl Richard — Thu, 26 Nov 2020 13:34:44 GMT

Hi!

Apologies for the delay. We have been awaiting a response from the developers, whom are now working on the issue. Could you try to get the log (similarily to your other case), so that we may get some more insight into what's happening. The prj.conf is as follows:

CONFIG_LOG_IMMEDIATE=y
CONFIG_NET_LOG=y
CONFIG_NET_SOCKETS_LOG_LEVEL_DBG=n
CONFIG_NET_HTTP_LOG_LEVEL_DBG=y

Thank you for your patience and best regards,
Carl Richard

RE: serial_lte_modem app gets stuck after multiple AT#XHTTPCREQ commands

Michal Vacha — Fri, 13 Nov 2020 13:09:07 GMT

I was unable to enable debug as the app then started crashing on stack overflow exception. So I've just replaced DBG logs to INF.

This is the end of the log collected over RTT:

00> 23 58 48 54 54 50 43 43  4f 4e 3a 30 0d 0a       |#XHTTPCC ON:0..  
00> [00:06:11.588,409] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:06:11.594,573] <inf> at_host: RX
00> 41 54 23 58 48 54 54 50  43 43 4f 4e 3d 31 2c 22 |AT#XHTTP CCON=1,"
00> 69 6e 67 72 65 73 73 2e  64 65 76 2e 74 65 6d 70 |ingress. dev.temp
00> 65 67 67 2e 63 6f 6d 22  2c 38 30                |egg.com" ,80     
00> [00:06:11.594,940] <inf> httpc: Connect from http server
00> [00:06:11.595,062] <inf> at_host: RX_DISABLED
00> [00:06:12.105,895] <err> httpc: Failed to resolve hostname ingress.dev.tempegg.com on IPv6
00> [00:06:12.121,826] <inf> httpc: Attempting to connect over IPv4
00> [00:06:12.312,316] <inf> httpc: Connected to ingress.dev.tempegg.com
00> [00:06:12.312,316] <inf> httpc: Configuring socket timeout (10 s)
00> [00:06:12.312,377] <inf> at_host: TX
00> 23 58 48 54 54 50 43 43  4f 4e 3a 31 0d 0a       |#XHTTPCC ON:1..  
00> [00:06:12.329,528] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:06:12.337,188] <inf> at_host: RX_DISABLED
00> [00:06:12.339,050] <inf> at_host: RX
00> 41 54 23 58 48 54 54 50  43 52 45 51 3d 22 50 4f |AT#XHTTP CREQ="PO
00> 53 54 22 2c 22 2f 61 75  74 68 22 2c 22 43 6f 6e |ST","/au th","Con
00> 74 65 6e 74 2d 4c 65 6e  67 74 68 3a 20 35 31 0d |tent-Len gth: 51.
00> 0a 22 2c 35 31                                   |.",51            
00> [00:06:12.339,508] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:06:12.340,301] <inf> httpc: send header: 20 bytes
00> [00:06:12.358,306] <inf> at_host: TX
00> 23 58 48 54 54 50 43 52  45 51 3a 31 0d 0a       |#XHTTPCR EQ:1..  
00> [00:06:12.358,306] <inf> httpc: wait until payload is ready
00> [00:06:12.365,814] <inf> at_host: RX
00> 7b 22 64 61 74 61 22 3a  22 30 30 31 31 32 32 33 |{"data": "0011223
00> 33 34 34 35 35 36 36 37  37 38 38 39 39 41 41 42 |34455667 78899AAB
00> 42 43 43 44 44 45 45 46  46 30 30 30 30 30 30 30 |BCCDDEEF F0000000
00> 31 22 7d                                         |1"}              
00> [00:06:12.366,302] <inf> at_host: RX_DISABLED
00> [00:06:12.366,973] <inf> httpc: send 51 bytes payload
00> [00:06:12.367,004] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:06:12.367,431] <inf> at_host: TX
00> 23 58 48 54 54 50 43 52  45 51 3a 30 0d 0a       |#XHTTPCR EQ:0..  
00> [00:06:13.033,752] <inf> httpc: Partial data received (277 bytes)
00> [00:06:13.033,782] <inf> httpc: All the data received (277 bytes)
00> [00:06:13.033,843] <inf> at_host: TX
00> 23 58 48 54 54 50 43 52  53 50 3a 32 37 37 2c 30 |#XHTTPCR SP:277,0
00> 0d 0a                                            |..               
00> [00:06:13.042,541] <inf> at_host: TX
00> 48 54 54 50 2f 31 2e 31  20 32 30 30 20 4f 4b 0d |HTTP/1.1  200 OK.
00> 0a 43 6f 6e 74 65 6e 74  2d 4c 65 6e 67 74 68 3a |.Content -Length:
00> 20 38 36 0d 0a 43 6f 6e  74 65 6e 74 2d 54 79 70 | 86..Con tent-Typ
00> 65 3a 20 61 70 70 6c 69  63 61 74 69 6f 6e 2f 6a |e: appli cation/j
00> 73 6f 6e 3b 20 63 68 61  72 73 65 74 3d 75 74 66 |son; cha rset=utf
00> 2d 38 0d 0a 52 65 71 75  65 73 74 2d 43 6f 6e 74 |-8..Requ est-Cont
00> 65 78 74 3a 20 61 70 70  49 64 3d 63 69 64 2d 76 |ext: app Id=cid-v
00> 31 3a 66 32 66 61 65 37  35 66 2d 63 35 31 65 2d |1:f2fae7 5f-c51e-
00> 34 38 37 63 2d 61 37 34  30 2d 32 39 63 33 31 36 |487c-a74 0-29c316
00> 37 37 38 38 31 65 0d 0a  44 61 74 65 3a 20 54 68 |77881e.. Date: Th
00> 75 2c 20 31 32 20 4e 6f  76 20 32 30 32 30 20 32 |u, 12 No v 2020 2
00> 32 3a 31 34 3a 34 30 20  47 4d 54 0d 0a 0d 0a 7b |2:14:40  GMT....{
00> 22 64 61 74 61 22 3a 22  33 35 38 32 65 32 31 39 |"data":" 3582e219
00> 32 39 30 30 39 39 64 38  62 34 38 62 66 61 34 38 |290099d8 b48bfa48
00> 31 34 38 65 39 64 35 34  32 33 34 64 65 63 37 31 |148e9d54 234dec71
00> 65 37 62 34 30 34 62 38  32 38 64 66 33 66 66 36 |e7b404b8 28df3ff6
00> 32 62 34 35 37 65 63 32  22 2c 22 63 6d 64 22 3a |2b457ec2 ","cmd":
00> 6e 75 6c 6c 7d                                   |null}            
00> [00:06:13.075,561] <inf> at_host: RX_DISABLED
00> [00:06:13.081,695] <inf> at_host: RX
00> 41 54 23 58 48 54 54 50  43 43 4f 4e 3f          |AT#XHTTP CCON?   
00> [00:06:13.082,031] <inf> at_host: TX
00> 23 58 48 54 54 50 43 43  4f 4e 3a 20 31 2c 22 69 |#XHTTPCC ON: 1,"i
00> 6e 67 72 65 73 73 2e 64  65 76 2e 74 65 6d 70 65 |ngress.d ev.tempe
00> 67 67 2e 63 6f 6d 22 2c  38 30 0d 0a             |gg.com", 80..    
00> [00:06:13.085,998] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:06:13.090,118] <inf> at_host: RX
00> 41 54 23 58 48 54 54 50  43 43 4f 4e 3d 30       |AT#XHTTP CCON=0  
00> [00:06:13.090,362] <inf> httpc: Disconnect from http server
00> [00:06:13.090,362] <inf> httpc: Closing socket 1
00> [00:06:13.090,637] <inf> at_host: RX_DISABLED
00> [00:06:13.090,728] <inf> at_host: TX
00> 23 58 48 54 54 50 43 43  4f 4e 3a 30 0d 0a       |#XHTTPCC ON:0..  
00> [00:06:13.100,677] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:06:13.106,842] <inf> at_host: RX
00> 41 54 23 58 48 54 54 50  43 43 4f 4e 3d 31 2c 22 |AT#XHTTP CCON=1,"
00> 69 6e 67 72 65 73 73 2e  64 65 76 2e 74 65 6d 70 |ingress. dev.temp
00> 65 67 67 2e 63 6f 6d 22  2c 38 30                |egg.com" ,80     
00> [00:06:13.107,208] <inf> httpc: Connect from http server
00> [00:06:13.107,330] <inf> at_host: RX_DISABLED
00> [00:06:14.072,418] <err> httpc: Failed to resolve hostname ingress.dev.tempegg.com on IPv6
00> [00:06:14.325,744] <inf> httpc: Attempting to connect over IPv4
00> [00:06:14.427,337] <inf> httpc: Connected to ingress.dev.tempegg.com
00> [00:06:14.427,368] <inf> httpc: Configuring socket timeout (10 s)
00> [00:06:14.427,398] <inf> at_host: TX
00> 23 58 48 54 54 50 43 43  4f 4e 3a 31 0d 0a       |#XHTTPCC ON:1..  
00> [00:06:14.444,915] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:06:14.452,117] <inf> at_host: RX
00> 41 54 23 58 48 54 54 50  43 52 45 51 3d 22 50 4f |AT#XHTTP CREQ="PO
00> 53 54 22 2c 22 2f 61 75  74 68 22 2c 22 43 6f 6e |ST","/au th","Con
00> 74 65 6e 74 2d 4c 65 6e  67 74 68 3a 20 35 31 0d |tent-Len gth: 51.
00> 0a 22 2c 35 31                                   |.",51            
00> [00:06:14.452,575] <inf> at_host: RX_DISABLED
00> [00:06:14.452,606] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:06:14.453,399] <inf> httpc: send header: 20 bytes
00> [00:06:14.472,717] <inf> at_host: TX
00> 23 58 48 54 54 50 43 52  45 51 3a 31 0d 0a       |#XHTTPCR EQ:1..  
00> [00:06:14.472,747] <inf> httpc: wait until payload is ready
00> [00:06:14.479,919] <inf> at_host: RX
00> 7b 22 64 61 74 61 22 3a  22 30 30 31 31 32 32 33 |{"data": "0011223
00> 33 34 34 35 35 36 36 37  37 38 38 39 39 41 41 42 |34455667 78899AAB
00> 42 43 43 44 44 45 45 46  46 30 30 30 30 30 30 30 |BCCDDEEF F0000000
00> 31 22 7d                                         |1"}              
00> [00:06:14.480,407] <inf> at_host: RX_DISABLED
00> [00:06:14.481,475] <inf> httpc: send 51 bytes payload
00> [00:06:14.481,506] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:06:14.481,933] <inf> at_host: TX
00> 23 58 48 54 54 50 43 52  45 51 3a 30 0d 0a       |#XHTTPCR EQ:0..  
00> [00:06:24.451,202] <inf> at_host: RX_DISABLED
00> [00:06:25.460,388] <inf> at_host: RX
00> 41 54 23 58 48 54 54 50  43 43 4f 4e 3f          |AT#XHTTP CCON?   
00> [00:06:25.460,754] <inf> at_host: TX
00> 23 58 48 54 54 50 43 43  4f 4e 3a 20 31 2c 22 69 |#XHTTPCC ON: 1,"i
00> 6e 67 72 65 73 73 2e 64  65 76 2e 74 65 6d 70 65 |ngress.d ev.tempe
00> 67 67 2e 63 6f 6d 22 2c  38 30 0d 0a             |gg.com", 80..    
00> [00:06:25.464,691] <inf> at_host: TX
00> 4f 4b 0d 0a                                      |OK..             
00> [00:06:25.468,811] <inf> at_host: RX
00> 41 54 23 58 48 54 54 50  43 43 4f 4e 3d 30       |AT#XHTTP CCON=0  
00> [00:06:25.469,055] <inf> httpc: Disconnect from http server
00> [00:06:25.469,055] <inf> httpc: Closing socket 1
00> [00:06:25.469,329] <inf> at_host: RX_DISABLED

After that it reboots. Unfortunately I've found that the watchdog does not always work. When the ESP reboots, it sends garbage over the UART, nRF gets stuck, but never reboots.

RE: serial_lte_modem app gets stuck after multiple AT#XHTTPCREQ commands

Carl Richard — Fri, 13 Nov 2020 12:15:29 GMT

Thanks for the trace and the logs. From what I can see the modem crashes at some point, but I have forwarded the trace to the modem team so they can have a look. In the meantime, could you enable debug messages for the RTT log? Change/add the following to prj.conf:

CONFIG_AT_CMD_LOG_LEVEL_DBG=y
CONFIG_SLM_LOG_LEVEL_DBG=y

This could give some more insight to what's happening.

The watchdog solution is not ideal, but I'm glad that it works.

Best regards,
Carl Richard

RE: serial_lte_modem app gets stuck after multiple AT#XHTTPCREQ commands

Michal Vacha — Thu, 12 Nov 2020 23:55:55 GMT

So far my only workaround is to have a watchdog timer reboot the NRF chip when it gets stuck...

RE: serial_lte_modem app gets stuck after multiple AT#XHTTPCREQ commands

Michal Vacha — Thu, 12 Nov 2020 20:47:56 GMT

Hi, so I was finally able to capture the modem trace.

devzone.nordicsemi.com/.../trace_2D00_2020_2D00_11_2D00_12T20_2D00_37_2D00_10.321Z.bin

The moment the modem got stuck, the trace also stopped.

This is the log from the ESP:

LTEM - CONNECT - to: https://WWW.XXXYYY.ZZZ/auth
LTEM - DISCONNECT
ESP32->AT#XHTTPCCON?
NRF91->#XHTTPCCON: 1,"WWW.XXXYYY.ZZZ",80
NRF91->OK
----------------------------------
ESP32->AT#XHTTPCCON=0
NRF91->#XHTTPCCON:0
NRF91->OK
----------------------------------
ESP32->AT#XHTTPCCON=1,"WWW.XXXYYY.ZZZ",80
NRF91->#XHTTPCCON:1
NRF91->OK
----------------------------------
TEST REQ: 109
AT#XHTTPCREQ="POST","/auth","Content-Length: 51
",51
OK
#XHTTPCREQ:1
{"data":"00112233445566778899AABBCCDDEEFF00000001"}
7B 22 64 61 74 61 22 3A 22 30 30 31 31 32 32 33 33 34 34 35 35 36 36 37 37 38 38 39 39 41 41 42 42 43 43 44 44 45 45 46 46 30 30 
30 30 30 30 30 31 22 7D OK
#XHTTPCREQ:0
#XHTTPCRSP:277,0
Received http response:
HTTP/1.1 200 OK
Content-Length: 86
Content-Type: application/json; charset=utf-8
Request-Context: appId=cid-v1:f2fae75f-c51e-487c-a740-29c31677881e
Date: Thu, 12 Nov 2020 20:39:59 GMT

{"data":"3582e219290099d8b48bfa48148e9d54c74827431dd3fbb46cbaa7cc983d36ab","cmd":null}@
TEST RESPONSE:{"data":"3582e219290099d8b48bfa48148e9d54c74827431dd3fbb46cbaa7cc983d36ab","cmd":null}
==========================================================
LTEM - CONNECT - to: https://WWW.XXXYYY.ZZZ/auth
LTEM - DISCONNECT
ESP32->AT#XHTTPCCON?
NRF91->#XHTTPCCON: 1,"WWW.XXXYYY.ZZZ",80
NRF91->OK
----------------------------------
ESP32->AT#XHTTPCCON=0
NRF91->#XHTTPCCON:0
NRF91->OK
----------------------------------
ESP32->AT#XHTTPCCON=1,"WWW.XXXYYY.ZZZ",80
NRF91->
ERROR: not entire line received!
NRF91->
ERROR: not entire line received!
AT COMMAND FAILED: AT#XHTTPCCON=1,"WWW.XXXYYY.ZZZ",80
----------------------------------

Thanks.

RE: serial_lte_modem app gets stuck after multiple AT#XHTTPCREQ commands

Michal Vacha — Thu, 12 Nov 2020 13:40:08 GMT

Hi Carl,

thank you for your response.

I'm not able to colllect the modem trace easily, as the ESP+NRF91 is on a custom board. But I have both devkits on hand, so I'll try to run the code on them and collect the trace from nrf91DK.

BR, Michal V.

RE: serial_lte_modem app gets stuck after multiple AT#XHTTPCREQ commands

Carl Richard — Thu, 12 Nov 2020 13:16:44 GMT

Hi, Michal!

Thank you for the patience. I'm looking into the issue. You are correct about "Ready". It means that the modem/nrf9160 has rebooted, indicating some kind of critical error.

While I'm at it could you manage to do a modem trace?

Best regards,
Carl Richard

RE: serial_lte_modem app gets stuck after multiple AT#XHTTPCREQ commands

Michal Vacha — Tue, 10 Nov 2020 22:50:47 GMT

I have discovered that this error happens after the ESP sends the payload of the XHTTPCREQ command. Instead of receiving #XHTTPCREQ:0 we receive something else. I have captured two different instances of this problem:

Log #1 - Ready is received after the payload has been accepted (via OK) by the NRF. The Ready message signalizes (reboot? reinit?) of the modem, so nrf is not stuck.

ESP>NRF91:
AT#XHTTPCREQ="POST","/auth","Content-Length: 51
",51
NRF91>ESP:OK
NRF91>ESP:#XHTTPCREQ:1
ESP>NRF91:
{"data":"........."}
NRF91>ESP:OK�Ready

Log #2 - ERROR is received after the payload confirmation. Now the NRF modem is stuck, it never reconnects again.

ESP>NRF91:
AT#XHTTPCREQ="POST","/auth","Content-Length: 51
",51
NRF91>ESP:OK
NRF91>ESP:#XHTTPCREQ:1
ESP>NRF91:
{"data":".........."}
NRF91>ESP:OK@
NRF91>ESP:ERROR