SLM HTTP client hangs with large payload

RE: SLM HTTP client hangs with large payload

Håkon Alseth — Fri, 24 Sep 2021 06:36:38 GMT

Hi,

[quote userid="2831" url="~/f/nordic-q-a/79750/slm-http-client-hangs-with-large-payload/330925#330925"]Perhaps the slm_at_httpc.c code should remove the OK after partially sending data. The other clients (ftp, mqtt, tcp) don't do this, I have no idea why httpc should be different.[/quote]

I will mention this in the internal jira.

But, what I was also made aware is that it is recommended to use ncs v1.7.0 for SLM, mainly due to this bug fix:

https://github.com/nrfconnect/sdk-nrf/commit/87ba51dbed3a4988eece125d8953d38ea78d6ed1

Kind regards,

Håkon

RE: SLM HTTP client hangs with large payload

Anthony Ambuehl — Thu, 23 Sep 2021 20:45:15 GMT

Hello Håkon..

Perhaps the slm_at_httpc.c code should remove the OK after partially sending data. The other clients (ftp, mqtt, tcp) don't do this, I have no idea why httpc should be different.

RE: SLM HTTP client hangs with large payload

Håkon Alseth — Thu, 23 Sep 2021 11:40:24 GMT

Hi,

[quote user="anthony.asterisk"]

I'm also seeing this sequence which appears to be caused by a small change in timing from my system due to a new CRITICAL_SECTION i added.

00> [00:00:57.125,701] <inf> httpc: Setting up TLS credentials
00> [00:00:59.010,589] <inf> httpc: Connected to api.cloud.calahealth.com
00> [00:00:59.039,855] <inf> at_host: Enter datamode
00> [00:00:59.043,090] <inf> at_host: Raw send 1
00> [00:00:59.069,580] <inf> at_host: Raw send 255
00> [00:00:59.095,764] <inf> at_host: Raw send 256
00> [00:00:59.109,893] <inf> at_host: Raw send 136
00> [00:00:59.738,159] <inf> at_host: Exit datamode

The first "Raw send" with only 1 byte seems to really screw things up.. I always get an empty response XHTTPCRSP:0,1

[/quote]

I am glad to hear that you found the root cause of the issue, and a workaround for this.

[quote user="anthony.asterisk"]

#define UART_RX_TIMEOUT_MS 1

to be a larger number such as 5, the 1 byte raw_send goes away.

[/quote]

I will request that this is changed, and can be configurable by the application as a feature request to the SLM team.

Kind regards,

Håkon

RE: SLM HTTP client hangs with large payload

Anthony Ambuehl — Tue, 21 Sep 2021 22:23:12 GMT

I'm also seeing this sequence which appears to be caused by a small change in timing from my system due to a new CRITICAL_SECTION i added.

The first "Raw send" with only 1 byte seems to really screw things up.. I always get an empty response XHTTPCRSP:0,1

If I change slm_at_host.c line 36

#define UART_RX_TIMEOUT_MS 1

to be a larger number such as 5, the 1 byte raw_send goes away.

RE: SLM HTTP client hangs with large payload

Anthony Ambuehl — Tue, 21 Sep 2021 22:15:02 GMT

I'm attaching some debugging logs from SLMdevzone.nordicsemi.com/.../ncs_5F00_1.6.1_5F00_slm_5F00_at_5F00_httpc.log

My code was (intentionally) sending some extra bytes after the paylaod in this example, thus producing some additional warnings about unexpected payload.

RE: SLM HTTP client hangs with large payload

Håkon Alseth — Tue, 21 Sep 2021 11:10:02 GMT

Hi,

[quote user="anthony.asterisk"]

BTW, in slm_at_httpc.c there is code that returns values other than 0/1.. lines 510..512 which reports error from the socket send/recv. It would be a good idea to update the documentation, an external implementer like me would need to know that this could occur. In my current code this would be treated as an unexpected response, and an error reported. In production, those errors would cause a massive cascade of paperwork explaining why an error occurred, what the root cause was, what corrective actions were taken, etc (e.g. several days with tons of meetings with the quality team). On the other hand, if I expect these values could occur during normal operation, I can prevent that in the design phase with a few simple comments and an extra conditional clause in the code.

[/quote]

The socket calls will return the errno, which can be many different types of return, based on what the failure is. Please see errno.h (https://github.com/nrfconnect/sdk-zephyr/blob/master/lib/libc/minimal/include/errno.h) for more information.

[quote user="anthony.asterisk"]BTW, I found that debugging is working when I monitor with J-Link RTT viewer. For some reason, it does not work from SES.[/quote]

Could you share the debug log when the issue occurs?

Kind regards,

Håkon

RE: SLM HTTP client hangs with large payload

Anthony Ambuehl — Mon, 20 Sep 2021 20:29:11 GMT

BTW, I found that debugging is working when I monitor with J-Link RTT viewer. For some reason, it does not work from SES.

RE: SLM HTTP client hangs with large payload

Anthony Ambuehl — Mon, 20 Sep 2021 16:35:24 GMT

The OK's are coming after receiving +XHTTPCREQ=1 and before receiving +XHTTPCREQ=0, therefore the SLM is in datamode. I am not sending AT commands, and they should not be received while in datamode anyway.

After reviewing the code, it seems in datamode that every time data is sent to the network port by httpc, a "\r\nOK\r\n" is produced. There are 2 triggers for sending data according to the "Running in data mode" documentation (https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/applications/serial_lte_modem/doc/slm_data_mode.html?highlight=datamode) one of those triggers is a "time limit trigger" which explains the periodic "OK" when data is pending. This is fired from the inactivity_timer_handler in slm_at_handler.

It would be really helpful if the documentation mentioned that an OK would be received on the UART whenever data is sent on the network port. This behavior appears to be specific to httpc... its in httpc_datamode_callback. I reviewed other instances of datamode, and none of them include this behavior.

Finally, I'm still unable to get debug logs after the bootloader jumps to the application. Any help at all with the debug logs would be really really appreciated. For unmodified 1.6.1, changing the line in prj.conf did not have any effect.

Maybe you could post a prj.conf that you've confirmed works?

Also any special steps I need to take after modifying prj.conf?

RE: SLM HTTP client hangs with large payload

Håkon Alseth — Mon, 20 Sep 2021 10:56:16 GMT

Hi,

[quote user="anthony.asterisk"]Is it expected for the OK message to be resent every 27 milliseconds when in datamode?. Is this some sort of progress indication?[/quote]

This should be printed as a response to a generic AT command. Do you send a specific AT command every 27ms?

Were you able to get the log running? SPM logging is disabled by default, so there should not be any RTT block location issues.

[quote user="anthony.asterisk"]

It seems after starting an HTTPCREQ, the OK message should be ignored until the HTTPCRSP is received. I suspect there is also some case where HTTPCREQ return values other than 0 and 1, which is not documented.

[/quote]

It should not do this, it should split the packet as per this logic:

https://github.com/nrfconnect/sdk-nrf/blob/v1.6.1/applications/serial_lte_modem/src/http_c/slm_at_httpc.c#L358-L372

each fragment size is set to this define (default 576 bytes):

https://github.com/nrfconnect/sdk-nrf/blob/v1.6.1/applications/serial_lte_modem/src/http_c/slm_at_httpc.c#L26

Even if the actual size is larger, it will fragment the packet for you.

Kind regards,

Håkon

RE: SLM HTTP client hangs with large payload

Anthony Ambuehl — Fri, 17 Sep 2021 18:07:51 GMT

I worked on it some more this morning and identified an issue in my code, that is caused by an unusual behavior from the SLM.

While in datamode receiving the payload data, the SLM will periodically send the OK_STR ("OK"). My code was not expecting the OK to be sent until after the data was fully transferred so it proceeded into a state where waiting for the XHTTPCRSP message.

Is it expected for the OK message to be resent every 27 milliseconds when in datamode?. Is this some sort of progress indication?

RE: SLM HTTP client hangs with large payload

Anthony Ambuehl — Fri, 17 Sep 2021 16:24:32 GMT

I tried enabling logs by changing prj.cnf line 8 to:

CONFIG_LOG_DEFAULT_LEVEL=4

I ran cmake and the rebuilt merged.hex (F7)

The only log lines are produced by the bootloader and then it goes silent.

Is it possible that logging is disabled completely somehow or that the application has a different location for the RTT block vs the bootloader?

Perhaps there are 2 RTT blocks one in secure zone and one non-secure?

Is there a way to build and run where I will get debug logs for the application?

RE: SLM HTTP client hangs with large payload

Anthony Ambuehl — Fri, 17 Sep 2021 16:09:02 GMT

oh, just realized the link you posted highlighted a specific line to change, thanks for that!

RE: SLM HTTP client hangs with large payload

Anthony Ambuehl — Fri, 17 Sep 2021 15:43:39 GMT

CTS at my external MCU is driven by the nrf9160 SLM. The signal indicates the external mcu should stop sending data.

Regarding the debug logs... (my apologies for basic questions, I'm new to NCS and there are a lot of docs to search through)

1) After changing prj.conf do I need to run cmake from Project..Run cmake.. ?

2) Is it correct for me to put this in the prj.conf:

CONIG_SLM_LOG_LEVEL=4

3) What does CONFIG_SLM_LOG_LEVEL_INF=y do?

RE: SLM HTTP client hangs with large payload

Håkon Alseth — Fri, 17 Sep 2021 13:08:24 GMT

Hi,

[quote user=""]I find the CTS signal goes low after 274 bytes of data were sent. (There is some buffering in my mcu so probably less received at SLM uart) [/quote]

Just to clarify, from which side is it set low?

[quote user=""]Also any advice on how to get debug logging from SLM_AT_HOST and SLM_AT_HTTPC ? I'm running from SES with stock SLM example...[/quote]

If you set the log level to debug (4) here:

https://github.com/nrfconnect/sdk-nrf/blob/v1.6.1/applications/serial_lte_modem/prj.conf#L8

Then it should print out debug logs over RTT. Could you try this and attach the log showing the issue?

Kind regards,

Håkon