nRF9160 send larger MQTT payloads than 2048 bytes to save battery power (need to send 200KB)

[Johan]

Hi,

I want to optimize the transmission efficiency and thinking of sending larger packets than 2048 bytes of MQTT would help.

We have limited battery power, and looking at the MQTT traffic towards AWS I see that this is happening:

1) MQTT-packet is sent.

2) Waiting for response (several seconds)

3) Receiving an ACK.

4) goto 1 (is the fastest way I can do it now, send the next packet of max 2048 bytes).

If I can change the max mqtt packet size to for instance 16K, that would mean an improvement I'm guessing.

Changing it though, makes the example code crash, going somewhere into non program space. It doesn't seem to allow changing of that size, even though it's available in the menu <Project -> configure nRF SDK...etc -> searching for MQTT -> finding maximum MQTT PACKET SIZE set to 2048>

I tried 4096, 8192, 16384.

I get for bigger sizes, crash in the code.

For smaller increases like 4096, I get an error from the socket_write instead, so it's then in the modem software I'm guessing.

I really want to send bigger packages, to improve battery life.

Who can help with this? 

IS this bugs or crash by design, or fail by design, even though the MQTT PACKET SIZE is allowed to be changed by design?

Johan

[Martin Lesund]

Hi Johan,

It is a bug.

The TLS output buffer has a size of 2kB, and there is a bug that you do not get a notification that the messages are too big. (this will be fixed)

So the solution is to have shorter messages.

Best Regards,

Martin  L,.

[Johan]

Ok, looking forward to that bugfix.

Will you enable us to send larger packets later? Is there a limitation somewhere? Is there power to save to send more in a bigger chunk, than to send them separately?

So, how to get up speed when needing to send a large data buffer?

I'm thinking to split the big chunk of 300KB (example size), into like you suggest, smaller parts

* send part 1

* send part 2

...

* send part n

immediately after eachother.

There is no need to wait for the ACK for each part before sending the next I'm guessing.

[Johan]

I tried this and result was:

Trial 1: sent 2 packets of 1990 bytes.
Packet 1, sent ok.
Packet 2, sent ok.
Response came in, for packet 1, then for packet 2.
OK!

--------------

Trial 2: Sent 10 packets of 1990 bytes

Packet 1, sent ok.
Packet 2, sent ok.
PAcket 3, ---> mqtt.c:client_disconnect() -> MQTT_EVT_DISCONNECT (-95) 
Errno lookup gives this.

5

EIO

I/O error

Packet 4..10 - same failure.
Errno lookup gives this.

128

EKEYREVOKED

Key has been revoked

So there is no support for sending multiple packets in sequence, more than 2? Bug again or limitation of some kind?

I need to bring sending speed here, and optimize for transmission. Otherwise I get 300bytes/sec, by waiting for each ACK before I can send the next packet.

[Martin Lesund]

Hi Johan,

I would highly recommend that you test out the newest NCS v1.0.0-rc3 to see if that changes the behavior since there have been several fixes implemented.

cd ncs/nrf
git checkout master
git pull
git checkout v1.0.0-rc3
west update

Then rebuild and try your solution.

Also make sure to have the newest modem firmware: https://www.nordicsemi.com/Products/Low-power-cellular-IoT/nRF9160/Download#infotabs

Best regards,

Martin L.

[Johan]

I flashed the new modem firmware you sent me via email.
I tried it with first the v0.4.0 and it worked with 2048 buffer size like before (asset_tracker example, modified for our own cloud with our own keys)

I tried with buffer size 4096 and 8192. They both seem to go into writing - via the modem without an error,
but there is no response at all from the network. SUBACK is what I'm expecting via the MQTT-drivers.
Checking the cloud, there is no data uploaded either, so still only 2048 bytes works.

Using instead v1.0.0-rc3, and the standard unmodified asset_tracker, I get 

```

***** Booting Zephyr OS v1.14.99-ncs2-rc4 *****

Flash region     Domain     Per***** Hardware exception *****

Current thread ID = 0x20001870

Faulting instruction address = 0x30100

Fatal fault in ISR! Spinning...

```

Note how I'm building and uploading.
1. Building from menu, "Build Solution".
2. Menu "Target -> Connect J-Link".
3. Menu "Target-> Download file -> Download Intel Hex File -> and choosing the merged.hex

This is from how you told me earlier.

Perhaps something has changed again, so this way of flashing and running is wrong?

Johan

[Martin Lesund]

Hi Johan, 
Please confirm that the correct mfw is on the modem side by sending the AT command: AT+CGMR

Also, please clean the application flash in case there is some garbage data on it. 
'nrfjprog -e'

I would like you to test out the RC4 that was just released to see if that changes anything.

The buffer is restricted to 2048, so the new mfw that is coming should include a notification that the buffer you choose is too big if it is larger than 2048.

BR,

Martin L.

[nordic_red]

Hi Martin!

My name is Fredrik Ellner and I am a colleague to Johan. What I understand from Johan is that the when we use the nRF9160 with LTE-M cat1 it takes everything from 20s to 1 minute to successfully transmit 2048bytes (excluding hand shake, encryption, error handling etcetera). What our project need is ONE channel with maximum speed.

We need your assistance in this, because this won't do!

BR,

Fredrik

[Martin Lesund]

Hi Fredrik,

That is indeed poor performance. 

How is the network coverage in your area?

Have you tested with mqtt_simple or custom code?

Using NCS v1.0.0 and mfw v1.0.0 ?

[Johan]

Yes now using NCS v1.0.0 and mfw v1.0.0. Same performance.
I just finished creating a trace and I will send it to you over email.