Trouble receiving binary stream data with recv()

RE: Trouble receiving binary stream data with recv()

-Richard- — Tue, 07 Mar 2023 13:36:12 GMT

It is in fact the same app_update.bin, the first 4 bytes when you compile the firmware in a sort of identifier, so 3D B8 F3 96 is an ID for MCUBoot Application Image, so I think even if the app_update.bin were different it should still start with that.

I tried every single encoding type that the express server supported and none of them worked, but I have been in touch with Onomondo's support, and although they don't think they're the cause of the issue either, I tried disabling the connector and just using regular HTTP/s requests and that actually works as expected!

So for now I'll just have to use regular requests and avoid the connector system when sending binary data.

Thank you for the support!

RE: Trouble receiving binary stream data with recv()

Sigurd Hellesvik — Wed, 01 Mar 2023 15:25:30 GMT

To view your modem traces, I converted them to .pcapng using the Modem Trace tool.
Then I open .pcapng in Wireshark.

For the app_update.bin, I use the command-line tool hexyl to look at the data.

Here are two screenshots from Wireshark and hexyl:

The "broken" trace:

app_update.bin:

As you can see from the data, both start with "3d" and then differ slightly.
I suspect this is because you have sent me the wrong app_update.bin from the one used in your trace.
Either way, both have "02 00 00 00 00 00 etc" as well, so the start of both these look fine so far.

However, after a little bit, the trace repeats "ef bd bd ef bf bd" and so on.
When I inspect packets further down where the data start to repeat, I get all "ef bd db" repeating.
If you convert this to decimal, it is likely the data you get repeating.

I googled this and found a post on stack overflow: https://stackoverflow.com/questions/38018263/what-is-the-format-or-encoding-of-a-file-with-data-like-this.

From this, I suggest you have another look at how you encode the binary file when you transfer it.
I am not experienced in nodejs express server, but you put "utf-8" into the send function. Maybe do something else here?

From the working hex-string trace, I can read data which is not these "illegal utf-8" bytes from the pcapng.
I think this is because when you send a string, the string is "utf-8".

Does this help?

Regards,
Sigurd Hellesvik

RE: Trouble receiving binary stream data with recv()

-Richard- — Mon, 27 Feb 2023 21:32:35 GMT

Here's the bin file that's on the server

devzone.nordicsemi.com/.../4722.app_5F00_update.bin

This is how I read the data from the socket on the nrf9160, both string formatted and binary.

The IP being 1.2.3.4 is correct, we use Onomondo connectors to send and receive our data, and we configure on their dashboard where the data will be sent. You can read more about it here Onomondo Connectors.

Here's how the nodejs express server returns the bin currently, where response is just an array of the bytes.

Binary version

Hex string version

When I test this in Postman or download it directly to my computer through the API it works with the binary version, but when I try to download on the NRF9160 I get the wrong bytes, although when I convert it to a string in a hex format I then receive it correctly.

Modem trace with string formatting.

devzone.nordicsemi.com/.../trace_2D00_2023_2D00_02_2D00_27T21_2D00_26_2D00_23.459Z.bin

devzone.nordicsemi.com/.../trace_2D00_2023_2D00_03_2D00_01T12_2D00_15_2D00_56.056Z.bin

RE: Trouble receiving binary stream data with recv()

Sigurd Hellesvik — Mon, 27 Feb 2023 14:18:56 GMT

I have still been unable to find what this could be, so I will keep asking for info.

Can you send the file you are trying to transfer?
Can you list the functions on the nRF9160 you use to receive the data?

From your modem trace, it looks like your server has the IP 1.2.3.4. This seems a bit odd to me. Does your server actually have the IP 1.2.3.4, or is something else happening here?

Since it works when you send the data as strings, can you list how the send function looks with and without the string formatting?
And can you take a modem trace with the string formatting? It could be useful to compare the good VS bad modem traces.

Regards,
Sigurd Hellesvik

RE: Trouble receiving binary stream data with recv()

-Richard- — Mon, 27 Feb 2023 11:55:16 GMT

The modem firmware version is mfw_nrf9160_1.3.3

RE: Trouble receiving binary stream data with recv()

Sigurd Hellesvik — Thu, 23 Feb 2023 13:16:09 GMT

Hi,

Which modem firmware version do you use?

Regards,
Sigurd Hellesvik

RE: Trouble receiving binary stream data with recv()

-Richard- — Tue, 21 Feb 2023 16:16:56 GMT

devzone.nordicsemi.com/.../8357.modem_2D00_trace.zip

There's a modem trace for 2 of the ports, as 2 of them seemed to read something and I wasn't sure which port exactly was the right one.

RE: Trouble receiving binary stream data with recv()

Sigurd Hellesvik — Mon, 20 Feb 2023 14:31:21 GMT

Can you record and upload a modem trace?

This may give us more insight into the issue.

Regards,
Sigurd Hellesvik

RE: Trouble receiving binary stream data with recv()

-Richard- — Fri, 17 Feb 2023 10:28:36 GMT

Yes it returns success for both bytes, I recv chunks of 2048 bytes at a time, and in there I'll have a mix of both.

I use a NodeJS express server, and if I return a binary buffer, that's when the NRF9160 gets the wrong bytes, however if I take that buffer and say turn it into a string or a hex string on the server before returning it, then I receive the data in string format correctly on the NRF9160.

Appears to only happen when I try to recv a binary buffer on the NRF9160 from the express server, as I've tried with other formats such as JSON, and that gets handled correctly.

RE: Trouble receiving binary stream data with recv()

Sigurd Hellesvik — Fri, 17 Feb 2023 08:08:39 GMT

[quote user=""] I try to use the recv function[/quote]

Do you check the return value of the recv() function?
Does it return success both for the "good bytes" and the "bad bytes"?

[quote user=""] I am however able to return the bytes in a string format and then convert them on the device, although it's a very slow operation.[/quote]

Is this for the server or the nRF9160?

Regards,
Sigurd Hellesvik

RE: Trouble receiving binary stream data with recv()

Sigurd Hellesvik — Thu, 16 Feb 2023 15:30:34 GMT

Hi,

I will look into this and return with more information tomorrow.

Regards,
Sigurd Hellesvik