Best way to send data from many sensors

RE: Best way to send data from many sensors

rct42 — Tue, 30 May 2017 21:25:56 GMT

If you decide to go down the UART over BLE, one recommendation for a protocol is IBM Bisync (or something similar). However, if you're wanting to transfer 3K bytes in a go, how about YMODEM? (I haven't used YMODEM on NRF5x projects but have used the STM32F0xx_IAP for STM32 UART/RS485 bootloaders)

RE: Best way to send data from many sensors

endnode — Tue, 30 May 2017 08:37:59 GMT

But to conclude on NUS: it is good thing which should just allow data to flow over the GATT, basically no overhead (examples are expecting the strings to end with line delimiters or similar but I'm not sure if it is inherent limitation of NUS itself or just example implementation). Still it doesn't help you with designing your own protocol (= if you want to send some sensor data over the link you need to format them yourself, if you want to provide them on request - after the other peer asks and maybe even says what data it wants - then you need to design your command/response protocol yourself as well... we were maybe mixing these two things together in this Q&A too much.

RE: Best way to send data from many sensors

endnode — Tue, 30 May 2017 08:27:26 GMT

Hi both, I'm used to work with single Characteristic simply because it works well. The only added value of splitting Tx/Rx streams into two GATT Characteristics (I'm aware of) is that you get full-duplex capability aka you can "stream" data both ways at the same time and througput should be close to optimal while with single Characteristic some devices (I don't think it is limitation mentioned in BT SIG specification but I might be wrong) can use only one GATT method (= Write or Notify) on the same handle during one connection interval so sometimes when there are multiple PDUs pending on both sides to go over the radio they might get delayed. But because 99% of protocols I see in the wild are half-duplex (command-response not asynchronous data streams from both sides) I don't see this as major limitation.

RE: Best way to send data from many sensors

Olivier — Tue, 30 May 2017 07:19:20 GMT

Hi rct42, Thanks for your comment! Yes I'm considering starting with this example (infocenter.nordicsemi.com/index.jsp. That is why I don't understand very well why endnode says that I only need one Characteristic while this example uses 2 Characteristics (working with the nRF51+S130 SoftDevice).

RE: Best way to send data from many sensors

rct42 — Tue, 30 May 2017 01:54:18 GMT

Just wondering whether you've considered the BLE UART abstraction (ble_app_uart) as a method of transferring data and adding a binary protocol to encode/decode this? (I'm also weighing up a similar decision vs writing a specific BLE service)

RE: Best way to send data from many sensors

Olivier — Mon, 29 May 2017 13:13:20 GMT

Thank you for your help and for your suggestions!

RE: Best way to send data from many sensors

endnode — Mon, 29 May 2017 09:49:38 GMT

I'm struggling to understand added value of that second optional Characteristic, what will be Read by Client there vs. what would be received in Notify on the other Characteristic upon some Write (containing command data). But if you want both things and have some use for it then why not.

RE: Best way to send data from many sensors

Olivier — Mon, 29 May 2017 08:37:16 GMT

To resumed the situation I'm now thinking of 2 possibles solutions:

Having a new custom service with only one characteristic (mandatory) which has Notify and Write. I will implement a customized protocol to recognize the data piped within BLE. The values will be stored by the application
Having a new custom service with one characteristic (mandatory) which has Notify and write and an other one (optional) which has read and write. For the notify characteristic I will implement a customized protocol to recognize the data piped within BLE. The values will be stored by the application. For the second characteristic the data will be stored in the stack in the value field and will contain some settings if available which are readable and writable. The Notify will carry many data values but the write command will carry only some data values and some commands. In my opinion splitting the settings and the data in two different characteristic make more sense but I'm still not 100% sure. I understand that you prefer choice 1 but can you please give me some advantages of this solution in comparison with the second choice?

RE: Best way to send data from many sensors

Olivier — Mon, 29 May 2017 08:22:19 GMT

I have updated my question to give a better overview of your answers. Yes I'm designing a solution for one very specific use case.

RE: Best way to send data from many sensors

endnode — Mon, 29 May 2017 07:48:34 GMT

ATT_MTU size is by default 23 and can be negotiated once per connection to minimum from Server and Client max values. We are going into too much theoretical discussion with many assumptions and potential situations where one or another solution can provide better throughput. Does it mean that you are designing solution for one very specific use case of data granularity and througput or are we really talking about some general (limit) use cases?

RE: Best way to send data from many sensors

Olivier — Mon, 29 May 2017 07:33:57 GMT

A) The Notify method provides ATT_MTU-3 value while read blob response provides ATT_MTU-1 with should offer a better throughput. Doesn't it?

RE: Best way to send data from many sensors

endnode — Mon, 29 May 2017 07:33:36 GMT

B) Again all my comments are skipping this basic scenario of quasi-static value stored in Characteristic's Value handle. Yes, you can do Write and Read from Client side and async Notification from Server side but that leads exactly to the scheme you presented originally: load of single-purpose Characteristics on the server side. It can be fine for simple use cases but when scaling to dozens of things (like your case with dozens of sensors) and caring about "performance" in terms of connection time and throughput the experience is suggesting to design single "tube" (or pair of Tx/Rx buffers) on top of some Characteristic Value and then define more condensed protocol with Write/Notify methods. The worst thing which could happen is that you will have the same "performance" as if you would do Read + response, but typically you will be much better.

RE: Best way to send data from many sensors

Olivier — Mon, 29 May 2017 07:30:52 GMT

C) Yes the Notification maximal size is ATT_MTU-3 but what is the size of ATT_MTU? Is it defined by the Server or by the Client?

RE: Best way to send data from many sensors

endnode — Mon, 29 May 2017 07:26:02 GMT

A) Reading values over (G)ATT methods in BLE makes sense for some read-only or similar informative string exposed by GATT Server. It's rarely used. For any real communication on application layer Notify methods are providing much better throughput so everyone uses that way.

RE: Best way to send data from many sensors

Olivier — Mon, 29 May 2017 07:09:42 GMT

B) If I understand you right, with the read command I will read the data written with the write command but the content of notification will not be the same as the written data? In my understanding a Service with notification has a Characteristic with the propertie (read, notify, etc.) in its value, a Characteristic with the value being advertised and a Characteristic format with the status of the notification/indication (enable or disable) in its value. For example the Battery Service is read and notify and has only one Characteristic Value, so I assume that an application reading the Battery level will get the same value as the one being notified. For this reason I don't understand why should the write command write at a different place as the read command read from? In this example I differentiate the notification enable and the write because notification enable write a 1 in the Characteristic Format bit0 which is notification enable.

RE: Best way to send data from many sensors

Olivier — Mon, 29 May 2017 07:07:07 GMT

A) Why do you say that the Read methods is kind of old-fashion "pull" model for people who are not able to understand or use "sync" way? What is the sync way if it isn't read?

RE: Best way to send data from many sensors

endnode — Wed, 24 May 2017 15:06:13 GMT

This is probably just implementation detail but normally (= with Nordic SoftDevice) you get event callback and in event data structure you get pointer to the data buffer. Then you should take the data and work with them, if you store them in one or million buffers inside your application code/logic is your business. Once event callback processing ends you should probably not assume that data are left somewhere for you (they actually are as I wrote but you should do little testing if it matters to you, I was never investigating this because I just use Write and Notify so there is no one who would "read" any data which are left in some Value data buffer).

RE: Best way to send data from many sensors

Olivier — Wed, 24 May 2017 15:01:26 GMT

B) If I write in a characteristic and I notify the same characteristic (say the characteristic is Write+Notify) will the data be written in the same memory region? I'm not referencing to the notification enable which enable the notification but is actually written in the CCCD characteristic.

RE: Best way to send data from many sensors

endnode — Wed, 24 May 2017 14:08:51 GMT

I don't want to be impolite but looking to your assumptions about how BLE works I'd recommend to go through some basic courses like this one from BT SIG. Maybe twice;) It's pretty detailed and explanatory to understand basic concepts of all layers. It's missing few enhancements coming with BT 4.2 and 5.0 but unfortunately there is no such material to cover these known to me, maybe just few blog posts by Nordic guys here cover some parts.

RE: Best way to send data from many sensors

endnode — Wed, 24 May 2017 13:34:22 GMT

Sorry, one more to Notifications and CCCD: once you allow Notification as method for given Characteristic then GATT Server (= the stack under the API aka Soft Device in Nordic case) automatically creates CCCD handle for you. You just need to remember its handle number so then if write arrives to it you know what is it (and can check data which might mean that Client wants to activate Notifications or anything else). You can try to blindly use API and just catch if there is some error return from SD (like "invalid operation (because Notifications are not enabled)") but that doesn't look like proper way of doing SW development;)

RE: Best way to send data from many sensors

endnode — Wed, 24 May 2017 13:28:47 GMT

E) The size limits I mention are linked to ASN.1 BER-TLV notation (length encoding to be specific) but that's completely independent on BLE so you can forget about it if you don't plan to use it;)

RE: Best way to send data from many sensors

endnode — Wed, 24 May 2017 13:27:27 GMT

D) Yes, that's what I would do.

RE: Best way to send data from many sensors

endnode — Wed, 24 May 2017 13:27:02 GMT

C) In Nordic SD API there are GATT calls how to send data out through Notify method but note that Notifications (and Indications) by nature cannot be larger then ATT_MTU size! (again this is my understanding, maybe good to double check with other sources on this forum and internet). The usual solution is to implement custom fragmentation/reassembly protocol on top of GATT (bellow your APP data) which then allows you to transport theoretically unlimited sizes of data strings.

RE: Best way to send data from many sensors

endnode — Wed, 24 May 2017 13:24:18 GMT

B) your understanding of GATT Characteristics seems to be wrong. There is "Value" handle attached to each Char and that is used for Write/Read methods (= data are "stored" there). However on top of the same "Value" handle you can support Notify or Indicate methods which are kind of "separate" (one time) data transfers not interfering with data in actual "Value" buffer. Or at least this is my understanding and also implementation in Nordic Stack (but please validate this experimentally). All methods must be "allowed" on given Characteristic explicitly but you can combine it so one single Char you can allow Notifications as well as Write methods and let the data flow bi-directionally (and you can allow even more things of course). The only limitation seems to be that methods cannot be combined within one connection interval (so it's kind of half-duplex).

RE: Best way to send data from many sensors

endnode — Wed, 24 May 2017 13:19:07 GMT

(continue...)

A) As I said I was lost in your Client/Server architecture so in general from Client to Server you transport data through Write methods (two types) and from Server to Client through Notify/Indicate methods (this is usual "async" model used by most of modern Profiles) or through Read methods (kind of old-fashion "pull" model for people who are not able to understand or use "sync" way). Again I don't understand all the details of your application data flow so this was general remark and you will need to decide yourself.