nRF9160 mqtt event message formatting for Azure IoT Hub

[prolo_isaac]

Hello,

I am trying to use the nRF9160 DK to publish some sensor data to an Azure IoT Hub. I have been able to successfully connect to the mqtt broker and publish simple strings as an D2C event message, but I am wanting to publish a json structure instead to take advantage of existing handling on our IoT Hub that I am not involved with in terms of development.

I have successfully recieved this d2c message:

{
    "event": {
        "origin": "{device-id}",
        "payload": "a"
    }
}

I want this:

{
    "event": {
        "origin": "{device-id}",
        "payload": {
            "published_at": "XXXX-XX-XXXXX:XX:XX.XXXX",
            "device_id": "{device-id}",
            "data": "0.710180:1.796526:48.254000:58.799999:988.430000",
            "event": "demo"
        }
    }
}

But I am getting this:

{
    "event": {
        "origin": "{device-id}",
        "payload": "{\n\"published_at\": \"XXXX-XX-XXXXX:XX:XX.XXXX\",\n\"device_id\": \"{device-id}\"\n,\"data\":\"0.710180:1.796526:48.254000:58.799999:988.430000\",\n\"event\": \"demo"
    }
}

Does anybody more familiar with zephyr's mqtt library or Azure's IoT Hub know what my options would be for formatting my message going into the mqtt_publish_param in mqtt.h? I feel like my hands are tied since the program is configured to send messages composed as type 'u8_t*'.

/** @brief Abstracts binary strings. */
struct mqtt_binstr {
	u8_t *data;             /**< Pointer to binary stream. */
	u32_t len;              /**< Length of binary stream. */
};

/** @brief Parameters for a publish message. */
struct mqtt_publish_message {
	struct mqtt_topic topic;     /**< Topic on which data was published. */
	struct mqtt_binstr payload; /**< Payload on the topic published. */
};

/** @brief Parameters for a publish message. */
struct mqtt_publish_param {
	/** Messages including topic, QoS and its payload (if any)
	 *  to be published.
	 */
	struct mqtt_publish_message message;

	/** Message id used for the publish message. Redundant for QoS 0. */
	u16_t message_id;

	/** Duplicate flag. If 1, it indicates the message is being
	 *  retransmitted. Has no meaning with QoS 0.
	 */
	u8_t dup_flag : 1;

	/** Retain flag. If 1, the message shall be stored persistently
	 *  by the broker.
	 */
	u8_t retain_flag : 1;
};

I included a copy of my project (mqtt_simple.zip), leaving out any specific information about my device or iot hub hostname if it would help anybody. Just find and replace all mentions of '{device-id}' with your proper device-id, replace '{hostname}' with the proper hostname of your iot hub, and replace '{shared-access-token}' with your generated token for your device. Do this for '\src\main.c', '\Kconfig', and '\prj.conf'.

I am using the command-line tool for Azure IoT Hub to monitor events:

az iot hub monitor-events --hub-name {hostname} -t 0 --cg {consumer-group}

In the command to monitor events, replace '{hostname}' with the proper hostname for your IoT Hub and {consumer-group} with a proper consumer group on your IoT Hub.

If I left out any important information that would be helpful, please let me know and I will provide it ASAP.

Thanks!

Isaac

[Håkon Alseth]

Hi,

 

It sounds like you want help encoding (and possibly decoding?) json format. Do I understand correctly? Please correct me if I've misunderstood.

Have you looked at cjson? This can be added by setting CONFIG_CJSON_LIB=y (and including cJSON.h)

 

There's a link in this thread to a stackoverflow post that explained how to use it:

https://devzone.nordicsemi.com/f/nordic-q-a/57484/properly-format-json-message-for-mqtt-transfer-to-aws-iot

 

Kind regards,

Håkon

[prolo_isaac]

I think I'm on the right track for using cJSON in my program, but I'm still running into a similar issue, and that thread you linked doesn't seem to have been verified to work as a solution by the user, or at least there was no working implementation acknowledged or given.

This is what I am doing with cJSON:

void main(void)
{

   char *out;
   cJSON *root, *cars, *car;

   /* create root node and array */
   root = cJSON_CreateObject();
   cars = cJSON_CreateArray();

   /* add cars array to root */
   cJSON_AddItemToObject(root, "cars", cars);

   /* add 1st car to cars array */
   cJSON_AddItemToArray(cars, car = cJSON_CreateObject());
   cJSON_AddItemToObject(car, "CarType", cJSON_CreateString("BMW"));
   cJSON_AddItemToObject(car, "carID", cJSON_CreateString("bmw123"));

   /* add 2nd car to cars array */
   cJSON_AddItemToArray(cars, car = cJSON_CreateObject());
   cJSON_AddItemToObject(car, "CarType", cJSON_CreateString("mercedes"));
   cJSON_AddItemToObject(car, "carID", cJSON_CreateString("mercedes123"));

   /* print everything */
   out = cJSON_Print(root);
   printf("%s\n", out);
   free(out);

   /* free all objects under root and root itself */
   cJSON_Delete(root);
   
	int err;

	printk("The MQTT simple sample started\n");

	modem_configure();

#if defined(CONFIG_PROVISION_CERTIFICATES)
	provision_certificates();
#endif /* CONFIG_PROVISION_CERTIFICATES */

	client_init(&client);

	err = mqtt_connect(&client);
	if (err != 0) {
		printk("ERROR: mqtt_connect %d\n", err);
		return;
	}

	err = fds_init(&client);
	if (err != 0) {
		printk("ERROR: fds_init %d\n", err);
		return;
	}

    err = poll(&fds, 1, K_SECONDS(CONFIG_MQTT_KEEPALIVE)); 
    if (err < 0) {
            printk("ERROR: poll %d\n", errno);
    }

    if ((fds.revents & POLLIN) == POLLIN) {
            err = mqtt_input(&client);
            if (err != 0) {
                    printk("ERROR: mqtt_input %d\n", err);
            }
    }

    while(1){
            err = data_publish(&client, MQTT_QOS_0_AT_MOST_ONCE, (u8_t*)out, (size_t) strlen(out)); // This works out
            printk("err=%d\n", err);
			printk("sizeof: %d", strlen(out));
            printk("Send reset message\n");
            k_sleep(30000);
    }
}

The object printed to the terminal correctly:

{

	"cars":	[{

			"CarType":	"BMW",

			"carID":	"bmw123"

		}, {

			"CarType":	"mercedes",

			"carID":	"mercedes123"

		}]

}

This code succesfully creates a cJSON object, but when I attempt to publish it via mqtt, it is still not formatted correctly.

{
    "event": {
        "origin": "{device-id}",
        "payload": "{\n\t\"cars\":\t[{\n\t\t\t\"CarType\":\t\"BMW\",\n\t\t\t\"carID\":\t\"bmw123\"\n\t\t}, {\n\t\t\t\"CarType\":\t\"mercedes\",\n\t\t\t\"carID\":\t\"mercedes123\"\n\t\t}]\n}"
    }
}

I'm not exactly sure where I should be looking to solve this, because I'm not sure what exactly the problem is. Any advice for where to start?

[prolo_isaac]

I now have the messages posting in json format using cJSON and mqtt. It turns out that you have to append the publishing topic in mqtt with tags denoting that the message is of the application/json type and it is UTF-8.

I changed the publishing topic in prj.conf to the following:

CONFIG_MQTT_PUB_TOPIC="devices/{device-id}/messages/events/$.ct=application%2Fjson&$.ce=utf-8"

And now it works.

Thanks for the help!