SAADC sampling causes periodic/repeated failure with I2C communication stream

[cor10]

System Details:

• Board: Custom
• Chip: nRF52840/nRF52833
• PCA Number: PCA10056/10100
• SoftDevice: S140, v7
• nRF5 SDK: Version 16.0.0
  
• mdk version: 8.46.0
• nrfjprog version: 10.15.4 external
• JLinkARM.dll version: 7.64d

Hello,

I have a BLE UART app that streams on-board sensor data to my central BLE device (laptop) every 50ms (20hz). I visualize this incoming data continuously, and I noticed that every 5 seconds, there is an I2C/TWI communication failure and all sensor data momentarily goes to 0s as the I2C lines become unavailable.

I was able to confirm that this periodic error comes from my code that performs SAADC sampling to read the on-board battery level. When I disabled this SAADC module, the error goes away completely. For context, my code uses a timer, which triggers the SAADC sampling by calling nrf_drv_saadc_sample(). This timer is triggered every BATTERY_SAMPLE_PERIOD milliseconds. I began experimenting with this timer frequency, and I noticed that if this timer is triggered every 900ms (or longer), then the I2C communication error will arise. However, if I set this BATTERY_SAMPLE_PERIOD to something shorter than that, say 800, then this issue never occurs. However, the end result is never actually calculated in only 700ms...generally takes a couple seconds.

Ideally, we want to be able to sample this battery level every 20 seconds or something, but it seems this communication error arises anytime we set the value above 900. My hunch makes me feel like this issue comes about when the SAADC module finishes sampling & sends an interrupt to the main program thread, which blocks the i2c/twi communication somehow. Is there some explanation for this behavior? Would you recommend a way to deal with these types of SAADC timing-related issues?

Here is my battery.c file which contains the battery sampling code for your reference:

#include "app_error.h"
#include "app_scheduler.h"
#include "app_timer.h"
#include "battery.h"
#include "board.h"
#include "nrf_drv_saadc.h"
#include "nrf_log.h"
#include "nrf_gpio.h"

#define BATTERY_SAMPLE_PERIOD                   700

#define SAADC_BUFFER_SIZE                       3

#define ADC_VOLTAGE_DIV_COMPENSATION            2 // 10M / (10M + 10M)
#define ADC_REF_VOLTAGE_IN_MILLIVOLTS           600
#define ADC_PRE_SCALING_COMPENSATION            6
#define ADC_RESULT_IN_MILLI_VOLTS(ADC_VALUE)    ((((ADC_VALUE) * ADC_REF_VOLTAGE_IN_MILLIVOLTS) / 1023) * ADC_PRE_SCALING_COMPENSATION * ADC_VOLTAGE_DIV_COMPENSATION)

APP_TIMER_DEF(m_batteryTimerId);

static nrf_saadc_value_t m_buffer_pool[2][SAADC_BUFFER_SIZE];
static uint16_t m_vBat;

void saadc_callback(nrf_drv_saadc_evt_t const * p_event)
{
    ret_code_t err_code;
    uint16_t adc_sum_value = 0;
    uint16_t adc_average_value;
    uint16_t adc_result_millivolts;

    if (p_event->type == NRF_DRV_SAADC_EVT_DONE)
    {
        int i;
        for (i = 0; i < SAADC_BUFFER_SIZE; i++)
        {
            adc_sum_value += p_event->data.done.p_buffer[i];
        }
        adc_average_value = adc_sum_value / p_event->data.done.size;
        //NRF_LOG_INFO("[battery] ADC avg value: %d \r\n", adc_average_value);
				
        adc_result_millivolts = ADC_RESULT_IN_MILLI_VOLTS(adc_average_value);
        m_vBat = adc_result_millivolts;

        NRF_LOG_INFO("[battery] ADC result: %dmV\r\n", adc_result_millivolts);

        err_code = nrf_drv_saadc_buffer_convert(p_event->data.done.p_buffer, SAADC_BUFFER_SIZE);
        APP_ERROR_CHECK(err_code);
    }
}

static void vBatSampleHelper(void)
{
    // Trigger sampling
    ret_code_t ret = nrf_drv_saadc_sample();
    APP_ERROR_CHECK(ret);
}

uint16_t batteryGetVbat(void)
{
    return m_vBat;
}

void batteryTimerHandler(void * p_context)
{
    app_sched_event_put(NULL, 0, (app_sched_event_handler_t)vBatSampleHelper);
}

void batterySamplingStart(void)
{
    // Start timer
    uint32_t ret = app_timer_start(m_batteryTimerId, APP_TIMER_TICKS(BATTERY_SAMPLE_PERIOD), NULL);
    APP_ERROR_CHECK(ret);

#ifdef NRF52832_XXAA
    // Initialize SAADC hardware to produce correct VBAT readings
    nrf_gpio_pin_set(PIN_VBAT_EN);
#endif
}

void batterySamplingStop(void)
{
    app_timer_stop(m_batteryTimerId);
#ifdef NRF52832_XXAA
    nrf_gpio_pin_clear(PIN_VBAT_EN);
#endif
}

void batteryInit(void)
{
    ret_code_t ret;

#ifdef NRF52832_XXAA
    /* Configure PIN_VBAT_EN as an OUTPUT */
    nrf_gpio_cfg_output(PIN_VBAT_EN);
    /* Write PIN_VBAT_EN to a LOW value (disables SAADC VBAT Reading Voltage Divider until BLE connects) */
    nrf_gpio_pin_clear(PIN_VBAT_EN);
#endif

    // Configure SAADC   
    nrf_saadc_channel_config_t channel_config = NRF_DRV_SAADC_DEFAULT_CHANNEL_CONFIG_SE(PIN_VBAT_SENSE);  // SELECT CHANNEL
    channel_config.reference = NRF_SAADC_REFERENCE_INTERNAL; // _REFERENCE_VDD4;
    channel_config.gain = NRF_SAADC_GAIN1_6; // _GAAIN1_4;

    ret = nrf_drv_saadc_init(NULL, saadc_callback);
    APP_ERROR_CHECK(ret);
    ret = nrf_drv_saadc_channel_init(0, &channel_config);
    APP_ERROR_CHECK(ret);
    ret = nrf_drv_saadc_buffer_convert(m_buffer_pool[0], SAADC_BUFFER_SIZE);
    APP_ERROR_CHECK(ret);
    ret = nrf_drv_saadc_buffer_convert(m_buffer_pool[1], SAADC_BUFFER_SIZE);
    APP_ERROR_CHECK(ret);

    // Create timer for periodic battery sampling
    ret = app_timer_create(&m_batteryTimerId, APP_TIMER_MODE_REPEATED, batteryTimerHandler);
    APP_ERROR_CHECK(ret);
}

Many thanks in advance,

Corten

[Edvin]

Hello,

I assume that your I2C/TWI is used to visualize the data?

How is your TWI set up? Is it blocking or non-blocking?

Perhaps you can share some snippets on that as well?

here is an I2C/TWI communication failure and all sensor data momentarily goes to 0s as the I2C lines become unavailable.

So the 0's are on the receiver of the TWI data?

I don't really see a reason why there would be a limit around 8-900ms. The SAADC generates an interrupt, and will go through the saadc_callback. If this is blocking the TWI, then you should consider spending as little time as possible within this callback. The conversion of the buffers are necessary, but perhaps you don't need to log (NRF_LOG_INFO()) inside the callback? You can store the data in a variable and print it from elsewhere. I see that you store it in m_vBat, so perhaps you can print this to the log from somewhere else. I guess the logging doesn't really take that much time unless you are using the UART backend for the log and you turned off deferred logging.

The device that is receiving the data over TWI. How does it behave if one message is delayed. The TWI operations you are doing may be interrupted, but if a message is sent (buffer is ready and you start the send task) then the message itself will not be interrupted. However, if you send multiple messages, and you need to receive all of those to update the visualization, then that may be the issue. So can you please show me a bit more how you are doing the TWI, and how does the TWI (slave) device behave if a message is delayed? What sort of device is it? A monitor/display?

BR,

Edvin

[cor10]

Hi Edvin, let me explain in more detail - I realize I left out some helpful information.

My TWI/I2C is non-blocking, as far as I can tell from the docs online (aka when I initialize the twi module, I provide a twiCallback event handler to the nrf_drv_twi_init() function call). I am including the initialization code as well as the event handler here for your reference:

static void twiCallback(nrf_drv_twi_evt_t const * p_event, void * p_context)
{
    switch (p_event->type)
    {
        case NRF_DRV_TWI_EVT_DONE:
        {
            m_xfer_status = TWI_XFER_STATUS_SUCCESS;
            break;
        }
        case NRF_DRV_TWI_EVT_ADDRESS_NACK:
        {
            if (expectedNACK)
            {
                m_xfer_status = TWI_XFER_STATUS_SUCCESS;
                //NRF_LOG_INFO("NRF DRV TWI: Expected Address NACK!");
            } else {
                m_xfer_status = TWI_XFER_STATUS_ERROR;
                NRF_LOG_INFO("NRF DRV TWI ADDRESS NACK ---> ERROR!");
            }
            break;
        }
        case NRF_DRV_TWI_EVT_DATA_NACK:
        {
            m_xfer_status = TWI_XFER_STATUS_ERROR;
            NRF_LOG_INFO("NRF DRV TWI DATA NACK ---> ERROR!");
            break;
        }
        default:
            break;
    }
}

void twiInit(uint8_t sda, uint8_t scl)
{
    ret_code_t err_code;

    // Configure and initialize TWI
    const nrf_drv_twi_config_t twi_config = 
    {
       .scl                = scl,
       .sda                = sda,
       .frequency          = NRF_DRV_TWI_FREQ_400K,
       .interrupt_priority = APP_IRQ_PRIORITY_HIGH,
       .clear_bus_init     = true,
    };

    err_code = nrf_drv_twi_init(&m_twi, &twi_config, twiCallback, NULL);
    APP_ERROR_CHECK(err_code);

    nrf_drv_twi_enable(&m_twi);	
}

The I2C/TWI is mostly used by the main nRF52833 mcu to read sensor data from the various ICs we have on our custom board (IMU, capacitive touch, etc). This is accomplished with a repeated timer (every 50ms) that reads all sensor data at once, and then streams it over BLE to a laptop upon which is an application that visualizes all the data. The only other functionality of my application is the app timer that triggers the SAADC battery reading, along with a Watch Dog Timer (every 300ms).

Edvin said:

So the 0's are on the receiver of the TWI data?

During each iteration of the repeated sensor data sampling timer I mentioned above, I zero-initialize all the data buffers. This means that if there is an issue in the I2C/TWI communication that prevents the nRF52833 from reading new sensor data, then the timer never actually overwrites the 0s in the data buffers, and that is what gets streamed to my visualization app. In other words, seeing a momentarily glitch of 0s in all the data visualizations/graphs is indicative of a failed read operation.

Edvin said:

I don't really see a reason why there would be a limit around 8-900ms.

Neither do I. I have a feeling something deeper is wrong with my code, and Im just hoping to understand how/why this SAADC timer is related, because maybe then I can fix whatever is broken. The SAADC timer I have seems to do its job just fine. I tested it and verified that it behaves as expected, even when I set the BATTERY_SAMPLE_PERIOD to something like 30seconds (30000ms). However, regardless of how large I make this sample period, this I2C/TWI failure always occurs every 5 seconds. The only thing I can do is to lower the sample period to something below 900ms, which prevents the error from arising, but is not ideal for our battery consumption.

Another strange behavior I noticed was the effect of commenting/uncommenting the NRF_LONG_INFO() function. When I was using BATTER_SAMPLE_PERIOD = 700ms and the NRF_LOG_INFO() was uncommented, everything was working and there were no TWI/I2C errors. However, when I used the same 700ms period, but with //NRF_LOG_INFO() commented out, the TWI/I2C errors showed up. This behavior is consistent/repeatable. This tells me things must be pretty unstable with our code, but I dont know where to look.

Edvin said:

unless you are using the UART backend for the log

From my config file: #define NRF_LOG_BACKEND_UART_ENABLED 0

What could be causing a 5 sec periodic failure? Any ideas what I should do next?

[Edvin]

Hello,

cor10 said:

as far as I can tell from the docs online (aka when I initialize the twi module, I provide a twiCallback event handler to the nrf_drv_twi_init() function call).

Yes. That should be non-blocking, then.

Interesting. I can't think of anything from the top of my head that should cause these glitches every 5 seconds. So when you clear all the buffers by setting the values to 0s. You are tringgering new reads every 50ms using a timer, right? Between the time that you are setting them to 0 and the time that you are sending the data over BLE. Are you sure that the TWI read commands are called? That is, can you also see the 0's when you are sending the data over BLE? I assume you are using notifications, so you can check the variables when you send them. Can you verify that the values are 0 when they are being sent?

So that leads to the next question. Why is it not updated from 0 to a value read from the sensor? Should the repeated timer trigger in the meantime? How often do you send data from the nRF to the central? How do you send that data (Notification or read requests)? What is your connection interval? Is it more or less than 50 ms?

Is there a way for me to reproduce this issue on a DK? I don't assume that there is a way to strip out the I2C part of your application, because they are essential in order to reproduce the bug, but did you test just writing dummy values in the timer timeout instead of actually reading the values, just to see whether the timer callback is called at all?

Best regards,

Edvin

[cor10]

Hi Edvin, thank you for the response. Let me respond to a couple things here.

Edvin said:

I assume you are using notifications

We are using the nus_data_send(data, data_len) function directly in the task that gets scheduled by the app scheduler when our repeated data sampling timer gets triggered. Below I have included the timer handler that schedules this task, sampleTimerHandler(), as well as the code that comprises the task being scheduled, sampleDataEventHandler(): 

static void sampleDataEventHandler(void)
{
    uint32_t err_code;
    uint8_t dataPacket[BLE_CHARACTERISTIC_MAX_LENGTH] = {0};
    uint8_t dataIndex = 0;

    if(m_dataTypes & DATA_TYPE_VBAT)
    {
        // Retrieve last sampled battery voltage data and update data array
        uint16_t batV = batteryGetVbat();
        
        memcpy(dataPacket + dataIndex, (uint8_t *)&batV, 2);
        dataIndex = dataIndex + 2;
    }

    if(m_dataTypes & DATA_TYPE_CAP_TOUCH) 
    {
        // Retrieve CAP TOUCH data, and update data array with new values
        uint8_t XYValues[98] = {0};
        capTouchReadTrackpad(XYValues);

        memcpy(dataPacket + dataIndex, XYValues, 98);
        dataIndex = dataIndex + 98;
    }

    if(m_dataTypes & DATA_TYPE_IMU_PRIMARY) 
    {
        // Retrieve primary IMU accelerometer and gyroscope data and update data array
        struct bmi2_sensor_data accData;
        struct bmi2_sensor_data gyroData;

        imuGetData(IMU_PRIMARY, &accData, &gyroData);

        // Copy accelerometer data
        memcpy(dataPacket + dataIndex, (uint8_t *)&accData.sens_data.acc, 6);
        dataIndex = dataIndex + 6;

        // Copy gyro data
        memcpy(dataPacket + dataIndex, (uint8_t *)&gyroData.sens_data.gyr, 6);
        dataIndex = dataIndex + 6;
    }

    if(m_streamBLE) 
    {
        err_code = nus_data_send(dataPacket, dataIndex);
        if ((err_code != NRF_ERROR_INVALID_STATE) &&
                (err_code != NRF_ERROR_RESOURCES) &&
                (err_code != NRF_ERROR_NOT_FOUND))
        {
            APP_ERROR_CHECK(err_code);
        }
    } 
    else
    {
        // When BLE hardware is disabled, but we want to see sensor output
        for (uint8_t i = 0; i < dataIndex; i++) 
        {
            NRF_LOG_INFO("%d, ", dataPacket[i]);
        }
        NRF_LOG_INFO("\n");
        NRF_LOG_FLUSH();
    }
}

/**@brief Function to read sensor values and stream them via BLE to backend server
 *
 *@Details This function is called repeatedly at DATA_SAMPLING_PERIOD using the nRF Timer infrastructure
 */
static void sampleTimerHandler(void * p_context) 
{
    // Schedule data sampling
    app_sched_event_put(NULL, 0, (app_sched_event_handler_t)sampleDataEventHandler);
}

In this code above, you can see that I can directly view the data being sent out without even needing the Bluetooth hardware....but when the hardware is present/enabled, I have placed print statements before the call to nus_data_send() and can verify the entire data packet going from non-zero to zero momentarily.

A new observation: When digging into this issue, I enabled more debugging statements throughout my code, and have observed that every time the zero-data glitch occurs, so too does a "Communication Error" when I call my IMU sensor's API to retrieve sensor data. Specifically, I have a function called imuGetData() from the code above, and this function is the one that calls bmi2_get_sensor_data() from the BMI 270 IMU API. See the code below, which shows BMI2_E_COM_FAIL (the error that shows up for me exactly every time the glitch occurs):

void imuGetData(imuInstance_t instance, struct bmi2_sensor_data * accData, struct bmi2_sensor_data * gyroData)
{
    int8_t rslt;
    uint16_t int_status;
    struct bmi2_dev * p_dev;

    if (instance == IMU_PRIMARY)
    {
        p_dev = &m_imuPrimary;
    }
    else if (instance == IMU_SECONDARY) 
    {
        p_dev = &m_imuSecondary;
    }

    accData->type = BMI2_ACCEL;
    gyroData->type = BMI2_GYRO;

    // Read acc sensor data from BMI270
    rslt = bmi2_get_sensor_data(accData, 1, p_dev);
    imuPrintRslt(rslt);

    // Read gyro sensor data from BMI270
    rslt = bmi2_get_sensor_data(gyroData, 1, p_dev);
    imuPrintRslt(rslt);
}

/*!
 *  @brief Prints the execution status of the API calls.
 *
 *  @param[in] rslt     : error code returned by the API whose execution status has to be printed.
 *
 *  @return void.
 */
void imuPrintRslt(int8_t rslt)
{
    switch (rslt)
    {
        case BMI2_OK:

            /* Do nothing */
            break;
        case BMI2_E_NULL_PTR:
            NRF_LOG_INFO("BMI270 Error   [%d] : Null pointer\r\n", rslt);
            break;
        case BMI2_E_COM_FAIL:
            NRF_LOG_INFO("BMI270 Error   [%d] : Communication failure\r\n", rslt);
            break;
        case BMI2_E_DEV_NOT_FOUND:
            NRF_LOG_INFO("BMI270 Error   [%d] : Device not found\r\n", rslt);
            break;
        case BMI2_E_INVALID_SENSOR:
            NRF_LOG_INFO("BMI270 Error   [%d] : Invalid sensor\r\n", rslt);
            break;
        case BMI2_E_SELF_TEST_FAIL:
            NRF_LOG_INFO("BMI270 Warning [%d] : Self test failed\r\n", rslt);
            break;
        case BMI2_E_INVALID_INT_PIN:
            NRF_LOG_INFO("BMI270 Warning [%d] : Invalid int pin\r\n", rslt);
            break;
        case BMI2_E_CONFIG_LOAD:
            NRF_LOG_INFO("BMI270 Error   [%d] : BMI device Config Load error\r\n", rslt);
            break;
        default:
            NRF_LOG_INFO("BMI270 Error   [%d] : Unknown error code\r\n", rslt);
            break;
    }
    NRF_LOG_FLUSH();
}

Edvin said:

Should the repeated timer trigger in the meantime?

I would maybe think this would be related if the glitch was happening faster than every 5 seconds. The 5 second frequency is throwing me off, but could the triggering of the timer/scheduling of a task really cause this?

Edvin said:

How often do you send data from the nRF to the central? How do you send that data (Notification or read requests)?

As you saw in the first code snippet I shared, the data from the nRF is sent to the central at the end of the data-sampling event that is triggered by our 50ms repeated timer. So, the data is sent at the same frequency with which it is read. If this code I shared is not the way you would recommend to stream data, any feedback or advice on best-practices would be much appreciated. I cant help but thinking that I am doing things incorrectly somewhere, which is creating this glitch-bug.

Edvin said:

What is your connection interval?

Here is a snippet of code with some defines that I think are related:

#define MIN_CONN_INTERVAL               MSEC_TO_UNITS(15, UNIT_1_25_MS)             /**< Minimum acceptable connection interval (0.1 seconds). */
#define MAX_CONN_INTERVAL               MSEC_TO_UNITS(30, UNIT_1_25_MS)  

Edvin said:

Is there a way for me to reproduce this issue on a DK?

Technically, yes there is a way you can reproduce this, but it would require some work on your end...unless you have the same BMI270 sensor IC (or perhaps an LPS22HB barometer, or IQS572 capacitive touch ic)...but if not, you would have to wire up a sensor IC that is i2c-enabled and then write some driver code to initialize the sensor and perform register reads, and then hook that into the data-sampling timer code we have.....this is a bit of effort. Maybe there is a sensor we both have, and I could write that driver for you to test/reproduce on your end?

Thank you so much for your help.

[Edvin]

sorry for the late reply. I have had some unplanned days out of office, and this typically builds up the backlog.

What does the bmi2_get_sensor_data() look like? What would lead to this returning BMI2_E_COM_FAIL (whatever this value is)?

[cor10]

Here is the code from that API call, with several related function definitions (most notably the very last function definition):

/*!
 * @brief This API gets the sensor/feature data for accelerometer, gyroscope,
 * auxiliary sensor, step counter, high-g, gyroscope user-gain update,
 * orientation, gyroscope cross sensitivity and error status for NVM and VFRM.
 */
int8_t bmi2_get_sensor_data(struct bmi2_sensor_data *sensor_data, uint8_t n_sens, struct bmi2_dev *dev)
{
    /* Variable to define error */
    int8_t rslt;

    /* Variable to define loop */
    uint8_t loop;

    /* Variable to get the status of advance power save */
    uint8_t aps_stat = 0;

    /* Null-pointer check */
    rslt = null_ptr_check(dev);
    if ((rslt == BMI2_OK) && (sensor_data != NULL))
    {
        /* Get status of advance power save mode */
        aps_stat = dev->aps_status;
        for (loop = 0; loop < n_sens; loop++)
        {
            /* Disable Advance power save if enabled for feature
             * configurations
             */
            if (sensor_data[loop].type >= BMI2_MAIN_SENS_MAX_NUM)
            {
                if (aps_stat == BMI2_ENABLE)
                {
                    /* Disable advance power save if
                     * enabled
                     */
                    rslt = bmi2_set_adv_power_save(BMI2_DISABLE, dev);
                }
            }
            if (rslt == BMI2_OK)
            {
                switch (sensor_data[loop].type)
                {
                    case BMI2_ACCEL:

                        /* Get accelerometer data */
                        rslt = get_accel_sensor_data(&sensor_data[loop].sens_data.acc, BMI2_ACC_X_LSB_ADDR, dev);
                        break;
                    case BMI2_GYRO:

                        /* Get gyroscope data */
                        rslt = get_gyro_sensor_data(&sensor_data[loop].sens_data.gyr, BMI2_GYR_X_LSB_ADDR, dev);
                        break;
                    case BMI2_AUX:

                        /* Get auxiliary sensor data in data mode */
                        rslt = read_aux_data_mode(sensor_data[loop].sens_data.aux_data, dev);
                        break;
                    default:
                        rslt = BMI2_E_INVALID_SENSOR;
                        break;
                }

                /* Return error if any of the get sensor data fails */
                if (rslt != BMI2_OK)
                {
                    break;
                }
            }

            /* Enable Advance power save if disabled while
             * configuring and not when already disabled
             */
            if ((aps_stat == BMI2_ENABLE) && (rslt == BMI2_OK))
            {
                rslt = bmi2_set_adv_power_save(BMI2_ENABLE, dev);
            }
        }
    }
    else
    {
        rslt = BMI2_E_NULL_PTR;
    }

    return rslt;
}




/*!
 * @brief This internal API gets the accelerometer data from the register.
 */
static int8_t get_accel_sensor_data(struct bmi2_sens_axes_data *data, uint8_t reg_addr, const struct bmi2_dev *dev)
{
    /* Variable to define error */
    int8_t rslt;

    /* Array to define data stored in register */
    uint8_t reg_data[BMI2_ACC_GYR_NUM_BYTES] = { 0 };

    /* Read the sensor data */
    rslt = bmi2_get_regs(reg_addr, reg_data, BMI2_ACC_GYR_NUM_BYTES, dev);
    if (rslt == BMI2_OK)
    {
        /* Get accelerometer data from the register */
        get_acc_gyr_data(data, reg_data);

        /* Get the re-mapped accelerometer data */
        get_remapped_data(data, dev);
    }

    return rslt;
}


/*!
 * @brief This internal API gets the gyroscope data from the register.
 */
static int8_t get_gyro_sensor_data(struct bmi2_sens_axes_data *data, uint8_t reg_addr, const struct bmi2_dev *dev)
{
    /* Variable to define error */
    int8_t rslt;

    /* Array to define data stored in register */
    uint8_t reg_data[BMI2_ACC_GYR_NUM_BYTES] = { 0 };

    /* Read the sensor data */
    rslt = bmi2_get_regs(reg_addr, reg_data, BMI2_ACC_GYR_NUM_BYTES, dev);
    if (rslt == BMI2_OK)
    {
        /* Get gyroscope data from the register */
        get_acc_gyr_data(data, reg_data);

        /* Get the compensated gyroscope data */
        comp_gyro_cross_axis_sensitivity(data, dev);

        /* Get the re-mapped gyroscope data */
        get_remapped_data(data, dev);

    }

    return rslt;
}


/*!
 * @brief This internal API gets the accelerometer/gyroscope data.
 */
static void get_acc_gyr_data(struct bmi2_sens_axes_data *data, const uint8_t *reg_data)
{
    /* Variables to store msb value */
    uint8_t msb;

    /* Variables to store lsb value */
    uint8_t lsb;

    /* Variables to store both msb and lsb value */
    uint16_t msb_lsb;

    /* Variables to define index */
    uint8_t index = 0;

    /* Read x-axis data */
    lsb = reg_data[index++];
    msb = reg_data[index++];
    msb_lsb = ((uint16_t) msb << 8) | (uint16_t) lsb;
    data->x = (int16_t) msb_lsb;

    /* Read y-axis data */
    lsb = reg_data[index++];
    msb = reg_data[index++];
    msb_lsb = ((uint16_t) msb << 8) | (uint16_t) lsb;
    data->y = (int16_t) msb_lsb;

    /* Read z-axis data */
    lsb = reg_data[index++];
    msb = reg_data[index++];
    msb_lsb = ((uint16_t) msb << 8) | (uint16_t) lsb;
    data->z = (int16_t) msb_lsb;
}


/*!
 * @brief This API reads the data from the given register address of bmi2
 * sensor.
 *
 * @note For most of the registers auto address increment applies, with the
 * exception of a few special registers, which trap the address. For e.g.,
 * Register address - 0x26, 0x5E.
 */
int8_t bmi2_get_regs(uint8_t reg_addr, uint8_t *data, uint16_t len, const struct bmi2_dev *dev)
{
    /* Variable to define error */
    int8_t rslt;

    /* Variable to define temporary length */
    uint16_t temp_len = len + dev->dummy_byte;

    /* Variable to define temporary buffer */
    uint8_t temp_buf[temp_len];

    /* Variable to define loop */
    uint16_t index = 0;

    /* Null-pointer check */
    rslt = null_ptr_check(dev);
    if ((rslt == BMI2_OK) && (data != NULL))
    {
        /* Configuring reg_addr for SPI Interface */
        if (dev->intf == BMI2_SPI_INTERFACE)
        {
            reg_addr = (reg_addr | BMI2_SPI_RD_MASK);
        }
        if (dev->aps_status == BMI2_ENABLE)
        {
            rslt = dev->read(dev->dev_id, reg_addr, temp_buf, temp_len);
            dev->delay_us(450);
        }
        else
        {
            rslt = dev->read(dev->dev_id, reg_addr, temp_buf, temp_len);
            dev->delay_us(20);
        }

        if (rslt == BMI2_OK)
        {
            /* Read the data from the position next to dummy byte */
            while (index < len)
            {
                data[index] = temp_buf[index + dev->dummy_byte];
                index++;
            }
        }
        else
        {
            rslt = BMI2_E_COM_FAIL;
        }
    }
    else
    {
        rslt = BMI2_E_NULL_PTR;
    }

    return rslt;
}

As you can see in the last function definition, bmi2_get_regs() is the function at the root of this API call which actually performs the Register Read operation to extract the IMU data (3-axes of data for both the accelerometer + gyroscope). You can also see in this function the BMI2_E_COM_FAIL which occurs as a result of a non-zero return value from performing dev->read(), which is defined in the imuInit() function, which I am attaching here for your reference:

void imuInit(imuInstance_t instance)
{
    int8_t rslt;
    struct bmi2_dev * p_dev;

    if (instance == IMU_PRIMARY)
    {
        p_dev = &m_imuPrimary;
        p_dev->dev_id = BMI2_I2C_PRIM_ADDR;
    }
    else if (instance == IMU_SECONDARY)
    {
        p_dev = &m_imuSecondary;
        p_dev->dev_id = BMI2_I2C_SEC_ADDR;
    }

    p_dev->read = imuRegRead;
    p_dev->write = imuRegWrite;
    p_dev->delay_us = delay_us;
    p_dev->read_write_len = 128;
    p_dev->intf = BMI2_I2C_INTERFACE;
    p_dev->config_file_ptr = NULL;

    uint8_t buff[1] = {0};
    if (instance == IMU_PRIMARY)
    {
        imuRegRead(BMI270_PRIM_ADDR, BMI270_ID_ADDR, &buff[0], 1);
        NRF_LOG_INFO("BMI270 Primary Chip ID: 0x%x", buff[0]);
    }
    else if (instance == IMU_SECONDARY)
    {
        imuRegRead(BMI270_SEC_ADDR, BMI270_ID_ADDR, &buff[0], 1);
        NRF_LOG_INFO("BMI270 Secondary Chip ID: 0x%x", buff[0]);
    }

    rslt = bmi270_init(p_dev);
    imuPrintRslt(rslt);

    imuEnable(instance);
}


static int8_t imuRegRead(uint8_t dev_addr, uint8_t reg_addr, uint8_t * data, uint16_t len)
{
    uint32_t ret = twiRegRead(dev_addr, reg_addr, data, len);
    if (ret != NRF_SUCCESS)
    {
        return BMI2_E_COM_FAIL;
    }

    return BMI2_OK;
}


uint32_t twiRegRead(uint8_t slaveAddr, uint8_t regAddr, uint8_t * pdata, uint16_t length)
{
    uint32_t ret;

    nrf_drv_twi_xfer_desc_t xfer = NRF_DRV_TWI_XFER_DESC_TXRX(slaveAddr, &regAddr, 1, pdata, length);
    m_xfer_status = 0;

    ret = nrf_drv_twi_xfer(&m_twi, &xfer, 0);


    if (NRF_SUCCESS != ret)
    {
        return ret;
    }

    // Wait for response for 5 ms
    for(uint32_t k = 0; k <= 5000; k++)
    {
        if(m_xfer_status != 0)
        {
            nrf_delay_ms(1);
            break;
        }
        nrf_delay_us(1);
    }

    if(m_xfer_status == TWI_XFER_STATUS_SUCCESS)
    {
        return NRF_SUCCESS;
    }
    else
    {
        return NRF_ERROR_INTERNAL;
    }
}

To be sure, this communication error is clearly happening across more than just this IMU sensor. I see this all-zero glitch occur in our Barometer data stream as well, and sometimes with our capacitive touch sensor (although less common for all cap touch values to go to zero for some reason). I am only going a bit deeper on this IMU case because it has a nice API to reference here in this discussion.

[Edvin]

Ok, so what I can see from this is that rslt = BMI2_OK, meaning that both "dev" and "data" are non-null pointers.

Then you read the same register, but the delay depends on dev->aps_status. What is dev->aps_status in the case where it fails?

And regardless of what dev->read() returns, as long as it is not BMI2_OK, you will return BMI_E_COM_FAIL. Did you check what dev->read() returns in the failing case? That is, BMI2_E_COM_FAIL will be returned by the imuRegRead as long as twiRegRead() doesn't return NRF_SUCCESS. But what does twiRegRead return when it doesn't return NRF_SUCCESS?