Suspending a thread also suspends another randomly

RE: Suspending a thread also suspends another randomly

Hossein Kasaei — Tue, 18 Feb 2025 13:59:52 GMT

I couldn't find the main issue. I decided not to suspend these threads, alternatively I used a flag and delay for them to do nothing when sending data is not requested.

RE: Suspending a thread also suspends another randomly

Susheel Nuguru — Wed, 03 Jan 2024 12:47:01 GMT

Happy New Year Hossein,

[quote user="Hossein Kasaei"]By the way I think the problem might be the fact that I'm suspending thread0 from the outside of itself.[/quote]

I do not think that this alone can be the issue here. It is allowed and legal to call k_thread_suspend from outside the thread and that is also the reason why it takes the thread_id as the argument.

[quote user="Hossein Kasaei"]maybe suspending it when it was in the middle of these operations is not a good practice, and maybe that's why this issue happens randomly[/quote]

This should not matter as well in the RTOS concept in itself. You should be able to suspend and resume thread when ever you see fit.

RE: Suspending a thread also suspends another randomly

Hossein Kasaei — Tue, 02 Jan 2024 12:22:38 GMT

Hi susheel,

Happy new year!

After a long time, I tried to figure out this issue. I used thread analyzer as you suggested but couldn't find it useful for this issue, or maybe I used it in a wrong way. I enabled it in the configuration file then placed the thread_analyzer_print() in the thread_main(), so it would log every second. I couldn't find anything weird in it maybe it's better to take a look at it just before the issue happens and the main loop log will end:

[00:00:46.198,730] --- 3 messages dropped --- [00:00:46.198,852] <inf> thread_analyzer:  0x20002370 [00:00:46.198,852] <inf> thread_analyzer: [00:00:46.199,096] <inf> thread_analyzer:  0x20002478 [00:00:46.199,096] <inf> thread_analyzer: [00:00:46.199,401] <inf> thread_analyzer:  0x20001c60 [00:00:46.199,432] <inf> thread_analyzer: [00:00:46.199,768] <inf> thread_analyzer:  0x20001b78 [00:00:46.199,768] <inf> thread_analyzer: [00:00:46.200,164] <inf> thread_analyzer:  0x20002b10 [00:00:46.200,164] <inf> thread_analyzer: [00:00:46.200,347] <inf> thread_analyzer:  0x200027f0 [00:00:46.200,347] <inf> thread_analyzer: [00:00:46.200,531] <inf> thread_analyzer:  0x20002268 [00:00:46.200,531] <inf> thread_analyzer: [00:00:46.200,622] <inf> thread_analyzer:  0x20002180 [00:00:46.200,622] <inf> thread_analyzer: [00:00:46.200,744] <inf> thread_analyzer:  0x20002940 [00:00:46.200,775] <inf> thread_analyzer: [00:00:46.201,019] <inf> thread_analyzer:  ISR0 [00:00:46.201,049] <dbg> main_c_log: thread_main:

[00:00:47.202,606] <inf> thread_analyzer: --- 2 messages dropped --- [00:00:47.202,728] <inf> thread_analyzer:  0x20002370 [00:00:47.202,728] <inf> thread_analyzer: [00:00:47.203,002] <inf> thread_analyzer:  0x20002478 [00:00:47.203,002] <inf> thread_analyzer: [00:00:47.203,277] <inf> thread_analyzer:  0x20001c60 [00:00:47.203,308] <inf> thread_analyzer: [00:00:47.203,613] <inf> thread_analyzer:  0x20001b78 [00:00:47.203,643] <inf> thread_analyzer: [00:00:47.203,979] <inf> thread_analyzer:  0x20002b10 [00:00:47.204,010] <inf> thread_analyzer: [00:00:47.204,193] <inf> thread_analyzer:  0x200027f0 [00:00:47.204,193] <inf> thread_analyzer: [00:00:47.204,345] <inf> thread_analyzer:  0x20002268 [00:00:47.204,376] <inf> thread_analyzer: [00:00:47.204,437] <inf> thread_analyzer:  0x20002180 [00:00:47.204,467] <inf> thread_analyzer: [00:00:47.204,589] <inf> thread_analyzer:  0x20002940 [00:00:47.204,589] <inf> thread_analyzer: [00:00:47.204,864] <inf> thread_analyzer:  ISR0 [00:00:47.204,864] <dbg> main_c_log: thread_main:

[00:00:48.206,420] <inf> thread_analyzer:  0x20002370 --- 3 messages dropped --- [00:00:48.206,420] <inf> thread_analyzer: [00:00:48.206,665] <inf> thread_analyzer:  0x20002478 [00:00:48.206,665] <inf> thread_analyzer: [00:00:48.206,970] <inf> thread_analyzer:  0x20001c60 [00:00:48.206,970] <inf> thread_analyzer: [00:00:48.207,275] <inf> thread_analyzer:  0x20001b78 [00:00:48.207,305] <inf> thread_analyzer: [00:00:48.207,672] <inf> thread_analyzer:  0x20002b10 [00:00:48.207,672] <inf> thread_analyzer: [00:00:48.207,855] <inf> thread_analyzer:  0x200027f0 [00:00:48.207,855] <inf> thread_analyzer: [00:00:48.208,007] <inf> thread_analyzer:  0x20002268 [00:00:48.208,038] <inf> thread_analyzer: [00:00:48.208,129] <inf> thread_analyzer:  0x20002180 [00:00:48.208,129] <inf> thread_analyzer: [00:00:48.208,251] <inf> thread_analyzer:  0x20002940 [00:00:48.208,251] <inf> thread_analyzer: [00:00:48.208,526] <inf> thread_analyzer:  ISR0 [00:00:48.208,557] <dbg> main_c_log: thread_main:

[00:00:49.210,083] <inf> thread_analyzer:  0x20002370 --- 3 messages dropped --- [00:00:49.210,113] <inf> thread_analyzer: [00:00:49.210,327] <inf> thread_analyzer:  0x20002478 [00:00:49.210,357] <inf> thread_analyzer: [00:00:49.210,632] <inf> thread_analyzer:  0x20001c60 [00:00:49.210,662] <inf> thread_analyzer: [00:00:49.210,968] <inf> thread_analyzer:  0x20001b78 [00:00:49.210,998] <inf> thread_analyzer: [00:00:49.211,334] <inf> thread_analyzer:  0x20002b10 [00:00:49.211,364] <inf> thread_analyzer: [00:00:49.211,517] <inf> thread_analyzer:  0x200027f0 [00:00:49.211,547] <inf> thread_analyzer: [00:00:49.211,700] <inf> thread_analyzer:  0x20002268 [00:00:49.211,700] <inf> thread_analyzer: [00:00:49.211,791] <inf> thread_analyzer:  0x20002180 [00:00:49.211,822] <inf> thread_analyzer: [00:00:49.211,914] <inf> thread_analyzer:  0x20002940 [00:00:49.211,944] <inf> thread_analyzer: [00:00:49.212,219] <inf> thread_analyzer:  ISR0 [00:00:49.212,219] <dbg> main_c_log: thread_main:

[00:00:50.052,642] <dbg> ble_service:

[00:00:50.212,310] <dbg> main_c_log: thread_main:

[00:00:50.213,470] <dbg> main_c_log: thread_main:

[00:00:55.102,752] <dbg> ble_service:

[00:01:00.052,886] <dbg> ble_service:

[00:01:05.053,009] <dbg> ble_service:

[00:01:10.103,118] <dbg> ble_service:

[00:01:15.153,625] <dbg> ble_service:

[00:01:20.153,350] <dbg> ble_service:

[00:01:25.103,485] <dbg> ble_service:

[00:01:30.103,607] <dbg> ble_service:

[00:01:35.153,717] <dbg> ble_service:

[00:01:40.153,839] <dbg> ble_service:

<inf> thread_analyzer:       : Total CPU cycles used: 1 : STACK: unused 272 usage 688 / 960 (71 %); CPU: 0 % : Total CPU cycles used: 90 : STACK: unused 1000 usage 536 / 1536 (34 %); CPU: 0 % : Total CPU cycles used: 65 : STACK: unused 1428 usage 556 / 1984 (28 %); CPU: 0 % : Total CPU cycles used: 3898 : STACK: unused 1600 usage 384 / 1984 (19 %); CPU: 0 % : Total CPU cycles used: 3854 : STACK: unused 1720 usage 264 / 1984 (13 %); CPU: 0 % : Total CPU cycles used: 13 : STACK: unused 616 usage 344 / 960 (35 %); CPU: 0 % : Total CPU cycles used: 250 : STACK: unused 504 usage 736 / 1240 (59 %); CPU: 0 % : Total CPU cycles used: 92 : STACK: unused 48 usage 656 / 704 (93 %); CPU: 17 % : Total CPU cycles used: 270147 : STACK: unused 228 usage 92 / 320 (28 %); CPU: 81 % : Total CPU cycles used: 1231154 : STACK: unused 1528 usage 584 / 2112 (27 %) main loop running... : Total CPU cycles used: 1 : STACK: unused 272 usage 688 / 960 (71 %); CPU: 0 % : Total CPU cycles used: 90 : STACK: unused 1000 usage 536 / 1536 (34 %); CPU: 0 % : Total CPU cycles used: 65 : STACK: unused 1336 usage 648 / 1984 (32 %); CPU: 0 % : Total CPU cycles used: 3991 : STACK: unused 1520 usage 464 / 1984 (23 %); CPU: 0 % : Total CPU cycles used: 4370 : STACK: unused 1720 usage 264 / 1984 (13 %); CPU: 0 % : Total CPU cycles used: 13 : STACK: unused 616 usage 344 / 960 (35 %); CPU: 0 % : Total CPU cycles used: 250 : STACK: unused 504 usage 736 / 1240 (59 %); CPU: 0 % : Total CPU cycles used: 92 : STACK: unused 48 usage 656 / 704 (93 %); CPU: 17 % : Total CPU cycles used: 276681 : STACK: unused 228 usage 92 / 320 (28 %); CPU: 81 % : Total CPU cycles used: 1256429 : STACK: unused 1528 usage 584 / 2112 (27 %) main loop running... : STACK: unused 272 usage 688 / 960 (71 %); CPU: 0 % : Total CPU cycles used: 90 : STACK: unused 1000 usage 536 / 1536 (34 %); CPU: 0 % : Total CPU cycles used: 65 : STACK: unused 1336 usage 648 / 1984 (32 %); CPU: 0 % : Total CPU cycles used: 4077 : STACK: unused 1520 usage 464 / 1984 (23 %); CPU: 0 % : Total CPU cycles used: 4891 : STACK: unused 1720 usage 264 / 1984 (13 %); CPU: 0 % : Total CPU cycles used: 13 : STACK: unused 616 usage 344 / 960 (35 %); CPU: 0 % : Total CPU cycles used: 250 : STACK: unused 504 usage 736 / 1240 (59 %); CPU: 0 % : Total CPU cycles used: 92 : STACK: unused 48 usage 656 / 704 (93 %); CPU: 17 % : Total CPU cycles used: 283216 : STACK: unused 228 usage 92 / 320 (28 %); CPU: 81 % : Total CPU cycles used: 1281732 : STACK: unused 1528 usage 584 / 2112 (27 %) main loop running... : STACK: unused 272 usage 688 / 960 (71 %); CPU: 0 % : Total CPU cycles used: 90 : STACK: unused 1000 usage 536 / 1536 (34 %); CPU: 0 % : Total CPU cycles used: 65 : STACK: unused 1336 usage 648 / 1984 (32 %); CPU: 0 % : Total CPU cycles used: 4163 : STACK: unused 1520 usage 464 / 1984 (23 %); CPU: 0 % : Total CPU cycles used: 5414 : STACK: unused 1720 usage 264 / 1984 (13 %); CPU: 0 % : Total CPU cycles used: 13 : STACK: unused 616 usage 344 / 960 (35 %); CPU: 0 % : Total CPU cycles used: 250 : STACK: unused 504 usage 736 / 1240 (59 %); CPU: 0 % : Total CPU cycles used: 92 : STACK: unused 48 usage 656 / 704 (93 %); CPU: 18 % : Total CPU cycles used: 289502 : STACK: unused 228 usage 92 / 320 (28 %); CPU: 81 % : Total CPU cycles used: 1307285 : STACK: unused 1528 usage 584 / 2112 (27 %) main loop running... comm_write_cb: received byte : 0 cancel any calculation suspending thread0 comm_write_cb: received byte : 2 comm_write_cb: received byte : 0 comm_write_cb: received byte : 1 comm_write_cb: received byte : 0 comm_write_cb: received byte : 2 comm_write_cb: received byte : 0 comm_write_cb: received byte : 1 comm_write_cb: received byte : 0 comm_write_cb: received byte : 2 comm_write_cb: received byte : 0

By the way I think the problem might be the fact that I'm suspending thread0 from the outside of itself. In this thread BLE sending and data fetching from sensors happens and maybe suspending it when it was in the middle of these operations is not a good practice, and maybe that's why this issue happens randomly. So I decided to suspend thread0 in itself and the problem seems to be solved! I'm never facing this issue again. What is your opinion about this? cause I'm a beginner in RTOS and the operation system concept.

RE: Suspending a thread also suspends another randomly

Susheel Nuguru — Mon, 04 Dec 2023 08:12:28 GMT

I think we might need to rule out if this is related to the stack overflow or not. The reason I am still sticking to this debug path is because i2c_burst_read_dt does not seem to be related to the threads suspending and unrelated random issue most likely happen with memory corruptions.

Please run the Thread Analyzer on your app to see if there are any other threads that are running dangerously close to using up allocated stack for them.

Otherwise, I do not not know or have any theory on how this can happen.

RE: Suspending a thread also suspends another randomly

Hossein Kasaei — Fri, 01 Dec 2023 13:38:30 GMT

[quote user="Aryan"]What is the context in which i2c_burst_read_dt is called? thread0 or some interrupt context?[/quote]

Yes it is running in the thread0 context.

[quote user="Aryan"]try to experiment with the stack sizes of your application created threads[/quote]

I doubled the thread0 stack size from 2048 to 4096, didn't fix it.
Also doubled the thread_main stack size to 4096, still nothing.
I used CONFIG_MAIN_STACK_SIZE = 4096, didn't fix the problem.

Something I noticed is that when i2c read take some time, like using i2c_burst_read_dt or using i2c_reg_read_dt in a for loop, then suspending will occur sooner, but when I read just one byte using i2c_reg_read_dt it take some time to face the problem. I have to say that I'm not getting any error log.

RE: Suspending a thread also suspends another randomly

Susheel Nuguru — Fri, 01 Dec 2023 11:33:07 GMT

This might be caused by buffer overflow or stack overflow.

What is the context in which i2c_burst_read_dt is called? thread0 or some interrupt context? I would try to experiment with the stack sizes of your application created threads just to see if this is some overflow issue.

No, Unfortunately I do not have that sensor, so I won't be able to reproduce the issue.

RE: Suspending a thread also suspends another randomly

Hossein Kasaei — Fri, 01 Dec 2023 10:26:49 GMT

Hi,

I tried to shrank down the bug and I ended up to the max86150 library. It's so weird but the i2c_burst_read_dt is causing the problem I don't know how it is related to the threads do you have any idea?

To clarify, I myself wrote the library for max86150, I have a function called max86150_fetch_data which I'm calling whenever data is needed for example in the thread0, for ppg calculation this function is called :

if (meas_value_current == 0x01) { // PPG + Force
	max86150_fetch_data(&max86150_dev, &MAX86150_config, &MAX86150_data);

	uint16_t ir_data = (MAX86150_data.channel_data[0]) >> 2;
	// uint16_t ir_data = 0;
	uint16_t red_data = (MAX86150_data.channel_data[1]) >> 2;
	// uint16_t red_data = 0;

	float filtered_force = 0, sum_force = 0;
	float new_force = read_Force();
	force_array[force_idx++] = new_force;
	force_idx %= FORCE_WINDOW_SIZE;

	for (int j=0; j<FORCE_WINDOW_SIZE; j++) {
		sum_force += force_array[j];
	}
	filtered_force = (float)sum_force / FORCE_WINDOW_SIZE;
	

	// filtered_force = new_force;

	epf_buf[epf_count++] = red_data;
	epf_buf[epf_count++] = red_data >> 8;
	epf_buf[epf_count++] = ir_data;
	epf_buf[epf_count++] = ir_data >> 8;
	epf_buf[epf_count++] = 0;
	epf_buf[epf_count++] = 0;
	epf_buf[epf_count++] = (char) filtered_force;
	epf_buf[epf_count++] = (filtered_force - epf_buf[epf_count-1]) * 100;;

	if (epf_count >= EPF_BUF_SIZE) {
		send_notify(epf_buf, sizeof(epf_buf));
		epf_count = 0;
	}
	k_msleep(5);
}

Inside max86150_fetch_data I'm using i2c_burst_read_dt :

int max86150_fetch_data(const struct i2c_dt_spec *dev_i2c, struct max86150_config *config, 
            struct max86150_data *data) {
    
    int num_bytes = 3 * data->num_channels;            
    uint8_t buffer[num_bytes];
    uint32_t fifo_data;
    int fifo_chan;
    int i;

    
    if (i2c_burst_read_dt(dev_i2c, MAX86150_REG_FIFO_DATA, buffer, num_bytes)) {
        LOG_ERR("Data register burst read failed");
        return -1;
    }

    fifo_chan = 0;
    for (i = 0; i < num_bytes; i += 3) {
        fifo_data = (buffer[i] << 16) | (buffer[i+1] << 8) | (buffer[i+2]);
        // if (config->slot[i/3] != 9) fifo_data &= MAX86150_PPG_FIFO_DATA_MASK; // if data is ecg don't mask it

        data->channel_data[fifo_chan++] = fifo_data;
    }

    return 0;
}

when I comment i2c_burst_read_dt, the problem will disappear. Any idea?

If you still want to reproduce, you have to have max86150 module if it is available to you let me know then I can provide you the proper code working for nRF52 DK.

RE: Suspending a thread also suspends another randomly

Susheel Nuguru — Fri, 01 Dec 2023 06:27:12 GMT

Hi Hosssein,

Can you please attach your minimalistic project to reproduce this. I can try to debug how suspending one thread causes the other to suspend. Have not seen such behavior before.

RE: Suspending a thread also suspends another randomly

Hossein Kasaei — Thu, 30 Nov 2023 19:13:09 GMT

[quote user="Hossein Kasaei"]Waking up thread0 is done in a timer callback, now I see maybe that's the problem huh?[/quote]

I investigated this and figured out this is not the problem. I placed a flag in the timer callback instead of calling k_thread_suspend in it, and used this function and check the flag in the main thread and suspend the thread there but the problem still persist.

RE: Suspending a thread also suspends another randomly

Hossein Kasaei — Thu, 30 Nov 2023 14:49:45 GMT

Hi Susheel,

Thanks for your response. I'm sorry I send you the wrong project file I'll send the right one.

[quote user="Aryan"]How do you know that suspending one thread is suspending main thread aswell? I mean how did you conclude that the main was suspended ?[/quote]

I'm checking the log on the RTT console, the printk("main loop running...\n") line is not show up anymore also changing the request code and shutting down the device doesn't work.

Not sure if the main thread is suspended but I know that it won't run anymore maybe it is blocked somehow?

[quote user="Aryan"]How are you waking up thread0? I do not see any related code to resume thread0 after it is suspended[/quote]

Waking up thread0 is done in a timer callback, now I see maybe that's the problem huh?
Actually I turn on this timer in the main thread as you can see below and after 1 second the timer will resume the thread.

void thread0_timer_cb(struct k_timer *timer) {
	printk("thread0 timer cb\n");
	k_thread_resume(thread0_id);
}

as an example when ppg is requested the timer will be turned on so thread0 can be resumed:

else if (meas_value == 0x01) {
	printk("cal PPG\n");
	MAX86150_config.slot[0] = MAX86150_SLOT_IR_LED1;
	MAX86150_config.slot[1] = MAX86150_SLOT_RED_LED2;
	MAX86150_config.slot[2] = 0;
	MAX86150_config.slot[3] = 0;
	MAX86150_config.IR_LED_PA = 0x1E;
	MAX86150_config.RED_LED_PA = 0x1E;
	max86150_init(&max86150_dev, &MAX86150_config, &MAX86150_data);
	// max86150_wakeup(&max86150_dev);
	k_timer_start(&thread0_timer, K_MSEC(1000), K_MSEC(0));
	meas_value_current = 0x01; 
	wake_to_sleep_time = 0;
}

[quote user="Aryan"]Are you sure that thread0_id value is not corrupted over time when the issue happens?[/quote]

It might be the case but I don't know where, maybe in the timer callback?

If you want to reproduce it's not that easy I may have to change the code cause I'm not using the nRF52 DK, we have a custom made board.

RE: Suspending a thread also suspends another randomly

Susheel Nuguru — Thu, 30 Nov 2023 11:57:28 GMT

Hi Hosssein,

The attached code and the code snippet seems to be a bit different. In your attached code I still cannot see the logic of where you are suspending thread0 (even though you showed this in your code snippet in this thread).

There are few clarification that I seek

How do you know that suspending one thread is suspending main thread aswell? I mean how did you conclude that the main was suspended ?
How are you waking up thread0? I do not see any related code to resume thread0 after it is suspended
Are you sure that thread0_id value is not corrupted over time when the issue happens?

It looks like you have stripped away most of the code in the attached zip, but it does not show any more details that can aid me to understand or reproduce this issue.