Board reboots when a function is executed in TF-M

RE: Board reboots when a function is executed in TF-M

dejans — Thu, 01 Dec 2022 10:09:06 GMT

Hi Pablo,

Your issue might be related to some kind of a bug in your project. I would suggest that you start with simple sample and start developing from there one step at the time. Make sure to test along the way and do not introduce many simultaneous changes into your project during the development.

Best regards,
Dejan

RE: Board reboots when a function is executed in TF-M

dejans — Thu, 24 Nov 2022 12:13:10 GMT

Hi Pablo,

I have made an internal inquiry related to your issue. I'll come back to you when I get new information, hopefully till the end of next week.

Best regards,
Dejan

RE: Board reboots when a function is executed in TF-M

Pablosanserr — Tue, 22 Nov 2022 12:21:20 GMT

Hi Dejan,

Thank you for the clarification. However, my problem is not that alloca fails with a large number. As I commented in my previous post, when I increase stack_size the way you told me it doesn't seem to have had an effect.

Using alloca with 4500 bytes and with a stack size of 0x800 works correctly. However, duplicating the number of bytes for alloca (9000 bytes) and greatly increasing the stack_size to 0x8000 does not work.

Best regards,
Pablo

RE: Board reboots when a function is executed in TF-M

dejans — Tue, 22 Nov 2022 11:59:23 GMT

Hi Pablo,

I have tested your sample project and managed to reproduce your issue.
In your code, inside CheckMemory() function, you should be careful how you use array and alloca() function.
The size of the array could potentially be as you have set it. However, there is no guarantee that this number of bytes would be available at run-time. As a result, you may not have enough stack space. Allocation on the stack with function alloca() may produce stack overflow with very large values. Alloca() man page specifies "The alloca() function returns a pointer to the beginning of the allocated space. If the allocation causes stack overflow, program behavior is undefined." Therefore, you may consider using alloca() with fewer bytes.
I have also modified and tested your sample with char c[10] and void *c=alloca(10) and everything was fine. I would suggest that you use less bytes in your array declaration as well as in the alloca() function.

Best regards,
Dejan

RE: Board reboots when a function is executed in TF-M

Pablosanserr — Sun, 13 Nov 2022 10:07:59 GMT

Hi Dejan,

Here you have an updated version of the sample project, with warnings removed. There are only 2 warnings, related with unused variables. These variables are used for testing VLA and alloca. There should be no problem with these warnings.

devzone.nordicsemi.com/.../6874.tfm_5F00_secure_5F00_partition_5F00_stacksize.zip

I think you need to enable the "Enable debug options" option for the build. This should allow you to see the TF-M log that I have shown in my last message.

To reproduce the error, you must modify the size given as a parameter in the CheckMemory function, which is used in dummy_partition/dummy_partition.c. With a value of 9000 there is an error, but with a value of 4500 there is no error.

printf("Let's check memory using malloc, VLA and alloca\r\n");
for(int i = 0; i <= 2; i++){
	CheckMemory(i, 9000);
}

You can also check, as I said in the previous message, that modifying the stack size in dummy_partition/tfm_dummy_partition.yaml has no effect.

"stack_size": "0x8000"

Best regards,
Pablo

RE: Board reboots when a function is executed in TF-M

dejans — Wed, 09 Nov 2022 16:16:58 GMT

Hi Pablo,

There are many warnings in your provided sample.

The results of testing your current sample tfm_secure_partition_stacksize are shown below.

Application log

TF-M log

Best regards,
Dejan

RE: Board reboots when a function is executed in TF-M

Pablosanserr — Wed, 09 Nov 2022 10:03:28 GMT

Hi Dejan,

Increasing stack memory didn't work for me, so I used a sample project for you to check it. I don't know what I am doing wrong. I'm using nRF SDK v2.0.0 and nRF5340DK.

devzone.nordicsemi.com/.../tfm_5F00_secure_5F00_partition_5F00_stacksize.zip

CheckMemory is a function that allocate memory using malloc, VLA or alloca. This function is executed from secure partition.

static void CheckMemory(int location, int bytes){
	if(location == 0){
		// HEAP
		printf("Allocating %d bytes using malloc (HEAP)...\r\n", bytes);
		void * m = malloc(bytes);
		printf(" -> HEAP: OK\r\n");
		free(m);
	}else if(location == 1){
		printf("Allocating %d bytes using VLA (STACK)...\r\n", bytes);
		// STACK 1
		char c[bytes];
		printf(" -> STACK 1: OK\r\n");
	}else if(location == 2){
		printf("Allocating %d bytes using alloca (STACK)...\r\n", bytes);
		// STACK 2
		void * c = alloca(bytes);
		printf(" -> STACK 2: OK\r\n");
	}
}

Using CheckMemorySIze with bytes = 4500:

Using CheckMemorySize with bytes = 9000:

As you can see, VLA success if bytes = 4500, but it fails if bytes = 9000. Alloca fails too. VLA and alloca uses stack, so increasing stack size should be enough. Following your recommendation, I increased stacksize from 0x800 to 0x8000 (tfm_dummy_partition.yaml), but it didn't worked for me. It fails again:

P.S: By default, stack size is 0x800 (2048 bytes), but I can use VLA and alloca with 4500. Does it make sense?

Best regards,
Pablo

RE: Board reboots when a function is executed in TF-M

dejans — Mon, 31 Oct 2022 10:12:53 GMT

Hi Pablo,

There is this line in the tfm_secure_peripheral_partition.yml which specifies the size of the stack.

Best regards,
Dejan

RE: Board reboots when a function is executed in TF-M

Pablosanserr — Sun, 30 Oct 2022 10:27:34 GMT

Hi Dejans,

As you can see in the following TF-M log, I also get an UsageFault error. The value of UFSR is 0x10.

I have looked at the meaning of the UFSR fields in the ARM documentation. The hexadecimal value 0x10 is equivalent to bit 2 with value 1. According to the ARM documentation, this is an error with name STKOF: "Stack overflow flag. Sticky flag indicating whether a stack overflow error has occurred".

As I guessed, it seems that there is not enough size in the safe partition stack to run this function. The following issue from the blst repo reports a failure with the same function. In the end, it concludes that it was a stack size problem. It worked for him by allocating more than 20 kB. Do I have a way to see the available stack size or increase it, for the execution of the functions in the secure partition?

Best regards,
Pablo

RE: Board reboots when a function is executed in TF-M

dejans — Fri, 28 Oct 2022 14:09:11 GMT

Hi Pablo,

From my TF-M log, UsageFault Status Register (UFSR) has the value 0x10. You can check various fields of UFSR in the ARM documentation.

Best regards,
Dejan

RE: Board reboots when a function is executed in TF-M

dejans — Mon, 24 Oct 2022 08:46:17 GMT

Hi Pablo,

I have reproduced your board resetting issue. Using the same approach as have been suggested to you in your other case, I have got FATAL ERROR: UsageFault. It seems that the problem occurs when jumping to non-secure code with jump_to_ns_code(). I have reported the issue internally.

Best regards,
Dejan

RE: Board reboots when a function is executed in TF-M

Pablosanserr — Tue, 11 Oct 2022 10:49:17 GMT

Hi Dejans,

I have updated the repo. Can you clone the repo again, do the build and launch the application please? I just tried cloning it and I can reproduce the problem, as you can see below (it restarts the application).

Regards,
Pablo

RE: Board reboots when a function is executed in TF-M

dejans — Tue, 11 Oct 2022 10:24:26 GMT

Hi Pablo,

I have tested again with the same setup using your main branch (with 2 lines that you mentioned uncommented) and got the same result as before. I could not observe board resets.

Best regards,
Dejan

RE: Board reboots when a function is executed in TF-M

Pablosanserr — Mon, 10 Oct 2022 20:13:43 GMT

Hi again, Dejans,

As you can see in the repo, I have put back the function that causes the reboot, so you can reproduce the error again.

Also, I've added a function to give me an idea of how much space is available in memory. It is not the most accurate method to obtain the available memory, but it is useful to get an estimate.

static uint32_t GetFreeMemorySize()
{
  uint32_t  i;
  uint32_t  len;
  uint8_t*  ptr;
 
  for(i=1;;i++)
  {
    len = i * 1024;
    ptr = (uint8_t*)malloc(len);
    if(!ptr){
      break;
    }
    free(ptr);
  }
 
  return i;
}

I execute this function just before the signature, the function that causes the reboot. This is what I get at the output, doing a debug with the "Enable debug options" option enabled:

I thought it would be a good idea to compare it with the result of the original project, bls-hsm, which uses SPM for secure-partition and NCS 1.8.0. This is what I get at the output, doing a debug with the "Enable debug options" option enabled:

The difference in available space is significant: it is approximately 5.6 times larger for the original version, using SPM (NCS 1.8.0), compared to the new version, using TF-M (NCS 2.0.0).

I don't know if the error might be related to that, but I thought this might be of interest.

Best regards,
Pablo

RE: Board reboots when a function is executed in TF-M

Pablosanserr — Mon, 10 Oct 2022 16:07:47 GMT

Hi Dejans,

It is normal that you could not reproduce the error. In one of the last commits, by mistake, I commented the program line that caused the board to reboot.

In order to reproduce the error, lines 53 and 54 (shown below), must not be commented. This allows you to reproduce the error, which you were able to test at the time.

char msg[] = "5656565656565656565656565656565656565656565656565656565656565656";
signature(pk, msg, buffer);

This change must be made in the mycli/src/main.c file. You can try to remove the comment to those two lines and launch the application. As soon as I can, I will put it back in the repository, so you can reproduce the error directly.

I also use my computer with Windows 10. The use of the virtual machine was for the purpose of obtaining the TF-M logs.

Best regards,
Pablo

RE: Board reboots when a function is executed in TF-M

dejans — Mon, 10 Oct 2022 11:50:11 GMT

Hi Pablo,

Thank you for the logs. I have tested your sample (from main) using NCS v2.0.0, nRF5340-DK and Windows 10, but I could not reproduce your issue. I didn't get any board resets. You could try using system without virtual machine.

Best regards,
Dejan

RE: Board reboots when a function is executed in TF-M

Pablosanserr — Thu, 06 Oct 2022 08:11:20 GMT

Hi Dejans,

I have connected pins P0.25 and P0.26 to a USB-to-serial converter and managed to get the TF-M log. Here are the logs obtained from TF-M and the application output:

Without "enable debug options"

Application output

TF-M log

With "enable debug options"

Application output

TF-M log

It seems to be no difference whether the "enable debug configuration" option is enabled or not. I would like to remind you that the board restarts automatically when the error occurs. This is the reason why you see the same thing 3 times in each log image.

I have not been able to observe any log of the application, from the non-secure partition. My understanding was that I should see it in VCOM1 in VS Code. I have also tried using PuTTY, and got nothing. I also tried with VCOM0, but nothing happened. VCOM2 is the application output.

I thought that, maybe, there was nothing to show this log. Therefore, I tried adding the following configuration:

CONFIG_LOG_PRINTK=y

This should dump the printk messages from my code to the log. However, I still couldn't manage to see those messages in the log.

Note: I get the TF-M log output from a linux virtual machine, as I was having problems with my USB-to-serial converter drivers. So I tried to look at the application's log from there, but I didn't get any result either.

I will keep trying to get the application log. If I succeed, I will post it on the forum. Meanwhile, this is all I have.

Regards,
Pablo

RE: Board reboots when a function is executed in TF-M

dejans — Mon, 03 Oct 2022 11:52:03 GMT

Hi Pablo,

The first log is ordinary UART application log. If UART logging is enabled in your application, you could use any terminal emulation program to see the log as shown in testing and debugging application. To see application log, you would need to connect to COM port corresponding to the application (VCOM1 in VS Code).

As mentioned before, documentation explains TF-M logging. I'll provide more details here. If you want to see TF-M log, you would need to take log from UART1. If you have 2 nRF5340-DKs (v2.0.0), you could connect them together to get required logs. First, flash the project to the first DK (DK-A) and recover the second board (DK-B). Connect pins P0.25 and P0.26 on DK-A to pins P1.00 (RXD) and P1.01 (TXD) on DK-B. Using terminal emulation program, connect to the COM ports on both devices. One of the COM ports (VCOM1 in VS Code) on the DK-A will show the application log and one of the COM ports (VCOM0 in VS Code) on DK-B will show the TF-M log.

In case that you have only one nRF5340-DK board, you could connect USB to serial converter cable directly to the DK-A using pins P0.25 and P0.26. You would then have 2 COM ports and 1 USB port open. You could then connect to the second VCOM port (VCOM1) shown in VS Code to see application log and to the COM port which corresponds to UART Serial port to see TF-M log.

Best regards,
Dejan

RE: Board reboots when a function is executed in TF-M

Pablosanserr — Sun, 02 Oct 2022 10:43:17 GMT

Hi,

I have removed the warnings that appeared when making the build. The problem has not been solved.

I have made a build with and without the "Enable debug option" and another one with it enabled. In order to enable the "Enable debug options" option, I had to increase the TF-M size. It is 0x50000 without enabling the option and 0x60000 by enabling the option. This can be checked in the prj.conf file.

Again, I show what you get when you run the following command

west build -t partition_manager_report

Build without debug options enabled:

Build with debug options enabled:

It is not clear to me which logs you need, so I would be grateful if you could specify which ones you need if they are not the ones I have given you.

Regards,
Pablo

RE: Board reboots when a function is executed in TF-M

dejans — Thu, 29 Sep 2022 08:37:01 GMT

Hi,

I have tested your sample from non_secure branch and it was successful. I also tested your sample from the main branch and it still gives many warnings during the build process. Could you please remove the warnings from main branch?

TF-M uses 2 UART interfaces for logging, one for secure part (TF-M and MCUboot) and one for the application. Documentation provides more information about logging. With regard to secure logging, you can refer to the TF-M logging section.
Could you send both application and TF-M output logs with and without "Enable debug options"?

Best regards,
Dejan

RE: Board reboots when a function is executed in TF-M

Pablosanserr — Wed, 28 Sep 2022 17:39:31 GMT

Hi,

I build for nrf5340dk_nrf5340_cpuapp_ns. My build configuration doesn't include 'Enable debug options'. Should I enable that?

Regards,

Pablo

RE: Board reboots when a function is executed in TF-M

dejans — Wed, 28 Sep 2022 17:05:34 GMT

Hi,

For which board did you build your sample?

Did your build configuration include "Enable debug options"?

Best regards,
Dejan

RE: Board reboots when a function is executed in TF-M

Pablosanserr — Tue, 27 Sep 2022 11:45:32 GMT

Hi,

I have a couple of updates about the project.

First of all, I have fixed some warnings that appear when doing the build. As I expected, this didn't solve the problem, but it's good to get rid of them.

Secondly, I have added to the repository a branch called "non_secure", in which I have imported the static library and run the functions in the non-secure partition. The function that gives me problems in tf-m works correctly in non secure partition. To make it work, I have added the following configuration to prj.conf, to allocate spare memory to the main stack:

CONFIG_MAIN_STACK_SIZE=104080

Although I am not using the tf-m functions in this branch, I have not removed it from the configuration.

When building, the following memory usage data is displayed:

In case it is useful, this is the log obtained when executing the command:

west build -t partition_manager_report

Regards,

Pablo

RE: Board reboots when a function is executed in TF-M

Pablosanserr — Mon, 26 Sep 2022 11:25:51 GMT

I narrowed down the problem by doing a step-by-step debug. I found that it is in that line of code that the board reboot occurs.

As for the warnings I get when doing the build, they don't seem to be related to the problem. I will still try to fix them

Executing the following command inside bls-hsm-2/mycli, i get this log:

west build -t partition_manager_report

The reboot of the board made me think that it might be a RAM problem. To do so, I set the value of the RAM assigned to the safe partition to an extremely high value, by writing the following in prj.conf:

CONFIG_PM_PARTITION_SIZE_TFM_SRAM=0x63000

This did not solve the problem. Running the same command, having added this configuration inside prj.conf to the build gives the following log:

Are them the logs you need?

Regards,

Pablo

RE: Board reboots when a function is executed in TF-M

dejans — Fri, 23 Sep 2022 15:05:04 GMT

Hi,

I have looked into your project. I noticed that there are many warnings during build process. I would suggest you to go through these warnings and try to fix them. There seems to be a problem with your project that would need to be investigated further.

Could you provide more information on how you narrowed down the problem to the blst_sign_pk_in_g1 function from lib/bls_hsm.h?

It would be beneficial if you could provide some logs. Could you show application log and secure partition log?

Have you seen any hard fault errors?

Bes regards,
Dejan