QEMU with board qemu_x86 crashes in snprintf()- source code missing, hard to debug

RE: QEMU with board qemu_x86 crashes in snprintf()- source code missing, hard to debug

Johan — Mon, 03 Jul 2023 10:51:06 GMT

I already used addr to line. Thats how I knew it was in snprinf the crash happened.

I didn't get any close because I only had the .a-precompiled files.

Not an issue anymore since switching to a 64bit qemu board solves it. See below.

RE: snprintf crash - source code missing, hard to debug

Johan — Sun, 02 Jul 2023 11:47:22 GMT

Switching to the board "qemu_x86_64" fixed the issue. The "qemu_x86" board seems to cause a crash.

RE: snprintf crash - source code missing, hard to debug

Sigurd Hellesvik — Fri, 30 Jun 2023 14:45:52 GMT

Second answer today:

A colleague of me suggested this:

"The standard library might be distributed as a compiled library. Especially for x86_64, and not one of the embedded targets. They might have to build the Zephyr SDK themselves, and figure out what library is used"

Looks like a plausible reason to what you are seeing. I hope it can be helpful

Regards,
Sigurd Hellesvik

RE: snprintf crash - source code missing, hard to debug

Sigurd Hellesvik — Fri, 30 Jun 2023 14:28:46 GMT

For debugging, CONFIG_NO_OPTIMIZATIONS=y is the best suggestion I got, but I see that you already got that.

However,maybe you can use addr2line to figure out where in the code the error is?

[quote user=""]So it's somewhere inside the snprintf it crashes with fault or double fault. I'm not sure where to find the source code and map what c-code line this offset of 0x00105f88 maps to. Can someone give me some helpful advice on the above generally? [/quote]

Have you tried this?

 arm-none-eabi-addr2line -e build/zephyr/zephyr.elf -a <fault address>

Regards,
Sigurd Hellesvik

RE: snprintf crash - source code missing, hard to debug

Johan — Thu, 29 Jun 2023 14:53:16 GMT

No difference with CONFIG_FPU=y

I wonder how I can debug from "source" of zephyr-sdk instead of precompiled .a-files.

How do you suggest I do that with your Connect SDK?

To see where it "dies" in snprintf.

RE: snprintf crash - source code missing, hard to debug

Sigurd Hellesvik — Thu, 29 Jun 2023 14:29:52 GMT

[quote user="Johan Sandgren"]Saw this: not sure if related.

github.com/.../304[/quote]

Could look related.

A comment here suggests CONFIG_FPU=y, which could look related to your error, as it is in soft-float.

[quote user=""]snprintf 0x0000000000105f40 0xae C:/k/np/toolchains/v2.2.0/opt/zephyr-sdk/x86_64-zephyr-elf/x86_64-zephyr-elf/lib/32/soft-float[/quote]

So if we are lucky CONFIG_FPU is all you need?

[quote user="Johan Sandgren"]Perhaps we need to switch to Connect SDK 2.4.0, if that is using the latest Zephyr SDK 0.16.1, possibly.
I see Connect SDK 2.4.0 uses 0.16.0, so quite new.[/quote]

Switching could be worth a try as well.

RE: snprintf crash - source code missing, hard to debug

Johan — Thu, 29 Jun 2023 14:20:07 GMT

Perhaps we need to switch to Connect SDK 2.4.0, if that is using the latest Zephyr SDK 0.16.1, possibly.
I see Connect SDK 2.4.0 uses 0.16.0, so quite new.

RE: snprintf crash - source code missing, hard to debug

Johan — Thu, 29 Jun 2023 14:12:07 GMT

The zephyr-sdk compiled in toolchain 2.2.0 is:

0.15.1

Quite old.

Latest is: Zephyr SDK 0.16.1

RE: snprintf crash - source code missing, hard to debug

Johan — Thu, 29 Jun 2023 14:09:18 GMT

Saw this: not sure if related.

github.com/.../304

RE: snprintf crash - source code missing, hard to debug

Johan — Thu, 29 Jun 2023 13:54:12 GMT

We use our own project, no sample code.
We use QEMU x86 to test our code, with controlled inputs and outputs.
Our code has e few threads running for example, and hardware interfaces (simulated).

build command is:
west build --build-dir _build_xyz_qemu_x86_ZTEST -b qemu_x86 -t run

.conf file for qemu x86 used contains this:

CONFIG_ASSERT=y

CONFIG_NEWLIB_LIBC=y

CONFIG_NEWLIB_LIBC_FLOAT_PRINTF=y

CONFIG_EVENTS=y

CONFIG_REBOOT=y

CONFIG_DEBUG=y

CONFIG_LOG=y

CONFIG_LOG_BACKEND_RTT=n

CONFIG_LOG_PRINTK=y

CONFIG_LOG_BUFFER_SIZE=256

# JSON

CONFIG_JSON_LIBRARY=y

#CONFIG_QEMU_ICOUNT=n

#CONFIG_ENTROPY_GENERATOR=y

#CONFIG_TEST_RANDOM_GENERATOR=y

# Network

CONFIG_NETWORKING=y

CONFIG_NET_NATIVE=n

CONFIG_NET_SOCKETS=y

CONFIG_NET_SOCKETS_OFFLOAD=y

CONFIG_NET_SOCKETS_POSIX_NAMES=y

# mbed TLS and security

CONFIG_MBEDTLS_PK_C=y

CONFIG_MBEDTLS_RSA_C=y

CONFIG_MBEDTLS_PKCS1_V15=y

CONFIG_MBEDTLS_ENABLE_HEAP=y

CONFIG_MBEDTLS_HEAP_SIZE=55936

CONFIG_MBEDTLS_TLS_LIBRARY=y

CONFIG_MBEDTLS_X509_LIBRARY=y

CONFIG_NRF_SECURITY_ADVANCED=y

CONFIG_NORDIC_SECURITY_BACKEND=y

# You can enable mbed TLS debug with

CONFIG_MBEDTLS_DEBUG_C=y

CONFIG_MBEDTLS_SHA1_C=y

# AT host library

# CONFIG_AT_HOST_LIBRARY=y

CONFIG_UART_INTERRUPT_DRIVEN=y

# Net related

CONFIG_NET_SOCKETS_SOCKOPT_TLS=n

CONFIG_NET_IPV4=y

CONFIG_NET_IPV6=n

CONFIG_DEBUG_OPTIMIZATIONS=n

# Heap and stacks

# Extended memory heap size needed for encoding nRF Cloud messages to JSON

CONFIG_HEAP_MEM_POOL_SIZE=4096

CONFIG_MAIN_STACK_SIZE=4096

CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096

CONFIG_NO_OPTIMIZATIONS=y

# settings

CONFIG_SETTINGS=y

CONFIG_SETTINGS_RUNTIME=y

CONFIG_STDOUT_CONSOLE=y

CONFIG_FLASH=y

CONFIG_FLASH_MAP=y

CONFIG_NVS=y

CONFIG_SETTINGS_NVS=y

CONFIG_SHELL_CMDS_SELECT=y

CONFIG_INIT_STACKS=y

CONFIG_THREAD_STACK_INFO=y

CONFIG_ENTROPY_GENERATOR=y

CONFIG_TEST_RANDOM_GENERATOR=y

CONFIG_THREAD_MONITOR=y

# ZTEST

CONFIG_ZTEST=y

CONFIG_ZTEST_NEW_API=y

# ZTEST enable test shuffling for more testing

#CONFIG_ZTEST_SHUFFLE=y

#CONFIG_ZTEST_SHUFFLE_SUITE_REPEAT_COUNT=0

#CONFIG_ZTEST_SHUFFLE_TEST_REPEAT_COUNT=0

# ZTEST more log output

CONFIG_ZTEST_VERBOSE_OUTPUT=y

CONFIG_ZTEST_VERBOSE_SUMMARY=y

RE: snprintf crash - source code missing, hard to debug

Sigurd Hellesvik — Thu, 29 Jun 2023 12:28:10 GMT

Hi Johan,

[quote user=""] I flash the ztest to the board, it works, but running qemu in windows x86, it crashes on that snprintf-line below.[/quote]

Do you use one of ther samples in the nRF Connect SDK for this?
If yes, can you give the command you use to build for qemu?

Regards,
Sigurd Hellesvik