Greetings!
Briefly summarized: I am trying to generate SPDX files for our project, but I hit some roadblocks while running the west spdx command. I found a way to workaround those roadblocks to get a spdx file that is considered valid by the online validation tool, however due to being un-experienced in the SPDX/SBOM area I am not sure if all of my workarounds are correct. I managed to replicate this bad behaviour with a wifi/shell sample in the NCS upstream. I will be happy to receive some support.
The problem
I have tried to generate the spdx files for our main application firmware, by following the instructions listed in the west spdx documentation.
The command would quickly stop with below error:
user@c54cb75cc7a7:~/workdir/project/app$ west spdx -d build/app
parsing CMake Cache file
parsing CMake Codemodel files
setting up SPDX documents
walking through targets
Traceback (most recent call last):
File "/home/user/ncs/toolchains/7cbc0036f4/usr/local/bin/west", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/user/ncs/toolchains/7cbc0036f4/usr/local/lib/python3.12/site-packages/west/app/main.py", line 1085, in main
app.run(argv or sys.argv[1:])
File "/home/user/ncs/toolchains/7cbc0036f4/usr/local/lib/python3.12/site-packages/west/app/main.py", line 244, in run
self.run_command(argv, early_args)
File "/home/user/ncs/toolchains/7cbc0036f4/usr/local/lib/python3.12/site-packages/west/app/main.py", line 505, in run_command
self.run_extension(args.command, argv)
File "/home/user/ncs/toolchains/7cbc0036f4/usr/local/lib/python3.12/site-packages/west/app/main.py", line 654, in run_extension
self.cmd.run(args, unknown, self.topdir, manifest=self.manifest,
File "/home/user/ncs/toolchains/7cbc0036f4/usr/local/lib/python3.12/site-packages/west/commands.py", line 194, in run
self.do_run(args, unknown)
File "/home/user/workdir/zephyr/scripts/west_commands/spdx.py", line 64, in do_run
self.do_run_spdx(args)
File "/home/user/workdir/zephyr/scripts/west_commands/spdx.py", line 113, in do_run_spdx
makeSPDX(cfg)
File "/home/user/workdir/zephyr/scripts/west_commands/zspdx/sbom.py", line 84, in makeSPDX
retval = w.makeDocuments()
^^^^^^^^^^^^^^^^^
File "/home/user/workdir/zephyr/scripts/west_commands/zspdx/walker.py", line 130, in makeDocuments
self.walkTargets()
File "/home/user/workdir/zephyr/scripts/west_commands/zspdx/walker.py", line 416, in walkTargets
self.collectTargetDependencies(cfgTargets, cfgTarget, pkg)
File "/home/user/workdir/zephyr/scripts/west_commands/zspdx/walker.py", line 617, in collectTargetDependencies
rd.ownerFileAbspath = pkg.targetBuildFile.abspath
^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'abspath'
After having a look at the mentioned walker.py file I added some logging statements to better see what was happening. Below is the patch file that can be applied by running git apply spdx_walker_with_logs.patch from zephyr's root dir.
7077.spdx_walker_with_logs.patch
The build log would then look like this:
user@c54cb75cc7a7:~/workdir/project/app$ west spdx -d build/app
parsing CMake Cache file
parsing CMake Codemodel files
setting up SPDX documents
walking through targets
WARNING: - collecting target dependencies for ..__nrf__lib__hw_id
WARNING: - adding pending relationship for driver_validation_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for syscall_list_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for device_api_ld_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for app_version_h
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for kobj_types_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for zephyr_generated_headers
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for tfm_api
WARNING: - adding pending relationship for cjson
WARNING: - adding pending relationship for ..__nrf__modules__azure-sdk-for-c
WARNING: - adding pending relationship for modules__hal_nordic__nrfx
WARNING: - adding pending relationship for modules__lvgl
WARNING: - adding pending relationship for nrf-wifi-shim
WARNING: - adding pending relationship for nrf70-buslib
WARNING: - adding pending relationship for modules__segger
WARNING: - adding pending relationship for kernel
WARNING: - collecting target dependencies for compiler
WARNING: - collecting target dependencies for compiler-cpp
WARNING: - collecting target dependencies for config-twister
WARNING: - collecting target dependencies for debug
WARNING: - collecting target dependencies for debugserver
WARNING: - collecting target dependencies for device_api_ld_target
WARNING: - collecting target dependencies for devicetree_target
WARNING: - adding pending relationship for driver_validation_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for syscall_list_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for device_api_ld_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for app_version_h
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for kobj_types_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for zephyr_generated_headers
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for mbedtls
WARNING: - adding pending relationship for mbedx509
WARNING: - adding pending relationship for mbedcrypto
WARNING: - adding pending relationship for nrf_security_utils
WARNING: - adding pending relationship for mbedcrypto_base
WARNING: - adding pending relationship for tfm
WARNING: - depAbspath wasn't set
WARNING: - collecting target dependencies for driver_validation_h_target
WARNING: - adding pending relationship for parse_syscalls_target
WARNING: - No targets, skipping
WARNING: - collecting target dependencies for drivers__adc
WARNING: - adding pending relationship for driver_validation_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for syscall_list_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for device_api_ld_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for app_version_h
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for kobj_types_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for zephyr_generated_headers
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for mbedtls
WARNING: - adding pending relationship for mbedx509
WARNING: - adding pending relationship for mbedcrypto
WARNING: - adding pending relationship for nrf_security_utils
WARNING: - adding pending relationship for mbedcrypto_base
WARNING: - adding pending relationship for tfm
WARNING: - depAbspath wasn't set
WARNING: - collecting target dependencies for drivers__charger
WARNING: - adding pending relationship for driver_validation_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for syscall_list_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for device_api_ld_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for app_version_h
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for kobj_types_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for zephyr_generated_headers
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for mbedtls
WARNING: - adding pending relationship for mbedx509
WARNING: - adding pending relationship for mbedcrypto
WARNING: - adding pending relationship for nrf_security_utils
WARNING: - adding pending relationship for mbedcrypto_base
WARNING: - adding pending relationship for tfm
WARNING: - depAbspath wasn't set
WARNING: - collecting target dependencies for drivers__clock_control
WARNING: - adding pending relationship for driver_validation_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for syscall_list_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for device_api_ld_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for app_version_h
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for kobj_types_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for zephyr_generated_headers
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for mbedtls
WARNING: - adding pending relationship for mbedx509
WARNING: - adding pending relationship for mbedcrypto
WARNING: - adding pending relationship for nrf_security_utils
WARNING: - adding pending relationship for mbedcrypto_base
WARNING: - adding pending relationship for tfm
WARNING: - depAbspath wasn't set
WARNING: - collecting target dependencies for drivers__console
WARNING: - adding pending relationship for driver_validation_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for syscall_list_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for device_api_ld_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for app_version_h
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for kobj_types_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for zephyr_generated_headers
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for mbedtls
WARNING: - adding pending relationship for mbedx509
WARNING: - adding pending relationship for mbedcrypto
WARNING: - adding pending relationship for nrf_security_utils
WARNING: - adding pending relationship for mbedcrypto_base
WARNING: - adding pending relationship for tfm
WARNING: - depAbspath wasn't set
WARNING: - collecting target dependencies for drivers__display
WARNING: - adding pending relationship for driver_validation_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for syscall_list_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for device_api_ld_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for app_version_h
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for kobj_types_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for zephyr_generated_headers
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for mbedtls
WARNING: - adding pending relationship for mbedx509
WARNING: - adding pending relationship for mbedcrypto
WARNING: - adding pending relationship for nrf_security_utils
WARNING: - adding pending relationship for mbedcrypto_base
WARNING: - adding pending relationship for tfm
WARNING: - depAbspath wasn't set
WARNING: - collecting target dependencies for drivers__entropy
WARNING: - adding pending relationship for driver_validation_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for syscall_list_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for device_api_ld_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for app_version_h
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for kobj_types_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for zephyr_generated_headers
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for mbedtls
WARNING: - adding pending relationship for mbedx509
WARNING: - adding pending relationship for mbedcrypto
WARNING: - adding pending relationship for nrf_security_utils
WARNING: - adding pending relationship for mbedcrypto_base
WARNING: - adding pending relationship for tfm
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for tfm_api
WARNING: - collecting target dependencies for drivers__ethernet
WARNING: - adding pending relationship for driver_validation_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for syscall_list_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for device_api_ld_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for app_version_h
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for kobj_types_h_target
WARNING: - depAbspath wasn't set
WARNING: - adding pending relationship for mbedtls
Traceback (most recent call last):
File "/home/user/ncs/toolchains/7cbc0036f4/usr/local/bin/west", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/user/ncs/toolchains/7cbc0036f4/usr/local/lib/python3.12/site-packages/west/app/main.py", line 1085, in main
app.run(argv or sys.argv[1:])
File "/home/user/ncs/toolchains/7cbc0036f4/usr/local/lib/python3.12/site-packages/west/app/main.py", line 244, in run
self.run_command(argv, early_args)
File "/home/user/ncs/toolchains/7cbc0036f4/usr/local/lib/python3.12/site-packages/west/app/main.py", line 505, in run_command
self.run_extension(args.command, argv)
File "/home/user/ncs/toolchains/7cbc0036f4/usr/local/lib/python3.12/site-packages/west/app/main.py", line 654, in run_extension
self.cmd.run(args, unknown, self.topdir, manifest=self.manifest,
File "/home/user/ncs/toolchains/7cbc0036f4/usr/local/lib/python3.12/site-packages/west/commands.py", line 194, in run
self.do_run(args, unknown)
File "/home/user/workdir/zephyr/scripts/west_commands/spdx.py", line 64, in do_run
self.do_run_spdx(args)
File "/home/user/workdir/zephyr/scripts/west_commands/spdx.py", line 113, in do_run_spdx
makeSPDX(cfg)
File "/home/user/workdir/zephyr/scripts/west_commands/zspdx/sbom.py", line 84, in makeSPDX
retval = w.makeDocuments()
^^^^^^^^^^^^^^^^^
File "/home/user/workdir/zephyr/scripts/west_commands/zspdx/walker.py", line 130, in makeDocuments
self.walkTargets()
File "/home/user/workdir/zephyr/scripts/west_commands/zspdx/walker.py", line 416, in walkTargets
self.collectTargetDependencies(cfgTargets, cfgTarget, pkg)
File "/home/user/workdir/zephyr/scripts/west_commands/zspdx/walker.py", line 613, in collectTargetDependencies
rd.ownerFileAbspath = pkg.targetBuildFile.abspath
^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'abspath'
Note: I have deleted some of the log lines as the contain filenames used in the project.
After getting to this the debugging started. In the above case the first problematic library was drivers_ethernet. Somehow this library was being enabled but our project wasn't using any driver file found under zephyr/drivers/ethernet. I disabled this library by setting CONFIG_ETH_DRIVER=n in the prj.conf. This problem disappeared, but then another one appeared. I somehow figured it out that the spdx script will crash if the library was enabled (by my understanding that cmake found a specific zephyr_library() line in the CMakelists.txt files) but no source files were added to it (since the dependencies for those source files weren't satisfied).
Most of the other errors that emerged were somehow related to our project Cmake structure, removing zepyhr_library() or zepyhr_library_amend() here and there resolved the errors in the walker.py and moved the progress further.
Then it got to the driver__wifi library. That one I couldn't resolve, our project doesn't amend this library or declared a new one, any change would require patching either NCS/Zephyr.
I then added the following if statement to the top of the collectTargetDependencies function in the walker.py:
def collectTargetDependencies(self, cfgTargets, cfgTarget, pkg):
log.wrn(f" - collecting target dependencies for {pkg.cfg.name}")
if not pkg.targetBuildFile:
return
That way the function would simply skip the whole processing. With this "fix" I would finally get spdx files.
The online SPDX validation tool confirmed that they are correct, but only after I have replaced word
proprietary with some valid arbitrary SDPX license marker. Replicating the issue
I was able to replace the issue in the in the ncs/sample/wifi/shell sample (building for the nrf7002dk), again by following the west spdx instructions. The script also stops at the drivers__wifi library.
Interestingly, the command west spdx works as expected in zephyr/samples/net/wifi/shell.
Questions
1. Since I discovered this issue in the NCS sample, I assume that my code is not at fault, something is probably badly configured on the CMake level. Could Nordic investigate this?
2. Today I learned that west spdx command can fail, even though that the west build command succeeds. The issue can lie either in the project code or in the NCS/Zephyr code. Besides ignoring the fact that the spdx walker code should probably print some friendly error info, what are reasonable debugging steps that one can take to determine how the project is badly configured? Is my current understanding "script is failing since there is a library without any source file" correct or is there something more to this?
3. The west spdx command generates several spdx files. The message in the commit that introduced the spdx support states that:
> 1) `app.spdx`: This contains the bill-of-materials for the application source files used for the build.
> 2) `zephyr.spdx`: This contains the bill-of-materials for the specific Zephyr source code files that are used for the build.
In the case of our project application I noticed that almost all project specific files were listed in the zephyr.spdx file and not in the app.spdx. I assume that this is due to the fact that we use zephyr_library and zephyr_library_sources for telling CMake to compile the files. In our project we chose same kind of Cmake organization that Bridle project uses, see the link for an example: https://github.com/tiacsys/bridle/tree/main/drivers
In other words: it is almost the same one as the Zephyr uses.
From the functional point we never hit any issue with this approach. Now, looking at the generated spdx output I am not sure if this is the correct approach. Also, since I don't enough about how the spdx files are later used, I don't even know if this really is an issue.
Can you comment on this? Do you have some suggestions?
I realize that last two points are more suitable for the Zephyr discord, I will also post them there, but I will be happy for an answer.
With best regards,
Marko Sagadin