Trouble updating bootloader

Our company has an existing product (snodar) which uses an nRF52840. When we created the snodar, it used a bootloader based on the `secure_bootloader` project. On top of that, we are using the s140 SD and have an APP which allows DFU over BLE. We have been able to update the snodar over BLE using the nordic DFU tools or our own custom Android/iOS apps.

We also have another product (gateway) which uses an nRF52840 + nRF9160. I've been working on mechanism to update the snodar by transferring the firmware over ESB using the gateway then doing an in-place DFU on the snodar. This works only if I can replace the bootloader on the snodar. To get the in-place DFU to work, I had to make a couple of changes to the bootloader project.

I needed to revise the bootloader to:
- revise sdk_config.h to add "#define NRF_BL_DFU_ALLOW_UPDATE_FROM_APP 1"
- revise the bootloader .ld file to increase RAM for my changes to the sdk
- revise the nrf_dfu_validation.c to add a function to copy cached flash data into ram
"In the lab" I can easily make a new merged hex and load onto a system over jtag using nrfprog. However, for field units, we update with DFU over BLE. I'm not clear on how I can update the bootloader for these systems.
The typical .zip just contains the SD + APP. It would be nice to have a one-off .zip which is BL + SD + ZIP. Reading through the nrfutil manual it indicates this is possible on page 9 but doesn't provide an example. Do I just add the --bootloader and --bootloader_version to the call which is making the zip now? Ideally, I'd almost like to just add the bootloader + softdevice and have the nrf5 flash erased elsewise so I can start fresh again.
Our bootloader was based on the 'nRF5_SDK_17.0.2\examples\dfu\secure_bootloader' project. We added our own dfu_public_key.c and then made some small customizations to the sdk_config.h so we couldn't do things like downgrade firmware. This bootloader was loaded on the units during assembly long before we had the idea to do this in-place DFU.
The only recent change to the bootloader's sdk_config.h was to add the line: "#define NRF_BL_DFU_ALLOW_UPDATE_FROM_APP 1". However, due the change necessary to that nrf_dfu_validation.c file, I had to increase the RAM some. The bootloader's .ld looks like:
MEMORY
{
  FLASH (rx) : ORIGIN = 0xf7000, LENGTH = 0x7000
  RAM (rwx) :  ORIGIN = 0x20005978, LENGTH = 0x3a688
  uicr_bootloader_start_address (r) : ORIGIN = 0x10001014, LENGTH = 0x4
  bootloader_settings_page (r) : ORIGIN = 0x000FF000, LENGTH = 0x1000
  uicr_mbr_params_page (r) : ORIGIN = 0x10001018, LENGTH = 0x4
  mbr_params_page (r) : ORIGIN = 0x000FE000, LENGTH = 0x1000
}
The FLASH was from 0xf8000 with len 0x6000 before the mod.
On the snodar side, the .ld had the following at the top:
MEMORY
{
  FLASH (rx) : ORIGIN = 0x27000, LENGTH = 0xc9000
  RAM (rwx) :  ORIGIN = 0x20006368, LENGTH = 0x39c98
  uicr_bootloader_start_address (r) : ORIGIN = 0x10001014, LENGTH = 0x4
}

SECTIONS
{
  . = ALIGN(4);
  .uicr_bootloader_start_address :
  {
    PROVIDE(__start_uicr_bootloader_start_address = .);
    KEEP(*(SORT(.uicr_bootloader_start_address*)))
    PROVIDE(__stop_uicr_bootloader_start_address = .);
  } > uicr_bootloader_start_address
}
If I attempted an in-place DFU with that setup, I would see:
00> dat len/crc = 142, b100ec1e
00> bin len/crc = 284644, 97038ff8
00> *** Trigger DFU ***
00> ~~ RTC ptimer disabled
00> CRC ok?
00> dat: b100ec1e - b100ec1e -> 1
00> bin: 97038ff8 - 97038ff8 -> 1
00> Starting DFU update: bin_addr = 3f81000, remaining = 284644
00> <info> app: NRF_DFU_EVT_DFU_STARTED
00>
00> <info> nrf_dfu_validation: Signature required. Checking signature.
00>
00> <info> nrf_dfu_validation: Calculating hash (len: 65)
00>
00> <info> nrf_dfu_validation: Verify signature
00>
00> <info> nrf_dfu_validation: Image verified
00>
00> <warning> nrf_dfu_settings: Settings write aborted since it tries writing to forbidden settings.
00>
00> <warning> nrf_dfu_serial: DFU request completed with result: 0xA
00>
00> <info> app: NRF_DFU_EVT_DFU_FAILED
So I modified the snodar's .ld to resemble:
MEMORY
{
  FLASH (rx) : ORIGIN = 0x27000, LENGTH = 0xc9000
  RAM (rwx) :  ORIGIN = 0x20006368, LENGTH = 0x39c98
  uicr_bootloader_start_address (r) : ORIGIN = 0x10001014, LENGTH = 0x4
  bootloader_settings_page (r) : ORIGIN = 0x000FF000, LENGTH = 0x1000
  uicr_mbr_params_page (r) : ORIGIN = 0x10001018, LENGTH = 0x4
  mbr_params_page (r) : ORIGIN = 0x000FE000, LENGTH = 0x1000
}

SECTIONS
{
  . = ALIGN(4);
  .uicr_bootloader_start_address :
  {
    PROVIDE(__start_uicr_bootloader_start_address = .);
    KEEP(*(SORT(.uicr_bootloader_start_address*)))
    PROVIDE(__stop_uicr_bootloader_start_address = .);
  } > uicr_bootloader_start_address
  . = ALIGN(4);
  .bootloader_settings_page(NOLOAD) :
  {
    PROVIDE(__start_bootloader_settings_page = .);
    KEEP(*(SORT(.bootloader_settings_page*)))
    PROVIDE(__stop_bootloader_settings_page = .);
  } > bootloader_settings_page
  . = ALIGN(4);
  .uicr_mbr_params_page :
  {
    PROVIDE(__start_uicr_mbr_params_page = .);
    KEEP(*(SORT(.uicr_mbr_params_page*)))
    PROVIDE(__stop_uicr_mbr_params_page = .);
  } > uicr_mbr_params_page
  . = ALIGN(4);
  .mbr_params_page(NOLOAD) :
  {
    PROVIDE(__start_mbr_params_page = .);
    KEEP(*(SORT(.mbr_params_page*)))
    PROVIDE(__stop_mbr_params_page = .);
  } > mbr_params_page
}
The top four lines in the memory part match the same in the bootloader. If I do this then the in-place dfu works.
The in-place dfu is weird and I don't fully understand it. During a normal ble dfu, when I start a dfu, the snodar switches to the bootloader and then does the update. I know this since the LEDs indicate what stage I am in. For this in-place dfu, we stay in the snodar fw and it is able to update itself -- it doesn't switch to the bootloader.
If I just use jtag and the hex file, both the bootloader and snodar work fine. I can update from one snodar fw to the next as long as the fw version is bigger. I also know that the in-place dfu will not work at all unless I use the newer bootloader. If I use the old bootloader, the validation fails, and it just reverts back to the current snodar fw.
When we use nrfutil we are doing a BL + SD + APP setup:
# "nrf52840_xxaa_s140.hex" is the generated bootloader
nrfutil settings generate --family NRF52840 --application "${APP_NAME}.hex" --application-version 0 --bootloader-version 1 --bl-settings-version 2 settings.hex
mergehex -m "${APP_NAME}.hex" *_softdevice.hex "nrf52840_xxaa_s140.hex" -o tmp.hex
mergehex -m tmp.hex settings.hex -o "${PROD_FW}_v${FW_VER}.hex"
rm tmp.hex
This hex works over jtag, but our field units are sealed up, so we need to use BLE DFU to update them. My understanding is that we need to make two files to update both. We are never changing the SD part. It is always the s140.
So my test bootloader generation was:
nrfutil pkg generate --hw-version 2 --bootloader nrf52840_xxaa_s140.hex --bootloader-version 1 --sd-req 0x0100 --key-file ../../../../../keys/private.key secure_bl_v1.zip
And the generated app was:
nrfutil pkg generate --hw-version "${HW_REV_VER}" --application-version-string "${FW_VER}" --application "${APP_NAME}.hex" --sd-req 0x0100 --key-file ../../../../../../keys/private.key "${SECURE_FW_NAME}.zip"
If I attempt to ble dfu of this, what I witness is that the snodar switches to some old version of the bootloader. The advertising name I see is "SNOdar BootloaderD" -- this is different from the new one. At this stage, I am able to then update the app firmware fine, but If I try to update this bootloader zip, it just gives me a bunch of "x" in the dfu app and really no feedback.
I had also tried adding the soft device to the bootloader generation:
nrfutil pkg generate --hw-version 2 --bootloader nrf52840_xxaa_s140.hex --bootloader-version 1 --softdevice "${SOFT_DEV}" --sd-req 0x0100 --key-file ../../../../../keys/private.key secure_bl_v1.zip
This has a similar result as before, but this time, the dfu app will show a "insufficient resources" at the bottom if I try to use this bootloader zip. The app update works fine.
Sorry for the data dump, but I just wanted to collect all my notes of things I've tried so far.
Is there any way to just wipe the flash and make a zip that just contains a BL + SD? over a BLE DFU? Do you have any other suggestions to update the bootloader?