As far as I can tell there is a sign error bug when setting the tx power to a negative value. ble_dtm.c uses an int32_t to represent the tx power so when calling dtm_set_txpower with -4 (0xFC) the resulting bits in m_tx_power will be 0xFFFFFFFC.
Later when dtm_init() is called, m_tx_power is validated in ble_dtm_hw_nrf51.c and checked against the valid values in nrf51_bitfields.h, all of which are defined as (0xXXUL). This comparison will always fail and dtm_init reports invalid configuration.
It can be solved by adding the following line of code in dtm_set_txpower:
m_tx_power &= 0xFF;
Am I missing something here? Is there another way that this is supposed to work?