Strange behavior Button 2 and Button 3 on nRF51 DK

Question

Updates are moved to the end of the question 
 I created a modified version of pca10028.h and boards.h and have preprocessors changed so that the following new config is loaded: 
 // Custom LEDs definitions for PCA10028
#define LEDS_NUMBER 1

//#define LED_START 21
#define LED_1 21
#define LED_2 22
#define LED_3 23
#define LED_4 24
#define LED_5 25

#define LED_START 24
#define LED_STOP 25

//#define LEDS_LIST { LED_1, LED_2, LED_3, LED_4 }
#define LEDS_LIST { LED_4 }

//#define BSP_LED_0 LED_1
//#define BSP_LED_1 LED_2
//#define BSP_LED_2 LED_3
//#define BSP_LED_3 LED_4
#define BSP_LED_0 LED_4
#define BSP_LED_1 LED_5

#define BSP_LED_0_MASK (1<<BSP_LED_0)
#define BSP_LED_1_MASK (1<<BSP_LED_1)
//#define BSP_LED_2_MASK (1<<BSP_LED_2)
//#define BSP_LED_3_MASK (1<<BSP_LED_3)

//#define LEDS_MASK (BSP_LED_0_MASK | BSP_LED_1_MASK | BSP_LED_2_MASK | BSP_LED_3_MASK)
#define LEDS_MASK (BSP_LED_0_MASK | BSP_LED_1_MASK)
/* all LEDs are lit when GPIO is low */
#define LEDS_INV_MASK LEDS_MASK

#define BUTTONS_NUMBER 0

#define BUTTON_1 17
#define BUTTON_2 18
#define BUTTON_3 19
#define BUTTON_4 20
 
 I expected that this changed the BSP module to use pin 24 and 25 for LED display purpose. 
 Then in main.c I am setting some simple GPIO toggling behavior. Basically, at a button input interrupt event, the interrupt handler would set a flag, and in main() 's infinite loop, the according LED should be toggled. My code is basically exactly this 
 ...

static bool m_input_it_flag = false;

...

/** @brief Helper function to toggle output
 */
static void __forceinline toggle_output(void)
{
 nrf_gpio_pin_toggle(2);
 nrf_gpio_pin_toggle(LED_2);
}

/** @brief Input pin interrupt handler
 */
void group1_input_it_handler(nrf_drv_gpiote_pin_t pin, nrf_gpiote_polarity_t action)
{
 m_input_it_flag = true;
// toggle_output();
}

/** @brief GPIO initialization
 */
void gpio_init(void)
{
 nrf_drv_gpiote_in_config_t nrf_drv_gpiote_in_config;
 
 // Input ---------------------------------------------------------------------------------------
 // This could return either NRF_SUCCESS or NRF_ERROR_INVALID_STATE. However, NRF_ERROR_INVALID_STATE
 // simply means the GPIOTE module is already initialized, so that error could be ignored.
 nrf_drv_gpiote_init();
 
 memset(&nrf_drv_gpiote_in_config, 0, sizeof(nrf_drv_gpiote_in_config_t));
 nrf_drv_gpiote_in_config.sense = NRF_GPIOTE_POLARITY_TOGGLE;
 nrf_drv_gpiote_in_config.pull = NRF_GPIO_PIN_PULLUP;
 nrf_drv_gpiote_in_config.hi_accuracy = false;
 nrf_drv_gpiote_in_config.is_watcher = false;
 
 nrf_drv_gpiote_in_init(BUTTON_2, &nrf_drv_gpiote_in_config, group2_input_it_handler);
 nrf_drv_gpiote_in_event_enable(BUTTON_2, true);
 
 // Output --------------------------------------------------------------------------------------
 nrf_gpio_cfg_output(2);
 nrf_gpio_cfg_output(LED_2);
 
 nrf_gpio_pin_set(2);
 nrf_gpio_pin_clear(LED_2);
}

...

/**@brief Application main function.
 */
int main(void)
{
 ...
 gpio_init();
 ...
 
 for (;;)
 {
 ...
 if (m_input_it_flag)
 {
 toggle_output();
 m_input_it_flag = false;
 }
 ...
 
 }
}
 
 I have trimmed the full code and change variable name not to make this post lengthy unnecessarily. However in the original code I have that above code for both Button 1, 2 and 3. They should toggle LED 1, 2 and 3, respectively. 
 Now come the fun part:
For button 2 and 3, after I pressed them once, their output seem to adopt the behavior of LED 4 (blink while advertising), which is BSP LED 1. In particular, LED 2 and 3 starts to blink in sync with LED 4. Sometimes they blink with the same state. Sometime they blink with reversed state. The output just toggle in sync. 
 Button 1 does not cause this problem. 
 Also, if instead of setting flags and handle the job in main, I toggle the output right at the handler, then this problem does not happen. 
 I could post a video of this if requested, but that seems too much work right now. 
 Do anyone have any guesses on what is going on? 
 
 Edit 1: I am using nRF51 DK with SDK v10.0.0 and SoftDevice S110 v8.0.0. 
 
 Edit 2: 
After some discussion with Øyvind in the comment down below, I have found some further issue even when not using BSP. 
 main.c code: buggy_gpiote_in_flag_main.c 
 
 Edit 4: In Edit 2 and 3's attachment I forgot to remove some irrelevant includes and codes. Here is a better trimmed down main.c file buggy_gpiote_in_flag_main.c 
 I tested it and it should run and reproduce the issue simply by replacing main.c in SDK v10's ble_app_uart example. 
 At the beginning of the file are two macros DEV_CFG_USE_FLAG and DEV_CFG_USE_BSP . Setting them to 1 will change the code to use flag raising to run GPIO triggered code or to use BSP, respectively. Setting them to 0 will change the code to run event code directly in GPIO handler or to disable BSP. 
 Also here is the board support file that could be used with it to observe the weird behavior when used with BSP. It is basically the code I have in the post. pca10028_modded.h