Core Specific Controllers Problem When Disconnected

I’ve been using the new mapping multiple controllers (mapped port) feature in Retroarch 1.9.6 (Win64) to set up having a dedicated Gamecube controller for Dolphin. With this I have the following setup

Controller 1: Wii U Bluetooth controller (connected as an Xbox 360 controller in Retroarch with WinUsoft) Controller 2: No controller (potentially I could add an Xbox One controller here) Controller 3: Power A Gamecube controller (connected as a Switch Joypad in Retroarch) Controller 4-16: No controller

When Dolphin is active controller 3 is mapped to controller 1 as a core remap. All is good and I can play Gamecube games as player 1 with the Gamecube controller. With all my other cores I can play as player 1 with the WiiU controller.

The problem comes when I attempt to use open Retroarch when only one of the controllers is connected to the PC. When only the WiiU controller is connected Retroarch assigns it to controller 3. How do I keep the settings without switching on all my controllers every time?