The N64 autoconfigs are all user-supplied, so there may be better ways to map them than whoever submitted the profile came up with. For example, some of them use double-mapping for one of the buttons so it performs the a needed menu function while also performing the competing in-game function.
The problem lies with the default mapping of the cores: they’re set up for modern layouts, with N64’s B and A being mapped to retropad Y and B, respectively, but if you map them like that, the menu controls are wonky, since they’re hardcoded to retropad A and B. IIRC, the double-mapping puts retropad Y and A on N64 B and retropad B on N64 A.
There’s also the “independent c-button mapping” option, which throws all of the N64 buttons onto digital retropad buttons for easier remapping (i.e., instead of having the C buttons on the right-analog and the C-button modeswitch on R2).
Either way, you’re probably gonna want to do some core remapping for it.