Not sure what you mean by “better mask the upscaled textures”
But what I do is (because I also use waifux2 for some sprites):
If this is my original texture:
I first scale it up to this:
Then I use IEU to split the RGBAs.
After that I pull all the alphas in a new folder and use the batch mode in my standard imageviewer XnView to fix the alpha channel and correct the outlines as good as I can.
My settings are: White 128, Black 65, Contrast 127
So I come from this:
To this:
Then I just pull all the alphas back in IEU and merge them with the RGBs
So I end up with this:
Which is close to the HD Remaster:
The outlines in the HD Remaster are better, but for now I dont know how to make them better automatically. Probably I have to edit the details manually in Gimp.
Because the sprites look a bit pale ingame I also use a sharpen model in RTCWHQ with color adjustments afterwards.
In the end it looks like this way ingame:
Original
Upscale