Introduction
I made a script that is fairly portable that can automate building, signing, etc. all cores from the latest github versions of retroarch on the latest iOS versions. It generates a nice little IPA for you to just pop on into your cydia impactor or whatever else.
Script Advantages
The script is below, but to describe the advantages of running my script rather than libretro-build-ios-arm64.sh directly with no arguments…
- It attempts to compile all available cores with various versions of iOS. This means that some cores that are not included in the default
libretro-build-ios-arm64.sh, but which nonetheless compile properly and work great, will be included. Cores like quasi88, for example. - It also builds the IPA for you - no need to mess around with xcode beyond installation.
- It keeps an archive of the latest compiled versions of each core. This is important since some cores lose their compilability through updates to their code (temporary breaking, etc). With this, you will keep access to the core regardless.
- This will copy in other cores in the order of iOS10 > iOS9 > iOS Generic > iOS-theos in case they don’t compile in arm64. The cores won’t work, mind you, but hey they’ll be there if you want 'em? There may be a way to get them to work, so please tell me if you happen to know.
- Fetches are done only immediately before building, which I guess makes it slightly more likely that you get the most bleeding edge core possible compiled compared to running
libretro-fetch.shand thenlibretro-build-ios-arm64.sh. Hurrah? - Fetches and compiles actually work for weird cores that are not pulled in normally by running
libretro-fetch.shwithout options. For some reason, for example, Play is not fetched by that script unless you explicitly runlibretro-fetch.sh play. Not that Play compiles currently, mind you. But it might some day!
Script Setup
- Install the latest OSX (I am using an install of an OSX guest on VMWare via a Windows host, so it does compile fine on that)
- Install xcode (ideally the latest developer version, as that’s what I’m testing against). Make sure the right xcode is set properly by running
xcode-select -p - Use git to check out libretrosuper and retroarch
- Figure out your provisioning profile name and developer certificate name (check https://developer.apple.com/account/resources/certificates/list and https://developer.apple.com/account/resources/profiles/list).
- Install xcprovisioner by running
sudo gem install xcprovisioner - Create a directory to store the output
- Edit the variables at the start of the script to reflect the directories you created and information you obtained.
- Download http://buildbot.libretro.com/assets/frontend/assets.zip and put it in
pkg/applein yourretroarchdirectory. - Make all the other .sh files referenced executable (chmod a+x)
- Run
sudo easy_install pip; sudo pip install tinydb; sudo pip install gitpython
There may be a better way to handle signing via CLI.
You may also need to run open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg and follow the installation.
I am going to assume that you know how to do all those steps above. If you don’t, I probably won’t be providing support.
Cores that do not compile currently, but which do compile with some tweaks
bsnes
Run the command CMAKE=cmake VERBOSE=1 SINGLE_CORE=bsnes FORCE=YES EXIT_ON_ERROR=1 ./libretro-buildbot-recipe.sh recipes/apple/cores-ios-arm64-generic
bsnes2014
Run the command CMAKE=cmake VERBOSE=1 SINGLE_CORE=bsnes2014 FORCE=YES EXIT_ON_ERROR=1 ./libretro-buildbot-recipe.sh recipes/apple/cores-ios-arm64-generic
craft
EasyRPG
Install brew and then use that to install cmake and pkg-config. Then cd into libretro-easyrpg and run git submodule update --init. Each time you re-build, you will need to run the following before doing so:
git clean -xfd
git submodule foreach --recursive git clean -xfd
git reset --hard
git submodule foreach --recursive git reset --hard
git submodule update --init --recursive
emux
Add the following line to recipes/apple/cores-ios-arm64-generic
emux libretro-emux https://github.com/libretro/emux.git master YES GENERIC Makefile.ios-arm64 libretro | emux_nes:MACHINE=nes emux_sms:MACHINE=sms emux_chip8:MACHINE=chip8 emux_gb:MACHINE=gb
Then run the following command:
ENABLED=1 CMAKE=cmake VERBOSE=1 SINGLE_CORE=emux ./libretro-buildbot-recipe.sh recipes/apple/cores-ios-arm64-generic
flycast
Mupen64plus_next
Oberon
Openlara
PCSX Rearmed
Run CMAKE=cmake VERBOSE=1 SINGLE_CORE=pcsx_rearmed_interpreter FORCE=YES EXIT_ON_ERROR=1 ./libretro-buildbot-recipe.sh recipes/apple/cores-ios-arm64-generic
play
Run the command CMAKE=cmake VERBOSE=1 SINGLE_CORE=play FORCE=YES EXIT_ON_ERROR=1 ./libretro-buildbot-recipe.sh recipes/apple/cores-ios-arm64-generic
sameboy
install brew (https://brew.sh) and then install rgbds via brew install rgbds.
To get the core to fully work currently, you will need to run the build script twice. The first time will likely fail due to ‘Bad CPU type in executable’ errors. But if you re-run it, it will compile.
test
Run this command CMAKE=cmake VERBOSE=1 SINGLE_CORE=test FORCE=YES EXIT_ON_ERROR=1 ./libretro-buildbot-recipe.sh recipes/apple/cores-ios-arm64-generic
thepowdertoy
Add the following line to recipes/apple/cores-ios-arm64-generic
thepowdertoy libretro-thepowdertoy https://github.com/libretro/ThePowderToy.git master YES CMAKE Makefile build -DCMAKE_TOOLCHAIN_FILE=../ios.cmake -DCMAKE_BUILD_TYPE="Release" --target thepowdertoy_libretro
Then run the following command:
ENABLED=1 CMAKE=cmake VERBOSE=1 SINGLE_CORE=thepowdertoy ./libretro-buildbot-recipe.sh recipes/apple/cores-ios-arm64-generic
TIC-80
Add the following line to recipes/apple/cores-ios-arm64-generic
tic80 libretro-tic80 https://github.com/jet082/TIC-80.git master YES CMAKE Makefile build2 -DCMAKE_TOOLCHAIN_FILE=../ios.cmake -DCMAKE_BUILD_TYPE="Release"
Then run the following command:
ENABLED=1 CMAKE=cmake VERBOSE=1 SINGLE_CORE=tic80 ./libretro-buildbot-recipe.sh recipes/apple/cores-ios-arm64-generic
Cores I’ve compiled for arm64
2048
3dengine
4do
81
atari800
basilisk2
bluemsx
bnes
bsnes
bsnes2014
bsnes_accuracy
bsnes_balanced
bsnes_cplusplus98
bsnes_hd
bsnes_mercury_accuracy
bsnes_mercury_balanced
bsnes_mercury_performance
bsnes_performance
cannonball
cap32
chailove
craft
crocods
daphne
desmume
dinothawr
dosbox
easyrpg
emux_chip8
emux_gb
emux_nes
emux_sms
fbalpha2012_cps1
fbalpha2012_cps2
fbalpha2012_cps3
fbalpha2012
fbalpha2012_neogeo
fbneo
fceumm
ffmpeg
flycast
fmsx
freechaf
freeintv
frodo
fuse
gambatte
gearboy
gearsystem
genesis_plus_gx
gme
gpsp
gw
handy
hatari
lutro
mame2000
mame2003
mame2003_plus
mame2010
mame2015
mame
mednafen_gba
mednafen_lynx
mednafen_ngp
mednafen_pce_fast
mednafen_pcfx
mednafen_psx
mednafen_saturn
mednafen_snes
mednafen_supergrafx
mednafen_vb
mednafen_wswan
melonds
mesen-s
mesen
meteor
mgba
mrboom
mu
mupen64plus_next
nekop2
nestopia
np2kai
nxengine
o2em
oberon
openlara
parallel_n64
pcsx_rearmed_interpreter
picodrive
play
pocketcdg
pokemini
prboom
prosystem
puae
px68k
quasi88
quicknes
reminiscence
sameboy
scummvm
snes9x2002
snes9x2005
snes9x2010
snes9x
squirreljme
stella2014
stella
stonesoup
test
tgbdual
theodore
thepowdertoy
tic80
tyrquake
uzem
vba_next
vbam
vecx
vice_x128
vice_x64
vice_xplus4
vice_xvic
virtualjaguar
xrick
yabause
Building for other OSX/iOS/tvOS systems
Just change the line finalPackage and listOfFallbacks to reflect what you want to build and fall back on.
If you are building for tvOS, open the RetroArch_iOS11.xcodeproj file once beforehand in xcode and set the team, etc. to yourself. Then make sure that eraseLocalChanges and manualSigning are set to False.
THE SCRIPT
import os, subprocess, glob, shutil, git, tinydb, re
from datetime import datetime
libretroSuperDir = os.path.expanduser("~/retroarch-stuff/libretro-super")
retroarchDir = os.path.expanduser("~/retroarch-stuff/libretro-super/retroarch")
outputDir = os.path.expanduser("~/retroarch-stuff/output")
coreArchiveDir = os.path.expanduser("~/retroarch-stuff/core-archive")
developerName = "DeveloperNameHere"
profileName = "ProfileNameHere"
teamName = "yourTeamNameHere"
finalPackage = "ios-arm64"
listOfFallbacks = ["ios10", "ios9"]
extension = "dylib"
debug = False
manualSigning = True
eraseLocalChanges = True
eraseBeforeFetching = False
directoryNameTranslation = {}
directoryNameTranslation["ios-theos"] = "theos_ios"
finalBuildTranslation = {}
finalBuildTranslation["ios-arm64"] = {"modules": "apple/iOS", "projectName": "apple/RetroArch_iOS11.xcodeproj", "targetName": "RetroArchiOS11", "endPlacement": "apple"}
finalBuildTranslation["tvos-arm64"] = {"modules": "apple/tvOS", "projectName": "apple/RetroArch_iOS11.xcodeproj", "targetName": "RetroArchTV", "endPlacement": "apple"}
finalBuildTranslation["ios10"] = {"modules": "apple/iOS", "projectName": "apple/RetroArch_iOS10.xcodeproj", "targetName": "RetroArchiOS10", "endPlacement": "apple"}
finalBuildTranslation["ios10"] = {"modules": "apple/iOS", "projectName": "apple/RetroArch_iOS9.xcodeproj", "targetName": "RetroArch iOS9", "endPlacement": "apple"}
finalBuildTranslation["ios"] = {"modules": "apple/iOS", "projectName": "apple/RetroArch_iOS8.xcodeproj", "targetName": "RetroArch iOS8", "endPlacement": "apple"}
finalBuildTranslation["ios-theos"] = {"modules": "apple/iOS", "projectName": "apple/RetroArch_iOS6.xcodeproj", "targetName": "RetroArch iOS6", "endPlacement": "apple"}
recipeTable = {}
recipeTable["ios-arm64"] = "apple/cores-ios-arm64-generic"
stupidInconsistentNameTranslation = {}
stupidInconsistentNameTranslation["mesen-s"] = "mesens"
stupidInconsistentNameTranslation["mednafen_pce"] = "mednafen_pce_fast"
stupidInconsistentNameTranslation["mupen64plus_next_gles3"] = "mupen64plus_next"
stupidInconsistentNameTranslation["test"] = "samples"
#stuffToRemove = ["vice_x64sc", "vice_xcbm2", "vice_xpet", "redbook", "bsnes", "flycast_wince", "bsnes_mercury", "snes9x2005_plus", "mednafen_psx_hw", "dosbox_svn_glide", "higan_sfc_balanced", "pcsx_rearmed_interpreter", "higan_sfc"]
stuffToRemove = []
coreArchiveDatabase = tinydb.TinyDB(os.path.join(outputDir, "buildDatabase.json"))
my_env = os.environ.copy()
def updateSomeDir(someDirectory):
someGitDir = git.Repo(someDirectory)
if eraseLocalChanges:
try:
someGitDir.git.clean("-xfd")
someGitDir.git.submodule("foreach", "--recursive", "git", "clean", "-xfd")
someGitDir.git.submodule("foreach", "--recursive", "git", "reset", "--hard")
someGitDir.git.reset("--hard")
someGitDir.git.stash()
someGitDir.git.stash("drop")
except:
pass
try:
someGitDir.git.pull()
someGitDir.git.submodule("update", "--init", "--recursive")
except:
pass
return someGitDir.head.object.hexsha
def file_as_bytes(file):
with open(file, "rb") as toSha:
return hashlib.sha256(toSha.read()).hexdigest()
def safeRemove(someFile):
if os.path.isfile(someFile):
os.remove(someFile)
def translateStupidMisnamedDirectories(someName):
if someName in directoryNameTranslation:
return directoryNameTranslation[someName]
else:
return someName
def runCommandWithDebug(somePopenArray):
with open(os.devnull, "w+") as devnull:
if debug:
result = subprocess.Popen(somePopenArray, env=my_env)
result.wait()
else:
result = subprocess.Popen(somePopenArray, env=my_env, stdout=devnull, stderr=devnull)
result.wait()
def tryToFetch(someBuildOption):
os.chdir(libretroSuperDir)
if someBuildOption in stupidInconsistentNameTranslation:
fetchAndDirectoryCommand = stupidInconsistentNameTranslation[someBuildOption]
else:
fetchAndDirectoryCommand = someBuildOption
theFetchedDirectory = os.path.join(libretroSuperDir, "libretro-" + fetchAndDirectoryCommand)
if eraseBeforeFetching and os.path.isdir(theFetchedDirectory):
shutil.rmtree(theFetchedDirectory)
runCommandWithDebug([os.path.join(libretroSuperDir, "libretro-fetch.sh"), fetchAndDirectoryCommand])
if not os.path.isdir(theFetchedDirectory):
print("Failed to fetch " + someBuildOption + " - RetroArch recipe problem!")
return False
return updateSomeDir(theFetchedDirectory)
def copySuccessfulCores(someBuildOption, someMethodToBuild, queryObject):
print("Successfully built " + someBuildOption + " with " + someMethodToBuild)
#Copy built core(s) to the archive
for someCompiledModule in glob.glob(os.path.join(libretroSuperDir, "dist", translateStupidMisnamedDirectories(someMethodToBuild), "*." + extension)):
copyPath = os.path.join(coreArchiveDir, translateStupidMisnamedDirectories(someMethodToBuild))
if not os.path.isdir(copyPath):
os.mkdir(copyPath)
shutil.copy2(someCompiledModule, copyPath)
safeRemove(someCompiledModule)
coreArchiveDatabase.update({"buildSuccess": True}, (queryObject.coreName == someBuildOption) & (queryObject.buildType == someMethodToBuild))
def tryToBuild(someBuildOption, someMethodToBuild):
print("Attempting to build " + someBuildOption + " with " + someMethodToBuild)
if someBuildOption in stupidInconsistentNameTranslation:
fetchAndDirectoryCommand = stupidInconsistentNameTranslation[someBuildOption]
else:
fetchAndDirectoryCommand = someBuildOption
if debug:
result = subprocess.Popen([os.path.join(libretroSuperDir, "libretro-build-" + someMethodToBuild + ".sh"), fetchAndDirectoryCommand], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
result.wait()
resultStdout, resultStderr = result.communicate()
else:
runCommandWithDebug([os.path.join(libretroSuperDir, "libretro-build-" + someMethodToBuild + ".sh"), fetchAndDirectoryCommand])
checkPath = translateStupidMisnamedDirectories(someMethodToBuild)
queryObject = tinydb.Query()
if len(glob.glob(os.path.join(libretroSuperDir, "dist", checkPath, someBuildOption + "*." + extension))) >= 1:
copySuccessfulCores(someBuildOption, someMethodToBuild, queryObject)
return True
else:
print("Failed to build " + someBuildOption + " with libretro-build-" + someMethodToBuild + ".sh. " + "Trying the buildbot method with " + someMethodToBuild)
my_env["CLEANUP"] = "YES"
my_env["ENABLED"] = "1"
my_env["VERBOSE"] = "1"
my_env["EXIT_ON_ERROR"] = "1"
my_env["SINGLE_CORE"] = fetchAndDirectoryCommand
my_env["CMAKE"] = "cmake"
runCommandWithDebug([os.path.join(libretroSuperDir, "libretro-buildbot-recipe.sh"), os.path.join(libretroSuperDir, "recipes", recipeTable[someMethodToBuild])])
if len(glob.glob(os.path.join(libretroSuperDir, "dist", checkPath, someBuildOption + "*." + extension))) >= 1:
copySuccessfulCores(someBuildOption, someMethodToBuild, queryObject)
return True
else:
print("Failed to build " + someBuildOption + " with " + someMethodToBuild)
if debug:
with open(os.path.join(outputDir, "failedLog.txt"), "a+") as failedLog:
for someStdOutLine in resultStdout.split("\n"):
if len(someStdOutLine) >= 1:
failedLog.write(someBuildOption + " (" + someMethodToBuild + "): " + someStdOutLine + "\n")
for someStdErrLine in resultStdout.split("\n"):
if len(someStdOutLine) >= 1:
failedLog.write(someBuildOption + " (" + someMethodToBuild + "): " + someStdOutLine + "\n")
coreArchiveDatabase.update({"buildSuccess": False}, (queryObject.coreName == someBuildOption) & (queryObject.buildType == someMethodToBuild))
return False
def cleanup():
for someExistingCore in glob.glob(os.path.join(libretroSuperDir, "dist/*/*." + extension)):
safeRemove(someExistingCore)
for someOldModule in glob.glob(os.path.join(retroarchDir, "pkg/*/*/modules/*." + extension)):
safeRemove(someOldModule)
safeRemove(os.path.join(outputDir, "failedLog.txt"))
for someOldBuild in glob.glob(os.path.join(retroarchDir, "pkg/*/build/*")):
shutil.rmtree(someOldBuild)
for someOldIpa in glob.glob(os.path.join(outputDir, "*.ipa")):
safeRemove(someOldIpa)
def init():
updateSomeDir(libretroSuperDir)
os.chdir(libretroSuperDir)
buildScripts = glob.glob(os.path.join(libretroSuperDir, '*.sh'))
for someScript in buildScripts:
os.chmod(someScript, 0o775)
listOfRecipes = set(glob.glob(os.path.join(libretroSuperDir, "recipes/*/cores-*"))) - set(glob.glob(os.path.join(libretroSuperDir, "recipes/*/cores-*.conf")))
masterRecipeSet = set()
for someRecipe in listOfRecipes:
with open(someRecipe) as f:
for line in f.readlines():
if line.split(' ')[0] != "\n":
masterRecipeSet.add(line.split(' ')[0])
with open(os.path.join(libretroSuperDir, "rules.d/core-rules.sh")) as coreRules:
for line in coreRules:
if re.findall(r"register_module core", line):
result = re.sub('.*"(.*)".*', r"\1", line, flags=re.DOTALL)
masterRecipeSet.add(result)
for someDuplicate in stuffToRemove:
if someDuplicate in masterRecipeSet:
masterRecipeSet.remove(someDuplicate)
return sorted(masterRecipeSet, key=str.lower)
def updateTheDatabase(toBuild, buildFor, coreGitVersion):
queryObject = tinydb.Query()
lastBuiltCoreInfo = coreArchiveDatabase.search((queryObject.coreName == toBuild) & (queryObject.buildType == buildFor))
sameVersion = False
successfulBuild = False
if len(lastBuiltCoreInfo) > 0:
if lastBuiltCoreInfo[0]["latestHash"] != coreGitVersion:
coreArchiveDatabase.update({"latestHash": coreGitVersion}, (queryObject.coreName == toBuild) & (queryObject.buildType == buildFor))
else:
sameVersion = True
successfulBuild = lastBuiltCoreInfo[0]["buildSuccess"]
else:
coreArchiveDatabase.insert({"coreName": toBuild, "latestHash": coreGitVersion, "buildSuccess": -1, "buildType": buildFor})
if (sameVersion and successfulBuild == True):
return 1
elif (sameVersion and successfulBuild == False):
return 2
else:
return 3
def tryToBuildTheCores(masterRecipeSet):
for toBuild in masterRecipeSet:
coreGitVersion = tryToFetch(toBuild)
alreadyBuilt = updateTheDatabase(toBuild, finalPackage, coreGitVersion)
if not alreadyBuilt == 1:
if alreadyBuilt == 2:
print("You've already tried and failed to build this version of " + toBuild + " for " + finalPackage)
success = False
else:
success = tryToBuild(toBuild, finalPackage)
if not success:
for someFallback in listOfFallbacks:
alreadyBuiltFallback = updateTheDatabase(toBuild, someFallback, coreGitVersion)
if not alreadyBuiltFallback == 1:
if alreadyBuiltFallback == 2:
print("You've already tried and failed to build this version of " + toBuild + " for " + someFallback)
success = False
else:
success = tryToBuild(toBuild, someFallback)
if success:
break
else:
print("You've already built this version of " + toBuild + " for " + someFallback + ". No need to rebuild...")
break
else:
print("You've already built this version of " + toBuild + " for " + finalPackage + ". No need to rebuild...")
def preppingForFinalBuild():
print("Okay we're all done building cores... Now we move some files...")
updateSomeDir(retroarchDir)
if not os.path.isdir(os.path.join(retroarchDir, "pkg", finalBuildTranslation[finalPackage]["modules"], "modules")):
os.mkdir(os.path.join(retroarchDir, "pkg", finalBuildTranslation[finalPackage]["modules"], "modules"))
#Copy archive to the modules for building in reverse
for someFallback in reversed(listOfFallbacks):
for someCompiledModule in glob.glob(os.path.join(coreArchiveDir, translateStupidMisnamedDirectories(someFallback), "*." + extension)):
shutil.copy2(someCompiledModule, os.path.join(retroarchDir, "pkg", finalBuildTranslation[finalPackage]["modules"], "modules"))
#Copy the final archive over to the modules for building
for someCompiledModule in glob.glob(os.path.join(coreArchiveDir, translateStupidMisnamedDirectories(finalPackage), "*." + extension)):
shutil.copy2(someCompiledModule, os.path.join(retroarchDir, "pkg", finalBuildTranslation[finalPackage]["modules"], "modules"))
def finalBuild():
print("And now, we build RetroArch itself...")
os.chdir(retroarchDir)
if manualSigning:
runCommandWithDebug(["xcprovisioner", "--target", finalBuildTranslation[finalPackage]["targetName"], "--configuration", "Release", "--specifier", profileName, "--identity", developerName, "--team", teamName, "--project", os.path.join(retroarchDir, "pkg", finalBuildTranslation[finalPackage]["projectName"])])
runCommandWithDebug(["xcodebuild", "-target", finalBuildTranslation[finalPackage]["targetName"], "-configuration", "Release", "-project", os.path.join(retroarchDir, "pkg", finalBuildTranslation[finalPackage]["projectName"])])
print("And now, we build your IPA...")
proc = subprocess.Popen(["git", "rev-parse", "HEAD"], stdout=subprocess.PIPE)
proc.wait()
gitVersion = proc.communicate()[0].replace("\n", "")
currentDate = datetime.utcnow().strftime("%Y_%m_%d_%I_%M_%p")
finalBuildDirectory = glob.glob(os.path.join(retroarchDir, "pkg", finalBuildTranslation[finalPackage]["endPlacement"], "build", "Release*/*.app"))[0]
os.mkdir(os.path.join(outputDir, "Payload"))
shutil.move(finalBuildDirectory, os.path.join(outputDir, "Payload"))
filename = os.path.join(outputDir, finalPackage + "-" + gitVersion + "-" + currentDate + ".ipa")
os.chdir(outputDir)
runCommandWithDebug(["zip", "-r", filename, "Payload"])
shutil.rmtree(os.path.join(outputDir, "Payload"))
print("All finished! Your IPA file is located at " + filename)
cleanup()
tryToBuildTheCores(init())
preppingForFinalBuild()
finalBuild()
