Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AppImage support #8019

Closed
skrzyp opened this issue Feb 23, 2020 · 6 comments
Closed

AppImage support #8019

skrzyp opened this issue Feb 23, 2020 · 6 comments

Comments

@skrzyp
Copy link

skrzyp commented Feb 23, 2020

Version of OpenTTD: 1.9.3
Expected result: Ability to drop a single binary for friends/coworkers in UNIX environment to be able to play together in ad-hoc LAN games without obligatory ceremony of installing/building.
Actual result: Right now you have to use your package manager in OS which pulls plenty of dependencies, or even build the source if your PM provides old or even no version of OpenTTD. This might increase the latency between idea and playback or even create a serious annoyance for Windows/Mac users yelling at UNIX environment or userbase.

Possible Solution

Use AppImage. It's a specification (not a "package manager" or "daemon" or anything) which allows to build standalone, system-independent executables that run on every modern Linux system.

Compared to snap/flatpak it doesn't enforce any 3rd-party tooling on destination machine, and can be considered like Mac DMGs or Windows ("portable") EXEs.

Under the hood it's just a SquashFS loopback filesystem with specified contents wrapped into ELF binary with actual code on top which mounts the FS as unprivileged user and spawn the application in it. It doesn't provide sandboxing (can be done with firejail on demand) in terms of security, just bundles everything what game/application needs in a single binary, thus being quite ideal for OSS games or "larger" self-contained applications like LibreOffice, Krita, and so on. Definitely not a thing for CLI apps, system tools and utiltiies (but no one prevents anyone from building these and they'll work just fine)

Steps to reproduce

At first I wanted to create a Pull Request for that, adding AppImage as a "bundle" (in Makefile.bundle.in) but I found the OpenTTD's build system quite complicated (maybe overcomplicated with that infamous GNU Autotools orchestra). So even if I can rip it out and inject my own inventions here and there, it would be a bit rude and unrespectful to do that.

Instead, I'd like to share the steps which I manually did to achieve working AppImage add my thoughts on it and how OpenTTD behaves:

  1. I set up a "most common denominator" OS environment at the time, which I think is Ubuntu 18.04.2 LTS. You can build AppImages on whatever OS you have, but doing so on oldest one you have is recommended.
    1.1. I've done that in Docker (Podman) container, but bare-metal, CI, chroot or VM will do.
    1.2. For reference, you can recreate the same environment in Docker using the following:
# "bionic" is codename for Ubuntu 18.04 LTS
docker run -v "$OPENTTD_SOURCES_DIR:/openttd" -ti ubuntu:bionic bash --login
  1. Got OpenTTD sources from Git in /openttd dir, checked out to 1.9.3 tag
  2. Installed the following Ubuntu packages resembling the dependencies needed for game:
build-essential libsdl1.2-dev zlib1g-dev liblzo2-dev libfluidsynth-dev libfreetype2-dev libicu-dev libfreetype6-dev libfontconfig1-dev libxdg-basedir-dev
  1. Configured the Autotools in the following way:
./configure --prefix-dir=/usr --binary-dir=bin 
  1. Did make -j8 to build the game.
  2. After I built the game, I needed it installed into the folder which can be directly used by AppImage's AppDir:
make install INSTALL_DIR="appdir"
  1. It's fine for now. We didn't needed any sort of source patching and no change in build systems. Time to build AppImage itself.
  2. The AppImage can be build manually, but there's a reference tool called LinuxDeploy which can be used to build AppImages without deep integration into software.
    8.1. The release binary is an AppImage itself, so you can curl or wget it in CI pipeline and use it right there
    8.2. The only thing linuxdeploy does in the actual openttd binary is RPATH to make it look for shared libraries in AppDir first, which is the probably the only "trick" making the AppImages work. Dead simple and quite elegant.
    8.3. I call linuxdeploy like this:
# "VERSION" is needed to indicate a version of software we build, in other cases it'll use a git/hg/cvs hash and store it in target filename and metadata (optional)
VERSION=1.9.3 ./bin/linuxdeploy--appdir appdir/ --output appimage
  1. linuxdeploy simply calls ldd on openttd, copies required *.sos for it into AppDir (with some exclusions, more on it later), finds *.desktop file in AppDir to lookup metadata and icon, packs that into SquashFS+payload and… its done!
    9.1. In CIs without fuse, you can run linuxdeploy (and other AppImages) via ./linuxdeploy --appimage-extract-and-run
  2. We can take a look at it:
$ ls *AppImage
OpenTTD-1.9.3-x86_64.AppImage
$  file OpenTTD-1.9.3-x86_64.AppImage
OpenTTD-1.9.3-x86_64.AppImage: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, for GNU/Linux 2.6.18, stripped
$ ldd OpenTTD-1.9.3-x86_64.AppImage
        not a dynamic executable

TL;DR version

docker run -v $HOME/src/openttd:/openttd -ti ubuntu:bionic bash --login
cd /openttd
apt install # ... dependencies ... #
./configure --prefix-dir=/usr --binary-dir=bin 
make -j8
make install INSTALL_DIR="appdir"
wget # ... linuxdeploy URL #
VERSION=1.9.3 ./linuxdeploy --appdir appdir --output appimage
file  OpenTTD-1.9.3-x86_64.AppImage

Issues I found

These are mostly cosmetic or formal ones, which can probably be addressed more officialy by someone not being dumb as me right now:

  • By default, openttd will fail with error about missing languages. strace shows it also misses all data in from its basedir.
    • It's quite understandable, as openttd will look up its data in $PWD or global directory (/usr/share) or local user dirs. AppImage does not fake or remap directories in running process and openttd can't figure out it has all its data stored in /tmp/.mount_<appimage_name>_<random uuid>/usr/share/games/openttd.
    • On the other hand, AppImage provides the "$APPDIR" environment variable for all child processes, which stores the absolute AppDir path. That's nice, but we don't want patch the game exclusively for AppImages, right?
    • By default, linuxdeploy symlinked openttd as AppRun which is the default entrypoint for AppImage. It's nice, but it don't need to be like that, the AppRun can be a script to so we don't need to fake binary paths, just use this as AppRun:
#!/usr/bin/env bash
cd "$APPDIR/usr/share/games/openttd" # go to basedir 
"$APPDIR/usr/bin/openttd" $* # "$*" is to pass argument list from AppImage binary call
  • I had to bundle OpenGFX/OpenMSX/OpenSFX inside the AppImage (in $APPDIR/usr/share/games/openttd)

    • This is my major pain point of the whole solution, and I can't figure it out.
    • The OpenTTD launches fine without it and asks user for BaNaNaS download, but after I click on "yes, download" button it only shows the progressbar for a fraction of second and it disappears, with completely gray game window.
    • I'm not sure if that's related to AppImage or OpenTTD or my Internet connection, but I'm quite sure the AppImage does not limit/sandbox/prevent filesystems for running binaries
      • As I said it's possible as an option with firejail, but should be invoked by user explicitly
      • However, you can make it more "portable", and let AppImage binaries not to pollute the host dirs
        • If there's a folder called <appimage filename without extension>.home in your $PWD or AppImage folder, it's being used as a $HOME environment variable. Simple and elegant solution.
        • Same goes for <appimage filename>.config, which is used as $XDG_CONFIG_HOME and that's why I used libxdg-basedir in build system
    • Probably that's easy fix or I've just accidently exposed a bug in BaNaNaS client downloader, but on the other side I would like to have a "full standalone" binary with assets combined to maximum convenience for external users, as long as it remains compatible with OpenTTD assets' license
  • I had issues with ICU libraries ABI change:

    • The linuxdeploy excludes the libicu*.so from integration into AppDir by default, explaining that's a "too internal" system library and it's not often directly called
    • The OpenTTD build system has a --static-icu option which I tried to use, but Ubuntu LTS doesn't have libsicu*.a libs available. It has, however, libicu*.a static libs in its libicu-dev packages, so I assume it's just matter of linker configuration and some Autotools magic
    • On the other hand, you can explicitly tell linuxdeploy to bundle these libs with you, and resulting AppImage has been proven working for us on a small network game this evening. For reference, you can call it like that:
# you don't need to address .so.XX.y directly, linuxdeploy is a little smarter about symlinks and ABIs
./linuxdeploy \
  --appdir appimage/ \
  -output appimage  \
  --library /usr/lib/x86_64-linux-gnu/libiculx.so  \
  --library /usr/lib/x86_64-linux-gnu/libicutu.so  \
  --library /usr/lib/x86_64-linux-gnu/libicuuc.so  \
  --library /usr/lib/x86_64-linux-gnu/libicuio.so  \
  --library /usr/lib/x86_64-linux-gnu/libicui18n.so  \
  --library /usr/lib/x86_64-linux-gnu/libicudata.so  \
  --library /usr/lib/x86_64-linux-gnu/libicu-le-hb.so
  • MIDI music in OpenTTD is non-portable
    • This is more a development / architectural problem than distribution, but it directly affets this issue.
    • Both FluidSynth and Timidity++ require a daemon running on host to play music which is quite problematic for AppImage and its stateless nature.
    • I would like to suggest some library-based software MIDI synthesizer, maybe WildMIDI or anything like that.
    • You can surely play without music, but I find it lovely and even bringing some nostalgia, as I play OpenTTD for around 15 years.

Why I'm doing this here, instead of maintaining it on my own?

AppImages are meant to be done by upstream. Why?

  • You can sign it to assume it's going from legitimate source and make users trust it. It's very optional, but welcoming for users and auxiliary tools can pick that up
  • You can provide update information. The AppImage can't update itself, but it can store HTTP/Github Releases/ZSync/FTP URL in metadata to be used in the following ways:
    • By auxiliary tooling like appimaged, which is optional but handy in a fleet of AppImages
    • By script in AppRun
    • By application itself in libappimage library if you're so kind to support that. The library can also determine if you're running in AppImage or not, so you can use other search paths for baseset.
  • You can put it in your CI pipelines, show the link on your website and call it a day: For 99% of the time, they act completely the same as regular Linux binaries, they're just called from lesser-common location and use bundled shared libs.

A little disclaimer

If you find this like forcing or advertising a new technology, it's not. I'm not affiliated in @probonopd or AppImage project in any way. The whole reasoning is that I wanted to quickly share an OpenTTD build for many different platforms and make it as most easy as possible, without polluting the host system. It worked pretty well and I've been looking forward to integrate such distribution form to other users.

P.S.: Dedicated server build can be done that way too, making them deployable in seconds.
P.P.S: Just stumbled upon the fact that your "cousin project" OpenRCT2, already does this and shares the AppImage build on their downloads page which is cool.

@nielsmh
Copy link
Contributor

nielsmh commented Feb 23, 2020

Don't put too much thought into integrating anything with the build system, it's being replaced with CMake soon.

@probonopd
Copy link

Maybe the AppImage creation should be revisited once the project has been migrated to the new CMake based build system, just to avoid double work.

@azubieta
Copy link

Hello folks!
A big fan of OpenTTD here (also a contributor of the AppImage project). I would love to help you packaging the game into an AppImage. Please let me know when the cmake migration is completed.
Cheers

@FLHerne
Copy link
Contributor

FLHerne commented Sep 16, 2020

The CMake transition was committed a while ago, in commit 56d54cf (and has been fixed-up in various ways since).

Thank you for the offer of help! It might be best if you join the IRC channel -- #openttd on irc.oftc.net , most active in EU evenings/weekends -- to discuss it. Or stick with GitHub comments, that works too. :-)

@azubieta
Copy link

azubieta commented Sep 25, 2020

I was a bit lazy to compile OpenTTD from sources, instead I used the latest binary you publish to build this AppImage. Nevertheless the recipe should work if you build it from sources and install it into an AppDir.

If you help me setup a workflow for the build, I could add the AppImage packaging bit 😃

Here is the link to my project: https://github.com/azubieta/openttd-appimage

azubieta added a commit to azubieta/OpenTTD that referenced this issue Oct 16, 2020
azubieta added a commit to azubieta/OpenTTD that referenced this issue Oct 16, 2020
azubieta added a commit to azubieta/OpenTTD that referenced this issue Oct 16, 2020
azubieta added a commit to azubieta/OpenTTD that referenced this issue Oct 16, 2020
azubieta added a commit to azubieta/OpenTTD that referenced this issue Oct 16, 2020
Note on the appiamge-builder usage:
Patching the deployed files is not supported yet, therefore we must
split it's execution and patch it manually.

This feature can be tracked at:
AppImageCrafters/appimage-builder#50
azubieta added a commit to azubieta/OpenTTD that referenced this issue Oct 16, 2020
Note on the appiamge-builder usage:
Patching the deployed files is not supported yet, therefore we must
split it's execution and patch it manually.

This feature can be tracked at:
AppImageCrafters/appimage-builder#50
azubieta added a commit to azubieta/OpenTTD that referenced this issue Oct 23, 2020
azubieta added a commit to azubieta/OpenTTD that referenced this issue Oct 23, 2020
azubieta added a commit to azubieta/OpenTTD that referenced this issue Oct 24, 2020
@TrueBrain
Copy link
Member

OpenTTD won't supply AppImage for now; but other awesome people in the community are supplying it: https://www.appimagehub.com/p/1425360/ ! See #8329 why we are not supplying it ourselves for now.

What we do have these days, is a linux-generic bundle. This contains all the binary files needed to run OpenTTD on any recent (read: 10 years) Linux platform. You can find them here: https://www.openttd.org/downloads/openttd-nightlies/latest.html

Both combined should address this issue, I believe :) So going to close it for now, but let me know if there is anything else to this issue! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants