Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add: support for emscripten (play-OpenTTD-in-the-browser) #8355

Merged
merged 6 commits into from Dec 15, 2020

Conversation

TrueBrain
Copy link
Member

@TrueBrain TrueBrain commented Dec 5, 2020

(depends on OpenTTD/bananas-server#39 before "online content" will work out-of-the-box)

(#7510 was used as inspiration for this PR. Different choices were made, as this PR is more self-contained and doesn't rely on custom Dockerfiles; there is nothing wrong with the approach taken in #7510, I just prefer a different one myself)


Running

To compile, first make a "build-host" folder in which you build "tools" for the host (we run this in the Docker container, to avoid library mismatches):

mkdir build-host
docker run -it --rm -v $(pwd):$(pwd) -u $(id -u):$(id -g) --workdir $(pwd)/build-host emscripten/emsdk cmake .. -DOPTION_DEDICATED=ON
docker run -it --rm -v $(pwd):$(pwd) -u $(id -u):$(id -g) --workdir $(pwd)/build-host emscripten/emsdk make -j5 tools

Next, use the following line in a new build folder to generate the WASM / JavaScript / HTML files:

mkdir build
docker run -it --rm -v $(pwd):$(pwd) -u $(id -u):$(id -g) --workdir $(pwd)/build emscripten/emsdk emcmake cmake .. -DHOST_BINARY_DIR=$(pwd)/build-host -DCMAKE_BUILD_TYPE=Release -DOPTION_USE_ASSERTS=OFF
docker run -it --rm -v $(pwd):$(pwd) -u $(id -u):$(id -g) --workdir $(pwd)/build emscripten/emsdk emmake make -j5

This produces the following files:

openttd.data
openttd.html
openttd.js
openttd.wasm

Which combined are the full game, which you can run in a browser. You do need a HTTP server to run it, even if used locally. Browsers don't allow XHR requests with file://. (pro-tip: python3 -m http.server)

Reasoning

The idea behind adding Emscripten is that we could produce these kind of binaries on Pull Request (after a trusted contributor okays it, to prevent abuse), so new features can be tested quickly, and bug-fixes can be reviewed all from within the browser. This Pull Request puts down the groundwork for this. It is also the reason no other external libraries are used, like for example there is no support for LZMA compression (emscripten doesn't have a port available for this).

If we would add LZMA support, we could extend this as a "Play Now" features on our website (and for example, reddit could do the same, and CityMania, or-whatever community wants to drag in players quickly). It creates a smooth and simple experience for new players to jump in.

Design choices

  • The SDL2 loop unrolling is strictly seen not required: emscripten can also run in a so called asyncify mode, where a sleep does a "context switch" to Javascript world, allowing for the canvas to be drawn. There are two significant issues with this approach:
  1. with asyncify, SDL2 is not v-sync'd, meaning mouse movement etc feels laggy and distorted. Frames are drawn when-ever. With the current Pull Request, it is v-sync'd, that is: when ever the browser indicates an animation frame should happen, OpenTTD does a tick and draws the screen. This feels significant less laggy.
  2. it has to unwind/rewind the stack, which is not always as trivial as it sounds. This can cause lockups (for example when you pause), or crashes. These problems are all solvable, but requires significant effort.
  • Only SDL2 is supported, although supporting SDL is also possible. But Emscripten support #7510 already mentions performance is worse with SDL, so no effort was taken to support it
  • 32bpp is used over 8bpp, as the performance is noticeable better with 32bpp.
  • The mouse is using relative-mode, as you can only do a "pointer lock" with emscripten, and not change the position of the cursor like we normally do.
  • OpenGFX is automatically loaded if it is detected that you haven't before. It is stored in the Indexed DB in your browser. This means that someone that plays more often, doesn't have to download 6MB every single time, but only once.
  • Emscripten has 2 bugs in their network stack that are blocking for us. Both are an upstream issue:
  1. accept(), getsockname(), getpeername(), and recvfrom() don't set addrlen in emulated mode emscripten-core/emscripten#12996
  2. getaddrinfo() doesn't set sin_zero in emulation emscripten-core/emscripten#12998
    Also applied is a commit that adds workarounds in our codebase for those bugs; they are very ugly, and should be considered a hack. But for now, it works, and once emscripten caught up with these issues, the hacks can be removed.
  • Listing servers from master-server and adding servers manually is disabled. To join a random server, someone needs to run an open proxy where WebSocket is translated into UDP/TCP from/to the game server. openttd.org will not be doing this because of the security aspects of such open proxy. As such, the buttons for this are disabled by default. However, communities like CityMania, Reddit, Discord, .. can build their own web-version with hard-coded servers in them, where they host themself a WebSocket proxy for those people to connect via. This can be done in pre.js and is documented in comments (openttd_server_list).
  • Start server is disabled, as you cannot run a server over WebSocket. It would be possible via a proxy to do this, but it feels a bit weird to allow this.

Known-bugs

  • Fast-Forward doesn't really work. That is to say, it strongly depends on your screens how fast the game can run. Because we are sync'd with Animation Frames, which browsers normally trigger on the refresh frequency of your screen, we cannot go faster than the browser allows us. If you have 144 Hz screens, it means Fast-Forward can go up to 144 fps. Using asyncify does not resolve this issue, but still restricts us to ~120 fps.

Example of preview

TrueBrain#19

@James103
Copy link
Contributor

James103 commented Dec 5, 2020

Would it be possible to add support for reading and writing data to/from the C:\Users\[user]\Documents\OpenTTD folder in addition to the browser's local storage? That way, you could have access to saved games and NewGRFs from your local installation (if you have any) rather than just having to use the browser's local storage.

@TrueBrain
Copy link
Member Author

TrueBrain commented Dec 5, 2020

Would it be possible to add support for reading and writing data to/from the C:\Users\[user]\Documents\OpenTTD folder in addition to the browser's local storage? That way, you could have access to saved games and NewGRFs from your local installation (if you have any) rather than just having to use the browser's local storage.

Not as far as I know. Browsers sandbox everything, and I would assume security wouldn't let access to the users disks. This is also one of the reasons I would love to add Cloud Saves, as that would solve this too :)

What is possible, how-ever, is to allow uploading of a savegame, and I am sure it is also possible to make a "Download savegame" button which uses the normal browser flow to download a file. But that is not really user-friendly :) (and currently also very low on the list to check out)

@James103
Copy link
Contributor

James103 commented Dec 5, 2020

Note that storing too much data in local storage may result in a warning or error from the browser, and the excess data may not be written correctly. The limit in this case can be as low as 5 MB (or even lower).

@TrueBrain
Copy link
Member Author

TrueBrain commented Dec 5, 2020

Note that storing too much data in local storage may result in a warning or error from the browser, and the excess data may not be written correctly. The limit in this case can be as low as 5 MB (or even lower).

https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Browser_storage_limits_and_eviction_criteria

There's also another limit called group limit — this is defined as 20% of the global limit, but it has a minimum of 10 MB and a maximum of 2 GB.

But local storage is an LRU, and files can be evicted for many reasons. But for now, it will do fine. But here too, Cloud Saves are kinda mandatory to really make this shine :)

@TrueBrain TrueBrain force-pushed the emscripten branch 3 times, most recently from 0800b47 to 9278e9f Compare December 6, 2020 15:00
src/settings.cpp Outdated Show resolved Hide resolved
@TrueBrain TrueBrain force-pushed the emscripten branch 5 times, most recently from bc94a9a to 9538d61 Compare December 7, 2020 20:56
@Milek7 Milek7 mentioned this pull request Dec 7, 2020
@TrueBrain TrueBrain added preview This PR is receiving preview builds and removed preview This PR is receiving preview builds labels Dec 8, 2020
@TrueBrain TrueBrain force-pushed the emscripten branch 7 times, most recently from b92d1a5 to b514738 Compare December 9, 2020 16:16
@TrueBrain TrueBrain marked this pull request as ready for review December 9, 2020 16:28
@TrueBrain TrueBrain force-pushed the emscripten branch 3 times, most recently from 52ba48d to 424bb55 Compare December 9, 2020 16:33
src/network/core/config.h Outdated Show resolved Hide resolved
@TrueBrain TrueBrain force-pushed the emscripten branch 2 times, most recently from 6d8b697 to b33c9f2 Compare December 14, 2020 10:59
@TrueBrain TrueBrain added candidate: yes This Pull Request is a candidate for being merged needs review size: large This Pull Request is large in size; review will take a while labels Dec 14, 2020
@TrueBrain TrueBrain force-pushed the emscripten branch 2 times, most recently from 0043db8 to 9954e9b Compare December 14, 2020 19:40
Copy link
Member

@LordAro LordAro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Told you i'd find things :)

.github/workflows/preview_build.yml Outdated Show resolved Hide resolved
.github/workflows/preview_build.yml Show resolved Hide resolved
.github/workflows/preview_build.yml Show resolved Hide resolved
.github/workflows/preview_build.yml Show resolved Hide resolved
CMakeLists.txt Show resolved Hide resolved
src/network/core/config.h Outdated Show resolved Hide resolved
src/network/core/os_abstraction.h Show resolved Hide resolved
src/network/network_gui.cpp Show resolved Hide resolved
src/saveload/saveload.cpp Outdated Show resolved Hide resolved
src/video/sdl2_v.cpp Show resolved Hide resolved
@TrueBrain TrueBrain force-pushed the emscripten branch 4 times, most recently from da07f25 to 47e7d4d Compare December 14, 2020 23:58
This commit prepares for the next commit, as Emscripten needs to
have a way to trigger a single iteration of the main loop. To
keep the real changes more clear, this commit only unrolls the
loop, and makes no changes to the logic itself.
TrueBrain and others added 5 commits December 15, 2020 12:58
Emscripten compiles to WASM, which can be loaded via
HTML / JavaScript. This allows you to play OpenTTD inside a
browser.

Co-authored-by: milek7 <me@milek7.pl>
This mode doesn't wrap the mouse constantly, but requests SDL
to lock the mouse pointer. This is needed, as with Emscripten
you are not allowed to change the mouse poisition (only to lock
it into place).
…wser

When a developer attaches the "preview" label, a build is created
and published on https://preview.openttd.org/. After that, new
pushes to the PR are automatically build (as long as the "preview"
label exists).

If a non-developer attaches the "preview" label, it will be
removed.
Instructions now suggest using build-host, etc. It is easier to
just ignore all build* folders.
Copy link
Member

@LordAro LordAro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh my.

@TrueBrain TrueBrain merged commit d6c54e7 into OpenTTD:master Dec 15, 2020
@TrueBrain TrueBrain deleted the emscripten branch December 15, 2020 14:46
@embeddedt
Copy link
Contributor

@TrueBrain Question: is it necessary to disable the "Add server" button on Emscripten? I understand that it can obviously only work with WebSocket-based servers, but what if the URL for my WebSocket server is unpredictable? It would be great to have a way of changing it without needing to rebuild the client every time.

@TrueBrain
Copy link
Member Author

The main issue is the common user. If we leave it enabled, they will be confused thinking they can join normal servers with it. They cannot. Ideally we would have WebSocket enabled for all servers. Or that the masterserver returns a list of servers that do have a WebSocket proxy. But we are far away from that :D

I can see two ways for you to solve your problem in the here and now:

  1. use a dynamic DNS provider, making your address predictable.
  2. from what I read, you are already building your own client: enable it again. Just make sure your users understand they can only use it to join your server; not some random other one ;)

This is also one of the reasons we still don't have a "Play Now" button on the frontpage. There is more work to do for Emscripten before it can go mainstream. Currently it really is to provide previews in Pull Requests :) PRs to improve that situation are very welcome ;)

@embeddedt
Copy link
Contributor

Makes sense. Thanks for the quick reply!

For now I will just re-enable the button. However, I would like to help work on a better solution if possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
candidate: yes This Pull Request is a candidate for being merged needs review size: large This Pull Request is large in size; review will take a while
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants