Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Game crashes trying to sell huge wagon chain #8295

Closed
VerlorenMind opened this issue Aug 12, 2020 · 19 comments
Closed

Game crashes trying to sell huge wagon chain #8295

VerlorenMind opened this issue Aug 12, 2020 · 19 comments
Labels
bug Something isn't working

Comments

@VerlorenMind
Copy link

VerlorenMind commented Aug 12, 2020

Version of OpenTTD

1.10.3 Windows 10 64-bit

Expected result

The game does not crash.

Actual result

The game crashes after roughly two months in-game. The exact date does not matter - you can wait for a monthly autosave, load it, and the game will crash in two months.

Steps to reproduce

  1. Load the save.
  2. Wait for two months to pass.
  3. Get the crash.

Granted I have NewGRFs installed, but the game shouldn't crash like that. In debug mode, there is no relevant errors prior to crash.

Savegame

@nielsmh
Copy link
Contributor

nielsmh commented Aug 12, 2020

Please attach the generated crash log to the issue too.

@VerlorenMind
Copy link
Author

@nielsmh Where is this log located on Windows?

@nielsmh
Copy link
Contributor

nielsmh commented Aug 12, 2020

By default, in your Documents\OpenTTD\ folder. The path is usually shown in the crash dialog.

If you get something other than OpenTTD's own crash dialog up, then please screenshot or more detailed describe the crash.

@VerlorenMind
Copy link
Author

@nielsmh There is no dialog and no log in the folder you mentioned. The game hangs for a second, and then closes. No dialogs, no errors, nothing.

@nielsmh
Copy link
Contributor

nielsmh commented Aug 12, 2020

Okay I reproduce it with that save. It seems to hit 100% CPU use (on a single core) and allocate a bunch more memory while it's doing that whatever.

@nielsmh
Copy link
Contributor

nielsmh commented Aug 12, 2020

This appears to be caused by an AI. When I collect a memory dump while it's frozen, I get a state where player 3 (RailwAI v25) is selling a rail wagon. I'm not sure why it hangs doing that, but two separate dumps taken at random times during the freeze points to the same place in the code with near-identical stacks.

Around here is where it is in my two dumps, but this may not be the direct cause in itself.
https://github.com/OpenTTD/OpenTTD/blob/1.10.3/src/vehicle.cpp#L2663-L2669

@JGRennison
Copy link
Contributor

Interestingly the depot at 130x784 (Bundtown, by train 8) has a free wagon chain of what appears to be 27,126 wagons.

@VerlorenMind
Copy link
Author

Interesting. In attempt to salvage the save game, I tried to delete 3rd AI via stop_ai 3. The game crashed immediately in the similar fashion.

@VerlorenMind
Copy link
Author

Apparently the 3rd AI player has bought an ungodly amount of wagons and then trying to sell them all at once, causing the crash.

@James103
Copy link
Contributor

James103 commented Aug 12, 2020

The game stack overflows upon loading the attached save and doing any of the following (all lead to similar crash reports):

  1. Run console command stopai 3, crashing the game in the process.
  2. Switch company cheat to company 3, then scrollto 130x784 and click the depot there. Click the "sell all vehicles" button and confirm. This crashes the game.
  3. Switch company cheat to company 3, then scrollto 130x784 and click the depot there. Drag one of the wagons anywhere else. This does not crash, but hangs the game for a few seconds.
  4. Switch company cheat to company 3, then scrollto 130x784 and click the depot there. Drag one of the wagons onto the 'sell wagon chain' button. This crashes the game.
  5. Let company 3 go bankrupt (max out loan, buy as many vehicles as you can that are expensive to run, and fast-forward a few months). The game will crash as soon as it ticks over to August 1991. Note that only part of the company's assets were sold off (including some, but not all of the company's vehicles) and that the company gains £4.311 quintillion (2^62) when the emergency savegame is reloaded.
Crash reason:
 Exception:  C00000FD

Within context:
     0: Vehicle::PreDestructor: veh: 13742: (Train 0, c:2, st:W, vs:HD, vf:, vcf:, gvf:, tf:, trk: 0x80, [tile: C4082 (130 x 784), type: 10 (MP_RAILWAY), height: 01, data: 02 0072 00 01 C2 00 00 0001])
     1: DoCommandP: tile: 0 (0 x 0), p1: 0x20002, p2: 0x0, company: 2 (Wreninghall Express), cmd: 0x57 (CmdCompanyCtrl), my_cmd: 1

A possible fix would be to optimize functions that handle wagon chains, preferably removing recursion from the process. If that isn't feasible, a workaround would be to just limit the length of wagon chains to a reasonable number, for example 256 or 1,024 wagons in a single chain.

@VerlorenMind
Copy link
Author

I managed to salvage the save on Linux. I loaded save with the game version for Debian 10 x64, and then used stop_ai 3. The game hanged for a second, but did not crash.

@nielsmh nielsmh changed the title The game crashes after two in-game months Game crashes trying to sell huge wagon chain Aug 14, 2020
@LordAro
Copy link
Member

LordAro commented Jan 1, 2021

Generated a profile of this with callgrind, and generated a nice pretty (huge) image with it

(
valgrind --tool=callgrind ./build/openttd -snull -mnull -g devsaves/bug8295.sav -vnull:ticks=500
&
gprof2dot -f callgrind -s callgrind.out.118484 | dot -Tpng -o bug8295.png
)

bug8295

@JGRennison
Copy link
Contributor

On the ReleaseDisastersTargetingVehicle issue mentioned earlier, the "easy" solution is to not do anything if there are no disaster vehicles at all, which is true the vast majority of the time. (Only strange users/savegames have disasters enabled, and these tend to be small games).
JGRennison/OpenTTD-patches@9354b5c
The performance cost is otherwise proportional to number of vehicles in chain x number of vehicles in game.

In this case you would still run out of stack even if that was removed.
You could probably get around that by iteratively running all the PreDestructors first, then iteratively unlinking the remaining downstream chain from ~Vehicle and deleting each once unlinked.

(Maintaining pre-filtered lists of vehicles by type/etc for faster iteration does have non-trivial performance benefits, and is useful in other areas. Using different pools would require a rethink of ID allocation and the savegame format.)

@LordAro
Copy link
Member

LordAro commented Jan 1, 2021

Wonder if we could do that, but without special-casing disaster vehicles and adding more global variables. An extension to pool (iterators), perhaps?

@michicc
Copy link
Member

michicc commented Jan 1, 2021

I would actually pose that this is a case of premature generalisation.

The only implemented disaster vehicle that targets any vehicles at all targets front road vehicles. Which makes checking for this on deleting a Train vehicle quite pointless.

@TrueBrain TrueBrain added the bug Something isn't working label Jan 1, 2021
@LordAro
Copy link
Member

LordAro commented Jan 1, 2021

Possible side-track, I'm not clear what PreDestructor is for at all. Why can't it just use the object's destructors like a normal class structure? Comment implies something about needing the virtual functions, but it doesn't make much sense to me

@JGRennison
Copy link
Contributor

PreDestructor is a method of Vehicle, but it does various things which require that the the members and virtual methods of its derived classes are still alive.
If you just called it from ~Vehicle it would be undefined behaviour and not work.

In the case of Train, after ~Train returns all of Trains members are destructed and the vtable reassigned before ~Vehicle is called.

@TrueBrain
Copy link
Member

TrueBrain commented Jan 11, 2021

We added code that prevents this situation from happening. Although we can make some performance enhancements to make removing such long chains quicker, the real problem of course that the AI managed to create such a long vehicle chain. This is not okay, and shouldn't happen :P So at least that is solved now :D

@LordAro : do you want to solve that the game doesn't close itself when deleting such huge chains? Or do you consider this mitigation sufficient to close the issue?

@2TallTyler
Copy link
Member

I vote that #8533 is sufficient mitigation and we can knock this off the issue tracker. 🙂

If anyone disagrees, feel free to reopen (or maybe open a more specific issue about how to best handle disaster vehicles in destructors).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

8 participants