-
Notifications
You must be signed in to change notification settings - Fork 69
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
History persistence lets .sfs files get very large #2400
Comments
More data:
|
https://drive.google.com/open?id=15bLvwZiRsIiOOg602r4tfFR7fna_4v4P 162MB sfs, of which ~155MB is from 2 vessels ~1500d old. |
I have managed to read the save and produce a The output file (776 MiB...) is on Google Drive and shared with @eggrobin. Good luck trying to read it in Mathematica. |
https://drive.google.com/open?id=1RDNwA2KRJU1miBiLIhgaaRehjXtOVdVf Notable (in both linked saves): simple zipping reduces the size by a good third. So if nothing else, different compression algorithms could be worth investigating (though higher compression might come at the cost of slower saves & loads). |
I noticed the same, but it mostly comes from the fact that the save uses base64, i.e., 6 bits per bytes. It stands to reason that zipping would save 25%. A better compression algorithm (zstd?) would probably not save more than ~5%. |
hypervelocity from KSP forums here, providing a hefty persistent file for your analysis. I tried uploading it to gist but website kept crashing before finishing upload. Let me know if anything else is needed or if there's any other way I can help. https://drive.google.com/file/d/1pxOinC2T4251Q8no2SX1sds2FfulqMsU/view?usp=sharing |
@damianreds: Thanks, I might try to see how the code in #2485 helps with your save. |
Running #2485 on the save dated Jan 31 above. Looking only at the Principia part, i.e., the part that's encoded in base64. Before (gipfeli, base64): 156'075'134 bytes. |
Running #2485 on the save provided by @damianreds. Looking only at the Principia part, i.e., the part that's encoded in base64. Before (gipfeli, base64): 83'633'693 bytes. Curiously this save is very slow to deserialize (135 s), irrespective of zfp. To be continued... |
@damianreds: Your pain comes from the fact that your game is in the year 2023 (I suspect that you started by warping all the way to 2020 or some such). On every scene change, Principia recomputes the history of the solar system from 1951 to 2023. That takes about 2 minutes. Obviously that's not very smart of us, but then the alternative of explicitly storing 72 years of history has its own drawbacks (it runs out of memory on my machine, for instance). This will need to be addressed in a future release, but it will take time. In the meantime, I recommend playing closer to the origin date of 1951 if possible. |
@pleroy thank you Pascal! I very much appreciate your insights. You are correct in your assumption, I did start by warping all the way through 2020. EDIT: @pleroy now that I think about it, is this issue first noticed now? I have been using Principia for a while now and always fast forwarded to roughly our present time and this is the first time I have experienced such significant delays. |
@damianreds: The "problem" was introduced in פרנקל. In that version we started to plot the trajectories of celestials. At the same time, we dropped the capability to forget the past histories; instead the setting of the history length hides the histories but doesn't delete them. Overall this seems useful because, well, all these trajectories look nice. We were aware that it would make saves bigger/slower, but the situation is not all that bad if you are a few years after 1951. We just didn't anticipate your use case, where 70 years would need to be integrated on each scene change. |
got it, thank you, very much appreciated! :) |
游戏加载的时候能直接从.sfs加载当前时间飞船的位置,而不是重新计算一遍吗?还有,能不能说让这些历史记录不放到内存里(如果必须的话放到页面文件里)来节约一些内存?我只有8G内存,加载RP-1就很耗时间、很占内存了,我不希望这个也占用我的内存,让我的游戏更卡了。 |
Is it possible to provide an option to forget the history over 2 weeks of some specified vehicle? I have some long-running vehicles, such as communication satellites around bodies. I don't necessarily care about their trajectories. Maybe making the history of them transient could make the size more manageable. (I would appreciate even a crappy one like making the history transient if the vehicles have some prefix in their names.) |
Slowing down scene changes is actually extremely harmful for the game play, as KSP does require a lot of scene changes to do even the most basic things. Having more memory usage instead of slow down is preferable. 32 / 64 GB machines are normal this days. 2-minute scene change waits quickly add-up to the time when it becomes more money efficient to simply buy more RAM (lost time is money). How about the following idea: have a configurable number of GB to allocate to the in-memory persistence of system state. If that number is exceeded, recompute the missing portion and drop a message somewhere "tune that particular number to get faster scene changes". This would allow people with sufficiently powerful machines(64 - 128 GB) allocate as much as they want and not worry about slowdowns, while smaller machines (16 gb or so) still both don't crash and people know how to address the problem (get more RAM). |
If you come here to comment on this issue, please provide facts: on my computer which has characteristics X, the attached save takes Y seconds to load and Z seconds to write. And please give us your save. Unless you are an astronomer or a Squad developer, your suggestions are not going to tell us anything that we didn't know about the problem. |
https://drive.google.com/file/d/1GB6LWSBCE67M0YhpufxOs5Gi44uhzTbK/view?usp=sharing This save is around 16 years into the game. It has a mix of interplanetary probes, earth sats, lunar sats, and lunar surface sats. |
Extra data point, this save is largely cleared of long term vessels (my PC can't cope), but there is one called Venus Orbiter 1 which has 934 days of history, mostly in a fairly boring Venus orbit. |
from discord discussion. Not a big problem so far, and too early to be proposing "fixes"; but worth tracking and gathering more data.
rp-1 career, ~15 active vessels including a ~5-year-old first orbit vessel; .sfs size was up to 280MB. Deleting just the first-orbit vessel reduced it by 60MB. Cleaning up all vessels from completed contracts brings it down to 24MB.
Main impact is on save/load speeds; saving was up to ~5s on SSD (presumably it's generating the data, not writing the file that's the bottleneck). Particularly annoying because this includes autosaves, so this can introduce long pauses the player doesn't control.
Secondary, minor impact is that big .sfs files are less easy to edit, share, etc.
The text was updated successfully, but these errors were encountered: