Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unexpected .sfs growth with #3200 test build #3203

Closed
lpgagnon opened this issue Nov 17, 2021 · 7 comments
Closed

Unexpected .sfs growth with #3200 test build #3203

lpgagnon opened this issue Nov 17, 2021 · 7 comments
Assignees
@pleroy

Comments

@lpgagnon
Copy link

(opening this as requested on discord, mostly as a place to put links to saves)

Seen while running test3200 build from https://discord.com/channels/319857228905447436/480397772248580098/909198803633053706
Unclear if it's related; growth seems much faster than before, but I had also just launched an extra 10 vessels.

.sfs file was ~70MB; warping ahead ~3 months brought it up to ~170MB; resetting history via Hack Gravity on all my "in orbit" vessels (13) - but leaving alone ~40 vessels on interplanetary paths - only brought it down to ~140MB. I confirmed it's principia data taking most of the space by manually cutting out the principa SCENARIO from the file (the rest was only 15MB)

I also now have godawful scene load times, which could be a consequence, a separate symptom of something wrong, or just a consequence of all those vessels in flight

0 - 1961.zip
The original 170MB .sfs

wip.zip
The 140MB .sfs after Hack Gravity trick

I later tried deleting the in-orbit vessels altogether; no meaningful difference. Deleting the oldest interplanetary vessel (0155 uranus 2, 1190d old) shaved off 4MB, don't know if that's within expectations. Deleting the newest (0291 pluto 3-2, 81d old) shaved off ~200KB, not obviously wrong either
@pleroy pleroy self-assigned this Nov 17, 2021
@lpgagnon
Copy link
Author

1 more month of timewarp, the 140MB has already gone back up to 185MB. Definitely something off.

Sorry, something went wrong.

@pleroy
Copy link
Member

pleroy commented Nov 17, 2021

I read the first save and put a few traces to see what is happening. There are many histories with ~700k points, which seems like a lot, but still some downsampling takes place. More analysis will be needed.

E1117 21:37:09.943651 15436 plugin_compatibility_test.cpp:95] Deserialization starting
E1117 21:37:19.674651  4912 vessel.cpp:506] Vessel: 0182 saturn 2
  Hist: 720704 points from 1958-02-12T01:52:04,8865470886230469 (TT) to 1961-01-01T00:02:34,1715040206909180 (TT)
  Psy: 2 points from 1961-01-01T00:02:34,1715040206909180 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:20.866652  4912 vessel.cpp:506] Vessel: 0220 saturn 3
  Hist: 720350 points from 1959-02-24T21:34:55,2637207508087158 (TT) to 1961-01-01T00:02:29,1923143863677979 (TT)
  Psy: 2 points from 1961-01-01T00:02:29,1923143863677979 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:21.225651  4912 vessel.cpp:506] Vessel: 0216 jupiter 3-1
  Hist: 723307 points from 1959-01-24T19:50:53,6011831760406494 (TT) to 1961-01-01T00:02:31,2343342304229736 (TT)
  Psy: 2 points from 1961-01-01T00:02:31,2343342304229736 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:21.530651  4912 vessel.cpp:506] Vessel: 0276 ceres 3
  Hist: 712964 points from 1960-05-15T18:57:15,7726237773895264 (TT) to 1961-01-01T00:02:29,7628207206726074 (TT)
  Psy: 2 points from 1961-01-01T00:02:29,7628207206726074 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:21.538651  4912 vessel.cpp:506] Vessel: 0176 jupiter 1
  Hist: 16011 points from 1959-05-12T07:47:01,1697094440460205 (TT) to 1961-01-01T00:02:27,6098301410675049 (TT)
  Psy: 2 points from 1961-01-01T00:02:27,6098301410675049 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:29.821651  4912 vessel.cpp:506] Vessel: 0246 uranus 3-2
  Hist: 716411 points from 1959-10-06T12:49:08,3284564018249512 (TT) to 1961-01-01T00:02:25,0941624641418457 (TT)
  Psy: 2 points from 1961-01-01T00:02:25,0941624641418457 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:29.929651  4912 vessel.cpp:506] Vessel: 0138 vesta 1
  Hist: 33498 points from 1960-01-26T21:56:00,5795731544494629 (TT) to 1961-01-01T00:02:30,5795731544494629 (TT)
  Psy: 2 points from 1961-01-01T00:02:30,5795731544494629 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:30.019651  4912 vessel.cpp:506] Vessel: 0283 mars 4 deimos
  Hist: 3865 points from 1960-09-15T16:43:12,9511308670043945 (TT) to 1961-01-01T00:02:31,7397813796997070 (TT)
  Psy: 2 points from 1961-01-01T00:02:31,7397813796997070 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:30.564651  4912 vessel.cpp:506] Vessel: 0152 uranus 1
  Hist: 720797 points from 1957-10-03T10:10:01,1869280338287354 (TT) to 1961-01-01T00:02:28,1821420192718506 (TT)
  Psy: 2 points from 1961-01-01T00:02:28,1821420192718506 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:31.208652  4912 vessel.cpp:506] Vessel: 0180 saturn 1
  Hist: 719604 points from 1958-02-16T02:36:13,5507624149322510 (TT) to 1961-01-01T00:02:28,1943747997283936 (TT)
  Psy: 2 points from 1961-01-01T00:02:28,1943747997283936 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:31.943651  4912 vessel.cpp:506] Vessel: 0257 jupiter 4-3
  Hist: 722776 points from 1960-02-28T20:28:24,1881446838378906 (TT) to 1961-01-01T00:02:27,6970996856689453 (TT)
  Psy: 2 points from 1961-01-01T00:02:27,6970996856689453 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:32.742651  4912 vessel.cpp:506] Vessel: 0262 saturn 4-1
  Hist: 718797 points from 1960-03-06T19:52:11,0954113006591797 (TT) to 1961-01-01T00:02:27,4058318138122559 (TT)
  Psy: 2 points from 1961-01-01T00:02:27,4058318138122559 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:34.032651  4912 vessel.cpp:506] Vessel: 0257 jupiter 4-2
  Hist: 714895 points from 1960-02-28T01:20:29,5126709938049316 (TT) to 1961-01-01T00:02:32,2697663307189941 (TT)
  Psy: 2 points from 1961-01-01T00:02:32,2697663307189941 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:34.311651  4912 vessel.cpp:506] Vessel: 0165 venus 2 relay
  Hist: 162865 points from 1960-01-26T21:59:31,0393724441528320 (TT) to 1961-01-01T00:02:31,0393724441528320 (TT)
  Psy: 2 points from 1961-01-01T00:02:31,0393724441528320 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:34.979651  4912 vessel.cpp:506] Vessel: 0159 pluto 1
  Hist: 718630 points from 1957-11-01T04:41:16,1445264816284180 (TT) to 1961-01-01T00:02:28,2907519340515137 (TT)
  Psy: 2 points from 1961-01-01T00:02:28,2907519340515137 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:35.580651  4912 vessel.cpp:506] Vessel: 0170 neptune 1
  Hist: 720750 points from 1957-12-23T06:31:58,1889255046844482 (TT) to 1961-01-01T00:02:24,5960624217987061 (TT)
  Psy: 2 points from 1961-01-01T00:02:24,5960624217987061 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:37.067651  4912 vessel.cpp:506] Vessel: 0246 uranus 3-1
  Hist: 716673 points from 1959-10-06T09:41:46,4892439842224121 (TT) to 1961-01-01T00:02:27,5149950981140137 (TT)
  Psy: 2 points from 1961-01-01T00:02:27,5149950981140137 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:37.071651  4912 vessel.cpp:506] Vessel: 0291 pluto 3-2
  Hist: 9552 points from 1960-10-11T12:50:29,0965554714202881 (TT) to 1961-01-01T00:02:31,8839211463928223 (TT)
  Psy: 2 points from 1961-01-01T00:02:31,8839211463928223 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:38.570652  4912 vessel.cpp:506] Vessel: 0257 jupiter 4-1
  Hist: 714910 points from 1960-02-27T20:30:52,8135318756103516 (TT) to 1961-01-01T00:02:33,0373151302337646 (TT)
  Psy: 2 points from 1961-01-01T00:02:33,0373151302337646 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:40.394651  4912 vessel.cpp:506] Vessel: 0246 uranus 3-3
  Hist: 714431 points from 1959-10-07T09:38:26,9977355003356934 (TT) to 1961-01-01T00:02:28,7757036685943604 (TT)
  Psy: 2 points from 1961-01-01T00:02:28,7757036685943604 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:40.745651  4912 vessel.cpp:506] Vessel: 0280 mars 3 deimos
  Hist: 4948 points from 1960-09-14T16:49:35,5553076267242432 (TT) to 1961-01-01T00:02:28,9152779579162598 (TT)
  Psy: 2 points from 1961-01-01T00:02:28,9152779579162598 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:45.422652  4912 vessel.cpp:506] Vessel: 0155 uranus 2
  Hist: 719470 points from 1957-09-28T11:01:43,6009993553161621 (TT) to 1961-01-01T00:02:30,7438416481018066 (TT)
  Psy: 2 points from 1961-01-01T00:02:30,7438416481018066 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:48.118651  4912 vessel.cpp:506] Vessel: 0174 neptune 2
  Hist: 724122 points from 1957-12-20T06:43:11,9348506927490234 (TT) to 1961-01-01T00:02:27,5655496120452881 (TT)
  Psy: 2 points from 1961-01-01T00:02:27,5655496120452881 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:48.465651  4912 vessel.cpp:506] Vessel: 0101 earthscan 1
  Hist: 234645 points from 1960-01-26T21:54:25,6596636772155762 (TT) to 1961-01-01T00:02:25,6596636772155762 (TT)
  Psy: 2 points from 1961-01-01T00:02:25,6596636772155762 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:48.839651  4912 vessel.cpp:506] Vessel: 0283 mars 4 phobos
  Hist: 5907 points from 1960-09-15T17:26:17,6998043060302734 (TT) to 1961-01-01T00:02:30,3196301460266113 (TT)
  Psy: 2 points from 1961-01-01T00:02:30,3196301460266113 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:48.842651  4912 vessel.cpp:506] Vessel: 0280 mars 3
  Hist: 7000 points from 1960-09-14T17:41:34,4755778312683105 (TT) to 1961-01-01T00:02:25,0353865623474121 (TT)
  Psy: 2 points from 1961-01-01T00:02:25,0353865623474121 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:48.844651  4912 vessel.cpp:506] Vessel: 0291 pluto 3-1
  Hist: 4312 points from 1960-10-10T09:55:14,2831954956054688 (TT) to 1961-01-01T00:02:29,9012224674224854 (TT)
  Psy: 2 points from 1961-01-01T00:02:29,9012224674224854 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:37:59.746651  4912 vessel.cpp:506] Vessel: 0250 neptune 3
  Hist: 722830 points from 1959-12-20T07:52:11,3250401020050049 (TT) to 1961-01-01T00:02:24,6063144207000732 (TT)
  Psy: 2 points from 1961-01-01T00:02:24,6063144207000732 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:01.058651  4912 vessel.cpp:506] Vessel: 0262 saturn 4-2
  Hist: 716493 points from 1960-03-07T02:20:33,1513204574584961 (TT) to 1961-01-01T00:02:29,6966600418090820 (TT)
  Psy: 2 points from 1961-01-01T00:02:29,6966600418090820 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:02.228652  4912 vessel.cpp:506] Vessel: 0216 jupiter 3-2
  Hist: 721304 points from 1959-01-25T19:53:06,7602052688598633 (TT) to 1961-01-01T00:02:27,7677195072174072 (TT)
  Psy: 2 points from 1961-01-01T00:02:27,7677195072174072 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:02.419651  4912 vessel.cpp:506] Vessel: 0289 pluto 2-1
  Hist: 4402 points from 1960-10-10T12:53:48,6652419567108154 (TT) to 1961-01-01T00:02:29,0994815826416016 (TT)
  Psy: 2 points from 1961-01-01T00:02:29,0994815826416016 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:02.444651  4912 vessel.cpp:506] Vessel: 0273 mercury 7
  Hist: 45779 points from 1960-05-08T04:27:26,3448286056518555 (TT) to 1961-01-01T00:02:29,6782045364379883 (TT)
  Psy: 2 points from 1961-01-01T00:02:29,6782045364379883 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:02.495651  4912 vessel.cpp:506] Vessel: 0098 moonscan 1
  Hist: 103059 points from 1960-01-26T21:55:22,4396095275878906 (TT) to 1961-01-01T00:02:26,2396059036254883 (TT)
  Psy: 2 points from 1961-01-01T00:02:26,2396059036254883 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:02.803651  4912 vessel.cpp:506] Vessel: 0237 venus 3 relay
  Hist: 212109 points from 1959-10-28T05:08:56,6247448921203613 (TT) to 1961-01-01T00:02:28,0645909309387207 (TT)
  Psy: 2 points from 1961-01-01T00:02:28,0645909309387207 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:03.242651  4912 vessel.cpp:506] Vessel: 0247 photo 4
  Hist: 279776 points from 1959-09-17T21:01:26,0176815986633301 (TT) to 1961-01-01T00:02:25,9409756660461426 (TT)
  Psy: 2 points from 1961-01-01T00:02:25,9409756660461426 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:04.145651  4912 vessel.cpp:506] Vessel: 0252 neptune 4
  Hist: 712313 points from 1959-12-20T15:26:21,7981102466583252 (TT) to 1961-01-01T00:02:32,1387295722961426 (TT)
  Psy: 2 points from 1961-01-01T00:02:32,1387295722961426 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:04.148651  4912 vessel.cpp:506] Vessel: 0283 mars 4
  Hist: 7381 points from 1960-09-15T17:28:47,6796612739562988 (TT) to 1961-01-01T00:02:25,4993486404418945 (TT)
  Psy: 2 points from 1961-01-01T00:02:25,4993486404418945 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:04.719651  4912 vessel.cpp:506] Vessel: 0260 sun probe
  Hist: 716886 points from 1960-02-29T01:16:18,7187833786010742 (TT) to 1961-01-01T00:02:30,6748249530792236 (TT)
  Psy: 2 points from 1961-01-01T00:02:30,6748249530792236 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:05.432651  4912 vessel.cpp:506] Vessel: 0267 vesta 2
  Hist: 721862 points from 1960-04-20T15:25:06,9838435649871826 (TT) to 1961-01-01T00:02:27,7443017959594727 (TT)
  Psy: 2 points from 1961-01-01T00:02:27,7443017959594727 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:05.495651  4912 vessel.cpp:506] Vessel: 0280 mars 3 phobos
  Hist: 4343 points from 1960-09-14T17:37:37,7558035850524902 (TT) to 1961-01-01T00:02:33,2352166175842285 (TT)
  Psy: 2 points from 1961-01-01T00:02:33,2352166175842285 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:05.675652  4912 vessel.cpp:506] Vessel: 0231 mercury 5
  Hist: 127008 points from 1959-05-12T06:13:47,2651028633117676 (TT) to 1961-01-01T00:02:28,1256213188171387 (TT)
  Psy: 2 points from 1961-01-01T00:02:28,1256213188171387 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:06.012651  4912 vessel.cpp:506] Vessel: 0239 venus 4
  Hist: 260609 points from 1959-06-25T02:30:29,1479229927062988 (TT) to 1961-01-01T00:02:34,2198612689971924 (TT)
  Psy: 2 points from 1961-01-01T00:02:34,2198612689971924 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:06.090651  4912 vessel.cpp:506] Vessel: 0201 mars return
  Hist: 172646 points from 1960-01-26T23:33:30,4094967842102051 (TT) to 1961-01-01T00:02:26,3076896667480469 (TT)
  Psy: 2 points from 1961-01-01T00:02:26,3076896667480469 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:06.094651  4912 vessel.cpp:506] Vessel: 0289 pluto 2-2
  Hist: 8546 points from 1960-10-11T09:50:38,9384946823120117 (TT) to 1961-01-01T00:02:32,1699721813201904 (TT)
  Psy: 2 points from 1961-01-01T00:02:32,1699721813201904 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:07.373651  4912 vessel.cpp:506] Vessel: 0262 saturn 4-3
  Hist: 716161 points from 1960-03-07T19:56:18,7487571239471436 (TT) to 1961-01-01T00:02:32,2927970886230469 (TT)
  Psy: 2 points from 1961-01-01T00:02:32,2927970886230469 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:08.030651  4912 vessel.cpp:506] Vessel: 0264 saturn  5
  Hist: 713898 points from 1960-03-08T02:16:06,0273890495300293 (TT) to 1961-01-01T00:02:30,1547536849975586 (TT)
  Psy: 2 points from 1961-01-01T00:02:30,1547536849975586 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
E1117 21:38:09.012651  4912 vessel.cpp:506] Vessel: 0255 neptune 5
  Hist: 716176 points from 1959-12-21T16:25:52,3797388076782227 (TT) to 1961-01-01T00:02:29,3827157020568848 (TT)
  Psy: 2 points from 1961-01-01T00:02:29,3827157020568848 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)
  Pred: 1 points from 1961-01-01T00:02:34,2872233390808105 (TT) to 1961-01-01T00:02:34,2872233390808105 (TT)

@pleroy
Copy link
Member

pleroy commented Nov 17, 2021

Each of the serialization::Vessel messages for the vessels with large histories takes about 5 MiB (after zfp and before gipfeli), consistent with the 4 MiB mentioned in the first comment. That's 7 bytes per point, so zfp is surprisingly efficient.

@eggrobin
Copy link
Member

zfp is surprisingly efficient.

also known as « we’re not downsampling ».
Looking at the history of 0182 saturn 2, the time between successive points (in seconds) as a function of time (in seconds, J2000 = 0) is
3203-time
which, as a function of index, is
3203-index
Starting from the 1022nd point, all points are 10 seconds apart, i.e., not downsampled.

@lpgagnon
Copy link
Author

As requested: JOURNAL.zip (~150MB)
From 1961-01-01 (as provided in 0 - 1961.zip), timewarped forward to 1961-02-01.

@eggrobin
Copy link
Member

Thanks for the journal, it greatly sped up tracking down the bug.

The explanation is that

Principia/physics/discrete_trajectory_segment_body.hpp

Lines 436 to 439 in df1e692

if (right_endpoints->empty()) {
number_of_dense_points_ = timeline_.empty() ? 0 : 1;
return absl::OkStatus();
}

is an incorrect translation of

Principia/physics/discrete_trajectory_body.hpp

Lines 682 to 684 in 3473dbc

if (right_endpoints->empty()) {
right_endpoints->push_back(dense_iterators.end() - 1);
}

In the old code, control then falls through to the code that actually does the downsampling. In the new code, if the entire dense segment could be fit by a single polynomial within the tolerances (as opposed to multiple polynomials), that is, if the trajectory is very smooth… we just don’t use that fit, we leave those ten thousand points lying around and move on.

@pleroy pleroy mentioned this issue Nov 19, 2021
Merged
@pleroy
Copy link
Member

pleroy commented Nov 20, 2021

Fixed by #3204.

@pleroy pleroy closed this as completed Nov 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pleroy pleroy

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@eggrobin @lpgagnon @pleroy