Normalize nixexpr{input,path} from builds to jobsetevals. #848
Conversation
Duplicating this data on every record of the builds table cost approximately 4G of duplication. Note that the database migration included took about 4h45m on an untuned server which uses very slow rotational disks in a RAID5 setup, with not a lot of RAM. I imagine in production it might take an hour or two, but not 4. If this should become a chunked migration, I can do that. Note: Because of the question about chunked migrations, I have NOT YET tested this migration thoroughly enough for merge.
grahamc
force-pushed
the
normalize-nixexprinputpath
branch
from
1fe84aa to
9516b25
Compare
There was a problem hiding this comment.
Seeing Hydra's poor track record on having the appropriate indices (per your earlier PR) I whole-heatedly endorse this. Hydra needs to prove it can follow the rules (well-planned joins) before it earns the right to break them (denormalization).
Thanks again for your work with this stuff, @grahamc.
|
For what it is worth I find Hydra's schema to be pretty good :). It has scaled to over a hundred million builds, and its largest table has 1.5 billion (1,500,000,000) rows. There are a few cleanups to be done, but all in it does a really great job for the amount of maintenance it requires |
FWIW, this is because the JobsetEvals table is a later addition (see 7daca03). There is some more cruft in the DB from the time that we didn't have JobsetEvals, like BuildInputs (which still exists but isn't maintained anymore). |
I meant to open this as a draft.
Duplicating this data on every record of the builds table cost
approximately 4G of duplication.
Note that the database migration included took about 4h45m on an
untuned server which uses very slow rotational disks in a RAID5 setup,
with not a lot of RAM. I imagine in production it might take an hour
or two, but not 4. If this should become a chunked migration, I can do
that.
Note: Because of the question about chunked migrations, I have NOT
YET tested this migration thoroughly enough for merge.