Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove huge packages #53092

Closed
wants to merge 5 commits into from
Closed

Conversation

matthewbauer
Copy link
Member

Motivation for this change

I'm going through some of the larger files in the Nixpkgs repo. Some of these may still be used by people, but their data files are taking up a lot of space.

Removes:

  • codimd (680K)
  • remarkjs (broken, 116K)
  • ue4 (unfree, broken, 468K)

Also:

  • Gets rid of herqq's gcc6.patch (unused, 56K)
  • Remove data files from pharo vm (72K)

Let me know if anyone is relying on these to work. This gets about 1.5M from the nixpkgs which is a about 1% of total repo space.

@Ma27
Copy link
Member

Ma27 commented Dec 31, 2018

I know from some people that they use codimd with Nix (including myself on my personal setup), so I'm 👎 at least for codimd.

cc @WilliButz

@WilliButz
Copy link
Member

WilliButz commented Dec 31, 2018

I agree with @Ma27 on this, having a Nix setup makes it a lot more comfortable to set up and maintain stuff like this.

As most of codimd's size comes from the generated node dependencies, maybe another approach would be more sensible. Instead of just removing the whole thing, has there ever been any attempt to move out the generated node-packages and then reference them via fetchurl or something similar? I know that this is not the optimal solution but it would certainly reduce the size of nixpkgs :)

@vcunat
Copy link
Member

vcunat commented Dec 31, 2018

Importing from a derviation would have its own issues, like tools not counting on builds happening during evaluation. In particular, we probably don't want that to happen on Hydra.

@vcunat
Copy link
Member

vcunat commented Dec 31, 2018

/cc other maintainers: @rickynils, @puffnfresh.

@veprbl
Copy link
Member

veprbl commented Dec 31, 2018

cc @ivan regarding ue4

@bhipple
Copy link
Contributor

bhipple commented Jan 1, 2019

For large generated package sets that are only used by a few users, or for large proprietary packages that we may not want to dedicate community resources to, something like @Mic92's NUR seems ideal:
https://discourse.nixos.org/t/introducing-nur-the-nix-user-repostory-share-all-your-nix-expressions/431

It lacks some of the convenience and discoverability of having all of the packages right there in NixPkgs, and as @vcunat mentions import-from-derivation brings a host of issues, but I could see this someday providing frequently updated channels for some of the huge, auto-generated package sets, such as the emacs, texlive, or nodejs ones.

For instance, texlive by itself is 2.1M, nearly all of which gets regenerated when we do an annual update. emacs has 6.1M of melpa packages as well, which get updated more frequently than the texlive packages.

@ivan
Copy link
Member

ivan commented Jan 1, 2019

I don't use ue4 and don't plan to fix it right now. It looks like it was added in #13110 but it's odd that it ever worked - I thought the entire repo was always set to Private since its inception.

@Mic92
Copy link
Member

Mic92 commented Jan 1, 2019

@Ma27 codimd should be integrated into our existing node-packages set so it can reuse existing definitions.

@danbst
Copy link
Contributor

danbst commented Jan 2, 2019

@vcunat if Hydra is main objection for IFD, than we can explicitly mark "no hydra" for IFD packages. And disable nixos test as well. We only have to assure, that meta is accessible without IFD, and this can be done with mkDerivation wrapper.

While for compiled languages (like Haskell) I see benefits of using Hydra builds VS compiling myself, for interpreted languages like JS/Python build cache isn't much required.

@danbst
Copy link
Contributor

danbst commented Jan 2, 2019

@vcunat actually, I can answer myself. Major downside for IFD here is that IFD plays bad with GC, so if you don't add GC root for each IFD you will redownload everything from scratch after nix-collect-garbage.

@7c6f434c
Copy link
Member

7c6f434c commented Jan 2, 2019

The GC pinning sounds feasible on the package level. I.e. when doing IFD we could pass the reference to that same derivation, then the package could optionally pin its own definition.

@globin
Copy link
Member

globin commented Jan 2, 2019

I'd prefer keeping codimd, as I know quite a few people using it.

@oxij
Copy link
Member

oxij commented Jan 2, 2019 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet