It would be good to collect some performance stats on this. Basically counting how long nix-instantiate emacsPackagesNg.magit and nix-env -qa -A emacsPackagesNg.

It would be good to collect some performance stats

I can get some, later. Right now I'm in the middle of a couple of rebuilds and don't wan't to GC. The PR should be in a perfectly usable state, if you want to take a look.

Subjectively, nix-instantiate performance seemed decent.

This is awesome work! Thanks!
I'm all for simplifying generation of melpa packages. As it is now it's not easy to use emacs2nix successfully.

I'd like to get an opinion on this from @ttuegel before feeling comfortable about moving away from emacs2nix.

@matthewbauer A completely unscientific test:

$ time nix-env -f '<nixpkgs>' -qa -A emacsPackagesNg

Master:

0.78user 0.08system 0:01.72elapsed 50%CPU (0avgtext+0avgdata 290204maxresident)k
0inputs+0outputs (0major+78099minor)pagefaults 0swaps

This PR:

0.69user 0.07system 0:01.38elapsed 55%CPU (0avgtext+0avgdata 279408maxresident)k
0inputs+0outputs (0major+73315minor)pagefaults 0swaps

$ time nix-instantiate '<nixpkgs>' -A emacsPackagesNg.magit

Master:

0.48user 0.05system 0:00.58elapsed 92%CPU (0avgtext+0avgdata 143172maxresident)k
0inputs+0outputs (0major+36232minor)pagefaults 0swaps

This PR:

0.38user 0.04system 0:00.47elapsed 90%CPU (0avgtext+0avgdata 122252maxresident)k
0inputs+0outputs (0major+29041minor)pagefaults 0swaps

While both of the results from this PR are faster it's imho a negligible difference.

While both of the results from this PR are faster it's imho a negligible difference.

Ha, seems like nix has a fast JSON reader built in ;-)

Any significant difference in performance would surprise me, since the computed nix expressions remained pretty much the same.

Here is a test script for timing nix-instantiate:

#!/bin/sh -ex

instantiate () {
    echo -n "Instantiated count "
    nix-instantiate -E "
      with import ./. { config = { allowBroken = true; }; };
      emacsPackagesNg.melpaPackages
    " 2>/dev/null | wc -l
}

(
    cd master
    echo "============================
          On master"
    nix-store --option keep-derivations false --gc 2>/dev/null
    time instantiate
)


(
    cd emacs-updater-elisp
    echo "============================
          On emacs-updater-elisp"
    nix-store --option keep-derivations false --gc 2>/dev/null
    time instantiate
)

the output on my machine is

============================================
          On master
12224 store paths deleted, 51.57 MiB freed
Instantiated count 4060

real    0m16.480s
user    0m4.346s
sys     0m0.331s
============================================
          On emacs-updater-elisp
12042 store paths deleted, 51.11 MiB freed
Instantiated count 4118

real    0m17.428s
user    0m4.518s
sys     0m0.363s

So it seems, we are a little bit slower, but times on my machine fluctuate heavily (15-25 seconds)
Differences in package count are due to this updater emitting broken packages as well.

ping @ttuegel

I'd like to get an opinion on this from @ttuegel before feeling comfortable about moving away from emacs2nix.

I am currently developing a new branch of emacs2nix with the aim of producing an overlay which updates automatically. I don't really like maintaining such a large set of generated packages in Nixpkgs itself. I also haven't been a very responsive maintainer (new baby, dissertation, new full-time job, upcoming new baby... excuses, excuses) so I'm not going to be offended if Nixpkgs moves away from emacs2nix. Right now the packages are not kept up-to-date, so I say: if this can distribute the work of package updates, and if it seems to work, then ship it!

@ttuegel thanks for your response. This PR isn't ready, yet, and as far as I'm concerned, it won't be before we can generate elpa and org packages as well. Now with your confirmation, I'll be glad to cherry-pick relevant commits into this PR.

I don't really like maintaining such a large set of generated packages in Nixpkgs itself.

This is a good point. The generator could have black-/whitelist capability, in order to maintain only a couple of necessary packages within nixpkgs and have the rest in overlays. Would this serve your purpose?

It sounds like you might still keep working on emacs2nix, if we were to make the switch ... would you consider adopting the .json format in that case?

This PR isn't ready, yet, and as far as I'm concerned, it won't be before we can generate elpa and org packages as well.

Is this really true? In my experience the hard part about emacs packages is generating melpa. Org and elpa packages always worked perfectly with emacs2nix.
I would rather see this PR move ahead as-is and iterate on it later.

Is this really true? In my experience the hard part about emacs packages is generating melpa. Org and elpa packages always worked perfectly with emacs2nix.

This may be true, but it would introduce another split (emacsPackages, emacsPackagesNg, melpa). Also, as I already mentioned, the integration of the json format from the nix side is a horrible hack and needs to be cleaned up.

To be perfectly blunt: The issue for me is also about shared ownership. Replacing a solution where @ttuegel is the sole maintainer with a solution where I'm the sole maintainer makes little sense to me, neither from a community standpoint, nor from a standpoint of my own free time.

So either somebody is going to jump in to help push this along (if this means pushing it out as is, so be it, at least I'll not be the only one receiving tickets about it), or it will have to wait until I get it to a state where I'm happy about releasing it, myself.

I've updated to use extracted libraries and to handle melpa-stable. I think the melpa part should be mostly ready, now. Please give it a whirl.

Does anybody care about some changed package names? e.g. _0blayout now needs to be referenced as emacsPackages."0blayout". I think this is justified, because there might be more renames, that the old scheme might not have caught either, e.g. @, and I don't feel like reverse engineering it all, given that the user benefit would be dubious at best ..

@bendlas I've rebased and re-ran this. The generation works but building my configuration currently fails on cider-mode not passing sha256.

I intend to make this usable (and hopefully closer to a mergable state) in the next couple of days.

My errors were simply a side effect of running the generation on an unstable internet connection.
I re-ran the generation and building my emacs config with 74 packages works great :)

@bendlas How do you feel about merging me merging this (providing ofborg gives thumbs up)?

I think we just want to copy the name mappings from emacs2nix https://github.com/ttuegel/emacs2nix/blob/9143b374be02bd6220fd6317f1b72837ce8254bf/names.nix before we do the merge.

cc'ing a few other known emacs users for testing
@matthewbauer @ttuegel @etu @talyz @tazjin

It seems I still have some evaluation errors to sort out. We're making headway!

edit: Bitbucket sources are broken. I believe this to be the only show stopper right now.
Everything seems to be working as intended.

Unless anyone requests changes I plan to merge this in 24 hours.

edit: I have been running off of this PR for the last 2 days without issues (the previous evaluation errors for bitbucket was not affecting my personal configuration).

I suggest cleaning up the commit history though, otherwise this could get annoying to run into while bisecting and it also doesn't adhere to the commit naming scheme we're trying to adhere to in nixpkgs

@infinisil That was always the plan :) In fact I'll do that now.

@bendlas Thanks for all your hard work in making this PR! I'm so happy about this change 😄 🎉

Awesome! Thanks for pushing this through! 🎉🎆