@zimbatm, thanks for your PR! By analyzing the history of the files in this pull request, we identified @bjornfor, @edolstra and @krgn to be potential reviewers.

I don't like the use of strings like fetchurl and requireFile. Why not use the functions directly? I doubt the actual fetchers used (and that need to listed as arguments) will need to change often, so this seems just as amenable to automated generation/updates:

src = {
  "x86_64-linux-demo" = fetchurl {
    url = "http://files.renoise.com/demo/Renoise_3_0_1_Demo_x86_64.tar.bz2";
    sha256 = "1q7f94wz2dbz659kpp53a3n1qyndsk0pkb29lxdff4pc3ddqwykg";
  };
  "x86_64-linux" = requireFile {
    url = "http://backstage.renoise.com/frontend/app/index.html#/login";
    name = "rns_3_0_1_linux_x86_64.tar.gz";
    sha256 = "1yb5w5jrg9dk9fg5rfvfk6p0rxn4r4i32vxp2l9lzhbs02pv15wd";
  };
  "i686-linux-demo" = fetchurl {
    url = "http://files.renoise.com/demo/Renoise_3_0_1_Demo_x86.tar.bz2";
    sha256 = "0dgqvib4xh2yhgh2wajj11wsb6xiiwgfkhyz32g8vnyaij5q8f58";
  };
  "i686-linux" = requireFile {
    url = "http://backstage.renoise.com/frontend/app/index.html#/login";
    name = "rns_3_0_1_reg_x86.tar.gz";
    sha256 = "1swax2jz0gswdpzz8alwjfd8rhigc2yfspj7p8wvdvylqrf7n8q7";
  };
}."${stdenv.system}${demoStr}";

I do like having a function that takes the multiple inputs and creates the all attribute for all the urls, however - that seems useful.

@aneeshusa on the longer run I want this block to be separated in a different file that can easily be parsed and updated by a script. I think that would work best if that block was composed of data only.

@zimbatm I wonder about having two components to your mechanism, one where we use requireFile / fetchurl / etc. directly (as @aneeshusa suggests,) and then an adapter to support the data-driven mechanism. This makes the simple case more obvious, and the complicated case a simpler addition. I think.

since nix is lazy, whats wrong with just using getAttr on an attributeset

it can even be part of the derivation and be accessible that way

I think you are missing vivaldi and google-music-manager

My concern with wanting to split out "the fetcher" from the build stuff is that often they're fairly entangled and can't be updated independently of one another. A patch that applied to one version might not apply after the version gets updated, and a e.g., a workaround configure flag might be needed after updating. So in some sense, I don't see the part of a package expression fetching sources to be a fundamentally different flavor of thing than the rest of the package build expression. Splitting them out so a machine can update one without updating the other feels like it'll lead to weirdness. But I'll come around later in this comment 😄

I also don't really grok why NixON is a desirable format for things. In most of my projects I've moved away from JSON towards things like Nix and Jsonnet precisely because I wanted function calls, and I think more and more people are realizing that a good model for configuration is to have some facilities for abstraction built into the language. I think Nix is miles ahead of the competition here, and it seems weird to me to arbitrarily cut out function calls and then simulate them poorly in a JSON-with-a-different-syntax. Do you have some writing on why that's desirable over pure JSON or straight Nix?

If your goal is to enable something like monitor.nixos.org to better update packages for us (and to run tentative builds with updates), I think that's a worthy one, but if we're defining NixON ourselves I'd probably just leave function calls in the spec, then write some manipulation functionality for Nix source files, and possibly use scopedImport to import NixON files with the fetchers already in scope. The big reason for splitting hashes out into a separate file as I understand it is that the file can be "managed by machine" without messing up formatting/comments/idiosyncrasies a human injected in the original package expression, right?

Sorry if I come across as overly negative but after a few years working with Ansible and CloudFormation and other horrible approximations of programing languages constrained to a JSON syntax, I have a negative reaction to the notion spreading.

Thanks for the feedback @copumpkin. I am still mulling over what you said but I think you might be right, I approached the problem the wrong way around. First I need to build the tool that helps with the packages upgrades, I am just unsure on how to parse the nix expression to do in-place transformation of the AST tree right now (without messing with the formatting and comments).

Well, I'm also possibly wrong so don't just take my word for it 😄

As far as manipulation of the AST goes, https://github.com/jwiegley/hnix might be helpful. Alternately, avoiding Nix AST manipulation altogether (since in some sense "the type is too damn big" and you might want a more explicit data model) an option might be to do something like your fetchOneOf (or fetchMulti) but only in the restricted context of automatic updates. That is to say, explicitly make it so that those functions aren't intended for humans. One possible structure might be:

1. Start a convention of using inherit (handleExternal ./autodeps.json) src in package expressions
2. Your tool scans the repo for autodeps.json, possibly merging it with metadata from the Nix side (or finds autodeps.json automatically by propagating attributes about it up to the Nix layer)
3. The handleExternal function implements some scheme that looks a lot like your fetchOneOf or fetchMulti, but uses them exclusively to interpret the JSON, and the JSON is explicitly autogenerated/managed by a machine.

But whichever way you go, I'd start with a hypothesis of what the whole "automatically managed version upgrades" story would look like so that the rest of us can think through whether we'd use it or not for our own packages. I'd also contrast it to what monitor is doing, or maybe outline how what you're doing builds on the monitor.

I'm mostly trying to avoid (you, us) spending a lot of time/work on something that's currently fairly speculative and that would have a large effect on everyone's packages, without getting a few of the "Nix core thinkers" (@edolstra, @shlevy, @domenkozar, etc.) on the same page for what that would look like. It makes me really sad to see tons of good thinking/work go into speculative work that never gets merged because people can't agree on how it should work.

I realize that my last two responses are sort of contradictory by the way. In the first one I was basically advocating Nix AST manipulation and in the second, I suggested not doing that. I think in my first reply, I was mostly reacting to NixON, and if it's going to be isomorphic to JSON, I'd probably just stick with JSON. I do think it can be nice to have an explicit "data model" for external sources we know how to update, especially if we can attach a notion of things to do when they change. For example, I'd love to be able to annotate a hack/workaround as being obviated by an upcoming release: "this patch/hack in the package expression goes away after 1.7.5" is currently expressed in human comments (if we're lucky). As another example, the data model can have nice ramifications for paranoid people wanting to check what sources their builds pull. Compare https://nixos.org/wiki/Download_all_sources, where we can predownload all the sources, but we can't easily track down where we got them from (which in many ways is more interesting from a reviewer's perspective). Your stuff could help with that, if done right.

If your system could systematize that (or at least have a consistent vision for how it would systematize it, even if it's not implemented at first), it would make me incredibly happy. So I do see the value in coming up with an explicit data model for what it means to pull something in from the outside.

Yeah I think the upgrade mechanism should work on a whole package or even distribution (eg: gnome) level. But I agree that it needs to be fleshed out.

What I'll do for now in this PR is just keep the switching bits to make it easy to use nix-prefetch-url.

@zimbatm sry for late reply. vacations got in the way :)

firefox-bin: i think per locale we valid case we will also have with other binary sources. eg. openoffice/libreoffice/skype/... no everybody defaults to english. trying to setup things for my family (slovenian) i can see that we are missing some and i expect we will need to fix them in future.

as for the NIXON. i dont really see the advantage of it over JSON. i actually see JSON as a format that every language can produce, while for NIXOS every language will have to create some sort of library. also JSON can be easily read by nix via readJSON. on the end it is just data and having it in NIXON or JSON deosn't really matter, what matters is more how much more work it gives us.

the most time consuming part of implementing this is going to be many updating scripts that we need to write and i see that holding the real value. also one person will never be able to do it. for this i think we will need to change contributions guide and require that everybody that writes an update or new package also writes and update script.

Thanks everyone for participating. Closing that discussion in favor of #20844