Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generated factorio mods #89799

Draft
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

priegger
Copy link
Contributor

@priegger priegger commented Jun 8, 2020

Motivation for this change

Simplify factorio mod management in NixOS.

Todos:

  • Handle incompatible dependencies (prefixed by "!").
  • Handle missing optional dependencies, i.e. optional dependencies that are not in factorio-mods.
  • Handle mods for stable (0.17) and experimental (0.18) factorio. Provide factorio-mods and factorio-mods-experimental, similar to factorio and factorio-experimental.
  • Handle/mention the licenses of the mods.
  • Squash (some) commits and submit PR.

Future improvements (not in this PR):

  • Include some more mods. In addition to Bob's mods I'd like to have Angel's mods, the AAI mods and RSO since I currently use them and they have a pretty high rating.
  • Get in touch with the factorio devs and ask them for sha256 checksums for the mods, so we can get rid of sha1.

Additional informaiton:

  • I use sha1 for veriying the downloaded mods. This is nice, because we get the sha1 sum from the mod portal api and I don't need to download all mods for each update but it is also not nice, because it's sha1.
  • I currently ignore hidden optional deps (prefixed with "(?)"). I'm not sure if it would make sense to handle them any other way than ignoring them.
  • Some mods are available on github (e.g. Angel's mods). Does it make sense to build them from source instead of using this "general solution"? => I go for the uniform solution and take all mods from the mod portal.
  • Recommended dependencies might make sense but AFAIK they are only a concept in texts (e.g. Bob's mods should be used together) and not available from the API. Should we remove them or manually add them to the json (and keep them when updating)? => I ignore them for now.
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@priegger
Copy link
Contributor Author

priegger commented Jun 8, 2020

@Baughn @elitak @erictapen: Is this the direction you wanted to go with factorio mod packaging?

@erictapen
Copy link
Member

I don't have the resources to review this atm and honestly I'd rather step down from maintainership for factorio, as I haven't played it for a long time now.

Unrelated: @priegger would you be interested in adding yourself as maintainer? I saw you submitting some updates for the factorio package and you seem to have an interest in the game.

@NickHu
Copy link
Contributor

NickHu commented Jun 9, 2020

Hi, thanks for your continued contributions! I've recently gotten back into this game and I'm glad someone is enthusiastically updating the nix infrastructure.

Some comments:

* I guess we need to handle mods for stable (0.17) and experimental (0.18) factorio. Does it make sense to have factorio-mods and factorio-mods-experimental, similar to factorio and factorio-experimental?

I don't see the harm in this, sounds like stable won't get updated to 0.18 until August when it becomes the 1.0 release. Can always change it again then.

* I use sha1 for veriying the downloaded mods. This is nice, because we get the sha1 sum from the mod portal api and I don't need to download all mods for each update but it is also not nice, because it's sha1.

In my opinion, this is much better than the trust-on-first-use model that most of nixpkgs is built around, so no need to change this.

* Which mods should be included? I currently added all mods that were in mods.nix before (bob's mods), but I'd like to add angel's mods and maybe some other mods I use.

I'm not opposed to having a massive JSON dump with all the mods, if there is a sufficiently nice programmatic way to get this data. Similar to how the Haskell module infrastructure works. In the grand scheme of things, several thousand mods aren't going to be an issue so long as we have an auto-update script like you have written :)

* I currently ignore hidden optional deps (prefixed with "(?)"). I'm not sure if it would make sense to handle them any other way than ignoring them.

Sounds reasonable - what's the rationale for an optional dep to be hidden?

* Some mods are available on github (e.g. Angel's mods). Does it make sense to build them from source instead of using this "general solution"?

Probably not - I would suggest that this infrastructure works uniformly for every mod.

* Where and how do we need to mention the licenses of mods? The mod portal api response contains license names and urls. Most of them are custom licenses, IIRC.

I think each mod should produce a derivation and then the license can be set in the meta attribute.

* Recommended dependencies might make sense but AFAIK they are only a concept in texts (e.g. Bob's mods should be used together) and not available from the API. Should we remove them or manually add them to the json (and keep them when updating)?

My suggestion is that you take the JSON API as the single source of truth and bug mod authors to fix their metadata on mods.factorio.com

@elitak
Copy link
Contributor

elitak commented Jun 9, 2020

This is based off what I committed to my branch years ago? I was close to finishing, but the major features that I wanted to request from the upstream devs was sha256 hashes in their JSON, and ideally a way to fetch the mod zips without authentication, maybe with some token they issued for all Nix users. Otherwise, I was going to create my own open mirror of the nixified mod zips that refreshed daily.

I think my plan was to not store any json/nix in the nixpkgs tree, because it changed so often. Unlike executable dependencies (for example python packages, whose APIs remain largely the same between versions), the binary .zip factorio mods are highly volatile and have very strict dependencies.

I haven't investigated Nix Flakes very much, but I suspect that mechanism would be ideal, as it addresses this very issue of having to "freeze" the dependency graph to get reproducible builds.

@priegger
Copy link
Contributor Author

priegger commented Jun 9, 2020

Edit: I mixed up some things here. I removed the part that does not make sense.

Hi @elitak, thanks for your reply!

I did not know about your branch, I wrote this from scratch.

  • To get sha256 in the json would be nice but I see that as a future improvement, not as a blocker. Is there any discussion (in the forums?) on this topic that I could follow up on?
  • Authentication for downloads is not that much of an issue from my point of view, since factorio has a token authentication and I don't need to provide my password anymore. Ideally I'd like to have pkgs.factorio.mods (and pkgs.factorio-experimental.mods), so I have to "override" only one package to add username and token. That would be a breaking change, but my guess is that nobody uses the current factorio-mods since they are pretty outdated.
  • For the mod versions: My plan was to provide always the latest versions compatible with factorio and factorio-experimental and refresh regularly. My hope was that this way we might always get mod versions compatible to each other and for more difficult cases, provide a simple way to add own mods with a fixed mod version. Creating a mirror is really difficult, since you have to make sure that you are allowed to distribute the mods you are distributing and AFAIK they can all have their own licenses and restrictions.

My current "workflow" when playing factorio is to use services.factorio and the client syncs with the server (so I think, we don't need to be able to provide mods for the non-headless version anymore in the factorio package, but that's another story). And with the mod sync it's less of a problem to have slightly outdated versions on the server, since it's fixed automatically when connecting to the game. At least that's my current observation and hope.

I would prefer to get something merged rather sooner than later and base the flakes version on it.

@jonringer
Copy link
Contributor

before making this ready for review, please squash all fixup commits, as per CONTRIBUTING.md

@elitak
Copy link
Contributor

elitak commented Jun 9, 2020

I think I committed 80% of the important stuff to that mods.nix and functions.nix that are in the master branch of nixpkgs. Anything beyond that was a tangled mess, which is part of the reason I abandoned the effort.

I maybe asked for sha256, but didn't follow up on it. If you post in the right forum it should get the lead dev's attention, they are pretty good at watching for requests.

I think the best solution is to provide users with the means to turn the JSON they can fetch with their credentials into a mods.nix that they can pull form, e.g. run nix-shell -p factorio --run 'factorio-build-modlist > /home/user/mods.nix, then in their configuration.nix: services.factorio.mods = with (import /home/user/mods.nix); [ modName-1.2 othermod-2.0 modWhoseVersionIWantTheNewestOf ];

( What follows is historical stuff I typed out that doesn't really matter anymore )

Sha256 was a bit of a stopper to me, because the way I saw it was that I would have to mirror & sha256-hash has the entire set of mods, then furnish a binary nix cache to the end users. Maybe I was naive, or maybe hashes were completely absent from the json, but I suppose you could use whichever hashing algorithm they have now, if you can adapt the verification function.

The other aspect is that rather than ship the mod.zip files I was creating derivations that may have involved unpacking them; I don't quite recall my reasoning on that. Maybe something to do with running the binary nix cache and having the sha256 hashes known.

As far as legal mirroring goes, my plan was to make a nixos module so that anyone could enable a service and have their own mirror, using their credentials. Whether they choose to disseminate that URL to friends or the public is out of our hands, but ought to be discouraged. Again though, running a mirror is far less than ideal.

I also had plans to parse the dependency format that the factorio mods uses to pull the proper versions as dependencies too. Simply using the HEAD of each is probably fine, in most cases.

@SuperSandro2000
Copy link
Member

Sha256 was a bit of a stopper to me, because the way I saw it was that I would have to mirror & sha256-hash has the entire set of mods, then furnish a binary nix cache to the end users. Maybe I was naive, or maybe hashes were completely absent from the json, but I suppose you could use whichever hashing algorithm they have now, if you can adapt the verification function.

Factorio mods use sha1.

The other aspect is that rather than ship the mod.zip files I was creating derivations that may have involved unpacking them; I don't quite recall my reasoning on that. Maybe something to do with running the binary nix cache and having the sha256 hashes known.

The Factorio devs recommend to not unzip the mods except if you are developing them to have a fast developing cycle.

As far as legal mirroring goes, my plan was to make a nixos module so that anyone could enable a service and have their own mirror, using their credentials. Whether they choose to disseminate that URL to friends or the public is out of our hands, but ought to be discouraged. Again though, running a mirror is far less than ideal.

Sharing Factorio mods unmodified is allowed as long as you don't proclaim that it is your work or modify them. The earlier MP versions uploaded mods quite often but I am not sure if MP still does that.

I also had plans to parse the dependency format that the factorio mods uses to pull the proper versions as dependencies too. Simply using the HEAD of each is probably fine, in most cases.

The version ranger in there are usually out of date and I would enjoy it more if you could just track latest or a specific version. No need to parse the info.json.

@stale
Copy link

stale bot commented Jul 20, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.status: merge conflict 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 10.rebuild-darwin: 0 10.rebuild-linux: 0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants