dwarf-fortress: Refactor, round 2: let the user pick which version of the lazy pack to install #43470

numinit · 2018-07-13T09:49:26Z

Motivation for this change

Dwarf Fortress unfuck, dfhack, and TWBT are tied to specific Dwarf Fortress versions, which makes the package harder to maintain and useless for people who want to run an older (or newer) version of the lazy pack. Let the user decide the Dwarf Fortress version, and we'll choose the versions of unfuck, dfhack, and TWBT for them based on the latest available for those versions.

Also, improve the launch behavior of Dwarf Therapist by creating a symlink to memory_layouts in the user's home directory, similar to how Dwarf Fortress already behaves.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

numinit · 2018-07-13T09:52:30Z

pkgs/games/dwarf-fortress/dwarf-therapist/dwarf-therapist.in

+cd "$install_dir/share/dwarftherapist"
+update_path memory_layouts
+
+QT_QPA_PLATFORM_PLUGIN_PATH="@qt_plugin_path@" \


Is there a better way to fix this? Seems to be the same as #37864.

Yeah it’s pretty much unavoidable. You could use wrapProgram though (it will generate a file that looks pretty close to this anyway).

numinit · 2018-07-13T09:53:02Z

pkgs/games/dwarf-fortress/dwarf-therapist/wrapper.nix

+      --subst-var-by stdenv_shell ${stdenv.shell} \
+      --subst-var-by install $out \
+      --subst-var-by therapist ${dwarf-therapist} \
+      --subst-var-by qt_plugin_path "${pkgs.qt5.qtbase}/lib/qt-5.11/plugins/platforms"


We shouldn't be hardcoding the version, need to fix this.

numinit · 2018-07-13T09:53:50Z

dfhack is currently broken with this, I don't think we're updating the MD5 correctly anymore. God this package is complicated...

Anton-Latukha · 2018-07-13T15:06:50Z

I wanted to say thanks.

I love Dwarf Fortress. But could not solve the puzzle fully, what all packages do, and how to make tilesets/LNP work.

Please, can you add comprehensive descriptions to packages? So people can understand what are those and how it works together.

numinit · 2018-07-13T17:02:35Z

Yeah, it could use some examples. The docs are a little (accidentally) misleading in some places.

Here's basically how to install the lazy pack with FPS enabled and a theme, assuming your reference to the unstable nixpkgs tree is called unstable:

      (unstable.dwarf-fortress-packages.dwarf-fortress-full.override {
        theme = "cla";
        enableFPS = true;
      })

matthewbauer · 2018-07-13T18:18:38Z

pkgs/games/dwarf-fortress/default.nix


    dwarf-fortress-full = callPackage ./lazy-pack.nix { };

-    dfhack = callPackage ./dfhack {


I am a little concerned about having too many versions of each package here. Does dfhack not support older versions of dwarf fortress?

Yep. If you check their release page, they dropped 0.44.11 support in the 0.44.12-alpha1 release. The base game sometimes changes too much for them to retain backwards compatibility.

Actually, looking at their release history, they always break backwards compatibility when Dwarf Fortress changes. Oi...

pkgs/games/dwarf-fortress/default.nix

matthewbauer · 2018-07-13T18:20:33Z

pkgs/games/dwarf-fortress/default.nix

+    value =
+      let
+        # I can't believe this syntax works. Spikes of Nix code indeed...
+        dwarf-fortress = callPackage ./game.nix {


I think it makes sense to put these in the callPackage wrapper.nix {...}. It should only be needed there. I think the same thing can be done for dfhack as well.

I've inlined those JSON files.

Maybe my knowledge of Nix is incomplete, but I think we might be stuck with the let ... in syntax here, because of the dependencies between packages?

matthewbauer · 2018-07-13T18:21:01Z

pkgs/games/dwarf-fortress/default.nix

@@ -59,7 +75,6 @@ let

    legends-browser = callPackage ./legends-browser {};

-    twbt = callPackage ./twbt {};


I was hoping this would be version independent. Is that not how it works currently?

Correct. Check out mifki/df-twbt:patches.hpp; there is a large if/elif defined(DF_$VERSION) chain. That version code is originally set in mifki's Makefile.

pkgs/games/dwarf-fortress/dfhack/dfhack.json

matthewbauer · 2018-07-13T18:24:00Z

pkgs/games/dwarf-fortress/dwarf-therapist/dwarf-therapist.in

+cd "$install_dir/share/dwarftherapist"
+update_path memory_layouts
+
+QT_QPA_PLATFORM_PLUGIN_PATH="@qt_plugin_path@" \


Yeah it’s pretty much unavoidable. You could use wrapProgram though (it will generate a file that looks pretty close to this anyway).

matthewbauer · 2018-07-13T18:24:26Z

pkgs/games/dwarf-fortress/dwarf-therapist/wrapper.nix

-    # therefore uses many relative paths.
-    wrapProgram $out/bin/dwarftherapist \
-      --run "cd $out/share/dwarftherapist"
+  buildCommand = ''


This looks wrong. buildCommand will override symlinkJoin behavior. Is there a reason to use this instead of postBuild?

When running in nix-shell, it seemed like the "unsymlink" code was trying to to stomp over a different package's init directory, and it complained about not being able able to delete init.txt due to permissions.

Specifically, here's the log when we use postBuild rebased against latest master. The symlink alias (dwarftherapist/DwarfTherapist) now fails.

I should also note that I'm trying to use it in a container.

building '/nix/store/5r6wa7f8hlymgbf2ms90c9ljmsxy5dph-dwarf-therapist-40.1.1.drv'... chmod: changing permissions of '/nix/store/q8pqivs0xh34vgil8hmh2xkzpwk18k0c-dwarf-therapist-40.1.1/bin/dwarftherapist': Operation not permitted builder for '/nix/store/5r6wa7f8hlymgbf2ms90c9ljmsxy5dph-dwarf-therapist-40.1.1.drv' failed with exit code 1 cannot build derivation '/nix/store/w1ld9xwxchf7zwydi9zd1wzf7b88pv7k-dwarf-fortress-full.drv': 1 dependencies couldn't be built cannot build derivation '/nix/store/id7vygj7inr00zlwzzvynbj8z182nsa2-system-path.drv': 1 dependencies couldn't be built cannot build derivation '/nix/store/5g52k9mq297c8allssnknxhmrzn4ma8f-nixos-system-df-18.03.132750.91b286c8935.drv': 1 dependencies couldn't be built cannot build derivation '/nix/store/bj4kbigg0x3cfy7kxy59axkn6ssgl3v4-etc-df.conf.drv': 1 dependencies couldn't be built cannot build derivation '/nix/store/r315hpn7122l851h15715vlvhak5lf4p-unit-container-df.service.drv': 1 dependencies couldn't be built cannot build derivation '/nix/store/pqkkn9h8qbaawqbla0yg5wadx0r67rpp-etc.drv': 1 dependencies couldn't be built cannot build derivation '/nix/store/v88s4vvcggplz3d6ik50v3z5qhbm3xsx-nixos-system-battlespire-18.03.132750.91b286c8935.drv': 1 dependencies couldn't be built error: build of '/nix/store/v88s4vvcggplz3d6ik50v3z5qhbm3xsx-nixos-system-battlespire-18.03.132750.91b286c8935.drv' failed

I changed it to use stdenv.mkDerivation. Is there a reason why symlinkJoin was necessary for the wrapper? It seems to work fine using stdenv.mkDerivation and preferLocalBuild like the DF wrapper.

matthewbauer · 2018-07-13T18:25:25Z

pkgs/games/dwarf-fortress/wrapper/default.nix

  };
 in

 stdenv.mkDerivation rec {
  name = "dwarf-fortress-${dwarf-fortress.dfVersion}";

-  compatible = lib.all (x: assert (x.dfVersion == dwarf-fortress.dfVersion); true) pkgs;
-


I think there may still be a use for this. Need to hear from other maintainers first on need for it though.

numinit · 2018-07-13T19:59:11Z

numinit · 2018-07-15T00:21:44Z

It looks like dfhack isn't working correctly because the wrapper's postBuild is never running. Any reason for this?

pkgs/games/dwarf-fortress/wrapper/default.nix

numinit · 2018-07-15T04:00:51Z

Missed that error this whole time... the MD5 fixup wasn't running in the correct environment.

numinit · 2018-07-15T07:42:59Z

Removing the [WIP] tag; most of the important changes are there now. This fixes the fact that you could potentially end up with an incompatible Dwarf Fortress/dfhack/Dwarf Therapist/unfuck combination before.

Mic92 · 2018-10-15T16:50:11Z

pkgs/games/dwarf-fortress/themes/default.nix

@@ -1,4 +1,4 @@
-{lib, fetchFromGitHub}:
+{lib, fetchFromGitHub, ...}:


Why the ...

Not sure, was there previously. Can remove it.

Edit: NVM, I added it. Will still remove it. :-)

Mic92 · 2018-10-15T16:52:42Z

pkgs/games/dwarf-fortress/dfhack/default.nix

-  dfVersion = "0.44.12";
-  version = "${dfVersion}-r1";
+  dfhack-releases = {
+    "0.43.05" = {


Based on what criteria are releases included here?

I would prefer to have less versions listed here, because it makes the build matrix one has to test rather big.

@matthewbauer added them in #43085. I agree, some of these versions are rather old.

numinit changed the title ~~dwarf-fortress: [WIP] refactor, round 2: support multiple unfuck/dfhack/TWBT versions~~ dwarf-fortress: [WIP] refactor, round 2: dwarf-therapist 40.1.1; dwarf-fortress 0.44.12; support multiple unfuck/dfhack/TWBT versions Jul 13, 2018

numinit commented Jul 13, 2018

View reviewed changes

GrahamcOfBorg added 8.has: clean-up 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-linux: 11-100 labels Jul 13, 2018

matthewbauer requested changes Jul 13, 2018

View reviewed changes

numinit force-pushed the df branch 2 times, most recently from b234377 to 558fbb1 Compare July 14, 2018 23:09

numinit force-pushed the df branch from 558fbb1 to b7de93a Compare July 15, 2018 01:10

numinit commented Jul 15, 2018

View reviewed changes

pkgs/games/dwarf-fortress/wrapper/default.nix Outdated Show resolved Hide resolved

numinit changed the title ~~dwarf-fortress: [WIP] refactor, round 2: dwarf-therapist 40.1.1; dwarf-fortress 0.44.12; support multiple unfuck/dfhack/TWBT versions~~ dwarf-fortress: [WIP] Refactor, round 2: let the user pick which version of the lazy pack to install Jul 15, 2018

numinit changed the title ~~dwarf-fortress: [WIP] Refactor, round 2: let the user pick which version of the lazy pack to install~~ dwarf-fortress: Refactor, round 2: let the user pick which version of the lazy pack to install Jul 15, 2018

numinit added 7 commits September 9, 2018 06:59

dwarf-fortress: Support multiple unfuck/dfhack/TWBT versions

4ada74e

Fix dfhack's Dwarf Fortress MD5 in the correct environment

f14d3b4

Let the user override dfVersion in dwarf-fortress-full

7a55215

Therapist needs to depend on the DF version, since it uses its MD5

fe847fc

Fix dangling DFHack "Git:" version text

870f05c

Use stdenv.mkDerivation for Therapist wrapper

5f1013d

Update twbt to 6.54

2cd2571

numinit force-pushed the df branch from 2c8aea4 to 2cd2571 Compare September 9, 2018 07:04

Inline JSON files

213b545

matthewbauer merged commit e4db8a9 into NixOS:master Sep 9, 2018

Mic92 reviewed Oct 15, 2018

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dwarf-fortress: Refactor, round 2: let the user pick which version of the lazy pack to install #43470

dwarf-fortress: Refactor, round 2: let the user pick which version of the lazy pack to install #43470

numinit commented Jul 13, 2018 •

edited

numinit Jul 13, 2018

matthewbauer Jul 13, 2018

numinit Jul 13, 2018

numinit Jul 14, 2018

numinit commented Jul 13, 2018

Anton-Latukha commented Jul 13, 2018

numinit commented Jul 13, 2018

matthewbauer Jul 13, 2018

numinit Jul 13, 2018

numinit Jul 13, 2018 •

edited

matthewbauer Jul 13, 2018

numinit Sep 9, 2018

matthewbauer Jul 13, 2018

numinit Jul 13, 2018 •

edited

matthewbauer Jul 13, 2018

matthewbauer Jul 13, 2018

numinit Jul 13, 2018

numinit Jul 14, 2018 •

edited

numinit Jul 15, 2018

matthewbauer Jul 13, 2018

numinit commented Jul 13, 2018 •

edited

numinit commented Jul 15, 2018 •

edited

numinit commented Jul 15, 2018

numinit commented Jul 15, 2018

Mic92 Oct 15, 2018

numinit Oct 15, 2018 •

edited

Mic92 Oct 15, 2018 •

edited

Mic92 Oct 15, 2018 •

edited

numinit Oct 15, 2018


		dwarf-fortress-full = callPackage ./lazy-pack.nix { };

		dfhack = callPackage ./dfhack {

		@@ -59,7 +75,6 @@ let

		legends-browser = callPackage ./legends-browser {};

		twbt = callPackage ./twbt {};

		@@ -1,4 +1,4 @@
		{lib, fetchFromGitHub}:
		{lib, fetchFromGitHub, ...}:

dwarf-fortress: Refactor, round 2: let the user pick which version of the lazy pack to install #43470

dwarf-fortress: Refactor, round 2: let the user pick which version of the lazy pack to install #43470

Conversation

numinit commented Jul 13, 2018 • edited

Motivation for this change

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

numinit commented Jul 13, 2018

Anton-Latukha commented Jul 13, 2018

numinit commented Jul 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

numinit Jul 13, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

numinit Jul 13, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

numinit Jul 14, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

numinit commented Jul 13, 2018 • edited

numinit commented Jul 15, 2018 • edited

numinit commented Jul 15, 2018

numinit commented Jul 15, 2018

Choose a reason for hiding this comment

numinit Oct 15, 2018 • edited

Choose a reason for hiding this comment

Mic92 Oct 15, 2018 • edited

Choose a reason for hiding this comment

Mic92 Oct 15, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

numinit commented Jul 13, 2018 •

edited

numinit Jul 13, 2018 •

edited

numinit Jul 13, 2018 •

edited

numinit Jul 14, 2018 •

edited

numinit commented Jul 13, 2018 •

edited

numinit commented Jul 15, 2018 •

edited

numinit Oct 15, 2018 •

edited

Mic92 Oct 15, 2018 •

edited

Mic92 Oct 15, 2018 •

edited