We definitely need this! Any reason for building using crystal instead of shards?

@peterhoeg I have zero experience with both crystal and shards, I just followed what the mint derivation was doing. Maybe crystal is preferable here because it also allows the compilation of the single crystal2nix.cr file, whereas shards I'd think would need a shard.yaml file.

shards seems to automatically know what file to build though, so maybe that would be nice to support too (in the future?).

I pushed a minor mint change for removing all buildInputs except openssl, because the others didn't end up being needed

I also added a commit for adding a Crystal section to the nixpkgs docs

whereas shards I'd think would need a shard.yaml file

When you do crystal init app foo, it will create a shard.yml file with (amongst other things), this:

targets:
  foo:
    main: src/foo.cr

Which is enough for you to build with shards build.

Except for crystal2nix.cr, I don't think anybody invokes the crystal compiler directly.

@peterhoeg Since we need this functionality anyways for crystal2nix, I'd say let's just leave it at this for now. Adding a way of using shards to build packages is probably not worth it right now because there's almost no crystal packages, can always be added later though.

Just added a commit to switch crystal to openssl_1_0_2 which makes crystal play work.

I'd say let's just leave it at this for now.

Fair enough. It's definite improvement on the existing situation. lgtm when the darwin test passes.

Hey, I wonder if there is a way to avoid needing crystal2nix. I would prefer if there story of someone using crystal on nix would not be that different from crystal on other platform.

When installing crystal dependencies described in a shard.yml a shard.lock is generated. That will have for example the git tag or the commit ref if the dependency was a branch.

When installing crystal dependencies with a shard.lock present that will be used. Nothing novel. I know that if the tag of a dependency is removed/changed the lock will be useless for reproducing.

If shards would include always the git commit in the lock file and install from it, would that be enough to avoid needing crystal2nix? Or what else would be needed?

using a lock file to pull in dependencies is something that's already done with rust and go packages. All it takes is someone implementing it.

Are there any requirements for those lock files to be enough for Nix? Eg: always include the explicit commit ref instead of tag.

generally there's some pull step in-which the dependencies defined in the lock file have to manifest themselves as a derivation which then can be consumed downstream.

For go, this is looking at the go.mod file, pulling all the related dependencies, then passing that to the actual build.

@bcardiff Almost transparent Nix support is possible with the following:

• The lock file format needs to be parseable with Nix. While custom parsers can be written, this is difficult and error-prone as Nix isn't made for parsing, there are also no parsing libraries. So ideally a builtin parser should be usable, which are currently only builtins.fromJSON and builtins.fromTOML. So the lock file being JSON or TOML would be best and fastest.
• The lock file needs to include content hashes for all dependencies. This is a bit of a problem though, because a lot of tarballs (e.g. the ones GitHub produces) change over time, which is why in Nix uses the hash of the unpacked directory (see here) instead. We can't hash directories though, so Nix actually first packs the directory into a deterministic NAR (Nix ARchive) format, from which it can compute and check the hash. So for hashes in a lock file to be usable by Nix, they need to be hashes over the NAR format. Since crystal probably shouldn't depend on Nix to compute these hashes, the NAR format could be implemented in crystal instead. Here is the reference implementation in Nix's source. Consequently, the hashes should then be under a narHash field in the lock file.
• The format of the hashes is ideally SRI, which Nix recently gained support for.

The only project I know of that has transparent Nix support is poetry using poetry2nix which only exposes a set of Nix functions for building poetry projects without any code generation necessary. This is also what allows poetry2nix to be used in nixpkgs by just checking in the project and lock file. For poetry I believe they don't need to mess with NAR hashes because all dependencies are downloaded from pypi's servers, which probably don't ever update tarballs (unlike e.g. GitHub).

Pinging @adisbladis as poetry2nix author, and who pushed poetry towards making their lock files Nix compatible.

@manveru is already working on this ref #89863 (comment)

@manveru I think your lock file transformer function wouldn't work for nixpkgs because of builtins.fetchGit, which does eval time fetching and would therefore slow down hydras evaluation.

And ideally we wouldn't have to do IFD or code generation also (which is the case with poetry2nix). IFD is no good for nixpkgs and code generation is more maintenance.

With the points I mentioned above both of these problems wouldn't be there.

Oh actually poetry2nix used builtins.fetchGit too, is that not a problem for nixpkgs? If it's really not then I guess that could be used, and the NAR hashing wouldn't be necessary.