Of course we are facing the same IFD issue with yarn2nix as with riot-desktop and #59111.
It seems like the only solution for now is including yarn.nix and package.json in nixpkgs.

Unfortunately it seems like this is a dead end:
moretea/yarn2nix uses builtins.fetchGit, because the yarn.lock file does not contain hashes for git dependencies. builtins.fetchGit doesn't work on Hydra/OfBorg, because it runs at eval-time in restricted mode and network access is not allowed.

I will try to use https://github.com/Profpatsch/yarn2nix tomorrow, which hashes the git dependencies when creating the nix expression, and then uses pkgs.fetchgit. Since we have to include a pregenerated Nix expression for the dependencies anyways (otherwise we get IFD problems), this is a small loss.

Can I do ...? @GrahamcOfBorg eval
As I understand it, you should be able to eval, but not build, unless you're a known or trusted user, in OfBorg terminology.

That's what I concluded too, but as it says in the OfBorg readme, there is no reason to call eval manually since it happens automatically.

After trying to integrate profpatsch/yarn2nix into nixpkgs, I give up on that. There are multiple broken Haskell packages required for profpatsch/yarn2nix to run.

My new plan is to add support for Hydra-enabled git dependencies to moretea/yarn2nix.

Hooray /o/

Thanks, I think I have implemented all your suggestions.

I don't know of a good way to add yarn2nix without forking the code base. When we had it in the tree it started diverging quickly with both sides receiving patches. Before adding it back we need to decide on how to resolve that issue. A few possibilities:

1. Add yarn2nix as a git submodule to nixpkgs, probably doesn't work with hydra
2. Use git substree to keep yarn2nix in sync. This needs a bit of a wrapper scripts action.
3. Archive upstream, declare nixpkgs to hold the canonical version.
4. ?

I don't know of a good way to add yarn2nix without forking the code base. When we had it in the tree it started diverging quickly with both sides receiving patches. Before adding it back we need to decide on how to resolve that issue. A few possibilities:

1. Add yarn2nix as a git submodule to nixpkgs, probably doesn't work with hydra
2. Use git substree to keep yarn2nix in sync. This needs a bit of a wrapper scripts action.
3. Archive upstream, declare nixpkgs to hold the canonical version.
4. ?

I agree it has to be discussed and well-thought-through.
Of the solutions you suggested, I prefer 3., but I have another idea:

Actually yarn2nix can be split into two parts, similar to bundix and bundlerEnv.

• One part is responsible for generating the nix expression and is maintained in a seperate repository. This command line tool can be provided in nixpkgs just like all other tools with fetchFromGitHub.
• The other part contains everything for building a node_modules folder from the generated nix expression and is kept in nixpkgs.

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/has-anyone-ran-mastodon-on-nixos/1440/9

Thanks for the work, I will test this on my server as soon as possible

My server is on 19.03 so I will need a bit of work to backport this it seems

My server is on 19.03 so I will need a bit of work to backport this it seems

It shouldn't be too much, if you have any questions let me know

My server is on 19.03 so I will need a bit of work to backport this it seems

I just tried to build the package with 19.03. You only have to cherry-pick #60693 and somehow the hashes of the two ruby dependencies that are pulled from git changed depending on what version of nixpkgs I used, so I needed to change those.

Can you please make sure the state directory is configurable?
It would help immensely for keeping control over user generated content, such as avatars, media, etc. which needs to be backed up regularly.
A typical name for this option in nix services is dataDir which could easily default to /var/lib/mastodon.

Thanks for your effort. 👍

Can you please make sure the state directory is configurable?
It would help immensely for keeping control over user generated content, such as avatars, media, etc. which needs to be backed up regularly.
A typical name for this option in nix services is dataDir which could easily default to /var/lib/mastodon.

Thanks for your effort. +1

@kampka can you please elaborate on what backup and control needs you have that won't work with /var/lib/mastadon?

... that was me trying a button without a label and clicking it twice because the UI didn't respond.

So far I think I have addressed all comments other than comments that contradict eachother (discussions on other topics). From my side this is ready to go. Sure, I can probably work on this for another 5 months, but this is good enough to be included in nixpkgs in my view.

I have lost track... what was the decision on node and yarn? Introduced back into nixpkgs?

yarn2nix was introduced back into nixpkgs as yarn2nix-moretea in #60429. node2nix was never really mentioned in this discussion, rather there were multiple competing projects with the name yarn2nix by Profpatsch and moretea.
There were concerns about the maintainability of yarn2nix-moretea in nixpkgs, because in the past we had lost track of upstream. As I understand this is now solved by an update script.

Out of people who are following this thread and have commit access: are there (m)any remaining holdups on this?

I'll have to insist at this point the modifications are dropped on postgresql simply because there is a larger conversation happening that isn't resolved. Sorry to be a hassle.

Okay, I will do the following:

• I will create a branch on my fork to preserve the current state with postgresql changes.
• I will remove the postgresql changes and all functionality that depends on it on this branch, even though some people will be unhappy about using the migrations on an empty database

Also... in the future keep in mind that you have been exceptionally accommodating to many people's requests in this PR. Someone stating a feature they think would be nice is not the same thing as a review that is critiquing code that has problems associated with it, or required missing features that you have overlooked. So don't feel bad to push back of you don't see value in a feature because after all it is you doing the work.

Except some people do state their (hard to fulfill) wishes as if they were requirements, quoting a comment from this PR: "i would feel bad about merging this if there isn't first-class support for custom forks of mastodon"

I have removed the postgresql changes and everything that depended on it. The version with those changes can be found in the branch feature/mastodon-postgres on my fork.

Note that the webserver test will not pass without the changes in the postgresql module, because manual intervention is required to initialize the database.

Actually it does work by using the migration chain, it's just quite slow.

to clarify my wishes about custom forks - i meant that more as a blanket statement about making it easy to update mastodon to any release, custom fork or not, because it seemed difficult to assemble all the required components and potentially easy to miss something. i was just advocating for an update script, really. sorry for confusion

Actually it does work by using the migration chain, it's just quite slow.

It's too slow for the amount of patience I have, so I found a way to test whether a postgresql database has no tables yet and use rake db:schema:load instead of rake db:migrate in that case. I've also ported the two mastodon tests to the new Python test driver. Both commits can be found in my mastodon-python-tests branch, which is based on this pull request rebased onto a recent master.

I have pushed your changes to my branch. Thanks!

I've now successfully created a Mastodon instance using this module and NixOps and a configuration that looks like this:

  tooter = { config, pkgs, ... }: {
    services.mastodon = {
      enable = true;
      configureNginx = true;
      localDomain = "example.com";
      smtp = {
        host = "smtp.example.org";
        port = 587;
        user = "user@mail.example.com";
        fromAddress = "Administrator <admin@mail.example.com>";
      };
    };
    networking.firewall.allowedTCPPorts = [ 80 443 ];
  };

I deployed it, copied my SMTP password over, restarted it, used its web interface to create an account, and then needed to use tootctl to promote the account to an administrator. But tootctl needs Mastodon's environment variables to be set up correctly, or it won't work. My solution to that problem can be found in my 60788-tootctl branch. It creates a shell script mastodon-env which sets up the environment for another command. With that done, making an administrator account became:

$ nixops ssh -d mastodon tooter
[root@tooter:~]# su - mastodon -s /bin/sh
[mastodon@tooter:~]$ mastodon-env tootctl accounts modify myaccount --role admin

This works, but I'm open to better solutions. One feature that might be added to mastodon-env is to make it optionally cd to the Mastodon package directory, which would be helpful if you want to use it to run any rake tasks.

This project has caused me too much stress and I'm not really interested in improving it any further. Maybe @happy-river can open a PR and continue this.

Don't worry, Take care of you @petabyteboy !

Thanks for everything you've done on this @petabyteboy ! I will continue to work on this and will create a new PR in a few days.

I actually got this working with a few monkeypatches. Been running it for almost a month. Seems like it shouldn't need much more work.

I've just created #78810 which contains this pull request with improvements.