cc @schmittlauch

Hey. This looks good to me. Is it feasible though to add mastodons npm packages to
our node-packages set though? We are already doing this for other packages like lumo or iosevka:

Hey. This looks good to me. Is it feasible though to add mastodons npm packages to
our node-packages set though? We are already doing this for other packages like lumo or iosevka:

nixpkgs/pkgs/development/node-packages/node-packages-v10.json

Line 80 in 8539d5f

, {"lumo-build-deps": "../interpreters/clojurescript/lumo" }

Afaik there are no official user-targeted mastodon npm packages. The webapp itself is not an npm package either, but it is exposed as a nix derivation in the nixpkgs attrset (as pkgs.mastodon.mastodon-assets). The yarn/npm dependencies used to build the client are exposed at pkgs.mastodon.mastodon-js-modules.
What would be the use case for exposing the build deps, which are not user-targeted, in node-packages?

@petabyteboy The idea is that we can share npm package source between different packages in nixpkgs instead of having 11.000 lines of generated nix expressions per package, which affects also evaluation time.

@petabyteboy The idea is that we can share npm package source between different packages in nixpkgs instead of having 11.000 lines of generated nix expressions per package, which affects also evaluation time.

Ah I see. But I doubt this would be possible since mastodon is using yarn and not npm.
It would be nice to see support for this kind of optimization in yarn2nix, possibly also sharing the node packages expression with node2nix.

Ah I see. But I doubt this would be possible since mastodon is using yarn and not npm.
It would be nice to see support for this kind of optimization in yarn2nix, possibly also sharing the node packages expression with node2nix.

Do both not use a packages.json. It thought that this could be read by npm as well.

Do both not use a packages.json. It thought that this could be read by npm as well.

Yes and no, the mastodon repository contains a yarn.lock file with extra version information that can not be described in the original package.json format. NPM added a package-lock.json file to support the same thing, but it is incompatible with the yarn.lock format.

Do both not use a packages.json. It thought that this could be read by npm as well.

Yes and no, the mastodon repository contains a yarn.lock file with extra version information that can not be described in the original package.json format. NPM added a package-lock.json file to support the same thing, but it is incompatible with the yarn.lock format.

So it won't capture the some indirect dependencies pinnings? Does this matter in our case though?

So it won't capture the some indirect dependencies pinnings? Does this matter in our case though?

I think so. Even if it is possible to package the dependencies with node2nix, the dependency versions could change every time the nix expression is generated. With such a complex application I think it is crucial to reproduce the dependencies as close to what upstream intends, and they clearly instruct users to use yarn when building the frontend.
I think it's more important to make sure that the application will work correctly before starting to optimize the evaluation time. Also it wouldn't be indirect dependencies only that are affected.

But on the other hand we also need to keep nixpkgs usuable. This packages adds 435.2KB of data to nixpkgs that everyone needs to download (Our node-packages.nix is 2.6M with all our packages). I get regularly complaints of people not being able to run nixpkgs-review with local evaluation because lack of RAM.

   expected: "sr-Latn"
        got: "sr"

Could this be leakage from your environment into the test environment? What is your i18n.defaultLocale?

   expected: "sr-Latn"
        got: "sr"

Could this be leakage from your environment into the test environment? What is your i18n.defaultLocale?

No, this output is from the NixOS test you wrote.

   expected: "sr-Latn"
        got: "sr"

Could this be leakage from your environment into the test environment? What is your i18n.defaultLocale?

No, this output is from the NixOS test you wrote.

On further investigation, it appears to be a bug in the Mastodon test which happens rarely due to the use of randomization in the test. I'll report it upstream.

After some testing I figured there should be an option to disable SMTP_LOGIN and SMTP_PASSWORD as it will block access to local smtp server without authentication.
Even with auth method to none like in the following example it will not work if the login and password variables are defined:

SMTP_AUTH_METHOD = "none";
SMTP_OPENSSL_VERIFY_MODE = "none";

After some testing I figured there should be an option to disable SMTP_LOGIN and SMTP_PASSWORD as it will block access to local smtp server without authentication.

Does this work?

services.mastodon.extraConfig = {
    SMTP_LOGIN = "";
    SMTP_PASSWORD = "";
};

If not, please share your local smtp server configuration and steps to reproduce the problem.

But it's not possible to merge those definition, no? It would be fine we could merge all those definitions some somehow...

Let's continue the discussion at #83499.

@happy-river I'm curious if you've had trouble generating user accounts? Using your module I repeatedly get "This email is invalid" messages when trying to sign up.

Curious what you did?

I'm using a mail server on a vps outside my lan.
But I've also attempted to use Gmail's smtp server too to no change.

I'm using this module for a long time now, what is blocking it from getting merged ?

Currently there's a merge conflict with all-tests.nix, likely since the line got modified in another commit. Should be easy to resolve.

Besides that are there any current blockers for this PR?

I just upgraded to v3.3.0. I tested this on a small instance and it seems to work. This time the upgrade included some manual steps, see the commit message for details.

I hope this is ok for everyone, that I push to this PR? If anybody uses this in production, you probably need to follow the migration instructions in the release notes.

I get this error when building:

Building native extensions. This could take a while...
ERROR:  Error installing /nix/store/2ysdwqd7a3ahp7dmwqxgizry3zpxfqx1-bcrypt-3.1.16.gem:
        ERROR: Failed to build gem native extension.

    current directory: /nix/store/asnccd5av7zqnhnxkx08yahylkfakg11-ruby2.7.2-bcrypt-3.1.16/lib/ruby/gems/2.7.0/gems/bcrypt-3.1.16/ext/mri
/nix/store/sxvw9ja9i1ndxslrbkird76m3jsamjsg-ruby-2.7.2/bin/ruby -I /nix/store/sxvw9ja9i1ndxslrbkird76m3jsamjsg-ruby-2.7.2/lib/ruby/2.7.0 -r ./siteconf20210111-8-3yq1qn.rb extconf.rb
creating Makefile

current directory: /nix/store/asnccd5av7zqnhnxkx08yahylkfakg11-ruby2.7.2-bcrypt-3.1.16/lib/ruby/gems/2.7.0/gems/bcrypt-3.1.16/ext/mri
make "DESTDIR=" clean
current directory: /nix/store/asnccd5av7zqnhnxkx08yahylkfakg11-ruby2.7.2-bcrypt-3.1.16/lib/ruby/gems/2.7.0/gems/bcrypt-3.1.16/ext/mri
make "DESTDIR="
make failedNo such file or directory - make "DESTDIR="

Gem files will remain installed in /nix/store/asnccd5av7zqnhnxkx08yahylkfakg11-ruby2.7.2-bcrypt-3.1.16/lib/ruby/gems/2.7.0/gems/bcrypt-3.1.16 for inspection.
Results logged to /nix/store/asnccd5av7zqnhnxkx08yahylkfakg11-ruby2.7.2-bcrypt-3.1.16/lib/ruby/gems/2.7.0/extensions/x86_64-linux/2.7.0/bcrypt-3.1.16/gem_make.out

Nothing I've tried is able to reproduce this other than just building the ruby-bcrypt derivation. breakpointHook works, but I can't find any more details on the issue anywhere in the filesystem.

I had a broken sandbox-paths in my nix.conf, not sure how I didn't catch that until now...

Make pkgs.mastodon.override { ... } work to build forks of Mastodon.

To make this a bit easier, perhaps update.sh should create an override.nix that's something like this?

{ mastodon, callPackage }:
mastodon.override {
  version = import ./version.nix;
  srcOverride = callPackage ./source.nix {};
  dependenciesDir = ./.;
}

Merge conflict :)

As nobody else joined in my initiative to get this finally merged (either by approving or merging it), I deduce consensus is not strong enough among the commiters. Still there seems to be great interest in having this. So let's have it as an out-of-tree module?

I started working on a Nix Flake mastodon-nixos, that currently somewhat mirrors the functionality of this PR. I intend to further maintain this Flake and add more functionality in the future (e.g. support for non-flake users). Goal is to move the project into the nix-community org, as I guess that'd be the right place for it. Could a member add me to the org so I can push the repo?

Also I will close this PR now. It is currently based on an commit from August 2020. That means that it uses an old version of ffmpeg and imagemagick, whose should have accumulated multiple security vulnerabilities by now. No one should use this branch without a rebase in production. Feel free to reopen if anyone wants to maintain it, but I currently see not much future in it.

Edit: Got it accepted into nix-community! https://github.com/nix-community/mastodon-nixos

Edit: as #112898 just got merged, there is no need for an out-of-tree module anymore. I'll delete the repository.

Oh gosh, thank you @erictapen ! A flake is a great idea

How to fix this error?:

mastodon-init-dirs-start[91340]: ERROR: Missing RAILS_ENV environment variable, please set it to "production", "development", or "test".

and

mastodon-init-db.service: Failed to load environment files: No such file or directory

So as it turns out there are some people from the nix-community IRC willing to merge this.
I rebased on current master. For that I had to open a new PR at #112898.