Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

funkwhale: init at 0.20.0 #53416

Closed
wants to merge 2 commits into from
Closed

funkwhale: init at 0.20.0 #53416

wants to merge 2 commits into from

Conversation

mmai
Copy link
Contributor

@mmai mmai commented Jan 4, 2019

Motivation for this change

This adds a new package and module which enable the Funkwhale music web service (https://funkwhale.audio/).

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.

Copy link
Member

@infinisil infinisil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All in all, this PR is far from done and needs a lot of work. Note that a lot of the comments I left also apply to many other places in your changes.

nixos/modules/services/web-apps/funkwhale/requirements.nix Outdated Show resolved Hide resolved
pkgs/servers/web-apps/funkwhale/default.nix Outdated Show resolved Hide resolved
pkgs/servers/web-apps/funkwhale/0001-changes.patch Outdated Show resolved Hide resolved
nixos/modules/services/web-apps/funkwhale/funkwhale.nix Outdated Show resolved Hide resolved
nixos/modules/services/web-apps/funkwhale/funkwhale.nix Outdated Show resolved Hide resolved
nixos/modules/services/web-apps/funkwhale/funkwhale.nix Outdated Show resolved Hide resolved
nixos/modules/services/web-apps/funkwhale/funkwhale.nix Outdated Show resolved Hide resolved
nixos/modules/services/web-apps/funkwhale/funkwhale.nix Outdated Show resolved Hide resolved
nixos/modules/services/web-apps/funkwhale/funkwhale.nix Outdated Show resolved Hide resolved
nixos/modules/services/web-apps/funkwhale/funkwhale.nix Outdated Show resolved Hide resolved
@matthiasbeyer
Copy link
Contributor

Any progress here?

@mmai mmai changed the title funkwhale: init at 0.18 funkwhale: init at 0.19 Jun 29, 2019
@ivan
Copy link
Member

ivan commented Sep 6, 2019

For the author, reviewers, and committers: this PR was scanned and appears to add a use of the deprecated types.string, which emits a warning as of #66346. Before merging, please change this to another type, possibly:

  • types.str for a single string where merging does not make sense, or cannot work
  • types.lines for multi-line configuration or scripts where merging is possible
  • types.listOf types.str for a mergeable list of strings

@FRidh
Copy link
Member

FRidh commented Sep 6, 2019

Nothing happening so closing.

@FRidh FRidh closed this Sep 6, 2019
@mmai
Copy link
Contributor Author

mmai commented Sep 7, 2019

@FRidh I'm not sure I understand. Do you expect more actions from me ? I've made all the requested changes for the initial pull request but my fixes didn't received any review.

@FRidh FRidh reopened this Sep 7, 2019
@mmai mmai changed the title funkwhale: init at 0.19.1 funkwhale: init at 0.20.0 Oct 7, 2019
@worldofpeace
Copy link
Contributor

@mmai I'll see about integrating the python packages later today.

@worldofpeace
Copy link
Contributor

Lol more like today now 🤣

@worldofpeace
Copy link
Contributor

Merged all python modules in #71262.

We need to drop all those commits from here, and the funkwhale commit needs a rewrite to not add you to the maintainer list. I had to do that separately for merging the modules 3895e09.

@mmai
Copy link
Contributor Author

mmai commented Oct 18, 2019

Thanks @worldofpeace . I've removed the python packages commits.

Copy link
Member

@infinisil infinisil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still some important things (and some less important things) to correct, but it's looking much better than initially!

installPhase = ''
mkdir $out
cp -R ./* $out
unzip ${srcs.frontend} -d $out
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In nixpkgs there's a structure for $out you need to follow. And since this is a python package, you should use buildPythonPackage to build, which will do a lot of things automatically. See section 10.15.2.2.2 in the Nixpkgs manual. This way you can also specify the dependencies in the package directly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a Django application and they usually do not provide any installer. While it would be nice if funkwhale would be usable without the NixOS module, given its a Django app it is typically managed using the manage.py script and various other executables are executed (gunicorn e.g.). Therefore, I think this approach is alright.

Note in another PR (https://github.com/NixOS/nixpkgs/pull/67951/files#diff-5e9f694ce07aa22a80eb0c049c03beaf) I recommended putting the contents of the Django application in a site-packages folder using buildPythonPackage because then the application could be added to the env (that would correspond to adding the funkwhale derivation to the Python env. That would be even nicer, especially if the Python env would be moved here!

group = cfg.group;
});

users.groups = optionalAttrs (cfg.group == "funkwhale") (singleton { name = "funkwhale"; });
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do users.groups.funkwhale = mkIf (cfg.group == "funkwhale") {} instead, and similarly for users.users

djangoSecretKey = mkOption {
type = types.str;
description = ''
Django secret key. Generate one using `openssl rand -base64 45` for example.
Copy link
Member

@infinisil infinisil Oct 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs aren't markdown but asciidoc docbook, so here you need <command>the command</command> instead

type = types.str;
default = "/srv/funkwhale/static";
description = ''
Where static files (such as API css or icons) should be compiled on your system ? Ensure this directory actually exists.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Where static files (such as API css or icons) should be compiled on your system ? Ensure this directory actually exists.
Where static files (such as API css or icons) should be compiled on your system. Ensure this directory actually exists.

Same for other descriptions where there's a random question mark.

type = types.str;
default = "consolemail://";
description = ''
Configure email sending. By default, it outputs emails to console instead of sending them. See https://docs.funkwhale.audio/configuration.html#email-config for details.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Syntax for links is <link xlink:href="https://example.com"/>

type = types.nullOr types.path;
default = "/run/postgresql";
defaultText = "/run/postgresql";
example = "/run/postgresql";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to specify defaultText when the default is the same. And no need to specify an example if it's the same as the default

port = mkOption {
type = types.int;
default = 5432;
defaultText = "5432";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also here, no need to set defaultText

let serviceConfig = {
User = "${cfg.user}";
WorkingDirectory = "${pkgs.funkwhale}";
EnvironmentFile = "${funkwhaleEnvFile}";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't be necessary since you set environment in the services already

};
};

musicDirectoryPath = mkOption {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe call it musicPath instead

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is more consistent with the funkwhaleEnvironment variable name (which is MUSIC_DIRECTORY_PATH), so IMO it should not be changed, because of consistency.

before = [ "funkwhale-init.service" ];
serviceConfig = {
User = "postgres";
ExecStart = '' ${config.services.postgresql.package}/bin/psql -d ${cfg.database.name} -c 'CREATE EXTENSION IF NOT EXISTS "unaccent";CREATE EXTENSION IF NOT EXISTS "citext";' '';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Split this into more lines, same for other long lines, try to keep them to ~80 chars wide

Copy link
Member

@FRidh FRidh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work. It's a very big application. I am worried though how we can keep it stable. I imagine that if the pythonEnv is moved to pkgs.funkwhale, that we can provide overrides there as needed. It might also be good to provide a NixOS test or lighter tests for the pkgs.funkwhale package so we know we don't break it. Maybe we should use the requirements.txt to verify deps as well?
Hook to test against a requirements.txt file. #71908

Group = "${cfg.group}";
};
script = ''
${pythonEnv}/bin/python ${pkgs.funkwhale}/manage.py migrate
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

${python.interpreter}

installPhase = ''
mkdir $out
cp -R ./* $out
unzip ${srcs.frontend} -d $out
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a Django application and they usually do not provide any installer. While it would be nice if funkwhale would be usable without the NixOS module, given its a Django app it is typically managed using the manage.py script and various other executables are executed (gunicorn e.g.). Therefore, I think this approach is alright.

Note in another PR (https://github.com/NixOS/nixpkgs/pull/67951/files#diff-5e9f694ce07aa22a80eb0c049c03beaf) I recommended putting the contents of the Django application in a site-packages folder using buildPythonPackage because then the application could be added to the env (that would correspond to adding the funkwhale derivation to the Python env. That would be even nicer, especially if the Python env would be moved here!

@mmai
Copy link
Contributor Author

mmai commented Nov 3, 2019

@FRidh @infinisil I didn't managed to set a structure resembling what you describe. Looking at the mailman package code and the documentation didn't help.

I'm really sorry to say that at this point I am not able to work on this anymore. I feel that my python and nixos abilities are to shallow and I invested way more time than initially planned, and I see that we are far from having a stable package (the master branch already breaks python dependencies).

Feel free to close the pull request.

@agateblue
Copy link

Hi, I am one of Funkwhale's maintainer, and I wasn't aware of this packaging effort, so I'm a bit late. I don't have any knowledge about NixOS, however, I'd be willing to help if you need some clarification or tweaks on Funkwhale side to make it easier to packagers.

I can also answer questions you have regarding our dependencies, for instance.

Many thanks to the people who worked on this!

@davidak
Copy link
Member

davidak commented Jan 13, 2020

@matthiasbeyer would you be able to finish it?

@matthiasbeyer
Copy link
Contributor

Holy cow. I doubt it. Mainly because I do not know how to properly write services and also because I have little time. I will try next weekend though, if I do not forget.

@matthiasbeyer
Copy link
Contributor

I'm not sure how to push directly to this PR, nor am I convinced I should. I have some patches at https://github.com/matthiasbeyer/nixpkgs/tree/funkwhale though.

@worldofpeace worldofpeace removed this from the 20.03 milestone Jan 27, 2020
@jbaum98
Copy link
Contributor

jbaum98 commented Jan 30, 2020

I'd also be happy to work on this. I've looked at the discussion, but can someone give a list of all the things that need to be changed or fixed to get this PR ready to merge?

@worldofpeace
Copy link
Contributor

@jbaum98 I believe #53416 (review) and maybe https://github.com/NixOS/nixpkgs/pull/53416/files#r338540545?

You should re PR and review can begin again.

@worldofpeace
Copy link
Contributor

We also have #53416 (comment) who responsive to questions so this can be transplanted into NixOS as easily as possible.
But honestly, a lot of the issues here will be ecosystem ones.

@worldofpeace
Copy link
Contributor

I'm going to close this since @mmai said they won't be working on it anymore, so it won't be merged. The code should still be available though.

@jbaum98
Copy link
Contributor

jbaum98 commented May 4, 2020

@FRidh I looked at the mailman PR, and I'm not quite clear on what structure we want for the Django app.

Also, this project has a pretty complicated architecture:

The project relies on the following components and services to work:

  • A web application server (Python/Django/Gunicorn)
  • A PostgreSQL database to store application data
  • A redis server to store cache and tasks data
  • A celery worker to run asynchronous tasks (such as music import)
  • A celery scheduler to run recurrent tasks
  • A ntp-synced clock to ensure federation is working seamlessly

It seems like we want package the API server separately from the frontend server, and then hook that together with all the other stuff in the NixOS module.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet