Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rutorrent: init at 3.10-beta #85613

Draft
wants to merge 11 commits into
base: master
Choose a base branch
from
Draft

Conversation

iv-nn
Copy link
Contributor

@iv-nn iv-nn commented Apr 20, 2020

Motivation for this change

Add rutorrent, a web frontend to rtorrent.

Things done

Add the package and the NixOS service. It's a beta but the last non-beta release doesn't work with the current rtorrent.
It depends on this PR #83287 for a rtorrent service.

A few things still need to be improved.

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

ln -sf ${pkgs.rutorrent}/conf/{access.ini,plugins.ini} ${cfg.home}/conf/
ln -sf ${rutorrentConfig} ${cfg.home}/conf/config.php

cp -r ${pkgs.rutorrent}/php ${cfg.home}/
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have to copy instead of symlink the php files because it tries to import files from the share directory using a relative path. Using symlink it search relative to the nix store path instead of the ${cfg.home}.

I find better to use symlink and tried to find a way to configure php so it would import files relative to the symlink and not it's target. I didn't find anything, is it possible? Otherwise I might end up patching the relative paths.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same things for the plugins.

This comment was marked as resolved.

Copy link
Member

@szlend szlend Jun 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iv-nn So I came up with a bit of a magical solution using bind mounts. This is probably even more hacky, but I kind of like it because it leaves the /var/lib/rutorrent clear of non-state clutter.

Basiaclly what I do is I set the nginx root = pkgs.rutorrent, so it points to the ruTorrent nix store directly. Then I mount the config.php file on top of it, along with the whitelisted plugins. I also set the $profilePath = '${cfg.home}'; so it points directly to the data dir. I do all of this in the phpfpm service file, so we don't even need the setup service.

Here's the modified phpfpm-rutorrent service:

phpfpm-rutorrent.serviceConfig = {
  ExecStartPre = [
    (pkgs.writeShellScript "setup-rutorrent" ''
      mkdir -p ${cfg.home}/{logs,settings,torrents,users}
      chown -R rutorrent:rutorrent ${cfg.home}/{logs,settings,torrents,users}
      chmod -R 755 ${cfg.home}/{logs,settings,torrents,users}
    '')
  ];
  TemporaryFileSystem = [ "${pkgs.rutorrent}/plugins" ];
  BindReadOnlyPaths = [
    "${rutorrentConfig}:${pkgs.rutorrent}/conf/config.php"
  ] ++ map (p: "${pkgs.rutorrent}/plugins/${p}") cfg.plugins;
};

We don't really need to mount the config file though. We could just replace it in the package with an env var pointing to the real config. Like this

phpConfig = writeText "config.php" ''
  <?php
    return require(getenv('RUTORRENT_CONFIG'));
  ?>
'';

installPhase = ''
  mkdir -p $out/
  cp -r . $out/
  cp -f ${phpConfig} $out/conf/config.php
'';

So I think this is only a problem with plugins.

What do you think? I think if mounting on top of the nix store is ugly, we could also mount the whole app to something like /var/run/rutorrent.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea of using bind mounts, it's clever. But I think I would rather mount the store files in the dataDir than the other way around.

But seeing your other comment I realise changing $profilePath should be enough.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plugins dir is the only issue really. With the modified config file you could easily have nginx point directly to /nix/store/... and load the settings through an env var, without any need for any bind mounts or symlinking.

Tbh, I would just skip the whole NixOS plugin management because you can manage plugins in the UI anyway. And it would greatly simplify the whole setup, though at the cost of unnecessary dependencies. Up to you though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kept the copy for now because I thought maybe it's an issue to tie the plugin setup to the php-fpm service. Shouldn't there be a way to use this service without setting nginx.enable = true, using some other web server?

Does using copy pose a real problem?

It even allow to use custom plugins, although a better way would be to allow adding them directly through the plugins option.

Copy link
Member

@szlend szlend Jan 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@iv-nn I think it's fine, forget it. I've seen Nix services do it both ways. Usually using the /nix/store/... as the vhost root when it's simple enough, or by copying/symlinking the application into the state directory in more complex cases. I think with plugins it might be complex enough to keep it as it is. Regardless, nginx.enable = true is not necessary as you can always point your custom vhost root to pkgs.rutorrent.

};

services.rtorrent.configText = ''
schedule = scgi_group,0,0,"execute.nothrow=chown,\":nginx\",(cfg.rpcsock)"
Copy link
Contributor Author

@iv-nn iv-nn Apr 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is dirty but I need a way to make rtorrent's rpc socket accessible by nginx if exposeInsecureRPC2mount is set. This would get overwritten if someone use mkForce to set it's rtorrent's configText. Maybe I should use some systemd post start hook to chown the socket.

This comment was marked as resolved.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is better since rutorrent doesn't actually need the rpc mount if it use one of the rpc or httprpc plugin. Also one could use rtorrent and the rpc mount with some other client (like flood or a mobile client).

Copy link
Member

@szlend szlend Jun 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, something related to this that I noticed in the documentation. It's recommended to add this line to rtorrent so that rutorrent plugins get initialized on startup. Otherwise the plugins don't start until you open the ruTorrent client manually.

execute2 = {sh,-c,/usr/bin/php /var/www/ruTorrent/php/initplugins.php tom &}

https://github.com/Novik/ruTorrent/wiki/Plugins#starting-plugins-with-rtorrent

Though, this is super annoying for a number of reasons:

  1. the user (tom) depends on the http auth username
  2. rtorrent now depends on rutorrent running

I'm not really sure how to approach this one. Maybe we could run it ourselves after ruTorrent starts? Though that still leaves the issue with the http auth user name which would have to be part of the config, or detected by a script scanning through the state dir (/var/lib/rutorrent).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a way to change rtorrent's socket group and change it if rutorrent is enabled. Should I make a separate PR for the rtorrent's change?

For initplugins.php the simplest way would be to let the user do it, just adding some documentation about it.

Or I could check if the nginx vhost has auth and add the config line directly to rtorrent config if rutorrent is enabled.

Copy link
Member

@szlend szlend left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome! I have some minor comments but I'll check it out in more detail as soon as I get the chance.

};

services.rtorrent.configText = ''
schedule = scgi_group,0,0,"execute.nothrow=chown,\":nginx\",(cfg.rpcsock)"

This comment was marked as resolved.

nixos/modules/services/web-apps/rutorrent.nix Outdated Show resolved Hide resolved
nixos/modules/services/web-apps/rutorrent.nix Outdated Show resolved Hide resolved
ln -sf ${pkgs.rutorrent}/conf/{access.ini,plugins.ini} ${cfg.home}/conf/
ln -sf ${rutorrentConfig} ${cfg.home}/conf/config.php

cp -r ${pkgs.rutorrent}/php ${cfg.home}/

This comment was marked as resolved.

Comment on lines 218 to 240
"${config.services.rtorrent.user}" = {
extraGroups = [ "rutorrent" ];
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this is needed if you manage the filesystem with ruTorrent? So rtorrent still has access to it?

Hm, personally I would run ruTorrent and rtorrent as the same user because otherwise it's gonna create an owner mess on the filesystem. Though I wonder if that would be a good default for NixOS too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that would be better.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I was testing this and I didn't notice anything weird going on with file ownership. It seems rutorrent tries to do as much as possible through rtorrent with XMLRPC. So I think it's actually fine to have a separate user as long as they're in each other's groups. Though in that case it would be nice to have the user and group configurable.

nixos/modules/services/web-apps/rutorrent.nix Outdated Show resolved Hide resolved
mediainfo = [ mediainfo ];
spectrogram = [ sox ];
screenshots = [ ffmpeg ];
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_cloudflare = [ python ];

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, when I was testing this it also requires some additional python libraries that are not in NixOS. Maybe we should just skip this plugin?


systemd = {
services = {
rtorrent.path = getPluginDependencies cfg.plugins;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is missing the php package. Either that or we need to add php to every plugin that requires it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what the issue is here, I don't get any error without it.

nixos/modules/services/web-apps/rutorrent.nix Outdated Show resolved Hide resolved
nixos/modules/services/web-apps/rutorrent.nix Outdated Show resolved Hide resolved
nixos/modules/services/web-apps/rutorrent.nix Outdated Show resolved Hide resolved
@iv-nn
Copy link
Contributor Author

iv-nn commented Jun 19, 2020

Thanks for you review, some great inputs. I've started making changes and should be done this week-end.

@Lassulus
Copy link
Member

soo, how is this coming along?

@iv-nn
Copy link
Contributor Author

iv-nn commented Jan 16, 2021

Well, it was a long week-end 😄.

I finally took some time to take care of this and made some progress.

@szlend
Copy link
Member

szlend commented Jan 16, 2021

@iv-nn awesome, looks great! I'll test it out in the next few days and see how it works on my end.

One thing I wanted to ask you about is what to do about initializing ruTorrent plugins? The problem is that plugins do not get loaded until you open the WebUI which is problematic for plugins like autotools. I currently run the linuxserver/rutorrent Docker image and what they do is execute initplugins.php on rtorrent startup.

See: https://github.com/linuxserver/docker-rutorrent/blob/5c56add7824f7199235fca6c7977567255a7126b/root/defaults/rtorrent.rc#L2

Maybe we could do something similar, but in the systemd service instead? I'll try it out and let you know.

As for rtorrent, If I'm not mistaken the changes will have to be part of a separate PR and certainly reviewed by rtorrent maintainers, so I would go ahead and open that in advance.

@iv-nn
Copy link
Contributor Author

iv-nn commented Jan 16, 2021

One thing I wanted to ask you about is what to do about initializing ruTorrent plugins? The problem is that plugins do not get loaded until you open the WebUI which is problematic for plugins like autotools. I currently run the linuxserver/rutorrent Docker image and what they do is execute initplugins.php on rtorrent startup.

See: https://github.com/linuxserver/docker-rutorrent/blob/5c56add7824f7199235fca6c7977567255a7126b/root/defaults/rtorrent.rc#L2

Maybe we could do something similar, but in the systemd service instead? I'll try it out and let you know.

As you said the issue is with the auth username. Maybe I can check if the user has configured some auth on the vhost and if so add that line with the configured username, otherwise it would be the use responsibility. But to be honest I don't really understand what the username is used for.

@szlend
Copy link
Member

szlend commented Jan 16, 2021

Ok, so a few minor things:

  1. If you override the default user/group, it tries to create it for you which caused configuration conflicts for me. The way I've seen other services do this is to wrap it into mkIf so it's only created when left to the default value. If you want a custom user you then have to configure it properly yourself.
users.users = mkIf (cfg.user == "rutorrent") {
  rutorrent = { ... };
};
  1. I've been wondering if we even need the rtorrent socket PR. What if we just added rtorrent to users.users.nginx.extraGroups?
  2. I tried to enable all the plugins and I got errors for _task and _cloudflare in the ruTorrent log tab. I was able to fix the _task errors by setting services.phpfpm.pools.rutorrent.phpEnv.PATH like NextCloud does. Probably there's a better way of doing this but I tried to set it to builtins.concatStringsSep ":" (getPluginDependencies cfg.plugins) and that didn't work for some reason.
  3. The target directory browser in "Add torrent" doesn't work when $topDirectory = '/' due to permission errors. We should probably make this configurable. Basically ruTorrent does not allow browsing directories that are not writable.

I'm done for today, but I'll try and see what I can come up with initplugins.php. We could possibly get the username(s) from the state directory (profiles/users/*) and execute initplugins.php for each of them. But this would have to be done after rtorrent starts because it has to communicate with rtorrent rpc. Maybe we could do it with ExecStartPost=?

@szlend
Copy link
Member

szlend commented Jan 17, 2021

Ok so I put some more thought into initplugins.php and I don't think there's a good way of doing it automagically. The main problem is that the user can change their basicAuth settings and the old share/users/* directories are gonna stay behind, meaning there is no way to figure out which one is correct. I think we should just make it explicit with a initPluginsForUser= configuration option or something like that. This option can be nullable in case no authentication is used. Then we can just add a oneshot service like so:

{
  rutorrent-plugin-init = {
    wantedBy = [ "rtorrent.service" ];
    after = [ "rtorrent.service" ];
    serviceConfig = {
      Type = "oneshot";
      ExecStart = [ "${pkgs.php}/bin/php ${cfg.dataDir}/php/initplugins.php ${cfg.initPluginsForUser}" ];
  };
}

Either that or use ExecStartPost= on rtorrent.

{
  systemd.services.rtorrent.serviceConfig.ExecStartPost =
    [ "${pkgs.php}/bin/php ${cfg.dataDir}/php/initplugins.php ${cfg.initPluginsForUser}" ];
}

What do you think?

@stale
Copy link

stale bot commented Jul 21, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 21, 2021
@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Oct 18, 2021
@stale
Copy link

stale bot commented Apr 17, 2022

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Apr 17, 2022
cp -r . $out/
'';

meta = with stdenv.lib; {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
meta = with stdenv.lib; {
meta = with lib; {

@@ -0,0 +1,25 @@
{ stdenv, fetchFromGitHub }:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
{ stdenv, fetchFromGitHub }:
{ stdenv, lib, fetchFromGitHub }:

@stale stale bot removed 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md labels Nov 8, 2022
@bolives-hax bolives-hax mentioned this pull request Feb 16, 2023
12 tasks
@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 18, 2023
@stale stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants