Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos/lighttpd: add Nextcloud sub-service #19703

Closed
wants to merge 2 commits into from

Conversation

bjornfor
Copy link
Contributor

@bjornfor bjornfor commented Oct 19, 2016

Motivation for this change

Add a simple module that enables Nextcloud + lighttpd + sqlite.

Things done
  • Tested using sandboxing
    (nix.useSandbox on NixOS,
    or option build-use-sandbox in nix.conf
    on non-NixOS)
  • Built on platform(s)
    • NixOS
    • OS X
    • Linux
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

Basic usage:

  services.lighttpd = {
    enable = true;
    nextcloud.enable = true;
  };

Visit http(s)://server.example/nextcoud to complete the setup. (Remember to
setup HTTPS in your webserver.)

@mention-bot
Copy link

@bjornfor, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @offlinehacker and @abbradar to be potential reviewers.

Copy link
Member

@fpletz fpletz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still feel we should make an abstraction for all webapps where configurations for different webservers can be defined. As my favourite httpd is nginx (though l have used lighty happily for many years :)), I would love to include an nginx vhost snippet but the current structure of this approach and of other webapps like those for apache doesn't really allow that. See #18977.

As for this PR, I vote for merging anyway because we don't have another Nextcloud module yet and it serves as an example how to get Nextcloud with php-fpm going. We can refactor later.

systemd.services.lighttpd.preStart =
''
echo "Setting up Nextcloud in ${cfg.installPrefix}/"
${pkgs.rsync}/bin/rsync -a "${cfg.package}/" "${cfg.installPrefix}/"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does Nextcloud really need its source/data to be writable? If not, I would prefer symlinks. For instance if files are removed upstream, they wouldn't be removed in the install prefix because rsync hasn't the --delete flag which you can't use anyway in this case without deleting the state/data.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I'll consider using symlinks and look at how upstream describes the upgrade procedure, in particular regarding stale files. (BTW, Nextcloud warns about extra/stale files in its admin UI, so at least it's easy to do manual fixup.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Serve files from ${nextcloud} and set the environment variable $NEXTCLOUD_CONFIG to a configuration directory, you don't need to copy or symlink anything.
See #18761 for an example 😃

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@schneefux: Do you know if Nextcloud is smart enough to output the $NEXTCLOUD_CONFIG path in the command it presents to the user to perform upgrades? (NEXTCLOUD_CONFIG=... ./occ [...])

Until it does that, I'd prefer the dumb rsync approach. That way the upgrades will be less painful I think.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't tried yet, but I doubt it. I use the web interface to update which works fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had problems with one upgrade and I think Nextcloud suggested to use ./occ in case the upgrade took so long that the webUI timed out (or in case of error). I don't remember the details :-/

listen = ${phpfpmSocketName}
listen.group = lighttpd
user = lighttpd
group = lighttpd
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The php-fpm pool of Nextcloud should probably have at least its own user for security.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What should that user be called? "phpfpm-lighttpd-nextcloud", "phpfpm-nextcloud" or just "nextcloud"?

@bjornfor
Copy link
Contributor Author

@fpletz: Thanks for the feedback. I agree about an abstraction for webapps. I've been thinking that an easy first step would be to move current webserver sub-services to separate webapps directory in nixpkgs and make each service have options for all webservers. Then there is the special webapp.<name>.backend option (being an enum of ["nginx" "lighttpd" "apache"]) that selects which webserver configuration to use.

@Mic92
Copy link
Member

Mic92 commented Oct 20, 2016

I think web server configuration and the web application itself should be different modules.
That way people can pick their favorite frontend server and can also deploy web server and application on different hosts/containers.

Basic usage:

  services.lighttpd = {
    enable = true;
    nextcloud.enable = true;
  };

Visit http(s)://server.example/nextcloud to complete the setup.
(Remember to setup HTTPS in your webserver.)
@bjornfor bjornfor force-pushed the nixos-nextcloud branch 3 times, most recently from 2d09c1d to ce994ec Compare February 17, 2017 20:49
Instead of the 'lighttpd' user. The result is better damage control in
case of a security breach.

/var/lib/nextcloud is now owned by nextcloud:nextcloud. lighttpd is
added to the nextcloud group so that it can read the web application
files, but not the user data (unless I made a mistake).
@bjornfor
Copy link
Contributor Author

While trying to use $NEXTCLOUD_CONFIG_DIR and serving Nextcloud from read-only /nix/store, Nextcloud says:

Cannot write into "apps" directory
This can usually be fixed by giving the webserver write access to the apps directory or disabling the appstore in the config file.

Is it worth it? Right now I'm thinking "no". With rsync there will be some left over files, but Nextcloud lists them in the UI, so they can at least be cleaned up manually.

@bjornfor
Copy link
Contributor Author

@bjornfor
Copy link
Contributor Author

I tested manually configuring the appstore and datadirectory to /var/lib/nextcloud (via directives in config.php) while having the remaining files in /nix/store. This is what happens now:

Feb 17 22:45:21 mini2 lighttpd[19015]: (mod_fastcgi.c.2543) FastCGI-stderr: PHP message: PHP Fatal error:  Uncaught Error: Call to a member function getLogger() on null in /nix/store/xbi9jxrpz4g1m7jpa55yhywpzvdml33w-nextcloud-11.0.1/index.php:61
Feb 17 22:45:21 mini2 lighttpd[19015]: (mod_fastcgi.c.2543) FastCGI-stderr: Stack trace:
Feb 17 22:45:21 mini2 lighttpd[19015]: (mod_fastcgi.c.2543) FastCGI-stderr: #0 {main}
Feb 17 22:45:21 mini2 lighttpd[19015]: (mod_fastcgi.c.2543) FastCGI-stderr:   thrown in /nix/store/xbi9jxrpz4g1m7jpa55yhywpzvdml33w-nextcloud-11.0.1/index.php on line 61
Feb 17 22:45:21 mini2 lighttpd[19015]: 127.0.0.1 localhost - [17/Feb/2017:22:45:21 +0100] "GET /nextcloud/index.php HTTP/1.1" 500 0 "http://localhost/nextcloud/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"

And index.php contains

try {

	require_once __DIR__ . '/lib/base.php';

	OC::handleRequest();

} catch ...
...
} catch (Error $ex) {
	\OC::$server->getLogger()->logException($ex, array('app' => 'index')); # <--- ERROR HERE
	OC_Response::setStatus(OC_Response::STATUS_INTERNAL_SERVER_ERROR);
	OC_Template::printExceptionErrorPage($ex);
}

@spiderbit
Copy link

Hi I am not shure its the approprietive place to ask that question, but I want to try out that packages idealy in a container, but thats not that important.

I checked out the nextcloud branch from @bjornfor in the container under /root/

and added according to the post:
http://anderspapitto.com/posts/2015-11-01-nixos-with-local-nixpkgs-checkout.html

this to the containers configuration.nix:

nix.nixPath = [ "/root" "nixos-config"=/etc/nixos/configuration.nix" ];

  services.lighttpd = {
    enable = true;
    nextcloud.enable = true;
  };

sudo nixos-container update nextcloud

I also tried with the absolute path with /var/lib/container... as nixpath.

I removed the " before the = looks not syntacticaly right. I get then:

error: The option services.lighttpd.nextcloud' defined in /var/lib/containers/nextcloud/etc/nixos/configuration.nix' does not exist.

Is that a container specific problem? or what am I doing wrong?

@maninalift
Copy link

I thought I would point your attention to a config for nextcloud on nginx that is out there https://github.com/eqyiel/deployments/blob/master/realms/tsumugi.rkm.id.au/nextcloud.nix

Pointing out a few differences: This config includes acme ssl and uses the existing nixos package (what it does with running the php from the package directory and using another directory for state has caused me some problems but doesn't seem like a bad idea) rather than defining one locally. It sets up a cron job.

I'm a nixos noob just trying to get something working at the moment.

@maninalift
Copy link

To clarify - the issues that I had related to not running the php from the same directory as the app state were firstly the apps directory which is mentioned above (I have checked the code of nextcloud and it is hard-coded to use /apps for this) and secondly finding a way to run the command-line setup script.

I still haven't managed to set up a working nextcloud config in nixos, so any progress on this very much appreciated. Thank you for your work.

PS you may also want to bump the nextcloud version to 11.0.2.

@bjornfor
Copy link
Contributor Author

@maninalift: Didn't that module you linked to(1) work? It seemed pretty advanced / complete.

(1): https://github.com/eqyiel/deployments/blob/master/realms/tsumugi.rkm.id.au/nextcloud.nix

@maninalift
Copy link

@bjornfor - in order to solve the app directory issue, to be able to run the occ setup script plus a couple of other issues, i modified the module to copy the entire package into /var/log/ñextcloud, do the same chmods as in your module and serve the site from that directory. After setting up the database and running the setup script i was able to log in. Things seem mostly to be working but on the admin page i am greeted with a host of messages telling me that headers are not correctly set which should be given the config (same origin etc). Not quite sure where to go next although i suspect the fix is simple. Also i.attempted to run the nextcloud update from within the web interface which complained it was not able to write to the directory.

Clearly the originator is using this config successfully, so I may ask him for help.

@maninalift
Copy link

Ps sorry, writing on phone with cracked screen.

@spiderbit
Copy link

whats still missing to solve this? remove this ids.nix conflict?

@bjornfor
Copy link
Contributor Author

bjornfor commented Apr 2, 2017

I was asked to not copy all of nextcloud into a writeable location, but rather serve directly from the read only nix store, while providing a separate writeable state directory.

I haven't managed to do that yet (see attempt above).

@schneefux
Copy link
Contributor

See this gist, you just set $NEXTCLOUD_CONFIG_DIR in PHP and serve from ${pkgs.nextcloud}: https://gist.github.com/schneefux/22b75d2bd3e4e754ba1684f1d1e93271

My config.php has this

  'datadirectory' => '/var/lib/nextcloud/data',
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/lib/nextcloud/apps',
      'url' => '/apps_addon',
      'writable' => true,
    ),
    1 =>
    array (
      'path' => OC::$SERVERROOT.'/apps',
      'url' => '/apps_internal',
      'writable' => false,
    ),
  ),

In my nginx.conf I have

                location ^~ /apps_addon {
                        alias /var/lib/nextcloud/apps;
                }
                location ^~ /apps_internal {
                        alias ${pkgs.nextcloud}/apps;
                }

@bjornfor
Copy link
Contributor Author

bjornfor commented Apr 2, 2017

Thanks for the example. I'll give it another try when master branch is usable again.

Did you manually edit config.php after deployment? I'm guessing "yes", so did you try putting the generated parts of the configuration in a separate file and then add it to config.php via the include directive? I'm guessing it's easier to inject a single line (include directive) than manipulating php arrays in-place.

@jluttine
Copy link
Member

jluttine commented Apr 2, 2017

I'm also working on Nextcloud sub-service but for Nginx, so perhaps we can try to help each other. Anyway, I'm not copying Nextcloud into writable location but using it from nix store. You need to create directories for config, data, apps and optionally for assets. Unfortunately, custom themes location isn't yet supported. config directory you set with environment variable NEXTCLOUD_CONFIG_DIR. For others: datadirectory, apps_paths and assetdirectory in config.php. I would use occ to modify configurations in config.php so you don't need to worry about injecting or manipulating a PHP file yourself directly. If it helps at all, here's my nix configuration (which currently just my own nix configuration and extremely unpolished and work in progress): https://github.com/jluttine/nixos-configuration/blob/master/extra-services/nginx/nextcloud.nix.

@bjornfor
Copy link
Contributor Author

@jluttine: I didn't know occ could be used to manipulate config.php. That sounds very useful. And thanks for the example code.

@devhell
Copy link
Contributor

devhell commented Jul 20, 2017

What's the status with this? It would be very useful to be able to simply enable nextcloud for lighttpd without any fuss. :)

@bjornfor
Copy link
Contributor Author

It has low priority for me, so I never seem to have time for it.

It basically works and I use this on my own system. But there was a request for running nextcloud directly from the nix store, unlike what's happening here -- running "rsync /nix/store/...-nextcloud /var/lib/nextcloud" on every start.

If anyone has time to work on this, please do.

@florianjacob
Copy link
Contributor

Announcement to avoid duplicate work: I'll take this over, trying to run nextcloud directly from the store. If anyone wants to help with this or test what I'll create, please tell me! 😄

@jluttine
Copy link
Member

jluttine commented Jan 7, 2018

@florianjacob FYI, here's something that I did: https://github.com/jluttine/nixos-configuration/tree/pilvi-nextcloud/extra-services/nextcloud

Not sure if that works as it is now, but some things to note:

  • Nextcloud is used from store
  • Stateful directories (and their permissions) are set properly
  • Runs Nextcloud upgrade migrations automatically (relevant when package is upgraded)
  • Installs Nextcloud automatically if it's not yet installed (e.g., first time it is enabled)

I tried to achieve the following:

  • Make database and socket configurations modular, that is, the user could easily use mysql or postgresql or define their own, or the user could choose between uwsgi and php-fpm or define their own. I guess this worked but it wasn't very pretty implementation..
  • Appstore disabled and thus apps are managed via nix. However, I wasn't able to build even the basic apps such as calendar and contacts. But otherwise the app paths were set properly to nginx configurations and config.php.
  • Support multiple Nextcloud installations. This started to make the configuration way too complex. I would drop this idea as it's not supported in most of the other webapps either.

Just in case those help. I spent quite many days on those (waaaay too many) so if they are useful, that's great.

@jluttine
Copy link
Member

jluttine commented Jan 7, 2018

Two more things I did:

  • Nextcloud keeps some state stuff in config.php. So I created an immutable nixos.config.php which contains configuration specified by the NixOS definition. Then, Nextcloud can mutate config.php as it sees fit.
  • Add properly wrapped occ executable so the user can easily manage the Nextcloud instance. In my default.nix it is defined by occPackage (though it adds a suffix to the executable name, which was to support multiple Nextcloud installations).

@gerbrand
Copy link

gerbrand commented Jan 7, 2018

I am using bjornfor:nixos-nextcloud module for some months/weeks, as well as nixos for some short time. I don't have enough experience with nixos yet to feel confident to contribute in patches. But I'll be happy to test as well as give mental support.

@bjornfor
Copy link
Contributor Author

Oooh, there is a nextcloud service in master already! (ebd3818)

@bjornfor bjornfor closed this Oct 12, 2018
@florianjacob
Copy link
Contributor

For reference: The now-merged module was introduced in #47159.

While @jluttine 's code was well-written, the task of ripping out all the modularization for multiple instances, different databases and different web servers was still too large to be handled in a reasonable time, so I settled with doing a simple, stateful nextcloud installation for 18.03 back then.

@jluttine I think the main problem with your code was that you essentially developed a full-blown universal web service definition like proposed in #22067, but intermingled with the concrete nextcloud module. As there was no progress on the universal web service definition side, your code is probably the most advanced prototype for this, and removing the nextcloud-specific parts to get to something reusable to write other web services indepentently of database and webserver might be a worthy endeavour.

@jluttine
Copy link
Member

@florianjacob Yes, I fully agree that my code was way too complex. I would remove the overly complex modularization and support for multiple instances, but these main contributions I really liked:

  • Nextcloud is used from store with separate stateful directories.
  • Nextcloud installation and upgrades are run automatically.
  • Optionally, appstore disabled and apps managed via nix.

I didn't check if (some of) these are part in the current service implementation. But great that there has been progress. 👍

@bjornfor bjornfor deleted the nixos-nextcloud branch January 10, 2024 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet