paperless: add package and service #44086

erikarvstedt · 2018-07-25T11:44:34Z

Notes for reviewers

Paperless has no setup.py, the main way to use it is to call manage.py in a properly setup
Python environment. That's what ${paperless}/bin/paperless does.

Paperless requires some Python packages that are missing from nixpkgs. Currently, they are defined in extra-python-packages.nix. Should I add them all to python-packages.nix? Do we need an extra git commit for each package?

Here's a sample script for testing:

dataDir=/tmp/paperless-data
rm -rf $dataDir

nix-build --out-link ./paperless - <<EOF
(import <nixpkgs> {}).paperless.withConfig {
  dataDir = $dataDir;
  config = {
    PAPERLESS_DISABLE_LOGIN = "true";
  };
}
EOF

# Setup DB
./paperless migrate

# Create a test document
$(nix-build --no-out-link '<nixpkgs>' -A imagemagick)/bin/convert \
  -size 400x40 xc:white -font "DejaVu-Sans" -pointsize 20 -fill black -annotate +5+20 \
  "hello world 16-10-2005" $dataDir/consume/doc.png

# Consume document
./paperless document_consumer --oneshot

# Start webinterface
./paperless runserver --noreload localhost:8000

# Cleanup
rm -rf $dataDir ./paperless

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

jfrankenau · 2018-07-30T18:28:58Z

Python packages should be globally available. Nowadays each package gets one directory/file. One commit per package would be nice but it may be ok to put them all in one.

Have you thought about creating a matching NixOS module?

dotlambda

Please create the packages besides paperless itself inside the pythonPackages package set. Some of them might already be packaged.

erikarvstedt · 2018-08-20T20:48:12Z

Back from vacation, please excuse the delay!

@dotlambda, done.

@jfrankenau, I've added a service module, but there are a few open issues:

The web interface (paperless-server) runs on the Django dev server. Should I use a proper web server like nginx or Apache?
By default, the services run with user nobody. Should I add a dedicated paperless user (via ids.nix)?
In a service preStart script im using ${pkgs.libuuid}/bin/runuser to run a cmd as the unprivileged user. Is there better, canonical way to do this? su is unavailable in the script env.

I'm using this script for testing the service.
Run this for a quick demo.

jfrankenau · 2018-08-21T07:33:30Z

@erikarvstedt, I guess for now using Django is OK.

/var/lib/paperless should be owned by a real user, so adding a new user to ids.nix is necessary.

What about adding another ExecStart line to run the unprivileged migration process? Or even extract it to another systemd oneshot service?

jfrankenau · 2018-08-21T07:36:26Z

nixos/modules/services/misc/paperless.nix

+
+    consumptionDir = mkOption {
+      type = with types; nullOr str;
+      default = null;


Set it to the default value mentioned in the description and remove the if-clause below.

Good catch! Fixed.

erikarvstedt · 2018-08-21T12:35:06Z

The migration has to run in a preStart script so that the server, which is only active when the consumer is active, is disabled during migrations.
I've also thought about a helper oneshot service, but I think that's even more complicated.

A dedicated paperless user would be no more 'real' than the nobody user.
The only advantage would be isolation from other processes that might be running as the nobody user. But is this worth the extra effort? Is there relevant NixOS documentation?

jfrankenau · 2018-08-22T13:21:27Z

Adding a new user is as you say for isolation purposes. Especially as /var/lib/paperless shouldn't be accessible to other users.

erikarvstedt · 2018-08-22T14:13:29Z

Ok, done.

globin · 2018-08-28T15:11:31Z

pkgs/top-level/python-packages.nix

@@ -284,7 +284,9 @@ in {

  fire = callPackage ../development/python-modules/fire { };

-  globus-sdk = callPackage ../development/python-modules/globus-sdk { };


You apparently accidentally deleted this line which causes the CI to fail during evaluation:
https://gist.github.com/GrahamcOfBorg/14241c580774a606e627f780e1809db9

Done. Thanks for digging into this, I thought that error came from upstream.

erikarvstedt · 2019-02-12T11:54:39Z

78bb5bc: Rebased to master, squashed fixup commits.

worldofpeace · 2019-04-24T20:54:47Z

@dotlambda Do you feel this is suitable?

Don't think it's fair to @erikarvstedt to leave this unmerged any longer.

erikarvstedt · 2019-04-25T11:03:18Z

I can understand if the review and maintanence burden of this feature seems unreasonable to you. I'd be fine with moving it to NUR then.

worldofpeace · 2019-04-25T11:12:57Z

I can understand if the review and maintanence burden of this feature seems unreasonable to you. I'd be fine with moving it to NUR then.

Oh no, I think this is perfectly suitable to have in nixpkgs.

dotlambda

Only some really minor stuff. And yes, this should definitely be merged.

nixos/modules/services/misc/paperless.nix

pkgs/applications/office/paperless/default.nix

dotlambda · 2019-04-26T11:55:04Z

pkgs/applications/office/paperless/default.nix

+  django_2_0 = pyPkgs: pyPkgs.django_2_1.overrideDerivation (_: rec {
+    pname = "Django";
+    version = "2.0.12";
+    name = "${pname}-${version}";


Suggested change

name = "${pname}-${version}";

For overrideDerivation , we have to update name manually.

dotlambda · 2019-04-26T11:58:08Z

pkgs/applications/office/paperless/withConfig.nix

+    set -e
+    ${setupEnv}
+    ${extraCmds}
+    exec python $paperlessSrc/manage.py "$@"


Suggested change

exec python $paperlessSrc/manage.py "$@"

exec ${paperless.python.interpreter} $paperlessSrc/manage.py "$@"

The correct form would be paperless.pythonEnv.interpreter.
setupEnv already handles all these details, so I'd rather not repeat them to keep it DRY.

erikarvstedt · 2019-04-29T09:58:03Z

b5651b3: Rebased to master

Updated djangoql to the latest release
Provide django-filter, django-crispy-forms as private implementations, because
we use unreleased versions and patches to make them compatible with the current nixpkgs
python package env.

nixos/modules/services/misc/paperless.nix

worldofpeace · 2019-05-07T23:46:30Z

@erikarvstedt If you can rebase/squash this I can merge this.

worldofpeace · 2019-05-08T21:19:59Z

Thank you @erikarvstedt for your patience (also very organized summary of changes which helps with reviews) and of course your contributions ✨

Likewise thanks to our reviewers 👍

FRidh · 2019-05-09T17:11:32Z

The attribute name is the attribute in the package set, and not the pname or name attribute in the expression.

erikarvstedt · 2019-05-09T20:07:35Z

Huge thanks for your help and feedback, @dotlambda, @worldofpeace!

@FRidh, you meant to post that in another thread, right?

GrahamcOfBorg added 8.has: package (new) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Jul 25, 2018

erikarvstedt force-pushed the paperless branch 2 times, most recently from a64d1e1 to 0e2ebd9 Compare July 25, 2018 12:44

dotlambda requested changes Jul 31, 2018

View reviewed changes

dotlambda added the 6.topic: python label Jul 31, 2018

erikarvstedt force-pushed the paperless branch from 0e2ebd9 to 416155a Compare August 20, 2018 20:32

erikarvstedt requested a review from FRidh as a code owner August 20, 2018 20:32

GrahamcOfBorg added 6.topic: nixos 8.has: module (update) labels Aug 20, 2018

erikarvstedt changed the title ~~paperless: init at 2.1.0~~ paperless: add package and service Aug 20, 2018

jfrankenau reviewed Aug 21, 2018

View reviewed changes

erikarvstedt force-pushed the paperless branch from 416155a to 960d8c9 Compare August 21, 2018 12:34

erikarvstedt force-pushed the paperless branch 2 times, most recently from f9f6874 to 07373c2 Compare August 22, 2018 14:37

globin requested changes Aug 28, 2018

View reviewed changes

erikarvstedt force-pushed the paperless branch from 07373c2 to d3cb6f7 Compare August 28, 2018 16:14

GrahamcOfBorg added the 2.status: merge conflict label Aug 28, 2018

erikarvstedt force-pushed the paperless branch from d3cb6f7 to 6d212ea Compare August 28, 2018 21:20

GrahamcOfBorg added 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 and removed 2.status: merge conflict 10.rebuild-darwin: 0 labels Aug 28, 2018

erikarvstedt force-pushed the paperless branch from 90298fd to 78bb5bc Compare February 12, 2019 11:54

GrahamcOfBorg added 11.by: package-maintainer and removed 2.status: merge conflict labels Feb 12, 2019

dotlambda approved these changes Apr 26, 2019

View reviewed changes

erikarvstedt added 6 commits April 29, 2019 11:41

pythonPackages.filemagic: init at 1.6

071f3b3

pythonPackages.langdetect: init at 1.0.7

72aec3e

pythonPackages.python-dotenv: init at 0.10.1

5e3e440

pythonPackages.pytest-env: init at 0.6.2

8646d2a

pythonPackages.pdftotext: init at 2.1.1

0babb25

pythonPackages.inotify-simple: init at 1.1.8

e401666

erikarvstedt force-pushed the paperless branch from 78bb5bc to 28b131f Compare April 29, 2019 09:52

pythonPackages.djangoql: init at 0.12.6

76218fd

erikarvstedt force-pushed the paperless branch from 28b131f to b5651b3 Compare April 29, 2019 09:56

worldofpeace reviewed Apr 29, 2019

View reviewed changes

nixos/modules/services/misc/paperless.nix Outdated Show resolved Hide resolved

erikarvstedt force-pushed the paperless branch from d0ed820 to 505f6f3 Compare April 29, 2019 16:59

erikarvstedt added 2 commits May 8, 2019 09:26

paperless: init at 2.7.0

0231273

paperless service: init

80c3ddb

erikarvstedt force-pushed the paperless branch from 505f6f3 to 80c3ddb Compare May 8, 2019 07:44

worldofpeace merged commit bb7e556 into NixOS:master May 8, 2019

c0bw3b mentioned this pull request May 9, 2019

Some python libraries, mostly scrapying-related #60775

Closed

10 tasks

tjni mentioned this pull request Nov 19, 2022

python310Packages.filemagic: remove #201900

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

paperless: add package and service #44086

paperless: add package and service #44086

erikarvstedt commented Jul 25, 2018 •

edited

jfrankenau commented Jul 30, 2018

dotlambda left a comment

erikarvstedt commented Aug 20, 2018 •

edited

jfrankenau commented Aug 21, 2018

jfrankenau Aug 21, 2018

erikarvstedt Aug 21, 2018

erikarvstedt commented Aug 21, 2018 •

edited

jfrankenau commented Aug 22, 2018

erikarvstedt commented Aug 22, 2018

globin Aug 28, 2018

erikarvstedt Aug 28, 2018

erikarvstedt commented Feb 12, 2019

worldofpeace commented Apr 24, 2019

erikarvstedt commented Apr 25, 2019

worldofpeace commented Apr 25, 2019

dotlambda left a comment

dotlambda Apr 26, 2019

erikarvstedt Apr 29, 2019

dotlambda Apr 26, 2019

erikarvstedt Apr 29, 2019

erikarvstedt commented Apr 29, 2019

worldofpeace commented May 7, 2019

worldofpeace commented May 8, 2019

FRidh commented May 9, 2019

erikarvstedt commented May 9, 2019

		@@ -284,7 +284,9 @@ in {

		fire = callPackage ../development/python-modules/fire { };

		globus-sdk = callPackage ../development/python-modules/globus-sdk { };

	exec python $paperlessSrc/manage.py "$@"
	exec ${paperless.python.interpreter} $paperlessSrc/manage.py "$@"

paperless: add package and service #44086

paperless: add package and service #44086

Conversation

erikarvstedt commented Jul 25, 2018 • edited

Notes for reviewers

Things done

jfrankenau commented Jul 30, 2018

dotlambda left a comment

Choose a reason for hiding this comment

erikarvstedt commented Aug 20, 2018 • edited

jfrankenau commented Aug 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikarvstedt commented Aug 21, 2018 • edited

jfrankenau commented Aug 22, 2018

erikarvstedt commented Aug 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikarvstedt commented Feb 12, 2019

worldofpeace commented Apr 24, 2019

erikarvstedt commented Apr 25, 2019

worldofpeace commented Apr 25, 2019

dotlambda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikarvstedt commented Apr 29, 2019

worldofpeace commented May 7, 2019

worldofpeace commented May 8, 2019

FRidh commented May 9, 2019

erikarvstedt commented May 9, 2019

erikarvstedt commented Jul 25, 2018 •

edited

erikarvstedt commented Aug 20, 2018 •

edited

erikarvstedt commented Aug 21, 2018 •

edited