Isso: init at 0.10.6 (plus dependencies) #34296

fgaz · 2018-01-26T15:25:28Z

Motivation for this change

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

Closes #13587

fgaz · 2018-01-26T15:26:26Z

I'll probably also write a nixos service once this is merged

dotlambda · 2018-01-27T00:06:27Z

Is this really a Python library? Otherwise it doesn't belong into python-packages.nix.

fpletz · 2018-01-27T00:13:01Z

Note that there is also #13587. Maybe some code can be reused. cc @Profpatsch

Profpatsch · 2018-01-27T00:21:36Z

Please, feel free to.

fgaz · 2018-01-27T08:17:36Z

@dotlambda I see several examples of non-libraries there, eg. mps-youtube. But you are right, it is even mentioned in the comment at the beginning of that file. I'll move it.

fgaz · 2018-01-27T09:00:45Z

@fpletz @Profpatsch Thanks! I should really search before opening prs and duplicating work...

The service part is definitely reusable!

fgaz · 2018-01-27T09:04:42Z

@Profpatsch Why did you choose to separate the js output? You don't seem to use that separation anywhere

Profpatsch · 2018-01-27T18:53:15Z

Good question. I’m not sure why I did that back then, isso itself seems to depend on the js folder, so it doesn’t make much sense.

Maybe I wanted to use only the js for my project.

FRidh · 2018-02-03T08:09:13Z

pkgs/top-level/python-packages.nix

@@ -5295,6 +5295,25 @@ in {
    };
  };

+  isso = buildPythonPackage rec {


This is an application and should therefore be called directly from all-packages.nix, and not via python-packages.nix.

is it ok if I do it like this?
https://github.com/NixOS/nixpkgs/pull/13587/files#diff-036410e9211b4336186fc613f7200b12R9553

Or should I move the whole buildPythonPackage in its own file in pkgs/applications (or pkgs/servers)?

Yes, please create a default.nix in pkgs/servers/isso/, and remember to use buildPythonApplication.
However, misaka (latest version) needs to be put in pkgs/development/python-modules and called from python-packages.nix.
You can then use overridePytonAttrs in isso's default.nix to specify a custom misaka version.

FRidh · 2018-02-03T08:10:42Z

pkgs/top-level/python-packages.nix

@@ -5667,6 +5686,24 @@ in {
    };
  };

+  # needed for isso 0.10.6
+  misaka_1 = buildPythonPackage rec {


This one will need to be added locally to the application isso, and not here in python-packages.nix.

so something like

let misaka_1 = buildPythonPackage {...}; in buildPythonPackage {name="isso-..."; propagatedBuildInputs = [misaka_1 ... ] ;...}

right?

Oh, and what about html5lib_0_9999999? Should I use the one in pythonModules? Should that be removed too?

You should better use overridePytonAttrs: https://github.com/NixOS/nixpkgs/search?q=overridePythonAttrs

fgaz · 2018-02-22T18:55:44Z

Added latest misaka to python modules, moved isso to all-packages with an override for misaka-1

fgaz · 2018-02-22T18:57:31Z

Wait, why is 940f182 there? I probably messed something up during a rebase

EDIT: fixed & rebased on master

dotlambda · 2018-02-22T23:02:02Z

pkgs/development/python-modules/misaka/default.nix

+{ lib, fetchPypi, buildPythonPackage, cffi }:
+buildPythonPackage rec {
+  pname = "misaka";
+  name = "${pname}-${version}";


dotlambda · 2018-02-22T23:02:20Z

pkgs/development/python-modules/misaka/default.nix

+
+  propagatedBuildInputs = [ cffi ];
+
+  doCheck = false;


Why? Add a comment.

...I actually don't remember why I did this. It tests fine. Removing it now.

well, that's why. re-disabling.

dotlambda · 2018-02-22T23:02:45Z

pkgs/servers/isso/default.nix

+let
+  misaka_1 = misaka.overridePythonAttrs (old: rec {
+    version = "1.0.2";
+    name = "${old.pname}-${version}";


I don't understand this. Should I redefine pname like this and use pname instead of old.pname?

pname = old.pname;

Or should I omit redefining name entirely?

If you leave out name in the misaka expression, there is no need to override any of name or pname.

dotlambda · 2018-02-22T23:03:18Z

pkgs/servers/isso/default.nix

+  misaka_1 = misaka.overridePythonAttrs (old: rec {
+    version = "1.0.2";
+    name = "${old.pname}-${version}";
+    src = fetchurl {


src = old.src.override { ... }

dotlambda · 2018-02-22T23:03:32Z

pkgs/servers/isso/default.nix

+  });
+
+in buildPythonApplication rec {
+  name = "isso-${version}";


dotlambda · 2018-02-22T23:03:42Z

pkgs/servers/isso/default.nix

+  name = "isso-${version}";
+  version = "0.10.6";
+
+  src = fetchurl {


fetchPypi

dotlambda · 2018-02-22T23:06:40Z

pkgs/servers/isso/default.nix

+, misaka, werkzeug, ipaddr, configparser, html5lib_0_9999999 }:
+
+let
+  misaka_1 = misaka.overridePythonAttrs (old: rec {


Better specify python as an argument and override as in https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/home-assistant/default.nix.
This will make sure that all dependencies use the old version of misaka as well (which is of course not necessary right now but might be in the future).

dotlambda · 2018-04-03T08:32:13Z

pkgs/top-level/all-packages.nix

@@ -9444,6 +9444,11 @@ with pkgs;
    stdenv = llvmPackages_4.stdenv;
  };

+  isso = callPackage ../servers/isso {
+    inherit (python2Packages)
+    buildPythonApplication fetchPypi misaka werkzeug ipaddr configparser html5lib_0_9999999;


Did you read #34296 (comment)?

dotlambda · 2018-04-03T08:32:53Z

@GrahamcOfBorg build python2.pkgs.misaka python3.pkgs.misaka

GrahamcOfBorg · 2018-04-03T08:33:21Z

Failure on x86_64-linux (full log)

Attempted: python2.pkgs.misaka, python3.pkgs.misaka

Partial log (click to expand)


For information on other options, you may wish to consult the
documentation at:

  https://setuptools.readthedocs.io/en/latest/easy_install.html

Please make the appropriate changes for your system and try again.

builder for '/nix/store/g6ici055vv4r47zc19phd47vnl9r2ifp-python3.6-misaka-2.1.0.drv' failed with exit code 1
�[31;1merror:�[0m build of '/nix/store/515z752rpa98xrwk3397if6jy15ybd2k-python2.7-misaka-2.1.0.drv', '/nix/store/g6ici055vv4r47zc19phd47vnl9r2ifp-python3.6-misaka-2.1.0.drv' failed

GrahamcOfBorg · 2018-04-03T08:33:47Z

Failure on aarch64-linux (full log)

Attempted: python2.pkgs.misaka, python3.pkgs.misaka

Partial log (click to expand)


For information on other options, you may wish to consult the
documentation at:

  https://setuptools.readthedocs.io/en/latest/easy_install.html

Please make the appropriate changes for your system and try again.

builder for '/nix/store/crvd0kj7zm83lmq7fqg5sbk4ah5rxkj7-python3.6-misaka-2.1.0.drv' failed with exit code 1
�[31;1merror:�[0m build of '/nix/store/57v21azxmmmgic88g8xv0bkgb6vn4q49-python2.7-misaka-2.1.0.drv', '/nix/store/crvd0kj7zm83lmq7fqg5sbk4ah5rxkj7-python3.6-misaka-2.1.0.drv' failed

fgaz · 2018-04-14T18:02:25Z

@dotlambda all done now

dotlambda · 2018-04-14T18:04:45Z

pkgs/top-level/python-packages.nix

@@ -1247,7 +1247,7 @@ in {

  blaze = callPackage ../development/python-modules/blaze { };

-  # Needed for bleach 1.5.0
+  # Needed for bleach 1.5.0 and isso 0.10.6


Could you override the html5lib version in the isso expression instead?

I will take care of getting rid of bleach 1.5.0

dotlambda · 2018-04-15T10:41:03Z

I added some small corrections, please have a look. Feel free to sqash the last two commits.

fgaz · 2018-04-16T14:12:23Z

Thanks! Looks good!

dotlambda · 2018-04-16T21:09:14Z

@GrahamcOfBorg build python2.pkgs.misaka python3.pkgs.misaka

GrahamcOfBorg · 2018-04-16T21:09:58Z

Success on aarch64-linux (full log)

Attempted: python2.pkgs.misaka, python3.pkgs.misaka

Partial log (click to expand)

post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/jjpm9klw56pnc5vizdrjjn9biybq9w62-python3.6-misaka-2.1.0
shrinking /nix/store/jjpm9klw56pnc5vizdrjjn9biybq9w62-python3.6-misaka-2.1.0/lib/python3.6/site-packages/misaka/_hoedown.abi3.so
strip is /nix/store/j7d4mr0ikv974ig7yzhknpsq288js4bs-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/jjpm9klw56pnc5vizdrjjn9biybq9w62-python3.6-misaka-2.1.0/lib  /nix/store/jjpm9klw56pnc5vizdrjjn9biybq9w62-python3.6-misaka-2.1.0/bin
patching script interpreter paths in /nix/store/jjpm9klw56pnc5vizdrjjn9biybq9w62-python3.6-misaka-2.1.0
checking for references to /build in /nix/store/jjpm9klw56pnc5vizdrjjn9biybq9w62-python3.6-misaka-2.1.0...
wrapping `/nix/store/jjpm9klw56pnc5vizdrjjn9biybq9w62-python3.6-misaka-2.1.0/bin/misaka'...
/nix/store/060a1bwy9q3qrn7gy9jx6wpl01kk7v9d-python2.7-misaka-2.1.0
/nix/store/jjpm9klw56pnc5vizdrjjn9biybq9w62-python3.6-misaka-2.1.0

dotlambda · 2018-04-16T21:29:17Z

@GrahamcOfBorg build isso

GrahamcOfBorg · 2018-04-16T21:34:24Z

Success on aarch64-linux (full log)

Attempted: isso

Partial log (click to expand)

warning: no files found matching 'isso/js/embed.dev.js'
warning: no files found matching 'isso/js/count.min.js'
warning: no files found matching 'isso/js/count.dev.js'
writing manifest file 'isso.egg-info/SOURCES.txt'
......................................................S............
----------------------------------------------------------------------
Ran 67 tests in 28.290s

OK (SKIP=1)
/nix/store/vzbxy4m5g2bh9fjzjjzslabz72zgsnr1-isso-0.10.6

GrahamcOfBorg · 2018-04-17T02:30:58Z

Success on x86_64-linux (full log)

Attempted: python2.pkgs.misaka, python3.pkgs.misaka

Partial log (click to expand)

post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/lpagqxkh49y835a1hi7ci7yhjdjl4sqj-python3.6-misaka-2.1.0
shrinking /nix/store/lpagqxkh49y835a1hi7ci7yhjdjl4sqj-python3.6-misaka-2.1.0/lib/python3.6/site-packages/misaka/_hoedown.abi3.so
strip is /nix/store/j75dgadrff2d1fyc4fczmcgqkid2imdx-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/lpagqxkh49y835a1hi7ci7yhjdjl4sqj-python3.6-misaka-2.1.0/lib  /nix/store/lpagqxkh49y835a1hi7ci7yhjdjl4sqj-python3.6-misaka-2.1.0/bin
patching script interpreter paths in /nix/store/lpagqxkh49y835a1hi7ci7yhjdjl4sqj-python3.6-misaka-2.1.0
checking for references to /build in /nix/store/lpagqxkh49y835a1hi7ci7yhjdjl4sqj-python3.6-misaka-2.1.0...
wrapping `/nix/store/lpagqxkh49y835a1hi7ci7yhjdjl4sqj-python3.6-misaka-2.1.0/bin/misaka'...
/nix/store/wgnda2v2v7jicg7sb29i4nj3q711p8qj-python2.7-misaka-2.1.0
/nix/store/lpagqxkh49y835a1hi7ci7yhjdjl4sqj-python3.6-misaka-2.1.0

GrahamcOfBorg · 2018-04-17T02:34:05Z

Success on x86_64-linux (full log)

Attempted: isso

Partial log (click to expand)

warning: no files found matching 'isso/js/embed.dev.js'
warning: no files found matching 'isso/js/count.min.js'
warning: no files found matching 'isso/js/count.dev.js'
writing manifest file 'isso.egg-info/SOURCES.txt'
......................................................S............
----------------------------------------------------------------------
Ran 67 tests in 10.377s

OK (SKIP=1)
/nix/store/52w954yrmbbcip7xlmkjhnrnn2bhv5rp-isso-0.10.6

dotlambda · 2018-04-17T07:01:15Z

Can you actually run isso? I'm getting

Traceback (most recent call last):
  File "/nix/store/52w954yrmbbcip7xlmkjhnrnn2bhv5rp-isso-0.10.6/bin/.isso-wrapped", line 8, in <module>
    from isso import main
  File "/nix/store/52w954yrmbbcip7xlmkjhnrnn2bhv5rp-isso-0.10.6/lib/python2.7/site-packages/isso/__init__.py", line 71, in <module>
    from isso.utils import http, JSONRequest, html, hash
  File "/nix/store/52w954yrmbbcip7xlmkjhnrnn2bhv5rp-isso-0.10.6/lib/python2.7/site-packages/isso/utils/html.py", line 19, in <module>
    import misaka
  File "/nix/store/lpagqxkh49y835a1hi7ci7yhjdjl4sqj-python3.6-misaka-2.1.0/lib/python3.6/site-packages/misaka/__init__.py", line 3, in <module>
    from .api import *
  File "/nix/store/lpagqxkh49y835a1hi7ci7yhjdjl4sqj-python3.6-misaka-2.1.0/lib/python3.6/site-packages/misaka/api.py", line 9, in <module>
    from ._hoedown import lib, ffi
ImportError: No module named _hoedown

fgaz · 2018-04-17T08:48:20Z

It works for me (even posting and getting comments). What did you do to make it crash?

edit: I tried it on both nixos and debian+nix

dotlambda · 2018-04-17T09:12:53Z

Maybe fetching from GitHub instead of PyPI changed something. Did you try it with my changes?
I only ran isso -h.

fgaz · 2018-04-17T09:21:18Z

Yup, I'm on 6d09fbf

dotlambda · 2018-04-17T09:56:48Z

That's strange: When using nix-review, I get the error above. It works however when building from your branch.

dotlambda · 2018-04-17T09:59:12Z

@fgaz Do you plan on adding a module in another PR?

fgaz · 2018-04-18T15:25:09Z

Yes, I do. The module part of #13587 seems still pretty good so I'll probably base it on that

fgaz requested a review from FRidh as a code owner January 26, 2018 15:25

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Jan 26, 2018

FRidh requested changes Feb 3, 2018

View reviewed changes

fgaz force-pushed the isso branch from 5c7d04f to 7cbd092 Compare February 22, 2018 18:54

GrahamcOfBorg added the 6.topic: python label Feb 22, 2018

GrahamcOfBorg added the 8.has: package (new) label Feb 22, 2018

fgaz force-pushed the isso branch from 7cbd092 to c482283 Compare February 22, 2018 19:02

dotlambda requested changes Feb 22, 2018

View reviewed changes

fgaz force-pushed the isso branch 2 times, most recently from dbf8c0b to f2115f8 Compare March 28, 2018 13:55

dotlambda reviewed Apr 3, 2018

View reviewed changes

fgaz force-pushed the isso branch 4 times, most recently from 7300df4 to 53f7340 Compare April 14, 2018 18:01

dotlambda reviewed Apr 14, 2018

View reviewed changes

python: misaka: init at 2.1.0

b532cc3

fgaz force-pushed the isso branch from 53f7340 to 96b1dd8 Compare April 15, 2018 08:48

python: isso: init at 0.10.6

6d09fbf

fgaz force-pushed the isso branch from 427ff5a to 6d09fbf Compare April 15, 2018 21:15

dotlambda merged commit c6a8b90 into NixOS:master Apr 17, 2018

fgaz deleted the isso branch April 17, 2018 10:57

davidak mentioned this pull request Aug 26, 2019

Package Commento and create NixOS module #67524

Closed

Isso: init at 0.10.6 (plus dependencies) #34296

Isso: init at 0.10.6 (plus dependencies) #34296

Conversation

fgaz commented Jan 26, 2018 • edited

Motivation for this change

Things done

fgaz commented Jan 26, 2018

dotlambda commented Jan 27, 2018

fpletz commented Jan 27, 2018

Profpatsch commented Jan 27, 2018

fgaz commented Jan 27, 2018

fgaz commented Jan 27, 2018

fgaz commented Jan 27, 2018

Profpatsch commented Jan 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fgaz commented Feb 22, 2018

fgaz commented Feb 22, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dotlambda commented Apr 3, 2018

GrahamcOfBorg commented Apr 3, 2018

GrahamcOfBorg commented Apr 3, 2018

fgaz commented Apr 14, 2018

dotlambda Apr 14, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dotlambda commented Apr 15, 2018

fgaz commented Apr 16, 2018

dotlambda commented Apr 16, 2018

GrahamcOfBorg commented Apr 16, 2018

dotlambda commented Apr 16, 2018

GrahamcOfBorg commented Apr 16, 2018

GrahamcOfBorg commented Apr 17, 2018

GrahamcOfBorg commented Apr 17, 2018

dotlambda commented Apr 17, 2018

fgaz commented Apr 17, 2018 • edited

dotlambda commented Apr 17, 2018 • edited

fgaz commented Apr 17, 2018

dotlambda commented Apr 17, 2018

dotlambda commented Apr 17, 2018

fgaz commented Apr 18, 2018

fgaz commented Jan 26, 2018 •

edited

fgaz commented Feb 22, 2018 •

edited

dotlambda Apr 14, 2018 •

edited

fgaz commented Apr 17, 2018 •

edited

dotlambda commented Apr 17, 2018 •

edited