pyprof2calltree: init at 1.4.3 #34379

SFrijters · 2018-01-29T10:56:30Z

Motivation for this change

New package: "Help visualize profiling data from cProfile with kcachegrind and qcachegrind"

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

dotlambda · 2018-01-29T11:04:44Z

pkgs/development/python-modules/pyprof2calltree/default.nix

+{ lib, buildPythonApplication, fetchPypi }:
+
+buildPythonApplication rec {
+  name = "${pname}-${version}";


Should this just be left out entirely? Is it automatically generated?
Apologies for any stupid questions - this is my first contribution.

Yes. Yes it is.

dotlambda · 2018-01-29T11:05:17Z

pkgs/development/python-modules/pyprof2calltree/default.nix

+    sha256 = "38a0774f7716d5303d4c57ff89ec24258154942666e4404558f6ac10f8bb1e52";
+  };
+
+  doCheck = false;


dotlambda · 2018-01-29T11:06:41Z

pkgs/development/python-modules/pyprof2calltree/default.nix

+    description = "Help visualize profiling data from cProfile with kcachegrind and qcachegrind";
+    homepage = https://pypi.python.org/pypi/pyprof2calltree/;
+    license = licenses.mit;
+  };


Would you mind adding yourself as maintainer?

Should this be added as a string in place, or should I add myself to maintainers.nix?

dotlambda · 2018-01-29T11:16:36Z

pkgs/top-level/python-packages.nix

@@ -316,6 +316,8 @@ in {
    callPackage = pkgs.callPackage;
  };

+  pyprof2calltree = callPackage ../development/python-modules/pyprof2calltree { };


That's the wrong location. Applications belong into all-packages.nix. See https://nixos.org/nixpkgs/manual/#building-packages-and-applications. Also, your derivation should probably be put into pkgs/tools/...

Well, it's a development tool, so maybe I should rather move the derivation to pkgs/development/tools/pyprof2calltree ?

SFrijters · 2018-01-29T13:54:40Z

I'm a bit lost w.r.t. the current error... I can take a look later, but would appreciate any hints.

dotlambda · 2018-01-29T14:37:09Z

pkgs/top-level/all-packages.nix

@@ -7780,6 +7780,8 @@ with pkgs;

  pprof = callPackage ../development/tools/profiling/pprof { };

+  pyprof2calltree = callPackage ../development/tools/profiling/pyprof2calltree { };


You need to use pythonPackages.callPackage or python3Packages.callPackage.

dotlambda · 2018-01-29T14:42:40Z

pkgs/development/tools/profiling/pyprof2calltree/default.nix

+    sha256 = "38a0774f7716d5303d4c57ff89ec24258154942666e4404558f6ac10f8bb1e52";
+  };
+
+  doCheck = false;


Don't forget to add checkInputs and enable tests or mention why they are disabled.

I'll look into it a bit more; if I run python setup.py test on a fresh pyprof2calltree clone the tests work fine, but the nix build freaks out. This is in a nix-shell by the way.

In pyprof2calltree clone:

$ nix-shell -p 'python27.withPackages(ps: with ps; [ setuptools ])' $ which python /nix/store/rmn08qyi17r6ngqvpa3d42c3bpg7bgk7-python-2.7.14-env/bin/python $ python setup.py test running test running egg_info writing pyprof2calltree.egg-info/PKG-INFO writing top-level names to pyprof2calltree.egg-info/top_level.txt writing dependency_links to pyprof2calltree.egg-info/dependency_links.txt writing entry points to pyprof2calltree.egg-info/entry_points.txt reading manifest file 'pyprof2calltree.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'pyprof2calltree.egg-info/SOURCES.txt' running build_ext test_direct_entries (test.test_integration.TestIntegration) ... ok test_pstats_data (test.test_integration.TestIntegration) ... ok ---------------------------------------------------------------------- Ran 2 tests in 0.001s

OK

In nixpkgs clone:

$ nix-build -A pyprof2calltree these derivations will be built: /nix/store/ydvv4c1725x997q34sygyz5fv7bwkli6-pyprof2calltree-1.4.3.drv building path(s) ‘/nix/store/z0n3xayrgxx6y2ybfaifw0ayajsgsx52-pyprof2calltree-1.4.3’ unpacking sources unpacking source archive /nix/store/s37xckpayjs978v3jq09abd2g47hp1ld-pyprof2calltree-1.4.3.tar.gz source root is pyprof2calltree-1.4.3 setting SOURCE_DATE_EPOCH to timestamp 1501275086 of file pyprof2calltree-1.4.3/setup.cfg patching sources configuring building running bdist_wheel running build running build_py creating build creating build/lib copying pyprof2calltree.py -> build/lib installing to build/bdist.linux-x86_64/wheel running install running install_lib creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/wheel copying build/lib/pyprof2calltree.py -> build/bdist.linux-x86_64/wheel running install_egg_info running egg_info writing pyprof2calltree.egg-info/PKG-INFO writing top-level names to pyprof2calltree.egg-info/top_level.txt writing dependency_links to pyprof2calltree.egg-info/dependency_links.txt writing entry points to pyprof2calltree.egg-info/entry_points.txt reading manifest file 'pyprof2calltree.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'pyprof2calltree.egg-info/SOURCES.txt' Copying pyprof2calltree.egg-info to build/bdist.linux-x86_64/wheel/pyprof2calltree-1.4.3-py2.7.egg-info running install_scripts creating build/bdist.linux-x86_64/wheel/pyprof2calltree-1.4.3.dist-info/WHEEL creating '/tmp/nix-build-pyprof2calltree-1.4.3.drv-0/pyprof2calltree-1.4.3/dist/pyprof2calltree-1.4.3-py2-none-any.whl' and adding '.' to it adding 'pyprof2calltree.py' adding 'pyprof2calltree-1.4.3.dist-info/DESCRIPTION.rst' adding 'pyprof2calltree-1.4.3.dist-info/entry_points.txt' adding 'pyprof2calltree-1.4.3.dist-info/metadata.json' adding 'pyprof2calltree-1.4.3.dist-info/top_level.txt' adding 'pyprof2calltree-1.4.3.dist-info/zip-safe' adding 'pyprof2calltree-1.4.3.dist-info/WHEEL' adding 'pyprof2calltree-1.4.3.dist-info/METADATA' adding 'pyprof2calltree-1.4.3.dist-info/RECORD' installing /tmp/nix-build-pyprof2calltree-1.4.3.drv-0/pyprof2calltree-1.4.3/dist /tmp/nix-build-pyprof2calltree-1.4.3.drv-0/pyprof2calltree-1.4.3 Processing ./pyprof2calltree-1.4.3-py2-none-any.whl Installing collected packages: pyprof2calltree Successfully installed pyprof2calltree-1.4.3 /tmp/nix-build-pyprof2calltree-1.4.3.drv-0/pyprof2calltree-1.4.3 post-installation fixup shrinking RPATHs of ELF executables and libraries in /nix/store/z0n3xayrgxx6y2ybfaifw0ayajsgsx52-pyprof2calltree-1.4.3 strip is /nix/store/mdyy001q67hiks0g24ra53z7ckm4jfr4-binutils-2.28.1/bin/strip stripping (with command strip and flags -S) in /nix/store/z0n3xayrgxx6y2ybfaifw0ayajsgsx52-pyprof2calltree-1.4.3/lib /nix/store/z0n3xayrgxx6y2ybfaifw0ayajsgsx52-pyprof2calltree-1.4.3/bin patching script interpreter paths in /nix/store/z0n3xayrgxx6y2ybfaifw0ayajsgsx52-pyprof2calltree-1.4.3 checking for references to /tmp/nix-build-pyprof2calltree-1.4.3.drv-0 in /nix/store/z0n3xayrgxx6y2ybfaifw0ayajsgsx52-pyprof2calltree-1.4.3... wrapping `/nix/store/z0n3xayrgxx6y2ybfaifw0ayajsgsx52-pyprof2calltree-1.4.3/bin/pyprof2calltree'... running install tests running test running egg_info writing pyprof2calltree.egg-info/PKG-INFO writing top-level names to pyprof2calltree.egg-info/top_level.txt writing dependency_links to pyprof2calltree.egg-info/dependency_links.txt writing entry points to pyprof2calltree.egg-info/entry_points.txt reading manifest file 'pyprof2calltree.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'pyprof2calltree.egg-info/SOURCES.txt' running build_ext Traceback (most recent call last): File "nix_run_setup", line 8, in <module> exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec')) File "setup.py", line 44, in <module> 'pyprof2calltree = pyprof2calltree:main', File "/nix/store/a4zb290zkx8avpand4a58ndci8ppczik-python2.7-bootstrapped-pip-9.0.1/lib/python2.7/site-packages/setuptools/__init__.py", line 129, in setup return distutils.core.setup(**attrs) File "/nix/store/87l8l9w3x8saxqz9sfcfw2i3jiilmxfi-python-2.7.14/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/nix/store/87l8l9w3x8saxqz9sfcfw2i3jiilmxfi-python-2.7.14/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/nix/store/87l8l9w3x8saxqz9sfcfw2i3jiilmxfi-python-2.7.14/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/nix/store/a4zb290zkx8avpand4a58ndci8ppczik-python2.7-bootstrapped-pip-9.0.1/lib/python2.7/site-packages/setuptools/command/test.py", line 226, in run self.run_tests() File "/nix/store/a4zb290zkx8avpand4a58ndci8ppczik-python2.7-bootstrapped-pip-9.0.1/lib/python2.7/site-packages/setuptools/command/test.py", line 248, in run_tests exit=False, File "/nix/store/87l8l9w3x8saxqz9sfcfw2i3jiilmxfi-python-2.7.14/lib/python2.7/unittest/main.py", line 94, in __init__ self.parseArgs(argv) File "/nix/store/87l8l9w3x8saxqz9sfcfw2i3jiilmxfi-python-2.7.14/lib/python2.7/unittest/main.py", line 149, in parseArgs self.createTests() File "/nix/store/87l8l9w3x8saxqz9sfcfw2i3jiilmxfi-python-2.7.14/lib/python2.7/unittest/main.py", line 158, in createTests self.module) File "/nix/store/87l8l9w3x8saxqz9sfcfw2i3jiilmxfi-python-2.7.14/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames suites = [self.loadTestsFromName(name, module) for name in names] File "/nix/store/87l8l9w3x8saxqz9sfcfw2i3jiilmxfi-python-2.7.14/lib/python2.7/unittest/loader.py", line 103, in loadTestsFromName return self.loadTestsFromModule(obj) File "/nix/store/a4zb290zkx8avpand4a58ndci8ppczik-python2.7-bootstrapped-pip-9.0.1/lib/python2.7/site-packages/setuptools/command/test.py", line 52, in loadTestsFromModule tests.append(self.loadTestsFromName(submodule)) File "/nix/store/87l8l9w3x8saxqz9sfcfw2i3jiilmxfi-python-2.7.14/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName module = __import__('.'.join(parts_copy)) File "/nix/store/87l8l9w3x8saxqz9sfcfw2i3jiilmxfi-python-2.7.14/lib/python2.7/test/regrtest.py", line 203, in <module> for module in sys.modules.itervalues(): RuntimeError: dictionary changed size during iteration builder for ‘/nix/store/ydvv4c1725x997q34sygyz5fv7bwkli6-pyprof2calltree-1.4.3.drv’ failed with exit code 1 error: build of ‘/nix/store/ydvv4c1725x997q34sygyz5fv7bwkli6-pyprof2calltree-1.4.3.drv’ failed

I've seen this error before but only seldom. I think the cause is that setuptools runs build_ext (for I don't know what reason), "building" the module in place, while it actually already exists. I can't think of another reason that sys.modules changes.

I think a workaround is to write

${python.interpreter} -m unittest discover -s test

Thanks. Not quite there yet though (same result in nix-shell and nixpkgs tree):

====================================================================== ERROR: test_integration (unittest.loader.ModuleImportFailure) ---------------------------------------------------------------------- ImportError: Failed to import test module: test_integration Traceback (most recent call last): File "/nix/store/87l8l9w3x8saxqz9sfcfw2i3jiilmxfi-python-2.7.14/lib/python2.7/unittest/loader.py", line 254, in _find_tests module = self._get_module_from_name(name) File "/nix/store/87l8l9w3x8saxqz9sfcfw2i3jiilmxfi-python-2.7.14/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name __import__(name) File "/tmp/nix-build-pyprof2calltree-1.4.3.drv-0/pyprof2calltree-1.4.3/test/test_integration.py", line 5, in <module> from .profile_code import top, expected_output ValueError: Attempted relative import in non-package

That's a valid bug in their sdist. They forgot to include __init__.py (and also profile_code.py).

Is doCheck = false; (with a comment) an option then? I can raise an issue upstream, but it's not my package.

Yes, it is. Raising an issue is a good idea, although I did notice they moved the test files around a bit since the current release.

dotlambda · 2018-01-29T17:34:16Z

@GrahamcOfBorg build pyprof2calltree

GrahamcOfBorg

Failure on aarch64-linux (full log)

Partial log (click to expand)

    from .profile_code import top, expected_output
ValueError: Attempted relative import in non-package


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)
builder for '/nix/store/7015ww1p2d3jw6l29mmr9rz3bqbzq68m-pyprof2calltree-1.4.3.drv' failed with exit code 1
error: build of '/nix/store/7015ww1p2d3jw6l29mmr9rz3bqbzq68m-pyprof2calltree-1.4.3.drv' failed

GrahamcOfBorg

Failure on x86_64-linux (full log)

Partial log (click to expand)

    from .profile_code import top, expected_output
ValueError: Attempted relative import in non-package


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)
builder for ‘/nix/store/lrjlma6y1mafk2w130sm4z8ppk5shspw-pyprof2calltree-1.4.3.drv’ failed with exit code 1
error: build of ‘/nix/store/lrjlma6y1mafk2w130sm4z8ppk5shspw-pyprof2calltree-1.4.3.drv’ failed

SFrijters · 2018-01-30T07:25:56Z

Ok, I now changed the derivation to use fetchFromGitHub, which does allow all tests to be run. Is this acceptable?

FRidh · 2018-01-30T09:06:18Z

@GrahamcOfBorg build pyprof2calltree

GrahamcOfBorg

Success on x86_64-darwin (full log)

Partial log (click to expand)

writing manifest file 'pyprof2calltree.egg-info/SOURCES.txt'
running build_ext
test_direct_entries (test.test_integration.TestIntegration) ... ok
test_pstats_data (test.test_integration.TestIntegration) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.002s

OK
/nix/store/rqxm4cd6853041mb2irfnvipch4s0jw0-pyprof2calltree-1.4.3

GrahamcOfBorg

Success on x86_64-linux (full log)

Partial log (click to expand)

writing manifest file 'pyprof2calltree.egg-info/SOURCES.txt'
running build_ext
test_direct_entries (test.test_integration.TestIntegration) ... ok
test_pstats_data (test.test_integration.TestIntegration) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK
/nix/store/0hwqqhs47d9v1cmlc7fvbs0jywzq74ym-pyprof2calltree-1.4.3

GrahamcOfBorg

Success on aarch64-linux (full log)

Partial log (click to expand)

writing manifest file 'pyprof2calltree.egg-info/SOURCES.txt'
running build_ext
test_direct_entries (test.test_integration.TestIntegration) ... ok
test_pstats_data (test.test_integration.TestIntegration) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.004s

OK
/nix/store/qdh2mvw6j8k4l35pmjr57mb9p2p3xh0d-pyprof2calltree-1.4.3

SFrijters · 2018-01-30T09:50:43Z

Thanks for helping me with my contribution!

dotlambda · 2018-01-30T09:54:55Z

Next time, it would be niced if you amended the commit instead of adding new ones though.
See https://nixos.org/nixpkgs/manual/#idm140737317041456.

pyprof2calltree: init at 1.4.3

ca47bba

SFrijters requested a review from FRidh as a code owner January 29, 2018 10:56

GrahamcOfBorg added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Jan 29, 2018

dotlambda reviewed Jan 29, 2018

View reviewed changes

SFrijters added 4 commits January 29, 2018 14:09

Add maintainer

870e395

Remove redundant name attribute

1df411c

Move derivation

92f0def

Fix maintainers

0dcd4f8

dotlambda reviewed Jan 29, 2018

View reviewed changes

SFrijters added 2 commits January 29, 2018 15:54

Fix callPackage

7cca2b4

Try workaround to make tests work (not successful yet)

b6bd9c1

GrahamcOfBorg reviewed Jan 29, 2018

View reviewed changes

SFrijters added 2 commits January 30, 2018 08:18

Disable check

b1aeda3

Fetch from GitHub instead

d4d8692

GrahamcOfBorg reviewed Jan 30, 2018

View reviewed changes

FRidh merged commit 6c744fc into NixOS:master Jan 30, 2018

Janik-Haag added the 12. first-time contribution label Jun 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pyprof2calltree: init at 1.4.3 #34379

pyprof2calltree: init at 1.4.3 #34379

SFrijters commented Jan 29, 2018

dotlambda Jan 29, 2018

SFrijters Jan 29, 2018

dotlambda Jan 29, 2018

dotlambda Jan 29, 2018

dotlambda Jan 29, 2018

SFrijters Jan 29, 2018

dotlambda Jan 29, 2018

dotlambda Jan 29, 2018

SFrijters Jan 29, 2018

dotlambda Jan 29, 2018

SFrijters commented Jan 29, 2018

dotlambda Jan 29, 2018

dotlambda Jan 29, 2018

SFrijters Jan 29, 2018 •

edited

SFrijters Jan 29, 2018 •

edited by FRidh

FRidh Jan 29, 2018

SFrijters Jan 29, 2018 •

edited by FRidh

FRidh Jan 29, 2018

SFrijters Jan 29, 2018

FRidh Jan 29, 2018

dotlambda commented Jan 29, 2018

GrahamcOfBorg left a comment

GrahamcOfBorg left a comment

SFrijters commented Jan 30, 2018

FRidh commented Jan 30, 2018

GrahamcOfBorg left a comment

GrahamcOfBorg left a comment

GrahamcOfBorg left a comment

SFrijters commented Jan 30, 2018

dotlambda commented Jan 30, 2018

		@@ -7780,6 +7780,8 @@ with pkgs;

		pprof = callPackage ../development/tools/profiling/pprof { };

		pyprof2calltree = callPackage ../development/tools/profiling/pyprof2calltree { };

pyprof2calltree: init at 1.4.3 #34379

pyprof2calltree: init at 1.4.3 #34379

Conversation

SFrijters commented Jan 29, 2018

Motivation for this change

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SFrijters commented Jan 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SFrijters Jan 29, 2018 • edited

Choose a reason for hiding this comment

SFrijters Jan 29, 2018 • edited by FRidh

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SFrijters Jan 29, 2018 • edited by FRidh

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dotlambda commented Jan 29, 2018

GrahamcOfBorg left a comment

Choose a reason for hiding this comment

GrahamcOfBorg left a comment

Choose a reason for hiding this comment

SFrijters commented Jan 30, 2018

FRidh commented Jan 30, 2018

GrahamcOfBorg left a comment

Choose a reason for hiding this comment

GrahamcOfBorg left a comment

Choose a reason for hiding this comment

GrahamcOfBorg left a comment

Choose a reason for hiding this comment

SFrijters commented Jan 30, 2018

dotlambda commented Jan 30, 2018

SFrijters Jan 29, 2018 •

edited

SFrijters Jan 29, 2018 •

edited by FRidh

SFrijters Jan 29, 2018 •

edited by FRidh