fwiw, in a failed attempt to get wxPython 4.1 to work for kicad, i copied the wxPython 4.0 package, checked the dependencies they listed for 4.1 and separated the ones not listed as a dependency

there seems to be a fairly large overlap between those and the ones you removed
evils@101d1e9#diff-78d98acc609d3ebd6964dfc105e7fe2dR24

@lopsided98 I’m not sure this is advisable. There indeed may be some unneeded packages but many of these are necessary. See https://github.com/wxWidgets/Phoenix#prerequisites

Despite what the documentation says, the lack of a reference to them basically proves they are not actually used by our build. I don't see how this PR can cause any new problems due to missing dependencies (unless they are needed for Darwin), because those dependencies are already missing from the build output.

Perhaps some configure flags or something else should be added to cause the build to actually make use of those dependencies, but that can be done in a different PR.

I used diffoscope to compare the output with and without this PR (changing gstreamer 0.10 to 1.0 to make it eval), and there were no differences other than a different $out.

this PR fixes kicad's scriptingSupport

I added another commit that removes the wrapPythonPrograms invocation from installPhase. This was causing double wrapping because wrapPythonPrograms automatically runs during the fixup phase.

@lopsided98 I expect that most of them are not statically linked. For example, without libjpeg wxPython is bound to crash as soon as anything involving a jpeg happens

I'm not sure what you mean about static linking. This PR has no effect on the package output; I verified this with diffoscope. Therefore any potential problem like you describe already happens without this PR.

for example, I expect import wx to fail with ImportError: libjpeg.so.8: cannot open shared object file: No such file or directory https://discuss.wxpython.org/t/wxpython-on-fedora-31/34483.

It's actually quite rare that devs take the time to specify all the binary dependencies of python packages, and I trust their judgment more than ours for what is needed...

Again, how can this start happening if there is no change to the package output?

None of those dependencies are propagated, and even if they were that would show up as a change in the output because propagatedBuildInputs are placed in the $out/nix-support/propagated-build-inputs file.

have you tried import wx? Note that doCheck = false.

It does seem that you identified a bunch of packages that should be listed in propagatedBuildInputs instead of buildInputs, but removing them outright will break the package.

If we want to go the route of removing dependencies that the devs say are necessary, proof isn't in the build succeeding but in the tests succeeding.

If you remove libjpeg from buildInputs, then libjpeg.so.8 is not going to be found. My understanding is that packages put in buildInputs are also available at run-time.

Yes, import wx succeeds.

In Nix, the only thing that really matters is the output. No change in the output means no change in the dependencies that are available at runtime. 'buildInputs` are not available at runtime unless they are referenced by the output.

As far as I can tell, the documentation seems to be listing the dependencies required to build both wxPython and wxGTK. wxGTK links against libjpeg, gstreamer and others, while wxPython only links against wxGTK.

at least for wxPython 4.1, it seems like building the version of wxWidgets (wxGTK) they include with it is required
i think the documentation indeed assumes you're building both, as this is the upstream expected build process

@lopsided98 Really appreciate the help maintaining this package, but you are wrong about buildInputs. They are available at runtime. See

Also:

test.nix:

with import <nixpkgs> {};
stdenv.mkDerivation {
    name = "test";
    buildInputs = [ git ];
}

> nix-shell test.nix
$ git
[...]

The wxPython Phoenix (4.0==first phoenix release) team is very clear that these dependencies are for Phoenix:

On Ubuntu the following development packages and their dependencies should be installed in order to build Phoenix. Other debian-like distros will probably also have these or similarly named packages available. Extrapolate other package names accordingly for other linux distributions or other unixes.

That works in nix-shell because nix-shell sets up the build time environment for the test derivation.

Nix has no concept of runtime dependencies other than those that are referenced by the output. No change in the output means no change in the runtime dependencies.

Maybe @FRidh can resolve this debate. In python packages, are buildInputs available at runtime?

@lopsided98 is indeed correct. If the output (using e.g. diffoscope) doesn't change when the libraries are removed, they are indeed not used at all. And buildInputs don't automatically become runtime dependencies, they need to actually be referenced in the final $out for that to be the case.

That said, even though all these libraries are removed, some of them are still referenced and used, such as libjpeg:

$ nix-shell -p 'python3.withPackages (p: [ p.wxPython_4_0 ])' -I nixpkgs=$PWD \
  --run 'strace python -c "import wx" 2>&1 | grep libjpeg.so'                                                                                                                          
openat(AT_FDCWD, "/nix/store/j8616aadakr67ynjndlnz2qf233svfmz-wxwidgets-3.0.4/lib/libjpeg.so.62", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[...]
openat(AT_FDCWD, "/nix/store/cx86r4i3nwvywqi0l04qfkcj1k07iyrf-libjpeg-turbo-2.0.4/lib/x86_64/libjpeg.so.62", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/cx86r4i3nwvywqi0l04qfkcj1k07iyrf-libjpeg-turbo-2.0.4/lib/libjpeg.so.62", O_RDONLY|O_CLOEXEC) = 3

and libtiff:

$ nix-shell -p 'python3.withPackages (p: [ p.wxPython_4_0 ])' -I nixpkgs=$PWD \
  --run 'strace python -c "import wx" 2>&1 | grep libtiff.so'
openat(AT_FDCWD, "/nix/store/j8616aadakr67ynjndlnz2qf233svfmz-wxwidgets-3.0.4/lib/libtiff.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[...]
openat(AT_FDCWD, "/nix/store/824zv3ch52nshl9yzgn7byrjgb36iib2-libtiff-4.1.0/lib/x86_64/libtiff.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/824zv3ch52nshl9yzgn7byrjgb36iib2-libtiff-4.1.0/lib/libtiff.so.5", O_RDONLY|O_CLOEXEC) = 3

Maybe more.

I think the best thing to do here is to figure out if wxPython's build would complain about missing libraries. Because if it does, then we don't have to worry, since the build here passes and can therefore find all necessary libraries.

I can also confirm that diffoscope doesn't show any significant changes before and after this PR, so this is essentially just a code cleanup that can't break anything. But I say let's take this opportunity to verify the package still works and has all deps it needs.

Huh, ok sounds like I am indeed wrong! Thank you both for educating me. I did not realize that Nix would not make the buildInputs available at runtime if the hash does not appear in the output. This actually explains a lot of problems in the Python and Julia ecosystem that rely on various versions of find_library. seems like for dynamically loaded libraries wrap_program is the main option.

wxPython itself uses find_library several times: https://github.com/wxWidgets/Phoenix/search?q=find_library

So apologies for mistakenly opposing these changes, I understand now why it doesn’t make it any worse off, but it does highlight for me that some of these libraries may indeed still be runtime dependencies that we are missing.

I looked into enabling the tests, but ran into various tooling related errors. I also tried pythonImportsCheck, but that claimed the module could not be found.

I don't really have time to look into fixing the tests. This was supposed to be a quick and safe fix to unbreak a relatively important dependency, so I would like to get it merged without making it more complex than it needs to be.

I looked into enabling the tests, but ran into various tooling related errors. I also tried pythonImportsCheck, but that claimed the module could not be found.

python by default will add the $PWD to sys.path, and can sometimes mess with imports because some compiled files won't be present.

I think it would be a good idea to make sure this package works before unbreaking it (even if just indirectly).

I ran the demo application included in the source and everything seemed to work.

I did notice that some more dependencies were listed in the requirements.txt file, so I added them to propagatedBuildInputs. They are only needed for certain modules, so maybe it makes more sense for downstream packages to pull in those dependencies if needed (which is probably what was done previously). For example, the demo application uses a module that needs six, but not ones that need numpy or pillow, so those dependencies would needlessly become part of the closure.

I'd love to see this PR being merged (or something being done to get wxpython4 working again)!
Is there anything i could help with?
Cheers!

After some more thought, I decided to remove my last change (adding the propagated dependencies from requirements.txt). These are not new dependencies, so they have probably already been added to downstream packages where needed, and I don't see a reason to change that as part of this PR.

I believe this PR is ready to be merged.

I guess we can always add more deps later if needed. This does at least unbreak the package partially for now

I'm confused, was this supposed to add pillow and numpy or not?
AFAICT It added them to the argument set in, but forgot to add them to the propagatedBuildInputs, and then they later got removed from the argument set because they were unused.

I'm building an application where in one of the phases pip tries and fails to find numpy and pillow.

Those packages were the ones I was talking about here: #94108 (comment). I initially added them, then removed them from propagatedBuildInputs but forgot to remove them from the argument set.

https://github.com/wxWidgets/Phoenix/blob/64e5d863f7833f10df6a0fbcf3221a730562224b/requirements/install.txt appears to impose runtime requirements for numpy and pillow.
What should be done?

They should probably be added, I just decided they were out of scope for this PR since they didn't seem to be needed by any dependents at the time.

Hm. https://discuss.wxpython.org/t/does-wxpython-4-0-6-must-also-install-numpy/32969/2
Of course it's never simple.. (that's also a pretty old post, IDK if things have changed)

I also don't quite understand why no one has run into the issue of buildPythonApplication trying to require numpy and pillow due to wxPython yet.

pip doesn't seem to provide any sane way of excluding deps pypa/pip#3090 , so I probably need to use some sort of indirect method if I deem them unnecessary.

Well, Robin (discuss.wxpython.org link above) does list --no-deps, but that seems rather shotgun.

The concrete example of these deps attempting to be pulled is this: f33d2a5#diff-9e9c51bfcf7e9562777c9c8d1b946f5a61995d0947955d188c7f1fc5b9a4c5c0R27

When run you get:

Executing pipInstallPhase
/build/source/dist /build/source
Processing ./WikidPad-2.4a1-py3-none-any.whl
Requirement already satisfied: wxpython>=4.0 in /nix/store/qy1ya8pcgjcjfhqvkm43pzb71w80122r-python3.7-wxPython-4.0.7.post2/lib/python3.7/site-packages (from WikidPad==2.4a1) (4.0.7.post2)
INFO: pip is looking at multiple versions of wikidpad to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement pillow (from wxpython)
ERROR: No matching distribution found for pillow

I'm entirely unsure what the best thing to do here is.
Apparently wxPython has numpy and pillow them as runtime dependencies because they're needed relatively often, but they're actually kind of optional?
Additionally there are "other packages" that are used less commonly, which are also runtime dependencies, but they aren't listed in the requirements because they're needed less often?

(Where does one find a list of these optional dependencies?)

related: wxWidgets/Phoenix#1932

Hi @RobinD42 , can you maybe clarify the dependency situation with numpy and pillow in wxPython?
Is there anything significant to pay attention to across versions?
Is there a list of other more optional packages somewhere? (see above post)

Specifically I would appreciate clarification on

Apparently wxPython has numpy and pillow them as runtime dependencies because they're needed relatively often, but they're actually kind of optional?
Additionally there are "other packages" that are used less commonly, which are also runtime dependencies, but they aren't listed in the requirements because they're needed less often?

in relation to https://discuss.wxpython.org/t/does-wxpython-4-0-6-must-also-install-numpy/32969/2 .

wxWidgets/Phoenix#1932 is already asking to make these dependencies optional, and they seem to have done a little work looking at what's needed where.

- and if I have your attention anyway, would you consider packaging wxPython with https://python-poetry.org/ ?
I hear good things about it. (I could open an issue on the Phoenix issue tracker I guess).
(It's not obvious whether the specific optional dependency handling is any better though at the time of https://stackoverflow.com/questions/60971502/python-poetry-how-to-install-optional-dependencies )

From a practical point of view, adding dependencies seems to be easier than removing them. - which contravariantly, means that the requirement should be removed from upstream.

I've filed #127768 , so I think any additional discussion should go there.