Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add scipy and pyyaml to FreeCAD, fix vtk8 build on gcc10 #108424

Closed
wants to merge 4 commits into from

Conversation

ppenguin
Copy link
Contributor

@ppenguin ppenguin commented Jan 4, 2021

Motivation for this change

Custom build of freecad (for additional python modules needed by additional Workbenches) fails due to vtk build failing on gcc10

Things done

FreeCAD PyrateWorkbench (possibly others) needs scipy and pyyaml, these were added to the py-packages in the freecad nix. The freecad dep vtk-8.2.0 fails to build with gcc10, modified the build file to require the gcc9 env.

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@tfmoraes
Copy link
Contributor

tfmoraes commented Jan 4, 2021

@ppenguin
Copy link
Contributor Author

ppenguin commented Jan 5, 2021

@ppenguin there is this patch to make VTK8 compiles with GCC10 https://gitweb.gentoo.org/repo/gentoo.git/tree/sci-libs/vtk/files/vtk-8.2.0-gcc-10.patch?id=c4256f68d3589570443075eccbbafacf661f785f

@tfmoraes Thanks, I actually found that one but since I just started (yesterday) with nix I haven't learnt how to integrate patches yet. But I agree it's better to integrate that patch. Will try...

@ppenguin
Copy link
Contributor Author

ppenguin commented Jan 5, 2021

Wow, it was so easy! nix is really, really elegant, kudos! (My reference is mainly experience with Gentoo, Manjaro/Arch, Homebrew, but I'm already a nix convert 🤓 )

Ok, so now the update to vtk became just the patch inclusion. In the future I'll make sure to separate PRs for different packages...

@tfmoraes
Copy link
Contributor

tfmoraes commented Jan 5, 2021

Cool @ppenguin!

@tfmoraes
Copy link
Contributor

tfmoraes commented Jan 5, 2021

I think You need add this other patch too to make VTK8 compile with QT5.15:

 {
      url = "https://gitweb.gentoo.org/repo/gentoo.git/plain/sci-libs/vtk/files/vtk-8.2.0-qt-5.15.patch?id=3ca9613d7ad604c93d714e29b116952561e4e41c";
      sha256 = "sha256-BFjoKws1hVD3Ly9RS4lGN62J6RTyI1E8ATHrZdzg7ds=";
}

@lheckemann
Copy link
Member

Hm, are these only in gentoo? They should probably be submitted upstream.

@tfmoraes
Copy link
Contributor

tfmoraes commented Jan 5, 2021

@lheckemann the Qt5.15 patch is already in VTK but there is not a VTK8 release with that https://gitlab.kitware.com/vtk/vtk/-/merge_requests/6943, maybe it's better to use this patch instead of the Gentoo one, the link is https://gitlab.kitware.com/vtk/vtk/-/merge_requests/6943.diff

}];
}
{
meta.description = "Fix compiling with gcc-10+";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Congrats on your first PR!
Note that the meta.description is something that should be used to describe a package as opposed to a patch for a package. You can drop that line.

@tfmoraes
Copy link
Contributor

tfmoraes commented Jan 6, 2021

@ppenguin the same patch you have applied to vtk8 com compile with gcc10 you will have to apply to vtk7. I tested here and it works with vtk7. Debian is applying a similar patch to vtk7 https://salsa.debian.org/science-team/vtk7/-/blob/master/debian/patches/115_support-gcc10.patch

@veprbl veprbl linked an issue Jan 6, 2021 that may be closed by this pull request
@eduardosm
Copy link
Contributor

Result of nixpkgs-review pr 108424 run on x86_64-linux 1

1 package marked as broken and skipped:
  • pcl
12 packages failed to build:
  • appcsxcad
  • libsForQt5.qcsxcad (libsForQt515.qcsxcad)
  • libsForQt512.qcsxcad
  • libsForQt514.qcsxcad
  • openems
  • python37Packages.python-csxcad
  • python37Packages.python-openems
  • python38Packages.python-csxcad
  • python38Packages.python-openems
  • python39Packages.python-csxcad
  • python39Packages.python-openems
  • vtkWithQt5
20 packages built:
  • cq-editor
  • csxcad
  • freecad
  • horizon-eda
  • inkcut
  • inkscape-extensions.inkcut
  • mirtk
  • opencascade
  • python37Packages.cadquery
  • python37Packages.enamlx
  • python37Packages.pythonocc-core
  • python37Packages.vtk_8
  • python38Packages.enamlx
  • python38Packages.pythonocc-core
  • python38Packages.vtk_8
  • python39Packages.enamlx
  • python39Packages.pythonocc-core
  • python39Packages.vtk_8
  • smesh
  • vtk (vtk_8)

@ppenguin
Copy link
Contributor Author

ppenguin commented Jan 10, 2021

Ok, I finally found some time to (hopefully) close this.... (thanks for your inputs all)

So I added the Qt5.15 patch to vtk8 and the gcc10 patch additionally to vtk7, all worked for me.

On a side note (remember, nix noob here), it took me a while to figure out to actually find how to have enableQt correctly evaluated. I did this by (in my nixpgs clone dir) executing:

nix-env -f . -iA vtkWithQt5

A few other issues for which I couldn't find clear answers:

  • after once successfully installing a package, I cannot force it to build from source anymore with nix-env and nix-build (even when removing and calling nix-collect-garbage)
  • is there a way to set a flag like enableQt true directly on the command line while invoking nix-env or nix-build? I tried with --arg but it apparently overrides everything and complains about all other needed envs not set.
  • I tried to read some about nix-shell, but didn't get it to work for experimental builds of vtk, and I didn't find a brief/focused enough doc/tutorial for my time budget (all docs have approaches involving different config files, and what is challenging in most docs is the fact that it is almost never clearly indicated to which files certain code-blocks belong). Any pointers?
  • I have seen that enableQt is set in all-packages, which is apparently included somewhere (indirectly) from default.nix
    • is this the preferred way to provide overrides?
    • how to specify a temporary override / temporarily include a .nix providing such an override along the normal default behaviour (default.nix) without explicitly including it in a top-level .nix file? (I would be making temporary changes to files even though I just want a one-time behaviour)

@tfmoraes
Copy link
Contributor

tfmoraes commented Jan 10, 2021

The only package not building is pcl because it's marked as broken.

❯ nix-shell -p nixpkgs-review --run "nixpkgs-review pr 108424"                                                                                              
$ git -c fetch.prune=false fetch --force https://github.com/NixOS/nixpkgs master:refs/nixpkgs-review/0 pull/108424/head:refs/nixpkgs-review/1
remote: Enumerating objects: 834, done.
remote: Counting objects: 100% (834/834), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 1324 (delta 823), reused 821 (delta 820), pack-reused 490
Receiving objects: 100% (1324/1324), 956.37 KiB | 1.48 MiB/s, done.
Resolving deltas: 100% (978/978), completed with 355 local objects.
From https://github.com/NixOS/nixpkgs
   4b0a2ac72ca..4d0a146f692  master                -> refs/nixpkgs-review/0
 + 4cd39c9c149...7305c0af96b refs/pull/108424/head -> refs/nixpkgs-review/1  (forced update)
$ git worktree add /home/thiago/.cache/nixpkgs-review/pr-108424-2/nixpkgs 4d0a146f69298aa2b943cd6cf4cd676a72b61c70
Preparing worktree (detached HEAD 4d0a146f692)
HEAD is now at 4d0a146f692 Merge pull request #108905 from marsam/update-flexget
$ git merge --no-commit 7305c0af96b44a458a682a755e8b1894e888284a
Automatic merge went well; stopped before committing as requested
$ nix --experimental-features nix-command build --no-link --keep-going --option build-use-sandbox relaxed -f /home/thiago/.cache/nixpkgs-review/pr-108424-2/build.nix

Link to currently reviewing PR:
https://github.com/NixOS/nixpkgs/pull/108424

1 package marked as broken and skipped:
pcl

53 packages built:
ants appcsxcad c3d cq-editor csxcad elastix ezminc freecad gdcm horizon-eda inkcut inkscape-extensions.inkcut itk itk4 libsForQt5.qcsxcad libsForQt512.qcsxcad libsForQt514.qcsxcad mirtk mrtrix opencascade openems python37Packages.cadquery python37Packages.dicom2nifti python37Packages.enamlx python37Packages.gdcm python37Packages.mayavi python37Packages.python-csxcad python37Packages.python-openems python37Packages.pythonocc-core python37Packages.vtk python37Packages.vtk_8 python38Packages.dicom2nifti python38Packages.enamlx python38Packages.gdcm python38Packages.mayavi python38Packages.python-csxcad python38Packages.python-openems python38Packages.pythonocc-core python38Packages.vtk python38Packages.vtk_8 python39Packages.dicom2nifti python39Packages.enamlx python39Packages.gdcm python39Packages.python-csxcad python39Packages.python-openems python39Packages.pythonocc-core python39Packages.vtk python39Packages.vtk_8 simpleitk smesh vtk vtkWithQt5 vtk_7

error: --- Error --------------------------------------------------------------------------------------------------------------------------------------- nix
build log of '/nix/store/fw2vh7jyr029c6n12fj2zn20y4zdqxsh-opencascade-oce-0.18.3.drv' is not available

@vojta001
Copy link
Contributor

Can we merge? This is currently blocking me from updating

@lheckemann
Copy link
Member

I'm currently rebasing/squashing and fixing the commit messages to push to master.

@ppenguin please refer to https://nixos.org/manual/nixpkgs/stable/#submitting-changes-making-patches for your commit messages in the future, that makes merging easier :)

@lheckemann
Copy link
Member

Regarding your questions:

On a side note (remember, nix noob here), it took me a while to figure out to actually find how to have enableQt correctly evaluated. I did this by (in my nixpgs clone dir) executing:

nix-env -f . -iA vtkWithQt5

A few other issues for which I couldn't find clear answers:

* after once successfully installing a package, I cannot force it to build from source anymore with `nix-env` and `nix-build` (even when removing and calling `nix-collect-garbage`)

This is probably because it's in an older generation of your profile, which is kept around so that you can roll back. You can delete these using e.g. nix-collect-garbage --delete-older-than 2d. That said, there isn't much point in rebuilding from source again, since the result will be the same — unless you've made changes to the expression, in which case the derivation will be different and nix will thus rebuild it :)

* is there a way to set a flag like `enableQt true` directly on the command line while invoking `nix-env`  or `nix-build`? I tried with `--arg` but it apparently overrides everything and complains about all other needed envs not set.

--arg as you used it here will pass arguments to the top-level nixpkgs function. The easiest way would probably be to build an expression, for example: nix-build -E '(import ./nixpkgs {}).vtk_7.override {enableQt = true;}.

nix-env behaves differently with -E and it's weird, so I'd suggest using nix-build followed by nix-env -i ./result, or writing your expression to a file and installing it with nix-env -f my-vtk.nix -i.

* I tried to read some about `nix-shell`, but didn't get it to work for experimental builds of `vtk`, and I didn't find a brief/focused enough doc/tutorial for my time budget (all docs have approaches involving different config files, and what is challenging in most docs is the fact that it is almost never clearly indicated to which files certain code-blocks belong). Any pointers?

If you want a shell containing vtk_8 from your nixpkgs, run nix-shell -p vtk_8 -I nixpkgs=path/to/your/nixpkgs. If you want a shell for building vtk manually, run nix-shell path/to/your/nixpkgs -A vtk_8.

* I have seen that `enableQt` is set in `all-packages`, which is apparently included somewhere (indirectly) from `default.nix`
  
  * is this the preferred way to provide overrides?
  * how to specify a temporary override / temporarily include a `.nix` providing such an override **along** the normal default behaviour (`default.nix`) **without** explicitly including it in a top-level `.nix` file? (I would be making temporary changes to files even though I just want a one-time behaviour)

See the nix-build command above, if you want this to apply globally you could use an overlay like self: super: { my_vtk = super.vtk_9.override { enableQt = true; }; }, then install my_vtk as you would otherwise install vtk_9 or similar.

@lheckemann
Copy link
Member

Also, IRC or discourse are probably better places to ask questions like this, lots of helpful people there and it's more likely to be seen than comments in a PR :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

vtk-8.2.0 fails to build on latest master
7 participants