Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kicad: fix #49089 by adding libraries files #57452

Merged
merged 4 commits into from Apr 30, 2019
Merged

Conversation

ThibautMarty
Copy link
Member

@ThibautMarty ThibautMarty commented Mar 11, 2019

This splits the KiCad package in several derivations:

  • original package
  • internationalization package
  • templates
  • schematic symbols libraries
  • PCB footprints libraries
  • 3D models libraries

From these derivations, 3 packages are proposed in top level:

  • kicad (original package + internationalization)
  • kicadWithLibraries (kicad + all libraries except 3D models) because 3D
    models can be heavy (currently 5.1GiB) and potentially unused
  • kicadWithLibraries3D (kicad + all libraries)
Motivation for this change

KiCad's i18n files and libraries where missing. We needed to download the libraires manually and configure paths in the software. This PR adds them in the package output.

Things done

The different top level packages are combined with symlinkJoin, which joins two outputs by creating symbolic links.
The advantage is that only one compilation of the main package is needed, as other packages are symlinked to the main one.
The drawback is that, by default, kicad looks for libraries in the /share directory of its binary.
Thus I added a wrapper that sets KICAD_SYMBOL_DIR and other environment variables to correctly point the libraries. This prevent user to use its own library (but this is still possible by using)

The internationalization works. The menu used to exist, but did not have effect.

I would like to have comments on:

  • Is symlinkJoin the best approach?
  • Is wrapping other binaries than kicad is needed? (e.g if we launch directly eeschema, environment variables will not be set)
  • Where should we put the meta attribute?
  • Is the top-level.nix approach correct?

CC @berce (maintener) and @crawford (opened #49089)

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@ThibautMarty
Copy link
Member Author

Other possible implementations include:

  • Give up about reusing main package. After all, users are expected to install only one of those, otherwise nix-store --optimise can do its jobs. This only matters for build time and distribution.
  • Use multi-output package, and patch the main CMakeLists.txt to add some "modules" between each install to each output
  • Manage to patch KiCad such that it can search files in another directory (without preventing the user to override these paths)

Also note that license for libraries is CC-BY-SA-4.0 and must be stated. The i18n does not seem to have any license.

I also suggest using the package with libraries by default, as it should be what most user expect.

@ThibautMarty
Copy link
Member Author

ThibautMarty commented Mar 13, 2019

OK, I finally used the callPackage's argument to enable libraries or 3D files, and used a similar mechanism to what symlinkJoin does but only for getting the libraries files in the derivation's output (with symlinks). The main KiCad package is not symlinked anymore, which fixes the issue /share-look up issue.

The 3 top-levels packages are now:

  • kicad
  • kicad-without-libs
  • kicad-with-3d

Note: interested people can diff between master and dd60fbb, as this commit mostly reverts changed I have done earlier.

I added the libraries license.
I tested the all binaries in /bin. They work except for the *.kiface files, which are actually shared object (they should not end here…)

Copy link
Member

@veprbl veprbl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes a lot of sense, especially after last iteration that simplifies things. I think we don't need a separate versions without libs or without 3d, those should be always provided.

pkgs/applications/science/electronics/kicad/default.nix Outdated Show resolved Hide resolved
@ThibautMarty
Copy link
Member Author

I don't fully agree for removing version without 3D. I agree that the version without libraries is now useless because the user can override the libraries paths, while it was not possible (I think) in my first draft because of the wrapper. Furthermore, they are mandatory for any basic usage, except for advanced users who could use custom/development libraries.
On the contrary, the 3D packages can be seen, IMHO, as a add-on that is not necessary. I don't feel comfortable adding 5GiB to the closure size for an add-on… For instance, the Archlinux packages split the libraries in two parts: 3d packages and the others.

@ThibautMarty
Copy link
Member Author

For the record, I tried enabling the tests and they fail.
I'm trying to update to 5.1.0 but this fails for now.

@veprbl
Copy link
Member

veprbl commented Mar 14, 2019

@ThibautMarty Wow. I did not realize there were so many 3d models available. Then, of course, this should be optional. Perhaps it would also make sense to set preferLocalBuild = true; so that Hydra doesn't put these huge files into the binary cache.

But you do agree on removing the version without libraries? Even if you work with with only custom component schematics you still likely may need the standard ones once in a while.

@ThibautMarty
Copy link
Member Author

ThibautMarty commented Mar 14, 2019

Agreed.

I'm not sure if preferLocalBuild is made for this purpose though. Not according to the doc.
I think there is an other mechanism to prevent Hydra building a package.

But the easiest solution may be to remove kicad-with-3d from all-packages.nix and let user build it with an overlay or an override with the enable3d option ? (That's not obvious for users though)
By the way enable3d may not be the best name for that option (KiCad does use 3D acceleration).

@ThibautMarty
Copy link
Member Author

ThibautMarty commented Mar 14, 2019

Maybe something like clearing hydraPlatforms.

@ThibautMarty
Copy link
Member Author

OK, I think we're done. I can squash the commits and rebase if people are OK to merge.
I'll then try to fix the 5.1.0 compilation and open another PR.

@ThibautMarty
Copy link
Member Author

Hum, on second thought, it may be good to have a package with the 3D packages.
It's useless to recompile KiCad just to add data libraries, but "necessary" with Nix for the binary to find them. Installing 3D packages is basically extracting the tarball. "Advanced" users can point KiCad (in preferences) to this package if they want it.

@ThibautMarty
Copy link
Member Author

Hum, on second thought, it may be good to have a package with the 3D packages.
It's useless to recompile KiCad just to add data libraries, but "necessary" with Nix for the binary to find them. Installing 3D packages is basically extracting the tarball. "Advanced" users can point KiCad (in preferences) to this package if they want it.

Oh wait, kicad.packages3d already does the job!

This splits the KiCad package in several derivations:
- original package (main KiCad package)
- internationalization package
- templates
- schematic symbols libraries
- PCB footprints libraries
- 3D models libraries

From these derivations, 2 packages are exposed in top level:
- `kicad` (main KiCad package + all libraries except 3D models)
- `kicad-with-3dpackages` (kicad + all libraries)

The 3D models can also be installed separately with `kicad.packages3d`.
This prevents a new compilation of KiCad, but the user must set the
`KISYS3DMOD` environment variable or option accordingly.
@ThibautMarty ThibautMarty changed the title [WIP] kicad: fix #49089 by adding libraries files kicad: fix #49089 by adding libraries files Mar 14, 2019
This moves module definition into passthru. Otherwise building kicad
would still require all modules to be built first even if they are not
used.

Also this drops preferLocalBuild from packages3d. hydraPlatforms should
do what we need to. preferLocalBuild would addtitionaly disable remote
builders, which is probably not what we want.
@veprbl
Copy link
Member

veprbl commented Mar 24, 2019

@ThibautMarty There was a small problem with your latest version of this PR: because packages3d was an attribute given to kicad derivation unconditionally, that package would need to be instantiated before kicad could be built. I had to move those things to passthru so that they are not required unless used.

I also did bunch of other changes on my way. Let me know if you think they are usable. I did some testing on this and was able to place some symbols and footprints just fine. I was not able to render any 3d components even though files are in place.

@ThibautMarty
Copy link
Member Author

I agree.

For the 3D rendering problem, there is another issue: #49090. I tried with the proposed workaround and it worked fine (before your changes, but they should not break things).

We may consider including the documentation, as requested by developers. This adds ~300MB to the package.

(A bit off topic) I tried to upgrade to 5.1.0, but got stuck with wxWidget/wxPython issues:

  • wxPython.h is not found anymore (because the CMake's detection scripts that changed)
  • If I put CMake's option -DKICAD_SCRIPTING_WXPYTHON_PHOENIX=ON, it builds (because it does not need and does not look up for wxPython.h). I don't know if this is ready or not, as it is disabled by default.

I did not have time to investigate further since I tried that.

Here is the current version:

{ wxGTK, lib, stdenv, fetchurl, fetchFromGitHub, cmake, libGLU_combined, zlib
, libX11, gettext, glew, glm, cairo, curl, openssl, boost, pkgconfig
, doxygen, pcre, libpthreadstubs, libXdmcp
, wrapGAppsHook
, oceSupport ? true, opencascade
, ngspiceSupport ? true, libngspice
, swig, python, pythonPackages
, lndir
}:

assert ngspiceSupport -> libngspice != null;

with lib;
let
  mkLib = version: name: sha256: attrs: stdenv.mkDerivation ({
    name = "kicad-${name}-${version}";
    src = fetchFromGitHub {
      owner = "KiCad";
      repo = "kicad-${name}";
      rev = "${version}";
      inherit sha256 name;
    };
    nativeBuildInputs = [
      cmake
    ];
  } // attrs);

in stdenv.mkDerivation rec {
  name = "kicad-${version}";
  series = "5.0";
  version = "5.1.0";

  src = fetchurl {
    url = "https://launchpad.net/kicad/${series}/${version}/+download/kicad-${version}.tar.xz";
    sha256 = "1g8cka71wy3mzwf4qgr9dqpskpxmwz0qpdxjkr6i8brpwy7037ks";
  };

  postPatch = ''
    substituteInPlace CMakeModules/KiCadVersion.cmake \
      --replace no-vcs-found ${version}
  '';

  cmakeFlags = [
    "-DKICAD_SCRIPTING=ON"
    "-DKICAD_SCRIPTING_MODULES=ON"
    "-DKICAD_SCRIPTING_WXPYTHON_PHOENIX=ON"
  ] ++ optionals (oceSupport) [ "-DKICAD_USE_OCE=ON" "-DOCE_DIR=${opencascade}" ]
    ++ optional (ngspiceSupport) "-DKICAD_SPICE=ON";

  nativeBuildInputs = [
    cmake
    doxygen
    pkgconfig
    wrapGAppsHook
    pythonPackages.wrapPython
    lndir
  ];
  pythonPath = [ pythonPackages.wxPython ];
  propagatedBuildInputs = [ pythonPackages.wxPython ];

  buildInputs = [
    libGLU_combined zlib libX11 wxGTK pcre libXdmcp glew glm libpthreadstubs
    cairo curl openssl boost
    swig python
  ] ++ optional (oceSupport) opencascade
    ++ optional (ngspiceSupport) libngspice;

  # this breaks other applications in kicad
  dontWrapGApps = true;

  passthru = {
    i18n = mkLib version "i18n" "1pyklms6qb03ppxh6c5wpfj8ncdxa2410pn2xs6ia723wk9x48by" {
      buildInputs = [
        gettext
      ];
      meta.license = licenses.gpl2; # https://github.com/KiCad/kicad-i18n/issues/3
    };
    symbols = mkLib version "symbols" "0czq7aaq5xdffrgfp7vc4mjbdihgzi2c1qc8d7a9q4cdb6qxh01y" {
      meta.license = licenses.cc-by-sa-40;
    };
    footprints = mkLib version "footprints" "1xdrw0y7ngkjlz17sxfvvxyjbjk4akh38xppi5px64jjw0ca9wvn" {
      meta.license = licenses.cc-by-sa-40;
    };
    templates = mkLib version "templates" "1nva4ckq0l2lrah0l05355cawlwd7qfxcagcv32m8hcrn781455q" {
      meta.license = licenses.cc-by-sa-40;
    };
    packages3d = mkLib version "packages3d" "0ni8bjkvab7fqr8q3ypc402yz1j630szs0rzzzbz44lz14mkykzw" {
      hydraPlatforms = []; # this is a ~1 GiB download, occupies ~5 GiB in store
      meta.license = licenses.cc-by-sa-40;
    };
  };

  modules = with passthru; [ i18n symbols footprints templates ];

  postInstall = ''
    mkdir -p $out/share
    for module in $modules; do
      lndir $module/share $out/share
    done
  '';

  preFixup = ''
    buildPythonPath "$out $pythonPath"
    gappsWrapperArgs+=(--set PYTHONPATH "$program_PYTHONPATH")

    wrapProgram "$out/bin/kicad" "''${gappsWrapperArgs[@]}"
  '';

  meta = {
    description = "Free Software EDA Suite";
    homepage = http://www.kicad-pcb.org/;
    license = licenses.gpl2;
    maintainers = with maintainers; [ berce ];
    platforms = with platforms; linux;
  };
}

@veprbl
Copy link
Member

veprbl commented Mar 27, 2019

@ThibautMarty I'm not an active use of kicad, so you'll need to help me here. Do you think this PR as is resolves #49090? Is this ready to be merged?

@gebner
Copy link
Member

gebner commented Apr 29, 2019

I just successfully used this for a PCB. It's a bit of a shame that we don't have 3D models without recompiling kicad, but at least we've got symbols and footprints now out of the box..

@ThibautMarty I'd like to merge this soon. Is there anything broken that I've missed? We can do the 5.1.2 upgrade and documentation afterwards as well.

Copy link
Member

@veprbl veprbl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any outstanding issues with this. Should be good to go as long as somebody thinks this is useful.

@gebner gebner merged commit a35df16 into NixOS:master Apr 30, 2019
@ThibautMarty ThibautMarty deleted the fix-kicad branch November 24, 2020 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants