Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

quartus: expose CLI executables + increase device support #83234

Merged
merged 2 commits into from Mar 25, 2020

Conversation

kwohlfahrt
Copy link
Contributor

@kwohlfahrt kwohlfahrt commented Mar 23, 2020

Motivation for this change

A user (I think @hpfr, apologies if I have the wrong person) requested support for Cyclone IV chipsets in quartus for remote lessons. This is addressed in f30f3bd

On the previous PR, @doronbehar requested exposing the CLI components of quartus rather than just the main GUI. That was a bit tricky in the context of buildFHSUserEnv but I have it working to my satisfaction now. This feature is added in e9075dd

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

Copy link
Contributor

@doronbehar doronbehar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kwohlfahrt I'm having trouble adding the downloaded Quartus files to my store:

$ nix-prefetch-url --type sha256 file:///home/doron/downloads/quartus/components/cyclone10lp-19.1.0.670.qdz
warning: dumping very large path (> 256 MiB); this may run out of memory
[265.7 MiB DL]
path is '/nix/store/03g98vrzc23nyicb105qj4ml2gvjv5hq-cyclone10lp-19.1.0.670.qdz'
1ccxq8n20y40y47zddkijcv41w3cddvydddr3m4844q31in3nxha
$ nix-prefetch-url --type sha256 file:///home/doron/downloads/quartus/QuartusLiteSetup-19.1.0.670-linux.run && nix-prefetch-url --type sha256 file:///home/doron/downloads/quartus/ModelSimSetup-19.1.0.670-linux.run
warning: dumping very large path (> 256 MiB); this may run out of memory
[2576.5 MiB DL]
path is '/nix/store/svq7aag9lc5mmbp0rr00qfqwk38hmf3k-QuartusLiteSetup-19.1.0.670-linux.run'
0wjsrcg3n3wn5i74xdjdcpxzjm92pril2r046ls2n8bjy9gxiy4s
warning: dumping very large path (> 256 MiB); this may run out of memory
[998.7 MiB DL]
path is '/nix/store/0bi68pph1sn7c1dgfg4d4dyvfglds1lv-ModelSimSetup-19.1.0.670-linux.run'
0j1vfr91jclv88nam2plx68arxmz4g50sqb840i60wqd5b0l3y6r

I also tried nix-store --add-fixed sha256 for the same files and it didn't work either.

pkgs/applications/editors/quartus-prime/default.nix Outdated Show resolved Hide resolved

qsysExecutables = (map (c: "quartus/sopc_builder/bin/qsys-${c}") [
"generate" "edit" "script"
]);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just out of curiosity, why not merge this list to quartusExecutables? It will ease on you in the shell script following this line.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No particular reason, I had two separate loops in some previous revisions so this is left over from that. That does remind me I wanted to add the ModelSim binary as well!


cd $out
chmod +x $EXECUTABLES
ln --symbolic --relative --target-directory ./bin $EXECUTABLES
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't tested this due to issues in adding the Quartus files I've downloaded to my store. I'm not sure I've managed to figure out what happens here eventually. Could you add some comments please?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. This just links all the executables into $out/bin, so if you do nix run quartus-prime-lite you get them added to your $PATH.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a comment to this last bit, is that what was missing or does anything else need commenting as well?

@kwohlfahrt
Copy link
Contributor Author

kwohlfahrt commented Mar 24, 2020

Most of the files seem to be OK - of the ones you posted, only QuartusLiteSetup doesn't have the correct hash. Does the original file have an md5 hash of 068DECCE4E35984AB8B49410FD47405B ? This is what mine has, and what is listed on Intel's download page (for me at least).

Are you getting errors for the other files as well?

@doronbehar
Copy link
Contributor

doronbehar commented Mar 24, 2020

only QuartusLiteSetup doesn't have the correct hash. Does the original file have an md5 hash of 068DECCE4E35984AB8B49410FD47405B ? This is what mine has, and what is listed on Intel's download page (for me at least).

You were right, my Quartus .run file had a wrong md5 hash as well. Do you think it might be possible to put this as an extra advice (check the md5 hash to correspond to the one on the website)? I mean, change this error message:

***
Unfortunately, we cannot download file QuartusLiteSetup-19.1.0.670-linux.run automatically.
Please go to https://fpgasoftware.intel.com/19.1/?edition=lite&platform=linux to download it yourself, and add it to the Nix store
using either
  nix-store --add-fixed sha256 QuartusLiteSetup-19.1.0.670-linux.run
or
  nix-prefetch-url --type sha256 file:///path/to/QuartusLiteSetup-19.1.0.670-linux.run

***

To, perhaps:

***
Unfortunately, we cannot download file QuartusLiteSetup-19.1.0.670-linux.run automatically.
Please go to https://fpgasoftware.intel.com/19.1/?edition=lite&platform=linux to download it yourself, and add it to the Nix store
using either
  nix-store --add-fixed sha256 QuartusLiteSetup-19.1.0.670-linux.run
or
  nix-prefetch-url --type sha256 file:///path/to/QuartusLiteSetup-19.1.0.670-linux.run

If this doesn't work please check that you download the files from the "Individual Files" tab.
You can also verify that the upstream md5 hashes correspond the files you downloaded.
Use the command `md5sum` to check this.
***

@kwohlfahrt
Copy link
Contributor Author

I'm a bit wary of custom messages here (last time we had issues with nix-store --add-fixed vs nix-prefetch-url). Most packages have no custom message. Of those that do, eight use nix-prefetch-url but Mathematica uses nix-store --add-fixed.

Unfortunately, I don't think it is possible to just add extra things to the end of the standard message, you have to replace it entirely, and copy/pasting the bulk of the message from trivial-builders.nix strikes me as a bit of a smell.

@doronbehar
Copy link
Contributor

doronbehar commented Mar 24, 2020

Unfortunately, I don't think it is possible to just add extra things to the end of the standard message, you have to replace it entirely, and copy/pasting the bulk of the message from trivial-builders.nix strikes me as a bit of a smell.

I see. Perhaps then it'd be nice to document that in the packages' notes, as in here: https://nixos.org/nixpkgs/manual/#sec-weechat and here: https://github.com/NixOS/nixpkgs/pull/77347/files#diff-f03a8f501d730eba83102a6be17115cb .. But don't feel I'm pressing you on this @kwohlfahrt - Nixpkgs has never been the best distro in terms of documentation :).

As for the rest of the improvements here @kwohlfahrt, I have some comments but overall, it seems you've done spectacular job 👏. I have 1 question though before I'll give a full review:

Where does the file $out/bin/quartus-prime-lite which has this content:

#! /nix/store/8gy2bmpz3qawxr3g8hqhfgkf32wb0wfl-bash-4.4-p23/bin/bash
exec /nix/store/aj1k6hxh2pkcp785kaa088a5sf2j4i42-chrootenv/bin/chrootenv /nix/store/d8444db34caddd5vw93z4rm306bkrlvy-quartus-prime-lite-init "$(pwd)" "$@"

Comes from? I think got it - it's what comes out of whatever's in runScript.

On a different topic, as for the modelsim executables, there are many:

drill           mc2_util        qhmake          sccom           vcom            verror          vmap            wlf2vcd
dumplog64       qhcvt           qhmap           scgenmod        vcover          vgencomp        vopt            wlfman
hm_entity       qhdel           qhsim           sdfcom          vdbg            vhencrypt       vovl            wlfrecover
jobspy          qhdir           qverilog        sm_entity       vdel            vlib            vrun            xml2ucdb
mc2com          qhgencomp       qvhcom          triage          vdir            vlog            vsim
mc2perfanalyze  qhlib           qvlcom          vcd2wlf         vencrypt        vmake           wlf2log

There are so many of them... Anyway, I've managed to find only this reference documenting them: https://www.intel.com/content/www/us/en/programmable/documentation/jeb1529967983176.html#mwh1410471006061 .

@kwohlfahrt
Copy link
Contributor Author

That file comes from buildFHSUserEnv. It creates $out/bin/${name} with the content

#! /nix/store/8gy2bmpz3qawxr3g8hqhfgkf32wb0wfl-bash-4.4-p23/bin/bash
exec /nix/store/aj1k6hxh2pkcp785kaa088a5sf2j4i42-chrootenv/bin/chrootenv ${runScript} "$(pwd)" "$@"

unsupportedDeviceIds = lib.filterAttrs (name: value:
!(lib.hasAttr name supportedDeviceIds)
) deviceIds;

quartus = stdenv.mkDerivation rec {
version = "19.1.0.670";
pname = "quartus-prime-lite";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it'll be clearer for someone from outside if:

Suggested change
pname = "quartus-prime-lite";
pname = "quartus-prime-lite-unwrapped";

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But then, the desktopItem's name attribute should be still quartus-prime-lite.

Comment on lines 71 to 72
"quartus_help"
"quartus_update"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curios: Why?

@@ -110,10 +138,41 @@ in buildFHSUserEnv {
xorg.libXrender
];

extraInstallCommands = ''
passthru = {
inherit quartus;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually this is what's done:

Suggested change
inherit quartus;
unwrapped = quartus;

See Neovim's derivation for reference.

@doronbehar
Copy link
Contributor

doronbehar commented Mar 24, 2020

Besides the small nitpicks above, I've had a few more ideas that should simplify the wrapping IMO. They are too broad to write in a review so allow me @kwohlfahrt to suggest a full patch here that includes some changes already suggested in the review. Feel free to apply / commit only some or none, it's just my diff but it is tested.

diff --git c/pkgs/applications/editors/quartus-prime/default.nix w/pkgs/applications/editors/quartus-prime/default.nix
index a222b63287d..ab7d0356736 100644
--- c/pkgs/applications/editors/quartus-prime/default.nix
+++ w/pkgs/applications/editors/quartus-prime/default.nix
@@ -25,7 +25,7 @@ let
 
   quartus = stdenv.mkDerivation rec {
     version = "19.1.0.670";
-    pname = "quartus-prime-lite";
+    pname = "quartus-prime-lite-unwrapped";
 
     src = let
       require = {name, sha256}: requireFile {
@@ -68,6 +68,7 @@ let
       copyComponent = component: "cp ${component} $TEMP/${component.name}";
       # leaves enabled: quartus, modelsim_ase, devinfo
       disabledComponents = [
+        # WHY?
         "quartus_help"
         "quartus_update"
         # not modelsim_ase
@@ -97,17 +98,17 @@ let
   };
 
   desktopItem = makeDesktopItem {
-    name = quartus.name;
+    name = "quartus-prime-lite";
     exec = "quartus";
     icon = "quartus";
     desktopName = "Quartus";
-    genericName = "Quartus FPGA IDE";
+    genericName = "Quartus Prime";
     categories = "Development;";
   };
 
 # I think modelsim_ase/linux/vlm checksums itself, so use FHSUserEnv instead of `patchelf`
 in buildFHSUserEnv rec {
-  name = "quartus-prime-lite";
+  name = "quartus-prime-lite"; # wrapped
 
   targetPkgs = pkgs: with pkgs; [
     # quartus requirements
@@ -139,40 +140,58 @@ in buildFHSUserEnv rec {
   ];
 
   passthru = {
-    inherit quartus;
+    unwrapped = quartus;
   };
 
   extraInstallCommands = let
-    quartusExecutables = (map (c: "quartus/bin/quartus_${c}") [
+    quartusExecutables = (map (c: "quartus_${c}") [
       "asm" "cdb" "cpf" "drc" "eda" "fit" "jbcc" "jli" "map" "pgm" "pow"
       "sh" "si" "sim" "sta" "stp" "tan"
-    ]) ++ [ "quartus/bin/quartus" ];
+    ]) ++ [ "quartus" ];
 
-    qsysExecutables = (map (c: "quartus/sopc_builder/bin/qsys-${c}") [
+    qsysExecutables = (map (c: "qsys-${c}") [
       "generate" "edit" "script"
     ]);
+    # Should we install all executables ?
+    modelsimExecutables = (map (c: c) [
+      "vsim" "vlog" "vlib"
+    ]);
   in ''
     mkdir -p $out/share/applications
-    cp ${desktopItem}/share/applications/* $out/share/applications
-
-    mkdir -p $out/quartus/bin
-    mkdir -p $out/quartus/sopc_builder/bin
+    ln -s ${desktopItem}/share/applications/* $out/share/applications
+    # handle icon - the size chosen here is roughly the size of the real
+    mkdir -p $out/share/icons/128x128
+    ln -s ${quartus}/licenses/images/dc_quartus_panel_logo.png $out/share/icons/128x128/quartus.png
 
+    mkdir -p $out/bin
+    # What's gonna come out of the runScript attribute
     WRAPPER=$out/bin/${name}
-    EXECUTABLES="${lib.concatStringsSep " " (quartusExecutables ++ qsysExecutables)}"
+    echo using wrapper script $WRAPPER
 
-    for executable in $EXECUTABLES; do
-        echo "#!${stdenv.shell}" >> $out/$executable
-        echo "$WRAPPER ${quartus}/$executable \$@" >> $out/$executable
-    done
+    # Every group of executables is located in a different path inside $quartus
 
-    cd $out
-    chmod +x $EXECUTABLES
-    # link into $out/bin so executables become available on $PATH
-    ln --symbolic --relative --target-directory ./bin $EXECUTABLES
+    for executable in ${lib.concatStringsSep " " (quartusExecutables)}; do
+        echo wrapping ${quartus}/quartus/bin/$executable
+        echo "#!${stdenv.shell}" >> $out/bin/$executable
+        echo "$WRAPPER ${quartus}/quartus/bin/$executable \$@" >> $out/bin/$executable
+        chmod +x $out/bin/$executable
+    done
+    for executable in ${lib.concatStringsSep " " (qsysExecutables)}; do
+        echo wrapping ${quartus}/quartus/sopc_builder/bin/$executable
+        echo "#!${stdenv.shell}" >> $out/bin/$executable
+        echo "$WRAPPER ${quartus}/quartus/sopc_builder/bin/$executable \$@" >> $out/bin/$executable
+        chmod +x $out/bin/$executable
+    done
+    for executable in ${lib.concatStringsSep " " (modelsimExecutables)}; do
+        echo wrapping ${quartus}/modelsim_ase/bin/$executable
+        echo "#!${stdenv.shell}" >> $out/bin/$executable
+        echo "$WRAPPER ${quartus}/modelsim_ase/bin/$executable \$@" >> $out/bin/$executable
+        chmod +x $out/bin/$executable
+    done
   '';
 
+  # This is the real wrapper
   runScript = writeScript "${name}-wrapper" ''
-    exec $@
+    exec -a "$1" $@
   '';
 }

BTW for reference, as for my original idea regarding implementing this idea of wrapping via symlinking $out/bin/${name} to the executables named as in ${quartus}/quartus/bin/ (#75561 (comment)), I've managed to realize it's not possible yet due to how buildFHSUserEnv calls what's in runScript. See #83296 .

@kwohlfahrt
Copy link
Contributor Author

quartus_help was disabled because it requires another download, so I felt it should be optional and I hadn't gone through the trouble of setting it up, no more reason than that.

The logo is a nice find, I'll include that. The ModelSim executables look reasonable, I will add them. However, I don't like having multiple loops with redundant elements. In this case, there is a bug here from copy/paste, specifically:

+    for executable in ${lib.concatStringsSep " " (modelsimExecutables)}; do
+        echo wrapping ${quartus}/modelsim_ase/bin/$executable
+        echo "#!${stdenv.shell}" >> $out/bin/$executable
+        echo "$WRAPPER ${quartus}/quartus/sopc_builder/bin/$executable \$@" >> $out/bin/$executable
+        chmod +x $out/bin/$executable
+    done

should be ${quartus}/modelsim_ase instead of ${quartus}/quartus/sopc_builder.

in my original version I had two copies of each executable, with the following reasoning:

  • the original paths ($out/quartus/bin/*, $out/quartus/sopc_builder/bin/*, $out/modelsim_ase/bin/*) in case people had scripted against a standard quartus install dir
  • links in $out/bin/* for a better user experience with nix run.

so I think it's useful to keep both.

Is the -a $1 in exec $@ necessary? Since the first item of $@ is passed as the command name anyway? In the change you proposed to buildFHSUserEnv it makes sense (since it's -a $0) but for -a $1 I think it's redundant.

buildFHSUserEnv does not currently support multiple binaries, so doing
this manually with wrappers.

Pass through original quartus derivation for debugging/overriding
@kwohlfahrt
Copy link
Contributor Author

I've pushed some of the changes, mostly adding the ModelSim executables. I think I explained above why I didn't merge any of the others, but let me know what you think.

ln -s ${desktopItem}/share/applications/* $out/share/applications
ln -s ${quartus}/licenses/images/dc_quartus_panel_logo.png $out/share/icons/128x128/quartus.png

mkdir -p $out/quartus/bin $out/quartus/sopc_builder/bin $out/modelsim_ase/bin
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @kwohlfahrt for accepting some of my changes. Personally, what I don't like most in the current way the executables are "installed" is that you create these directories. I can imagine myself a better way to integrate some the executables' names in a nix data type which will integrate nicely to the shell script, but I'm not experienced enough in nix to do it myself. Do you think you could write a shell script that would use something like this:

let
  executables = {
    quartus = {
      sharedPath = "quartus/bin";
      commands = (map (c: "quartus_${c}") [
        "asm" "cdb" "cpf" "drc" "eda" "fit" "jbcc" "jli" "map" "pgm" "pow"
        "sh" "si" "sim" "sta" "stp" "tan"
      ]) ++ [ "quartus/bin/quartus" ];
    };
    qsys = {
      sharedPath = "quartus/sopc_builder/bin";
	  commands = map (c: "qsys-${c}") [
	    "generate" "edit" "script"
      ];
	};
	modelsim = {
      sharedPath = "modelsim_ase/bin";
      commands = map (c: "${c}") [
        "vsim" "vlog" "vlib"
      ];
    };
  };
in

Perhaps?

This will also remove the need to link the directory at the end as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, that could definitely use some clean-up - I'll give that a go tomorrow. It was a deliberate choice to have both them and the links in $out/bin though - here is my reasoning:

Normally, quartus binaries get installed as $QUARTUS_INSTALL_DIR/quartus/bin/... (ans similarly for modelsim_ase and quartus/sopc_builder). So, some users might expect them to be in these relative directories instead of all in $out/bin. The way it is done now means that users can still use the same scripts for a normal quartus install and a nix build (just pointing $QUARTUS_INSTALL_DIR to the nix result). If we only keep the ones in $out/bin they will have to change their scripts to either look for these binaries in different places.

On the other hand, if we don't link to them from $out/bin, users installing quartus with home-manager or nix-env will not find the binaries on their $PATH as expected.

Does that seem a compelling enough use-case to keep both?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Normally, quartus binaries get installed as $QUARTUS_INSTALL_DIR/quartus/bin/... (ans similarly for modelsim_ase and quartus/sopc_builder) ...
...
Does that seem a compelling enough use-case to keep both?

I guess so. Just for the record, it's not possible to run these executables without the wrapper right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They still need to go through runScript, because that sets up the environment with /lib & friends mounted in the expected place if that's what you meant?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

@doronbehar
Copy link
Contributor

should be ${quartus}/modelsim_ase instead of ${quartus}/quartus/sopc_builder.

That is correct 😬 I've edited the comment almost right after I've sent it hoping you won't notice.

Is the -a $1 in exec $@ necessary? Since the first item of $@ is passed as the command name anyway? In the change you proposed to buildFHSUserEnv it makes sense (since it's -a $0) but for -a $1 I think it's redundant.

You were right, that was a mistake.

@Mic92 Mic92 merged commit fb00dea into NixOS:master Mar 25, 2020
@kwohlfahrt kwohlfahrt deleted the quartus branch June 5, 2020 10:28
@hpfr
Copy link

hpfr commented Jan 21, 2021

An extremely belated thanks to you two for your work on this! I was indeed the user you mentioned and it did work very well for my purposes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants