Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dotnet-sdk: Allow multiple packages #73262

Merged
merged 7 commits into from Jan 17, 2020

Conversation

baracoder
Copy link
Contributor

@baracoder baracoder commented Nov 12, 2019

Motivation for this change

Colleagues have switched some projects to version 3.0 of .NET Core. I want my development environment to work with all versions.
This changes allow to combine multiple SDKs and runtimes.

in configuration.nix

  dotnetCombined = with dotnetCorePackages; combinePackages {
    cli=sdk_3_1_preview;
    withSdks = [ sdk_2_2 sdk_3_0 sdk_3_1_preview ];
    withRuntimes = [ aspnetcore_2_1 sdk_3_0 sdk_2_2 sdk_3_1_preview  ];
  };

This setup worked so far. I am open for suggestions.

Using the runtime packages, it should also be possible to build smaller docker images for deployment.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @kuznero

Related issues

#73193
#72300
#69392
#69382
#69391

@baracoder baracoder changed the title dotnet core: Allow multiple packages dotnet-sdk: Allow multiple packages Nov 12, 2019
@ofborg ofborg bot requested a review from kuznero November 12, 2019 00:31
@jonringer
Copy link
Contributor

I do like the idea of being able to compose many sdk's. I will have to take a closer look when I get back from work

@ghost
Copy link

ghost commented Dec 4, 2019

I can confirm this is something a Dot Net developer could need : I'm currently working on some legacy projects (2.2) and some new projects that are working with 3.0 only features (Like some Generic Hosts extensions).

I'll will give a try to this PR this week.

@reset
Copy link

reset commented Dec 11, 2019

I've been using this work in a project of mine with no issues on Ubuntu, Windows WSL2, and a modified version of it for macOS. Excellent work and IMO exactly what I think the dotnet package should look like. @baracoder thank you!

Copy link
Contributor

@jonringer jonringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on the whole, this looks great, I'll try to compile a few projects, and if it works, then I'll merge it.

Sorry for shelving this so long, was focusing on python for the longest time

pkgs/development/compilers/dotnet/default.nix Outdated Show resolved Hide resolved
pkgs/development/compilers/dotnet/combinePackages.nix Outdated Show resolved Hide resolved
@felschr
Copy link
Member

felschr commented Dec 12, 2019

I've only tried it with v3.1 (stable) of the SDK yet but that has been working without any issues for quite a few project.

Since 3.1 is stable by now it should also be added to the list.
To save you some time this is what I've used:

sdk_3_1 = buildNetCoreSdk {
  version = "3.1.100";
  sha512 = "0hvshwsgbm6v5hc1plzdzx8bwsdna2167fnfhxpysqs5mz7crsa4f13m4cxhrbn64lasqz2007nhdrlpgaqvgll6q8736h884aaw5sj";
};

@baracoder
Copy link
Contributor Author

Thank you all for the suggestions. I have updated the branch with the changes and replaced sdk_3_1_preview with sdk_3_1.
When using Rider, I had to run dotnet clean and invalidate cache to catch up.

@evenbrenden
Copy link
Contributor

Great! Maybe update the default version in pkgs/top-level/all-packages.nix too?

-  dotnet-sdk = dotnetCorePackages.sdk_2_2;
+  dotnet-sdk = dotnetCorePackages.sdk_3_1;

@baracoder
Copy link
Contributor Author

I am not sure about this change. This could break some of the packages, which depend on the current version of dotnet-sdk:

% grep -irnl dotnet-sdk
..
pkgs/applications/blockchains/wasabiwallet/default.nix
pkgs/development/python-modules/dotnetcore2/default.nix
pkgs/development/tools/build-managers/msbuild/default.nix
pkgs/servers/nosql/eventstore/default.nix
pkgs/servers/jellyfin/default.nix
pkgs/top-level/python-packages.nix

@evenbrenden
Copy link
Contributor

I am not sure about this change. This could break some of the packages, which depend on the current version of dotnet-sdk:

% grep -irnl dotnet-sdk
..
pkgs/applications/blockchains/wasabiwallet/default.nix
pkgs/development/python-modules/dotnetcore2/default.nix
pkgs/development/tools/build-managers/msbuild/default.nix
pkgs/servers/nosql/eventstore/default.nix
pkgs/servers/jellyfin/default.nix
pkgs/top-level/python-packages.nix

Ah, right. Never mind then - I suppose updating (which includes checking each of these and possibly fixing them to 2.2) warrants a separate issue in any case.

@jonringer
Copy link
Contributor

jonringer commented Dec 13, 2019

@worldofpeace do you see anything out of the ordinary? it LGTM

the reason why such a construction is necessary is:

dotnet makes the assumption that it's able to download missing sdk's/runtimes to a directory relative to the executable, which ends up being in the nix store, but if the sdk/runtime is already available, then it skips this step. By allowing the sdks/runtimes to be composed through the dotnetPackages.combinePackages function, we can make sure the packages are available, even from mismatching versions (e.g. dotnet_3_1 sdk needs to build a package with dotnet-sdk_2_2).

@jonringer
Copy link
Contributor

@baracoder I still haven't forgot this PR, I just keep getting sidetracked with other things :(

Copy link
Contributor

@jonringer jonringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the only thing I noticed is that if i have 2 sdk's, and no runtimes, then this fails to build. It would also be nice to prefer runtimes from sdks if present, as that will reduce closure size

@jonringer
Copy link
Contributor

the problem with linking to sdk's as "runtimes" is that you don't really get any closure benefit, you bring in the whole other closure once you do that.

@baracoder
Copy link
Contributor Author

Thank you for testing and the input.

with dotnetCorePackages; combinePakcages { cli=sdk_3_1; withRuntimes=[ sdk_3_1 ]; withSdks=[ sdk_3_1 ]; }

Using the same sdk package in runtimes won't increase the closure size, as the directories are just linked.

I had the wrong assumption that explicitly specifying the runtime would decrease closure size for built packages, but this is false, because the cli from the combined package would be used and pull in all SDKs.
combinePackages is only be useful for development environments.

I will update the PR tomorrow.

@jonringer
Copy link
Contributor

also, I don't think the cli argument is necessary, could order the sdks and runtimes first by sdk, then by version, then just choose the first one.

@baracoder
Copy link
Contributor Author

I did not have time to update the pull request, but stumbled today on a nice nixpkgs function buildEnv. Which makes it look like I have tried to reinvent the wheel:

  latestCombined = buildEnv {
    name = "dotnet-combined-latest";
    paths = [ sdk_3_1 sdk_2_2 sdk_3_0 sdk_3_1 ];
    pathsToLink = [ "/host" "/packs" "/sdk" "/shared" "/templates" ];
    ignoreCollisions = true;
    postBuild = ''
      cp ${sdk_3_1}/dotnet $out/dotnet
      mkdir $out/bin
      ln -s $out/dotnet $out/bin/
    '';
  };

It works a little bit different but simplifies the process a lot.

Copy link
Contributor

@worldofpeace worldofpeace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created files are missing final newlines.

@snewell92
Copy link

This change pleases me greatly.

Little girl with blue cotton candy being so excited at a sporting event.

@baracoder
Copy link
Contributor Author

Sorry, for the delay, I forgot to ping.

I have updated the merge request. You can now get a combined package like this:

with dotnetCorePackages; combinePackages { packages = [ sdk_3_1 sdk_2_2 sdk_3_0 sdk aspnetcore_2_1 ]; }

The cli tool will be used from the first entry.

I have smuggled in another commit to fix the apphost binary, which is used to create executable files for projects with OutputType=exe. There is a catch though: If a library specifies a patch version of the environment like 3.0.1, it might download a nuget package containing a copy of apphost, which is not patched.

% find ~/.nuget/packages -iname apphost  
/home/bara/.nuget/packages/microsoft.netcore.app.host.linux-x64/3.0.1/runtimes/linux-x64/native/apphost

Copy link
Contributor

@jonringer jonringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

definite improvement, I have a few suggestions

pkgs/development/compilers/dotnet/combinePackages.nix Outdated Show resolved Hide resolved
pkgs/development/compilers/dotnet/combinePackages.nix Outdated Show resolved Hide resolved
pkgs/development/compilers/dotnet/default.nix Outdated Show resolved Hide resolved
pkgs/development/compilers/dotnet/combinePackages.nix Outdated Show resolved Hide resolved
pkgs/development/compilers/dotnet/combinePackages.nix Outdated Show resolved Hide resolved
@jonringer
Copy link
Contributor

I'm happy with this for the time being. If it needs to be revisited we can do so in another PR.

This is a much needed improvement to nixpkgs, thanks @baracoder for up-streaming this :)

@jonringer jonringer merged commit 6bc3acd into NixOS:master Jan 17, 2020
@baracoder baracoder deleted the dotnet-multi-runtime branch January 17, 2020 19:49
@jonringer
Copy link
Contributor

hydra built it https://hydra.nixos.org/eval/1565649#tabs-new . Once a successful release is done, we'll be able to use this on unstable :)

@nyanloutre
Copy link
Member

I made a pull request to add the missing runtimes and use it with Jellyfin : #77961

@garrett-hopper
Copy link

How would I go about using this combinePackages from nix-env? Do I need to create a .nix file and install that or put it in my nix config instead?

@baracoder
Copy link
Contributor Author

baracoder commented Jan 19, 2020

@Jumblemuddle I am using nix repl to intsall it in my profile:

$ nix repl /path/to/nixpkgs
nix-repl> :i with dotnetCorePackages; combinePackages [ sdk_3_1 sdk_2_2 sdk_3_0 aspnetcore_2_1 ]

Don't know if it is possible with nix-env

@jonringer
Copy link
Contributor

I would probably use a shell.nix in the repository that you need it:

with import <nixpkgs> {};

mkShell rec {
  name = "dotnet";
  buildInputs = [
    (with dotnetCorePackages; combinePackages [ sdk_3_1 sdk_2_1 ])
  ];
}

doing nix-shell will then read in the shell.nix and give you a dotnet with:

$ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   3.1.100
 Commit:    cd82f021f4

Runtime Environment:
 OS Name:     nixos
 OS Version:  20.03pre209250.7184df6beb8
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /nix/store/cdns908qgng8rk0ngm1dqrd5q1qqnhqa-dotnet-sdk-3.1.100/sdk/3.1.100/

Host (useful for support):
  Version: 3.1.0
  Commit:  65f04fb6db

.NET Core SDKs installed:
  2.1.509 [/nix/store/sdsv8i00h54iii9j8wsrxkjplvmc8z3m-dotnet-core-combined/sdk]
  3.1.100 [/nix/store/sdsv8i00h54iii9j8wsrxkjplvmc8z3m-dotnet-core-combined/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.13 [/nix/store/sdsv8i00h54iii9j8wsrxkjplvmc8z3m-dotnet-core-combined/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.13 [/nix/store/sdsv8i00h54iii9j8wsrxkjplvmc8z3m-dotnet-core-combined/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.0 [/nix/store/sdsv8i00h54iii9j8wsrxkjplvmc8z3m-dotnet-core-combined/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.13 [/nix/store/sdsv8i00h54iii9j8wsrxkjplvmc8z3m-dotnet-core-combined/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.0 [/nix/store/sdsv8i00h54iii9j8wsrxkjplvmc8z3m-dotnet-core-combined/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

@jonringer
Copy link
Contributor

once #77961 gets merged, and I write up some documentation, the I'll probably backport this to 19.09 so they can use it as well

@baracoder
Copy link
Contributor Author

🤦‍♂️ I have made an error in copying things from src: cp ./ will copy env-vars which pulls the source tar.gz package into the runtime closure

% nix why-depends /nix/store/8w7566287q3zalhn0xixl9610ssqirfx-dotnet-sdk-3.1.100 /nix/store/cc94mviqcb0wpqq5488p7a0ggp8v3hny-dotnet-sdk-3.1.100-linux-x64.tar.gz
/nix/store/8w7566287q3zalhn0xixl9610ssqirfx-dotnet-sdk-3.1.100
╚═══env-vars: …=".".declare -x src="/nix/store/cc94mviqcb0wpqq5488p7a0ggp8v3hny-dotnet-sdk-3.1.100-linux-x64.ta…
    => /nix/store/cc94mviqcb0wpqq5488p7a0ggp8v3hny-dotnet-sdk-3.1.100-linux-x64.tar.gz

noDumpEnvVars=true; should fix this. I will create a PR

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/why-cant-i-install-dotnet-in-my-rust-build-environment-shell/5786/1

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-are-your-goals-for-20-03/4773/16

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

10 participants