Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/master' into openssl-1.1
Browse files Browse the repository at this point in the history
  • Loading branch information
@globin
globin committed Aug 25, 2017
2 parents 828994f + 58c0d63 commit a74b7ff
Show file tree
Hide file tree
Showing 1,491 changed files with 37,476 additions and 32,021 deletions.
2 changes: 1 addition & 1 deletion .github/PULL_REQUEST_TEMPLATE.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

###### Things done

Please check what applies. Note that these are not hard requirements but merely serve as information for reviewers.
<!-- Please check what applies. Note that these are not hard requirements but merely serve as information for reviewers. -->

- [ ] Tested using sandboxing
([nix.useSandbox](http://nixos.org/nixos/manual/options.html#opt-nix.useSandbox) on NixOS,
Expand Down
6 changes: 6 additions & 0 deletions .travis.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,21 @@ matrix:
script:
- ./maintainers/scripts/travis-nox-review-pr.sh nixpkgs-verify nixpkgs-manual nixpkgs-tarball nixpkgs-unstable
- ./maintainers/scripts/travis-nox-review-pr.sh nixos-options nixos-manual
env:
- BUILD_TYPE="Test Nixpkgs evaluation & NixOS manual build"
- os: linux
sudo: required
dist: trusty
before_script:
- sudo mount -o remount,exec,size=2G,mode=755 /run/user
script: ./maintainers/scripts/travis-nox-review-pr.sh nox pr
env:
- BUILD_TYPE="Build affected packages (Linux)"
- os: osx
osx_image: xcode7.3
script: ./maintainers/scripts/travis-nox-review-pr.sh nox pr
env:
- BUILD_TYPE="Build affected packages (macOS)"
env:
global:
- GITHUB_TOKEN=5edaaf1017f691ed34e7f80878f8f5fbd071603f
Expand Down
2 changes: 1 addition & 1 deletion doc/languages-frameworks/haskell.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -827,7 +827,7 @@ the work to be licensed" under the terms of the LGPL (including for free).

The LGPL licensing for GMP is a problem for the overall licensing of binary
programs compiled with GHC because most distributions (and builds) of GHC use
static libraries. (Dynamic libraries are currently distributed only for OS X.)
static libraries. (Dynamic libraries are currently distributed only for macOS.)
The LGPL licensing situation may be worse: even though
[The Glasgow Haskell Compiler License](https://www.haskell.org/ghc/license)
is essentially a "free software" license (BSD3), according to
Expand Down
46 changes: 37 additions & 9 deletions doc/languages-frameworks/python.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -340,7 +340,7 @@ other packages we like to have in the environment, all specified with `propagate
Indeed, we can just add any package we like to have in our environment to `propagatedBuildInputs`.

```nix
with import <nixpkgs>;
with import <nixpkgs> {};
with pkgs.python35Packages;
buildPythonPackage rec {
Expand Down Expand Up @@ -423,7 +423,7 @@ and in this case the `python35` interpreter is automatically used.
### Interpreters

Versions 2.7, 3.3, 3.4, 3.5 and 3.6 of the CPython interpreter are available as
respectively `python27`, `python33`, `python34`, `python35` and `python36`. The PyPy interpreter
respectively `python27`, `python34`, `python35` and `python36`. The PyPy interpreter
is available as `pypy`. The aliases `python2` and `python3` correspond to respectively `python27` and
`python35`. The default interpreter, `python`, maps to `python2`.
The Nix expressions for the interpreters can be found in
Expand Down Expand Up @@ -469,7 +469,6 @@ sets are

* `pkgs.python26Packages`
* `pkgs.python27Packages`
* `pkgs.python33Packages`
* `pkgs.python34Packages`
* `pkgs.python35Packages`
* `pkgs.python36Packages`
Expand Down Expand Up @@ -546,6 +545,35 @@ All parameters from `mkDerivation` function are still supported.
* `catchConflicts` If `true`, abort package build if a package name appears more than once in dependency tree. Default is `true`.
* `checkInputs` Dependencies needed for running the `checkPhase`. These are added to `buildInputs` when `doCheck = true`.

##### Overriding Python packages

The `buildPythonPackage` function has a `overridePythonAttrs` method that
can be used to override the package. In the following example we create an
environment where we have the `blaze` package using an older version of `pandas`.
We override first the Python interpreter and pass
`packageOverrides` which contains the overrides for packages in the package set.

```nix
with import <nixpkgs> {};
(let
python = let
packageOverrides = self: super: {
pandas = super.pandas.overridePythonAttrs(old: rec {
version = "0.19.1";
name = "pandas-${version}";
src = super.fetchPypi {
pname = "pandas";
inherit version;
sha256 = "08blshqj9zj1wyjhhw3kl2vas75vhhicvv72flvf1z3jvapgw295";
};
});
};
in pkgs.python3.override {inherit packageOverrides;};
in python.withPackages(ps: [ps.blaze])).env
```

#### `buildPythonApplication` function

The `buildPythonApplication` function is practically the same as `buildPythonPackage`.
Expand Down Expand Up @@ -622,7 +650,7 @@ attribute. The `shell.nix` file from the previous section can thus be also writt
```nix
with import <nixpkgs> {};
(python33.withPackages (ps: [ps.numpy ps.requests])).env
(python36.withPackages (ps: [ps.numpy ps.requests])).env
```

In contrast to `python.buildEnv`, `python.withPackages` does not support the more advanced options
Expand Down Expand Up @@ -755,17 +783,17 @@ In the following example we rename the `pandas` package and build it.
```nix
with import <nixpkgs> {};
let
(let
python = let
packageOverrides = self: super: {
pandas = super.pandas.override {name="foo";};
pandas = super.pandas.overridePythonAttrs(old: {name="foo";});
};
in pkgs.python35.override {inherit packageOverrides;};
in python.pkgs.pandas
in python.withPackages(ps: [ps.pandas])).env
```
Using `nix-build` on this expression will build the package `pandas`
but with the new name `foo`.
Using `nix-build` on this expression will build an environment that contains the
package `pandas` but with the new name `foo`.

All packages in the package set will use the renamed package.
A typical use case is to switch to another version of a certain package.
Expand Down
10 changes: 3 additions & 7 deletions doc/languages-frameworks/ruby.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ $ cd sensu
$ cat > Gemfile
source 'https://rubygems.org'
gem 'sensu'
$ $(nix-build '<nixpkgs>' -A bundix)/bin/bundix --magic
$ $(nix-build '<nixpkgs>' -A bundix --no-out-link)/bin/bundix --magic
$ cat > default.nix
{ lib, bundlerEnv, ruby }:
Expand Down Expand Up @@ -82,7 +82,7 @@ versions available from various packages.
</para>

<para>Resulting derivations for both builders also have two helpful
attributes, <literal>env</literal> and <literal>wrapper</literal>.
attributes, <literal>env</literal> and <literal>wrappedRuby</literal>.
The first one allows one to quickly drop into
<command>nix-shell</command> with the specified environment present.
E.g. <command>nix-shell -A sensu.env</command> would give you an
Expand Down Expand Up @@ -110,13 +110,9 @@ the needed dependencies. For example, to make a derivation
in stdenv.mkDerivation {
name = "my-script";
buildInputs = [ env.wrapper ];
buildInputs = [ env.wrappedRuby ];
script = ./my-script.rb;
buildCommand = ''
mkdir -p $out/bin
install -D -m755 $script $out/bin/my-script
patchShebangs $out/bin/my-script
'';
Expand Down
4 changes: 2 additions & 2 deletions doc/multiple-output.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@

<varlistentry><term><varname>
$outputMan</varname></term><listitem><para>
is for man pages (except for section 3). They go to <varname>man</varname> or <varname>doc</varname> or <varname>$outputBin</varname> by default.
is for man pages (except for section 3). They go to <varname>man</varname> or <varname>$outputBin</varname> by default.
</para></listitem></varlistentry>

<varlistentry><term><varname>
Expand All @@ -83,7 +83,7 @@

<varlistentry><term><varname>
$outputInfo</varname></term><listitem><para>
is for info pages. They go to <varname>info</varname> or <varname>doc</varname> or <varname>$outputMan</varname> by default.
is for info pages. They go to <varname>info</varname> or <varname>$outputBin</varname> by default.
</para></listitem></varlistentry>

</variablelist>
Expand Down
91 changes: 63 additions & 28 deletions doc/overlays.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,59 +8,88 @@
overlays. Overlays are used to add layers in the fix-point used by Nixpkgs
to compose the set of all packages.</para>

<para>Nixpkgs can be configured with a list of overlays, which are
applied in order. This means that the order of the overlays can be significant
if multiple layers override the same package.</para>

<!--============================================================-->

<section xml:id="sec-overlays-install">
<title>Installing Overlays</title>
<title>Installing overlays</title>

<para>The list of overlays is determined as follows.</para>

<para>The set of overlays is looked for in the following places. The
first one present is considered, and all the rest are ignored:
<para>If the <varname>overlays</varname> argument is not provided explicitly, we look for overlays in a path. The path
is determined as follows:

<orderedlist>

<listitem>
<para>First, if an <varname>overlays</varname> argument to the nixpkgs function itself is given,
then that is used.</para>

<para>As an argument of the imported attribute set. When importing Nixpkgs,
the <varname>overlays</varname> attribute argument can be set to a list of
functions, which is described in <xref linkend="sec-overlays-layout"/>.</para>

<para>This can be passed explicitly when importing nipxkgs, for example
<literal>import &lt;nixpkgs> { overlays = [ overlay1 overlay2 ]; }</literal>.</para>
</listitem>

<listitem>
<para>Otherwise, if the Nix path entry <literal>&lt;nixpkgs-overlays></literal> exists, we look for overlays
at that path, as described below.</para>

<para>In the directory pointed to by the Nix search path entry
<literal>&lt;nixpkgs-overlays></literal>.</para>
<para>See the section on <literal>NIX_PATH</literal> in the Nix manual for more details on how to
set a value for <literal>&lt;nixpkgs-overlays>.</literal></para>
</listitem>

<listitem>

<para>In the directory <filename>~/.config/nixpkgs/overlays/</filename>.</para>
<para>If one of <filename>~/.config/nixpkgs/overlays.nix</filename> and
<filename>~/.config/nixpkgs/overlays/</filename> exists, then we look for overlays at that path, as
described below. It is an error if both exist.</para>
</listitem>

</orderedlist>
</para>

<para>For the second and third options, the directory should contain Nix expressions defining the
overlays. Each overlay can be a file, a directory containing a
<filename>default.nix</filename>, or a symlink to one of those. The expressions should follow
the syntax described in <xref linkend="sec-overlays-layout"/>.</para>
<para>If we are looking for overlays at a path, then there are two cases:
<itemizedlist>
<listitem>
<para>If the path is a file, then the file is imported as a Nix expression and used as the list of
overlays.</para>
</listitem>

<para>The order of the overlay layers can influence the recipe of packages if multiple layers override
the same recipe. In the case where overlays are loaded from a directory, they are loaded in
alphabetical order.</para>
<listitem>
<para>If the path is a directory, then we take the content of the directory, order it
lexicographically, and attempt to interpret each as an overlay by:
<itemizedlist>
<listitem>
<para>Importing the file, if it is a <literal>.nix</literal> file.</para>
</listitem>
<listitem>
<para>Importing a top-level <filename>default.nix</filename> file, if it is a directory.</para>
</listitem>
</itemizedlist>
</para>
</listitem>
</itemizedlist>
</para>

<para>To install an overlay using the last option, you can clone the overlay's repository and add
a symbolic link to it in <filename>~/.config/nixpkgs/overlays/</filename> directory.</para>
<para>On a NixOS system the value of the <literal>nixpkgs.overlays</literal> option, if present,
is passed to the system Nixpkgs directly as an argument. Note that this does not affect the overlays for
non-NixOS operations (e.g. <literal>nix-env</literal>), which are looked up independently.</para>

<para>The <filename>overlays.nix</filename> option therefore provides a convenient way to use the same
overlays for a NixOS system configuration and user configuration: the same file can be used
as <filename>overlays.nix</filename> and imported as the value of <literal>nixpkgs.overlays</literal>.</para>

</section>

<!--============================================================-->

<section xml:id="sec-overlays-layout">
<title>Overlays Layout</title>
<section xml:id="sec-overlays-definition">
<title>Defining overlays</title>

<para>Overlays are expressed as Nix functions which accept 2 arguments and return a set of
packages.</para>
<para>Overlays are Nix functions which accept two arguments,
conventionally called <varname>self</varname> and <varname>super</varname>,
and return a set of packages. For example, the following is a valid overlay.</para>

<programlisting>
self: super:
Expand All @@ -75,25 +104,31 @@ self: super:
}
</programlisting>

<para>The first argument, usually named <varname>self</varname>, corresponds to the final package
<para>The first argument (<varname>self</varname>) corresponds to the final package
set. You should use this set for the dependencies of all packages specified in your
overlay. For example, all the dependencies of <varname>rr</varname> in the example above come
from <varname>self</varname>, as well as the overridden dependencies used in the
<varname>boost</varname> override.</para>

<para>The second argument, usually named <varname>super</varname>,
<para>The second argument (<varname>super</varname>)
corresponds to the result of the evaluation of the previous stages of
Nixpkgs. It does not contain any of the packages added by the current
overlay nor any of the following overlays. This set should be used either
overlay, nor any of the following overlays. This set should be used either
to refer to packages you wish to override, or to access functions defined
in Nixpkgs. For example, the original recipe of <varname>boost</varname>
in the above example, comes from <varname>super</varname>, as well as the
<varname>callPackage</varname> function.</para>

<para>The value returned by this function should be a set similar to
<filename>pkgs/top-level/all-packages.nix</filename>, which contains
<filename>pkgs/top-level/all-packages.nix</filename>, containing
overridden and/or new packages.</para>

<para>Overlays are similar to other methods for customizing Nixpkgs, in particular
the <literal>packageOverrides</literal> attribute described in <xref linkend="sec-modify-via-packageOverrides"/>.
Indeed, <literal>packageOverrides</literal> acts as an overlay with only the
<varname>super</varname> argument. It is therefore appropriate for basic use,
but overlays are more powerful and easier to distribute.</para>

</section>

</chapter>
39 changes: 31 additions & 8 deletions doc/stdenv.xml
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@

<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xml:id="chap-stdenv">
Expand Down Expand Up @@ -1153,7 +1154,7 @@ makeWrapper $out/bin/foo $wrapperfile --prefix PATH : ${lib.makeBinPath [ hello

</listitem>
</varlistentry>


<varlistentry xml:id='fun-substitute'>
<term><function>substitute</function>
Expand Down Expand Up @@ -1312,7 +1313,7 @@ someVar=$(stripHash $name)

</para></listitem>
</varlistentry>


<varlistentry xml:id='fun-wrapProgram'>
<term><function>wrapProgram</function>
Expand Down Expand Up @@ -1342,12 +1343,34 @@ someVar=$(stripHash $name)
<variablelist>

<varlistentry>
<term>GCC wrapper</term>
<listitem><para>Adds the <filename>include</filename> subdirectory
of each build input to the <envar>NIX_CFLAGS_COMPILE</envar>
environment variable, and the <filename>lib</filename> and
<filename>lib64</filename> subdirectories to
<envar>NIX_LDFLAGS</envar>.</para></listitem>
<term>CC Wrapper</term>
<listitem>
<para>
CC Wrapper wraps a C toolchain for a bunch of miscellaneous purposes.
Specifically, a C compiler (GCC or Clang), Binutils (or the CCTools + binutils mashup when targetting Darwin), and a C standard library (glibc or Darwin's libSystem) are all fed in, and dependency finding, hardening (see below), and purity checks for each are handled by CC Wrapper.
Packages typically depend on only CC Wrapper, instead of those 3 inputs directly.
</para>
<para>
Dependency finding is undoubtedly the main task of CC wrapper.
It is currently accomplished by collecting directories of host-platform dependencies (i.e. <varname>buildInputs</varname> and <varname>nativeBuildInputs</varname>) in environment variables.
CC wrapper's setup hook causes any <filename>include</filename> subdirectory of such a dependency to be added to <envar>NIX_CFLAGS_COMPILE</envar>, and any <filename>lib</filename> and <filename>lib64</filename> subdirectories to <envar>NIX_LDFLAGS</envar>.
The setup hook itself contains some lengthy comments describing the exact convoluted mechanism by which this is accomplished.
</para>
<para>
A final task of the setup hook is defining a number of standard environment variables to tell build systems which executables full-fill which purpose.
They are defined to just be the base name of the tools, under the assumption that CC Wrapper's binaries will be on the path.
Firstly, this helps poorly-written packages, e.g. ones that look for just <command>gcc</command> when <envar>CC</envar> isn't defined yet <command>clang</command> is to be used.
Secondly, this helps packages not get confused when cross-compiling, in which case multiple CC wrappers may be simultaneous in use (targeting different platforms).
<envar>BUILD_</envar>- and <envar>TARGET_</envar>-prefixed versions of the normal environment variable are defined for the additional CC Wrappers, properly disambiguating them.
</para>
<para>
A problem with this final task is that CC Wrapper is honest and defines <envar>LD</envar> as <command>ld</command>.
Most packages, however, firstly use the C compiler for linking, secondly use <envar>LD</envar> anyways, defining it as the C compiler, and thirdly, only so define <envar>LD</envar> when it is undefined as a fallback.
This triple-threat means CC Wrapper will break those packages, as LD is already defined as the actually linker which the package won't override yet doesn't want to use.
The workaround is to define, just for the problematic package, <envar>LD</envar> as the C compiler.
A good way to do this would be <command>preConfigure = "LD=$CC"</command>.
</para>
</listitem>
</varlistentry>

<varlistentry>
Expand Down

0 comments on commit a74b7ff

Please sign in to comment.