Skip to content

Commit

Permalink
Merge branch 'staging'
Browse files Browse the repository at this point in the history
  • Loading branch information
@vcunat
vcunat committed Dec 19, 2017
2 parents b455192 + 69345ec commit 4a2340f
Show file tree
Hide file tree
Showing 68 changed files with 1,039 additions and 369 deletions.
55 changes: 43 additions & 12 deletions doc/stdenv.xml
View file
Expand Up @@ -1358,36 +1358,67 @@ someVar=$(stripHash $name)
<variablelist>

<varlistentry>
<term>CC Wrapper</term>
<term>Bintools Wrapper</term>
<listitem>
<para>
CC Wrapper wraps a C toolchain for a bunch of miscellaneous purposes.
Specifically, a C compiler (GCC or Clang), Binutils (or the CCTools + binutils mashup when targetting Darwin), and a C standard library (glibc or Darwin's libSystem) are all fed in, and dependency finding, hardening (see below), and purity checks for each are handled by CC Wrapper.
Packages typically depend on only CC Wrapper, instead of those 3 inputs directly.
Bintools Wrapper wraps the binary utilities for a bunch of miscellaneous purposes.
These are GNU Binutils when targetting Linux, and a mix of cctools and GNU binutils for Darwin.
[The "Bintools" name is supposed to be a compromise between "Binutils" and "cctools" not denoting any specific implementation.]
Specifically, the underlying bintools package, and a C standard library (glibc or Darwin's libSystem, just for the dynamic loader) are all fed in, and dependency finding, hardening (see below), and purity checks for each are handled by Bintools Wrapper.
Packages typically depend on CC Wrapper, which in turn (at run time) depends on Bintools Wrapper.
</para>
<para>
Dependency finding is undoubtedly the main task of CC wrapper.
Bintools Wrapper was only just recently split off from CC Wrapper, so the division of labor is still being worked out.
For example, it shouldn't care about about the C standard library, but just take a derivation with the dynamic loader (which happens to be the glibc on linux).
Dependency finding however is a task both wrappers will continue to need to share, and probably the most important to understand.
It is currently accomplished by collecting directories of host-platform dependencies (i.e. <varname>buildInputs</varname> and <varname>nativeBuildInputs</varname>) in environment variables.
CC wrapper's setup hook causes any <filename>include</filename> subdirectory of such a dependency to be added to <envar>NIX_CFLAGS_COMPILE</envar>, and any <filename>lib</filename> and <filename>lib64</filename> subdirectories to <envar>NIX_LDFLAGS</envar>.
The setup hook itself contains some lengthy comments describing the exact convoluted mechanism by which this is accomplished.
Bintools Wrapper's setup hook causes any <filename>lib</filename> and <filename>lib64</filename> subdirectories to be added to <envar>NIX_LDFLAGS</envar>.
Since CC Wrapper and Bintools Wrapper use the same strategy, most of the Bintools Wrapper code is sparsely commented and refers to CC Wrapper.
But CC Wrapper's code, by contrast, has quite lengthy comments.
Bintools Wrapper merely cites those, rather than repeating them, to avoid falling out of sync.
</para>
<para>
A final task of the setup hook is defining a number of standard environment variables to tell build systems which executables full-fill which purpose.
They are defined to just be the base name of the tools, under the assumption that CC Wrapper's binaries will be on the path.
They are defined to just be the base name of the tools, under the assumption that Bintools Wrapper's binaries will be on the path.
Firstly, this helps poorly-written packages, e.g. ones that look for just <command>gcc</command> when <envar>CC</envar> isn't defined yet <command>clang</command> is to be used.
Secondly, this helps packages not get confused when cross-compiling, in which case multiple CC wrappers may be simultaneous in use (targeting different platforms).
<envar>BUILD_</envar>- and <envar>TARGET_</envar>-prefixed versions of the normal environment variable are defined for the additional CC Wrappers, properly disambiguating them.
Secondly, this helps packages not get confused when cross-compiling, in which case multiple Bintools Wrappers may simultaneously be in use.
<footnote><para>
Each wrapper targets a single platform, so if binaries for multiple platforms are needed, the underlying binaries must be wrapped multiple times.
As this is a property of the wrapper itself, the multiple wrappings are needed whether or not the same underlying binaries can target multiple platforms.
</para></footnote>
<envar>BUILD_</envar>- and <envar>TARGET_</envar>-prefixed versions of the normal environment variable are defined for the additional Bintools Wrappers, properly disambiguating them.
</para>
<para>
A problem with this final task is that CC Wrapper is honest and defines <envar>LD</envar> as <command>ld</command>.
A problem with this final task is that Bintools Wrapper is honest and defines <envar>LD</envar> as <command>ld</command>.
Most packages, however, firstly use the C compiler for linking, secondly use <envar>LD</envar> anyways, defining it as the C compiler, and thirdly, only so define <envar>LD</envar> when it is undefined as a fallback.
This triple-threat means CC Wrapper will break those packages, as LD is already defined as the actually linker which the package won't override yet doesn't want to use.
This triple-threat means Bintools Wrapper will break those packages, as LD is already defined as the actual linker which the package won't override yet doesn't want to use.
The workaround is to define, just for the problematic package, <envar>LD</envar> as the C compiler.
A good way to do this would be <command>preConfigure = "LD=$CC"</command>.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term>CC Wrapper</term>
<listitem>
<para>
CC Wrapper wraps a C toolchain for a bunch of miscellaneous purposes.
Specifically, a C compiler (GCC or Clang), wrapped binary tools, and a C standard library (glibc or Darwin's libSystem, just for the dynamic loader) are all fed in, and dependency finding, hardening (see below), and purity checks for each are handled by CC Wrapper.
Packages typically depend on CC Wrapper, which in turn (at run time) depends on Bintools Wrapper.
</para>
<para>
Dependency finding is undoubtedly the main task of CC Wrapper.
This works just like Bintools Wrapper, except that any <filename>include</filename> subdirectory of any relevant dependency is added to <envar>NIX_CFLAGS_COMPILE</envar>.
The setup hook itself contains some lengthy comments describing the exact convoluted mechanism by which this is accomplished.
</para>
<para>
CC Wrapper also like Bintools Wrapper defines standard environment variables with the names of the tools it wraps, for the same reasons described above.
Importantly, while it includes a <command>cc</command> symlink to the c compiler for portability, the <envar>CC</envar> will be defined using the compiler's "real name" (i.e. <command>gcc</command> or <command>clang</command>).
This helps lousy build systems that inspect on the name of the compiler rather than run it.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term>Perl</term>
<listitem><para>Adds the <filename>lib/site_perl</filename> subdirectory
Expand Down
11 changes: 9 additions & 2 deletions pkgs/applications/video/mpv/default.nix
View file
@@ -1,4 +1,4 @@
{ stdenv, fetchurl, fetchFromGitHub, makeWrapper
{ stdenv, fetchurl, fetchFromGitHub, fetchpatch, makeWrapper
, docutils, perl, pkgconfig, python3, which, ffmpeg
, freefont_ttf, freetype, libass, libpthreadstubs
, lua, lua5_sockets, libuchardet, libiconv ? null, darwin
Expand Down Expand Up @@ -90,7 +90,14 @@ in stdenv.mkDerivation rec {
sha256 = "0746kmsg69675y5c70vn8imcr9d1zpjz97f27xr1vx00yjpd518v";
};

patchPhase = ''
patches = [
(fetchpatch {
url = "https://github.com/mpv-player/mpv/commit/2ecf240b1cd20875991a5b18efafbe799864ff7f.patch";
sha256 = "1sr0770rvhsgz8d7ysr9qqp4g9gwdhgj8g3rgnz90wl49lgrykhb";
})
];

postPatch = ''
patchShebangs ./TOOLS/
'';

Expand Down
40 changes: 40 additions & 0 deletions pkgs/build-support/bintools-wrapper/add-flags.sh
View file
@@ -0,0 +1,40 @@
# See cc-wrapper for comments.
var_templates_list=(
NIX+IGNORE_LD_THROUGH_GCC
NIX+LDFLAGS
NIX+LDFLAGS_BEFORE
NIX+LDFLAGS_AFTER
NIX+LDFLAGS_HARDEN
)
var_templates_bool=(
NIX+SET_BUILD_ID
NIX+DONT_SET_RPATH
)

declare -a role_infixes=()
if [ "${NIX_BINTOOLS_WRAPPER_@infixSalt@_TARGET_BUILD:-}" ]; then
role_infixes+=(_BUILD_)
fi
if [ "${NIX_BINTOOLS_WRAPPER_@infixSalt@_TARGET_HOST:-}" ]; then
role_infixes+=(_)
fi
if [ "${NIX_BINTOOLS_WRAPPER_@infixSalt@_TARGET_TARGET:-}" ]; then
role_infixes+=(_TARGET_)
fi

for var in "${var_templates_list[@]}"; do
mangleVarList "$var" "${role_infixes[@]}"
done
for var in "${var_templates_bool[@]}"; do
mangleVarBool "$var" "${role_infixes[@]}"
done

if [ -e @out@/nix-support/libc-ldflags ]; then
NIX_@infixSalt@_LDFLAGS+=" $(< @out@/nix-support/libc-ldflags)"
fi

if [ -e @out@/nix-support/libc-ldflags-before ]; then
NIX_@infixSalt@_LDFLAGS_BEFORE="$(< @out@/nix-support/libc-ldflags-before) $NIX_@infixSalt@_LDFLAGS_BEFORE"
fi

export NIX_BINTOOLS_WRAPPER_@infixSalt@_FLAGS_SET=1
53 changes: 53 additions & 0 deletions pkgs/build-support/bintools-wrapper/add-hardening.sh
View file
@@ -0,0 +1,53 @@
hardeningFlags=(relro bindnow)
# Intentionally word-split in case 'hardeningEnable' is defined in
# Nix. Also, our bootstrap tools version of bash is old enough that
# undefined arrays trip `set -u`.
if [[ -v hardeningEnable[@] ]]; then
hardeningFlags+=(${hardeningEnable[@]})
fi
hardeningLDFlags=()

declare -A hardeningDisableMap

# Intentionally word-split in case 'hardeningDisable' is defined in Nix.
for flag in ${hardeningDisable[@]:-IGNORED_KEY} @hardening_unsupported_flags@
do
hardeningDisableMap[$flag]=1
done

if (( "${NIX_DEBUG:-0}" >= 1 )); then
printf 'HARDENING: disabled flags:' >&2
(( "${#hardeningDisableMap[@]}" )) && printf ' %q' "${!hardeningDisableMap[@]}" >&2
echo >&2
fi

if [[ -z "${hardeningDisableMap[all]:-}" ]]; then
if (( "${NIX_DEBUG:-0}" >= 1 )); then
echo 'HARDENING: Is active (not completely disabled with "all" flag)' >&2;
fi
for flag in "${hardeningFlags[@]}"
do
if [[ -z "${hardeningDisableMap[$flag]:-}" ]]; then
case $flag in
pie)
if [[ ! ("$*" =~ " -shared " || "$*" =~ " -static ") ]]; then
if (( "${NIX_DEBUG:-0}" >= 1 )); then echo HARDENING: enabling LDFlags -pie >&2; fi
hardeningLDFlags+=('-pie')
fi
;;
relro)
if (( "${NIX_DEBUG:-0}" >= 1 )); then echo HARDENING: enabling relro >&2; fi
hardeningLDFlags+=('-z' 'relro')
;;
bindnow)
if (( "${NIX_DEBUG:-0}" >= 1 )); then echo HARDENING: enabling bindnow >&2; fi
hardeningLDFlags+=('-z' 'now')
;;
*)
# Ignore unsupported. Checked in Nix that at least *some*
# tool supports each flag.
;;
esac
fi
done
fi

0 comments on commit 4a2340f

Please sign in to comment.