Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libxnd, libndtypes: refactor add support for darwin #46578

Merged
merged 2 commits into from Sep 13, 2018

Conversation

costrouc
Copy link
Member

Motivation for this change

Prior build instructions were hacks because I did not understand that the configure script was choosing ld as the linker instead of using gcc to properly link. This commit fixes these issues and simplifies the build.

Things done

libxnd: refactor
libndtypes: refactor

Please ensure that this build works on darwin.

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Fits CONTRIBUTING.md.

@LnL7
Copy link
Member

LnL7 commented Sep 12, 2018

I'm a bit confused by the comments, cc is the compiler and ld is the linker. Does the build pass compiler flags to LD?

@costrouc
Copy link
Member Author

costrouc commented Sep 12, 2018

For some reason the configure script by default chooses ld as the linker. I have had this issue in several other builds I am working on. This causes errors because the Makefile assumes that gcc or clang is used for the linker. Additionally gcc and clang pass the appropriate linking flags to include glibc pthreads etc. Is there a more elegant way to do this? cc is symlinked to either gcc or clang depending on the OS.

@LnL7
Copy link
Member

LnL7 commented Sep 12, 2018

Yes that's very normal, gcc and clang are not linkers. Could you explain or post the error?

@costrouc
Copy link
Member Author

costrouc commented Sep 12, 2018

Here is the error if I do not manually set cc as the linker.

I am aware that gcc and clang are not linkers (they are wrappers around preprocessing, compiling, and linking). From what I know they can be used as linkers and call ld with appropriate arguments.

....
gcc -I.. -Wall -Wextra -std=c11 -pedantic -O2 -g -fPIC -c seq.c -o .objs/seq.o
gcc -I.. -Wall -Wextra -std=c11 -pedantic -O2 -g -fPIC -c symtable.c -o .objs/symtable.o
gcc -I.. -Wall -Wextra -std=c11 -pedantic -O2 -g -fPIC -c util.c -o .objs/util.o
gcc -I.. -Wall -Wextra -std=c11 -pedantic -O2 -g -fPIC -c values.c -o .objs/values.o
ld -shared -Wl,-soname,libndtypes.so.0 -o libndtypes.so.0.2.0dev3 .objs/alloc.o .objs/attr.o .objs/context.o .objs/copy.o .objs/encodings.o .objs/equal.o .objs/grammar.o .objs/io.o .objs/lexer.o .objs/match.o .objs/ndtypes.o .objs/parsefuncs.o .objs/parser.o .objs/seq.o .objs/symtable.o .objs/util.o .objs/values.o compat/.objs/bpgrammar.o compat/.objs/bplexer.o compat/.objs/import.o
/nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/ld: unrecognized option '-Wl,-soname,libndtypes.so.0'
/nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/ld: use the --help option for usage information
make[1]: *** [Makefile:57: libndtypes.so.0.2.0dev3] Error 1
make[1]: Leaving directory '/build/source/libndtypes'
make: *** [Makefile:23: default] Error 2
builder for '/nix/store/5gplhcy8rxmiabvw43agnwpkflrvz29l-libndtypes-0.2.0dev3.drv' failed with exit code 2
error: build of '/nix/store/5gplhcy8rxmiabvw43agnwpkflrvz29l-libndtypes-0.2.0dev3.drv' failed

'-Wl,-soname,libndtypes.so.0' is a argument that is passed to gcc or clang that is appropriately passed to ld. If I am understanding anything wrong please let me know. Or if it could be done in a nicer way.

@LnL7
Copy link
Member

LnL7 commented Sep 12, 2018

Ah I see, the build is passing compiler specific flags to ld like I thought. -Wl is a compiler flag to specify extra flags to pass to the linker. Could you rephrase the comment to avoid confusion.

That should be, so using LD=cc basically turns the incorrect usage into the first variant.

$CC -Wl,-soname,libndtypes.so.0 ...
# or
$LD -soname libndtypes.so.0 ...

Might be worth creating an upstream issue.

@costrouc
Copy link
Member Author

costrouc commented Sep 13, 2018

Ok I have updated the pull request with a better comment for why cc was used as a linker. Its ready to merge.

I still am not convinced this is incorrect usage. While in this case using ld would have been simple enough. For some complex libraries this is not always the case. Take for example mpicc, mpif90 these are wrappers around gcc with linker and special compilation flags. So similarly using gcc for linking simplifies the process by not having to worry about the location of glibc, lm, ld-linux-x86-64.so.2, etc. Using -Wl,--verbose you can print all of the work done by gcc for linking.

Here are the linking flags used by mpicc -showme:linker

mpicc -showme:linker

gcc -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include -I/usr/lib/x86_64-linux-gnu/openmpi/include -pthread -L/usr//lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi

@LnL7
Copy link
Member

LnL7 commented Sep 13, 2018

I only have a vague understanding of this, somebody like @Ericson2314 can probably explain this better. But we explicitly set LD=ld in the stdenv which is probably what exposed the issue.

makeFlags = [ "CONFIGURE_LDFLAGS='-shared'" ];
# Override linker with cc (symlink to either gcc or clang)
# Library expects to use cc for linking
configureFlags = [ "LD=cc" ];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

${stdenv.cc.targetPrefix}cc instead of cc, for sake of cross

makeFlags = [ "CONFIGURE_LDFLAGS='-shared'" ];
# Override linker with cc (symlink to either gcc or clang)
# Library expects to use cc for linking
configureFlags = [ "LD=cc" ];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@Ericson2314
Copy link
Member

Yeah it is done for cross on other things to force configure scripts to use the right linker. But then some configure scripts are crazy and have defaults like LD ?= gcc for this. I've seen a CCLD environment variable instead, which is a much better way to use a C compiler for linking but also be independent ofCC. That is what upstream ought to use.

@costrouc
Copy link
Member Author

@Ericson2314 I have made the changes that you requested for cross compatibiliy.

Just to make sure that I understand correctly. Using a variable CCLD is useful because it indicates that you are using a compiler to determine the linker. Setting LD=gcc can be confusing and is not good style. Is that correct?

@Ericson2314
Copy link
Member

@costrouc I'd change "compiler to determine the linker" to "using a compiler to do the linking". The compiler will modify how you link things in subtle ways to pick up the C/C++ runtime things, etc.

@costrouc
Copy link
Member Author

Should be ready for a build now

@Ericson2314 Ericson2314 merged commit 59be668 into NixOS:master Sep 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants