I think it would be good if can avoid patching like this. These patches usually end up breaking between version updates. I think we might be able to just add $out/lib to RPATH for $out/lib/libxmlsec.* and $out/bin/xmlsec, and put that in the postInstal...

Modifying the RPATH didn't work for me.

The xmlsec crypto libraries aren't linked dynamically, but get loaded with lt_dlopenext. This function uses either a libtool integrated loader, which can be influenced via lt_dladdsearchdir, LD_LIBRARY_PATH and LTDL_LIBRARY_PATH or it uses dlopen which can be influenced via LD_LIBRARY_PATH and the RPATH of the library that calls dlopen, which is libltdl.so in this case.

I tested the dlopen variant by appending $out/lib to the libxmlsec1.so library RPATH and straceing the xmlsec1 binary (without wrapProgram "$out/bin/xmlsec1"), which already includes $out/lib in it's RPATH. (strace log: https://pastebin.com/2ZBVWWbK)

It only tries to load libxmlsec1-openssl.so from /nix/store/xxx-glibc-2.27/lib/ (except the sane path, which is my LD_LIBRARY_PATH), which is the only RPATH entry of libltdl.so.
So dlopen and RPATH don't work together when used in a library.

These are the only options I could find so far.

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

I'm also trying to use python-xmlsec and it can't load the crypto library:

❯ python -c "import xmlsec"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
xmlsec.Error: (1, 'cannot load crypto library for xmlsec.')

@jtojnar in researching this problem I've seen your name pop up on related issues elsewhere.

Looks like pysaml2 also has trouble with xmlsec and does patching:

The changes look reasonable. Future updates can break any customizations and I prefer when the breakage is loud so that we can be aware of it and fix it. Since this was not caught by the project tests adding a derivation checking if this still works to passthru.tests would be a good idea.

Looks like xmlSecCryptoDLLoadLibrary only takes crypto library name like openssl or nss and then it constructs libxmlsec1-nss library name, which is lt_dlopened

1. https://github.com/lsh123/xmlsec/blob/c882d225f87194a8d457ad61ee23ff4befdb86c7/src/dl.c#L442
2. https://github.com/lsh123/xmlsec/blob/c882d225f87194a8d457ad61ee23ff4befdb86c7/src/dl.c#L451
3. https://github.com/lsh123/xmlsec/blob/c882d225f87194a8d457ad61ee23ff4befdb86c7/src/dl.c#L118
4. https://github.com/lsh123/xmlsec/blob/c882d225f87194a8d457ad61ee23ff4befdb86c7/src/dl.c#L252

so alternately, we could hard-code the absolute path within xmlSecCryptoDLLibraryConstructFilename. That might have the benefit that any future changes would complain loudly by breaking tests but it would require us to include the file extension too.

Looks like pysaml2 also has trouble with xmlsec and does patching:

pysaml2 only uses the xmlsec1 binary, so nothing would change there.

Since this was not caught by the project tests adding a derivation checking if this still works to passthru.tests would be a good idea.

The tests seem to only run the xmlsec1 binary and library loading is made possible by setting LD_LIBRARY_PATH in the Makefile.

Adding a passthru.tests check for library loading is a good idea. I'm currently working on such a test.

so alternately, we could hard-code the absolute path within xmlSecCryptoDLLibraryConstructFilename. That might have the benefit that any future changes would complain loudly by breaking tests but it would require us to include the file extension too.

I'm not sure if hardcoding the paths has an advantage over embedding the lib path. If the tests can ensure that library loading works, both should fail equally fast on a breaking change.

The benefit of embedding the lib path would be that dependent packages can still modify the search_dir and load their own libraries instead of the embedded ones.

@B4dM4n did you ever end up packaging python-xmlsec? I'm trying to get it built right now and running into dlopen issues

I'm also trying to use python-xmlsec and it can't load the crypto library:

❯ python -c "import xmlsec"
Traceback (most recent call last):
File "", line 1, in
xmlsec.Error: (1, 'cannot load crypto library for xmlsec.')

this is the exact issue I'm getting, even with this patch

never mind, I'm actually just running an older version of nixpkgs than when this was merged