Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

credstash: add standalone Python application #51807

Merged
merged 2 commits into from Dec 11, 2018

Conversation

ivanbrennan
Copy link
Member

@ivanbrennan ivanbrennan commented Dec 10, 2018

Motivation for this change

I want to use the credstash command-line application, but credstash is currently only offered as a Python library.

In order for this to work, I needed to remove the copy of the library that's placed in $out/bin and marked executable during the install phase. Other than the patched shebang and executable bit, it's identical to the library that's installed to $out/lib/python3.7/site-packages.

Before postFixup has run wrapPythonPrograms, bin/ contains two Python files -- credstash and credstash.py -- where bin/credstash is the executable you'd expect a user to invoke from the command-line and bin/credstash.py contains the credstash module, which bin/credstash imports.

After wrapPythonPrograms has run, bin/credstash is a shell wrapper around the bin/.credstash-wrapped python entrypoint, and bin/credstash.py is shell wrapper around the bin/.credstash.py-wrapped python module. Invoking bin/credstash execs bin/.credstash-wrapped, which attempts to import the credstash module from bin/credstash.py (now a shell script), rather than either bin/.credstash.py-wrapped or lib/python3.7/site-packages/credstash.py.

This leads to an error:

$ credstash get mykey
Traceback (most recent call last):
  File "/nix/store/hk6yma716w6141lcdh509d6qyyi7zm0i-python3.7-credstash-1.15.0/bin/.credstash-wrapped", line 8, in <module>
    from credstash import main
  File "/nix/store/hk6yma716w6141lcdh509d6qyyi7zm0i-python3.7-credstash-1.15.0/bin/credstash.py", line 2
    export PATH='/nix/store/6lm4gi5iv8fbf1b1mm6g3gfnnv63f1gn-python3-3.7.1/bin:/nix/store/hk6yma716w6141lcdh509d6qyyi7zm0i-python3.7-credstash-1.15.0/bin:/nix/store/2n13gf1zdr39ir5dynxlkqndxgy36g08-python3.7-setuptools-40.4.3/bin:/nix/store/mhnqwpa4y1l81zi4cwx989i8h8z9g67l-python3.7-jmespath-0.9.0/bin:/nix/store/qc6q3a2nv4211wyh7q319v6zzd3ab6pc-python3.7-docutils-0.14/bin'${PATH:+':'}$PATH
              ^
SyntaxError: invalid syntax

I was able to resolve this by removing bin/credstash.py before the postFixup phase has a chance to wrap anything. Now the executable imports the library correctly:

(shell wrapper)
 bin/credstash
       │      (python executable)
       └─> bin/.credstash-wrapped
                  │                        (python library)
                  └─> lib/python3.7/site-packages/credstash.py

I tried using dontWrapPythonPrograms as an alternative solution, but that leads to a different error because runtime dependency lookups fail:

$ credstash get mykey
Traceback (most recent call last):
  File "/run/current-system/sw/bin/credstash", line 7, in <module>
    from credstash import main
  File "/nix/store/8rmldlvlv1z1xl7w02dy7f5qhkzdrg8z-python3.7-credstash-1.15.0/bin/credstash.py", line 26, in <module>
    import boto3
ModuleNotFoundError: No module named 'boto3'

Fixes #47751

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.

credstash was only available as a library. Provide it as a standalone
application as well.

In order for this to work, I needed to remove the copy of
the library that's placed in $out/bin and marked executable
during the install phase. Other than the patched shebang and
executable bit, it's identical to the library that's installed to
$out/lib/python3.7/site-packages.

Before the postFixup has run `wrapPythonPrograms`, $out/bin contains
two Python files -- credstash and credstash.py -- where bin/credstash
is the executable you'd expect a user to invoke from the command-line
and bin/credstash.py contains the credstash module, which bin/credstash
imports.

After `wrapPythonPrograms` has run, bin/credstash is a shell
wrapper around the bin/.credstash-wrapped python entrypoint, and
bin/credstash.py is shell wrapper around bin/.credstash.py-wrapped.
Invoking bin/credstash execs bin/.credstash-wrapped, and that python
script attempts to import the credstash module from bin/credstash.py,
the shell wrapper, rather than either bin/.credstash.py-wrapped or
lib/python3.7/site-packages/credstash.py.

This leads to an error:

    $ credstash get mykey
    Traceback (most recent call last):
      File "/nix/store/hk6yma716w6141lcdh509d6qyyi7zm0i-python3.7-credstash-1.15.0/bin/.credstash-wrapped", line 8, in <module>
        from credstash import main
      File "/nix/store/hk6yma716w6141lcdh509d6qyyi7zm0i-python3.7-credstash-1.15.0/bin/credstash.py", line 2
        export PATH='/nix/store/6lm4gi5iv8fbf1b1mm6g3gfnnv63f1gn-python3-3.7.1/bin:/nix/store/hk6yma716w6141lcdh509d6qyyi7zm0i-python3.7-credstash-1.15.0/bin:/nix/store/2n13gf1zdr39ir5dynxlkqndxgy36g08-python3.7-setuptools-40.4.3/bin:/nix/store/mhnqwpa4y1l81zi4cwx989i8h8z9g67l-python3.7-jmespath-0.9.0/bin:/nix/store/qc6q3a2nv4211wyh7q319v6zzd3ab6pc-python3.7-docutils-0.14/bin'${PATH:+':'}$PATH
                  ^
    SyntaxError: invalid syntax

If we try using `dontWrapPythonPrograms` to resolve this, runtime
dependency lookups fail:

    $ credstash get mykey
    Traceback (most recent call last):
      File "/run/current-system/sw/bin/credstash", line 7, in <module>
        from credstash import main
      File "/nix/store/8rmldlvlv1z1xl7w02dy7f5qhkzdrg8z-python3.7-credstash-1.15.0/bin/credstash.py", line 26, in <module>
        import boto3
    ModuleNotFoundError: No module named 'boto3'

I was able to resolve things by simply removing bin/credstash.py before
the postFixup phase has a chance to wrap any executables. Now the
executable imports the library correctly:

 (shell wrapper)
  bin/credstash
        │      (python executable)
        └─> bin/.credstash-wrapped
                   │                        (python library)
                   └─> lib/python3.7/site-packages/credstash.py
Make it clear why we're removing this file. For a more in-depth
description of the issue it addresses, see the previous commit:
2fc870e
@dotlambda
Copy link
Member

@GrahamcOfBorg build credstash

@dotlambda
Copy link
Member

What is not clear to me is why this is even in pythonPackages? Isn't this only ever used on the command line nd never via Python's import?

@sean-bennett112
Copy link

@dotlambda credstash can also be used via Python's import. We're currently using it at work in this way.

@ivanbrennan
Copy link
Member Author

Same here. We have some internal tools that use the library to interact with credstash.

@dotlambda dotlambda merged commit ca8b128 into NixOS:master Dec 11, 2018
@ivanbrennan ivanbrennan deleted the credstash-application branch December 11, 2018 11:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Credstash not runnable as an application
4 participants