Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vim_configurable: restore ability to override python for modules #40920

Merged
merged 1 commit into from May 30, 2018

Conversation

Ma27
Copy link
Member

@Ma27 Ma27 commented May 22, 2018

Motivation for this change

It seems as Python will be fetched from $PATH in Vim 8.1:

stat("/home/ma27/bin/python", 0x7ffe57a317b0) = -1 ENOENT (No such file or directory)
stat("/run/wrappers/bin/python", 0x7ffe57a317b0) = -1 ENOENT (No such file or directory)
stat("/home/ma27/.nix-profile/bin/python", 0x7ffe57a317b0) = -1 ENOENT (No such file or directory)
stat("/nix/var/nix/profiles/default/bin/python", 0x7ffe57a317b0) = -1 ENOENT (No such file or directory)
stat("/run/current-system/sw/bin/python", {st_mode=S_IFREG|0555, st_size=291, ...}) = 0
readlink("/run/current-system/sw/bin/python", "/nix/store/ggjkqbvwnv7dflkmdgmmp"..., 4096) = 72

This breaks in cases where you want to use a modified Python derivation
for the VIM plugins you use in vim_configurable:

let
  vim_configurable' = vim_configurable.override {
    # python with modules for ensime
    python = python.withPackages (ps: with ps; [ sexpdata websocket_client ]);
  };
in
  vim_configurable'.customize {
    # ...
  }

With VIM 8.0 this worked perfectly fine, now it's necessary to install
the modified python in $PATH to actually use it, otherwise an error
like this arises:

[ensime] A dependency is missing, please `pip2 install sexpdata websocket-client` and restart Vim.
Press ENTER or type command to continue

However it should be possible to pass the modified Python to the
modules, the easiest workaround is to write a wrapper which prefixes
$PATH to have the Python derivation available.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

@Ma27
Copy link
Member Author

Ma27 commented May 22, 2018

I'll run nox-review locally just to be sure %)

EDIT:

lrwxrwxrwx 1 ma27 users 69 May 22 14:22 result -> /nix/store/3i6ghp533b3wi2r4190abwlz8g4p44q9-vim_configurable-8.1.0001
lrwxrwxrwx 1 ma27 users 57 May 22 14:22 result-2 -> /nix/store/377c9axaal9p25q7i0fl0hflsplwf0x3-ezquake-3.0.1
lrwxrwxrwx 1 ma27 users 67 May 22 14:22 result-3 -> /nix/store/agmfks6nk39kf046mwnd2xr3imx0qyxw-vim-plugin-names-to-nix
lrwxrwxrwx 1 ma27 users 69 May 22 14:22 result-4 -> /nix/store/aq0j0x9lp689qzsansm5pxlzq4fxs6bz-vim_configurable-8.1.0001

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: vim_configurable

Partial log (click to expand)

Try 'basename --help' for more information.
/nix/store/3i6ghp533b3wi2r4190abwlz8g4p44q9-vim_configurable-8.1.0001/share/vim/vim81/tools/demoserver.py: interpreter directive changed from "/usr/bin/python" to "/nix/store/l9j2jsc9flrbmpf799nw9wdq59gpwms8-python-2.7.14/bin/python"
/nix/store/3i6ghp533b3wi2r4190abwlz8g4p44q9-vim_configurable-8.1.0001/share/vim/vim81/tools/ref: interpreter directive changed from "/bin/sh" to "/nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/sh"
/nix/store/3i6ghp533b3wi2r4190abwlz8g4p44q9-vim_configurable-8.1.0001/share/vim/vim81/tools/vimspell.sh: interpreter directive changed from "/bin/sh" to "/nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/sh"
/nix/store/3i6ghp533b3wi2r4190abwlz8g4p44q9-vim_configurable-8.1.0001/share/vim/vim81/tools/vimm: interpreter directive changed from "/bin/sh" to "/nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/sh"
/nix/store/3i6ghp533b3wi2r4190abwlz8g4p44q9-vim_configurable-8.1.0001/bin/gvimtutor: interpreter directive changed from "/bin/sh" to "/nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/sh"
head: error writing 'standard output': Broken pipe
/nix/store/3i6ghp533b3wi2r4190abwlz8g4p44q9-vim_configurable-8.1.0001/bin/vimtutor: interpreter directive changed from " /bin/sh" to "/nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/sh"
checking for references to /build in /nix/store/3i6ghp533b3wi2r4190abwlz8g4p44q9-vim_configurable-8.1.0001...
/nix/store/3i6ghp533b3wi2r4190abwlz8g4p44q9-vim_configurable-8.1.0001

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: vim_configurable

Partial log (click to expand)

/nix/store/b1qa158dbkiygmy9jagx5k4yj5nmvwzh-vim_configurable-8.1.0001/bin/vimtutor: interpreter directive changed from " /bin/sh" to "/nix/store/l4w7xwjy2nmk31fl5kgyy7gg8z7l9n8z-bash-4.4-p19/bin/sh"
/nix/store/b1qa158dbkiygmy9jagx5k4yj5nmvwzh-vim_configurable-8.1.0001/share/vim/vim81/tools/vimspell.sh: interpreter directive changed from "/bin/sh" to "/nix/store/l4w7xwjy2nmk31fl5kgyy7gg8z7l9n8z-bash-4.4-p19/bin/sh"
/nix/store/b1qa158dbkiygmy9jagx5k4yj5nmvwzh-vim_configurable-8.1.0001/share/vim/vim81/tools/vimm: interpreter directive changed from "/bin/sh" to "/nix/store/l4w7xwjy2nmk31fl5kgyy7gg8z7l9n8z-bash-4.4-p19/bin/sh"
/nix/store/b1qa158dbkiygmy9jagx5k4yj5nmvwzh-vim_configurable-8.1.0001/share/vim/vim81/tools/ref: interpreter directive changed from "/bin/sh" to "/nix/store/l4w7xwjy2nmk31fl5kgyy7gg8z7l9n8z-bash-4.4-p19/bin/sh"
basename: invalid option -- 'w'
Try 'basename --help' for more information.
/nix/store/b1qa158dbkiygmy9jagx5k4yj5nmvwzh-vim_configurable-8.1.0001/share/vim/vim81/tools/demoserver.py: interpreter directive changed from "/usr/bin/python" to "/nix/store/vhz9s1dzzbndz032dysnixj8mxyfl30s-python-2.7.14/bin/python"
/nix/store/b1qa158dbkiygmy9jagx5k4yj5nmvwzh-vim_configurable-8.1.0001/share/vim/vim81/macros/less.sh: interpreter directive changed from "/bin/sh" to "/nix/store/l4w7xwjy2nmk31fl5kgyy7gg8z7l9n8z-bash-4.4-p19/bin/sh"
checking for references to /build in /nix/store/b1qa158dbkiygmy9jagx5k4yj5nmvwzh-vim_configurable-8.1.0001...
/nix/store/b1qa158dbkiygmy9jagx5k4yj5nmvwzh-vim_configurable-8.1.0001

@Mic92
Copy link
Member

Mic92 commented May 22, 2018

cc @LnL7 @FRidh

@Mic92
Copy link
Member

Mic92 commented May 22, 2018

In neovim this is done by python_host_prog and python3_host_prog but I suppose this is not possible with vim. There might be user who don't want the python passed to during build time and instead use the python set by virtualenv/nix-shell. Is there a better way to distinguish the python version used by vim internally from the python executable in $PATH?

@Ma27
Copy link
Member Author

Ma27 commented May 22, 2018

It seems as the issue doesn't exist in neovim as it allows an extra property named pythonDependencies to specify additional python modules (see pkgs/applications/editors/neovim/wrapper.nix line 33)

There might be user who don't want the python passed to during build time and instead use the python set by virtualenv/nix-shell. Is there a better way to distinguish the python version used by vim internally from the python executable in $PATH?

When you build the code for vim_configurbale I posted above with VIM 8.0 everything works perfectly fine which means that the python derivation from the override was used for vim modules previously.

First of all the question is when do we want to use Python from the derivation and when the one from $PATH? Unfortunately I don't know too much about Vim's internals, so I'm not sure if there's an easy way to fix this 😅

@LnL7
Copy link
Member

LnL7 commented May 22, 2018

There might be user who don't want the python passed to during build time and instead use the python set by virtualenv/nix-shell.

Yeah, I'm not sure if creating the wrapper unconditionally is a good idea. Creating a wrapper yourself is pretty straightforward, while disabling it would be pretty tricky.

@Ma27
Copy link
Member Author

Ma27 commented May 22, 2018

Yeah, I'm not sure if creating the wrapper unconditionally is a good idea. Creating a wrapper yourself is pretty straightforward, while disabling it would be pretty tricky.

I absolutely understand your point. The reason why I filed this patch was to retain the behavior with VIM 8.0 where the python argument from vim_configurable.override was used for VIM scripts.
Admittedly I'm not sure how to proceed here, so do you think it's easier/better to create the wrapper locally or should we somehow provide this functionality in nixpkgs? Unless I'd close this for now %)

@Ma27
Copy link
Member Author

Ma27 commented May 24, 2018

As I'm currently using an altered python in $PATH (which contains sixpdata and websocket_client modules for ensime) I stumbled across another issue: what if I'm in a nix-shell (with a different python) and I simply want to edit a file with my VIM? Then the setup breaks again as VIM uses the Python from $PATH and yields an error message before opening (and makes ensime unusable).

As I already said, I'm open for further discussion (we can even make this behavior optional, right?), I'm just not sure if we want to use the "new" behavior of VIM 8.1 or keep the behavior of VIM 8.0 where the python from vim_configurable.override { python = ... } was used for plugins.

@Ma27
Copy link
Member Author

Ma27 commented May 29, 2018

ping @LnL7

@LnL7
Copy link
Member

LnL7 commented May 29, 2018

Adding an option for the wrapper so it can be overridden sounds fine to me. Maybe we should keep it disabled by default since using PATH is now the default upstream, unless everybody thinks the wrapper makes more sense.

It seems as Python will be fetched from $PATH in Vim 8.1:

```
stat("/home/ma27/bin/python", 0x7ffe57a317b0) = -1 ENOENT (No such file or directory)
stat("/run/wrappers/bin/python", 0x7ffe57a317b0) = -1 ENOENT (No such file or directory)
stat("/home/ma27/.nix-profile/bin/python", 0x7ffe57a317b0) = -1 ENOENT (No such file or directory)
stat("/nix/var/nix/profiles/default/bin/python", 0x7ffe57a317b0) = -1 ENOENT (No such file or directory)
stat("/run/current-system/sw/bin/python", {st_mode=S_IFREG|0555, st_size=291, ...}) = 0
readlink("/run/current-system/sw/bin/python", "/nix/store/ggjkqbvwnv7dflkmdgmmp"..., 4096) = 72
```

This breaks in cases where you want to use a modified Python derivation
for the VIM plugins you use in `vim_configurable`:

```
let
  vim_configurable' = vim_configurable.override {
    # python with modules for ensime
    python = python.withPackages (ps: with ps; [ sexpdata websocket_client ]);
  };
in
  vim_configurable'.customize {
    # ...
  }
```

With VIM 8.0 this worked perfectly fine, now it's necessary to install
the modified `python` in $PATH to actually use it, otherwise an error
like this arises:

```
[ensime] A dependency is missing, please `pip2 install sexpdata websocket-client` and restart Vim.
Press ENTER or type command to continue
```

However it should be possible to pass the modified Python to the
modules, the easiest workaround is to write a wrapper which prefixes
$PATH to have the Python derivation available.
@Ma27 Ma27 force-pushed the vim-configurable-python-override branch from 4a0395b to f43446c Compare May 30, 2018 07:24
@Ma27
Copy link
Member Author

Ma27 commented May 30, 2018

@LnL7 you're right, the behavior is optional now %)

@GrahamcOfBorg
Copy link

Success on x86_64-linux (full log)

Attempted: vim_configurable

Partial log (click to expand)

/nix/store/za5xymm6hzll89apifwn3kjjblgjj8rr-vim_configurable-8.1.0001/share/vim/vim81/tools/demoserver.py: interpreter directive changed from "/usr/bin/python" to "/nix/store/xlc7ld1xd57ha3x9mbx11c84j290va0f-python-2.7.15/bin/python"
basename: invalid option -- 'w'
Try 'basename --help' for more information.
/nix/store/za5xymm6hzll89apifwn3kjjblgjj8rr-vim_configurable-8.1.0001/share/vim/vim81/tools/ref: interpreter directive changed from "/bin/sh" to "/nix/store/m47apl3hq3i52fzy2cz24378p0xn4lyx-bash-4.4-p19/bin/sh"
/nix/store/za5xymm6hzll89apifwn3kjjblgjj8rr-vim_configurable-8.1.0001/share/vim/vim81/tools/vimm: interpreter directive changed from "/bin/sh" to "/nix/store/m47apl3hq3i52fzy2cz24378p0xn4lyx-bash-4.4-p19/bin/sh"
/nix/store/za5xymm6hzll89apifwn3kjjblgjj8rr-vim_configurable-8.1.0001/share/vim/vim81/tools/vimspell.sh: interpreter directive changed from "/bin/sh" to "/nix/store/m47apl3hq3i52fzy2cz24378p0xn4lyx-bash-4.4-p19/bin/sh"
/nix/store/za5xymm6hzll89apifwn3kjjblgjj8rr-vim_configurable-8.1.0001/bin/vimtutor: interpreter directive changed from " /bin/sh" to "/nix/store/m47apl3hq3i52fzy2cz24378p0xn4lyx-bash-4.4-p19/bin/sh"
/nix/store/za5xymm6hzll89apifwn3kjjblgjj8rr-vim_configurable-8.1.0001/bin/gvimtutor: interpreter directive changed from "/bin/sh" to "/nix/store/m47apl3hq3i52fzy2cz24378p0xn4lyx-bash-4.4-p19/bin/sh"
checking for references to /build in /nix/store/za5xymm6hzll89apifwn3kjjblgjj8rr-vim_configurable-8.1.0001...
/nix/store/za5xymm6hzll89apifwn3kjjblgjj8rr-vim_configurable-8.1.0001

@GrahamcOfBorg
Copy link

Success on aarch64-linux (full log)

Attempted: vim_configurable

Partial log (click to expand)

/nix/store/azvwcm4fy9ng848ydha3wlw4zvwk6344-vim_configurable-8.1.0001/bin/vimtutor: interpreter directive changed from " /bin/sh" to "/nix/store/vnb8q2h7951gd551nm2vq2g6n8296g5b-bash-4.4-p19/bin/sh"
/nix/store/azvwcm4fy9ng848ydha3wlw4zvwk6344-vim_configurable-8.1.0001/share/vim/vim81/tools/vimspell.sh: interpreter directive changed from "/bin/sh" to "/nix/store/vnb8q2h7951gd551nm2vq2g6n8296g5b-bash-4.4-p19/bin/sh"
/nix/store/azvwcm4fy9ng848ydha3wlw4zvwk6344-vim_configurable-8.1.0001/share/vim/vim81/tools/vimm: interpreter directive changed from "/bin/sh" to "/nix/store/vnb8q2h7951gd551nm2vq2g6n8296g5b-bash-4.4-p19/bin/sh"
/nix/store/azvwcm4fy9ng848ydha3wlw4zvwk6344-vim_configurable-8.1.0001/share/vim/vim81/tools/ref: interpreter directive changed from "/bin/sh" to "/nix/store/vnb8q2h7951gd551nm2vq2g6n8296g5b-bash-4.4-p19/bin/sh"
basename: invalid option -- 'w'
Try 'basename --help' for more information.
/nix/store/azvwcm4fy9ng848ydha3wlw4zvwk6344-vim_configurable-8.1.0001/share/vim/vim81/tools/demoserver.py: interpreter directive changed from "/usr/bin/python" to "/nix/store/1xkxxrd2sd8q78icm687ac8si9miak45-python-2.7.15/bin/python"
/nix/store/azvwcm4fy9ng848ydha3wlw4zvwk6344-vim_configurable-8.1.0001/share/vim/vim81/macros/less.sh: interpreter directive changed from "/bin/sh" to "/nix/store/vnb8q2h7951gd551nm2vq2g6n8296g5b-bash-4.4-p19/bin/sh"
checking for references to /build in /nix/store/azvwcm4fy9ng848ydha3wlw4zvwk6344-vim_configurable-8.1.0001...
/nix/store/azvwcm4fy9ng848ydha3wlw4zvwk6344-vim_configurable-8.1.0001

@LnL7 LnL7 merged commit 3010d99 into NixOS:master May 30, 2018
@Ma27 Ma27 deleted the vim-configurable-python-override branch May 30, 2018 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants