EXPERIMENTAL: replace bash with oil #105233

zimbatm · 2020-11-28T14:46:58Z

http://www.oilshell.org/ is supposed to be compatible with Bash. Let's
find out if it's true!

Motivation for this change

Things done

http://www.oilshell.org/ is supposed to be compatible with Bash. Let's find out if it's true!

zimbatm · 2020-11-28T14:50:38Z

Don't mind me. https://twitter.com/thewayofbash/status/1332689420940701699?s=20

zimbatm · 2020-11-28T15:22:43Z

nix-build -A bash works.

$ nix path-info -Sshr .#bash
/nix/store/1b3308hdqqxzpjmn0d21d43ixwqv1iry-bash-oil           	760.0 	 39.2M
/nix/store/3g1qm2xf7ysfdnka7hb1mwgx8i4fgxl0-glibc-2.32         	 29.7M	 31.4M
/nix/store/a076ip3brcj4x5a1rnj2j5qisfqh9xgk-oil-0.8.4          	  4.1M	 39.2M
/nix/store/g4gk96z9i98hyyf2j4mzn8vvws6gb7i3-libidn2-2.3.0      	217.5K	  1.8M
/nix/store/ma8xwfjik16p74wxdj640d7mk8wj58z9-libunistring-0.9.10	  1.6M	  1.6M
/nix/store/ydziqgw4hb71mzwgigm641m1wm1isfa5-readline-6.3p08    	382.7K	 35.2M
/nix/store/zg0npll5jbm6fbq6pahva3bhj1krmds3-ncurses-6.2        	  3.4M	 34.8M

zimbatm · 2020-11-28T15:30:57Z

It looks like oil doesn't support all of the bash startup flags. --norc, --noprofile and --rcfile seem to be quite important for interactive shells.

zimbatm · 2020-11-28T15:34:45Z

$ nix-build -A perl
warning: unknown setting 'extra-sandbox-paths'
these 2 derivations will be built:
  /nix/store/li1mf14aqavy7i3v93v3768ahqkxk1qh-stdenv-linux.drv
  /nix/store/9rw4di8zixd6cmvy8mzmsq12alwnvv18-perl-5.32.0.drv
warning: unknown setting 'extra-sandbox-paths'
building '/nix/store/li1mf14aqavy7i3v93v3768ahqkxk1qh-stdenv-linux.drv'...
oil: Invalid applet name 'bash'.
error: --- Error --- nix-daemon
builder for '/nix/store/li1mf14aqavy7i3v93v3768ahqkxk1qh-stdenv-linux.drv' failed with exit code 2; last 1 log lines:
  oil: Invalid applet name 'bash'.
error: --- Error ------------------------------------------------------------------------------------------------------------------------------------------------------------------ nix-build
1 dependencies of derivation '/nix/store/9rw4di8zixd6cmvy8mzmsq12alwnvv18-perl-5.32.0.drv' failed to build

zimbatm · 2020-11-28T21:48:16Z

$ nix-build -A perl
these 2 derivations will be built:
  /nix/store/wmfabvifpi3ckzm0r125c1j22wpdd1dk-stdenv-linux.drv
  /nix/store/s6s8rz8fwbxjnpgnwjxchg8s112rs0pk-perl-5.32.0.drv
warning: unknown setting 'extra-sandbox-paths'
building '/nix/store/wmfabvifpi3ckzm0r125c1j22wpdd1dk-stdenv-linux.drv'...
error: --- Error --- nix-daemon
output '/nix/store/mn1zxrxvk2m2fmpq5sq0snpizjqsjlr8-stdenv-linux' is not allowed to refer to the following paths:
  /nix/store/6a1vdcysqv997m920i793iigwnrpmhrn-oil-0.8.4
  /nix/store/8dpaz0i0cz6p4cnjfmvdvh959zqrc0jq-readline-6.3p08
  /nix/store/ddak93ycd54x6m5d6mp5sl6m4x0rgc39-ncurses-6.2
error: --- Error ------------------------------------------------------------------- nix-build
1 dependencies of derivation '/nix/store/s6s8rz8fwbxjnpgnwjxchg8s112rs0pk-perl-5.32.0.drv' failed to build

zimbatm · 2020-11-28T21:51:37Z

$ nix-build -A perl
this derivation will be built:
  /nix/store/02rda66hvdjar8x6rkqhf3g3sy789v00-perl-5.32.0.drv
warning: unknown setting 'extra-sandbox-paths'
building '/nix/store/02rda66hvdjar8x6rkqhf3g3sy789v00-perl-5.32.0.drv'...
type: 'preHook' not found
type: 'addInputsHook' not found
      local -ri hostOffset="$2"
            ^~~
/nix/store/pbx08z51536fi4fp09k8rbqqj9j2d1zr-stdenv-linux/setup:356: 'local' doesn't accept flag -i
      local -ri hostOffset="$2"
      ^~~~~
/nix/store/pbx08z51536fi4fp09k8rbqqj9j2d1zr-stdenv-linux/setup:356: fatal: Exiting with status 2 (command in PID 1)
  source $stdenv/setup
  ^~~~~~
/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh:1: fatal: Exiting with status 2 (command in PID 1)
type: 'failureHook' not found
error: --- Error ------------------------------------------------------------------- nix-build
builder for '/nix/store/02rda66hvdjar8x6rkqhf3g3sy789v00-perl-5.32.0.drv' failed with exit code 2; last 10 log lines:
        local -ri hostOffset="$2"
              ^~~
  /nix/store/pbx08z51536fi4fp09k8rbqqj9j2d1zr-stdenv-linux/setup:356: 'local' doesn't accept flag -i
        local -ri hostOffset="$2"
        ^~~~~
  /nix/store/pbx08z51536fi4fp09k8rbqqj9j2d1zr-stdenv-linux/setup:356: fatal: Exiting with status 2 (command in PID 1)
    source $stdenv/setup
    ^~~~~~
  /nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh:1: fatal: Exiting with status 2 (command in PID 1)
  type: 'failureHook' not found

zimbatm · 2020-11-28T22:19:33Z

currently stuck on the local -i compatibility: oilshell/oil#864

grahamc · 2020-11-29T13:51:40Z

I think osh isn't intended to be interactive, so I guess that is a thing.

see oilshell/oil#864

happysalada · 2021-05-05T23:42:08Z

Interested in this!
From what I gather from the issue, I guess that won't be possible though right?
It would require removing all the -i flags.

zimbatm · 2021-05-06T07:57:59Z

Go for it. -i is relatively rare so if you manage to get past the stdenv, a large part of the tree should build just fine.

It might be worth submitting the -i removal to the stdenv as the -i semantic is weird (see oilshell/oil#864). declare -i foo=foo doesn't even throw an error, it just converts the value to zero.

happysalada · 2021-05-14T04:47:49Z

Ok, I was a little too optimistic. There is quite some work to do the beam at the moment. I have to put this in the back burner for now.

zimbatm · 2021-05-14T11:25:06Z

Did you gather some additional notes in the process?

Maybe that's something that can be tackled collectively if we all add a little stone to the road.

happysalada · 2021-05-14T11:50:57Z

unfortunately not, I went down a rabbit hole of researching oil, the new language andrew wants to make.
I thought instead of osh, why not go the full way into oil.
My take on it is that it's a little early to start thinking about switching out of bash.

happysalada · 2021-06-22T00:44:11Z

I'm giving this another go.
Here are things that could be interesting.

I'm using shellcheck to fix some lints https://www.shellcheck.net/
You definitely want to turn on debugging statements for the shell. There is a special NIX_DEBUG var, but I haven't found how to use it. a simple env NIX_DEBUG=10 nix-build -A perl does not work. What I do instead is use this as the line of osh execution exec ${oil}/bin/osh -x -O strict:all "\$@". -x to turn on debugging. I'm trying strict:all to see if more "bugs" wouldn't appear.
My rationale so far.
- Split this into 2 parts, the first part will just a bunch of fixes to .sh files. Those fixes would consist of a mix of shellcheck lints and some things that are necessary to run osh. If we include shellcheck lints, perhaps people would be less reticent in including some of the things that are only needed to make osh run. For example removing the -i flag on defining a variable. It definitely makes the intent less clear, so I'm thinking people might object to this.

nix-build -A bash does indeed work with no problems.
nix-build -A perl fails on stage 3. I am trying something else before I post the error. Every build takes about 2 hours on machine.
The next failure is related to a problematic reference on burning oil. The stdenv stages allowedRequisites attribute need to be modified I think.
I'm going to open a separate PR, I hope it will be clearer.

happysalada · 2021-06-22T01:36:22Z

just opened a PR here #127736
I will post the errors as I go along.

stale · 2022-01-08T23:07:00Z

I marked this as stale due to inactivity. → More info

happysalada · 2022-01-09T14:17:32Z

Still working on that, albeit, slowly.

adamcstephens · 2023-03-17T12:15:23Z

Is this still being worked on? Is it replaced by #131676 ?

happysalada · 2023-03-17T12:19:42Z

@adamcstephens thanks for coming around!
I've tried to replace bash with oil, I've even made an RFC about it. However the community didn't really want to go that route. They mostly felt that nix was complicated enough as it was and were scared that oil is small and could possibly stop being maintained.
While oil is getting better and better, I still don't think that the nixos community would be ready to accept it unfortunately (only my opinion though).

adamcstephens · 2023-03-17T12:26:10Z

@happysalada makes sense. Should we close this and the other attempts at that switch then?

happysalada · 2023-03-17T12:35:28Z

Depends on what the intent is, if the intent is to try to reduce the number of PRs open, then sure.
The only reason I left mine open was to leave some potential clues for people who ever work on bash. Also as a potential reference if this changes in the future (if oil gets some cool new feature).
Personally, I'm happy to close my PR.

zimbatm · 2023-05-06T19:41:22Z

closing as I'm not going to keep working on this, it was a fun experiment

EXPERIMENTAL: replace bash with oil

996682e

http://www.oilshell.org/ is supposed to be compatible with Bash. Let's find out if it's true!

zimbatm force-pushed the burning-oil branch from 6cd177b to 996682e Compare November 28, 2020 14:49

fix for NixOS

fe3c4f1

zimbatm added 2 commits November 28, 2020 16:44

try to fix 'oil: Invalid applet name 'bash'.'

ef4659c

fixup! try to fix 'oil: Invalid applet name 'bash'.'

d076858

stdenv: disable allowedRequisites for now

e1849bb

ofborg bot added the 6.topic: stdenv Standard environment label Nov 28, 2020

oil: 0.8.4 -> 0.8.5

8316cce

zimbatm mentioned this pull request Nov 28, 2020

local -i not supported oilshell/oil#864

Closed

zimbatm added 2 commits November 30, 2020 18:51

tree-wide: don't use 'local -i'

3706a31

see oilshell/oil#864

something is wrong with the stdenv

773e80f

SuperSandro2000 added the 2.status: merge conflict label Jan 18, 2021

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 8, 2022

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 9, 2022

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 10, 2022

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 17, 2023

infinisil added the significant Novel ideas, large API changes, notable refactorings, issues with RFC potential, etc. label Apr 19, 2023

zimbatm closed this May 6, 2023

zimbatm deleted the burning-oil branch May 6, 2023 19:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EXPERIMENTAL: replace bash with oil #105233

EXPERIMENTAL: replace bash with oil #105233

zimbatm commented Nov 28, 2020

zimbatm commented Nov 28, 2020

zimbatm commented Nov 28, 2020 •

edited

zimbatm commented Nov 28, 2020

zimbatm commented Nov 28, 2020

zimbatm commented Nov 28, 2020

zimbatm commented Nov 28, 2020

zimbatm commented Nov 28, 2020

grahamc commented Nov 29, 2020

happysalada commented May 5, 2021

zimbatm commented May 6, 2021

happysalada commented May 14, 2021

zimbatm commented May 14, 2021

happysalada commented May 14, 2021

happysalada commented Jun 22, 2021

happysalada commented Jun 22, 2021

stale bot commented Jan 8, 2022

happysalada commented Jan 9, 2022

adamcstephens commented Mar 17, 2023

happysalada commented Mar 17, 2023

adamcstephens commented Mar 17, 2023

happysalada commented Mar 17, 2023

zimbatm commented May 6, 2023

EXPERIMENTAL: replace bash with oil #105233

EXPERIMENTAL: replace bash with oil #105233

Conversation

zimbatm commented Nov 28, 2020

Motivation for this change

Things done

zimbatm commented Nov 28, 2020

zimbatm commented Nov 28, 2020 • edited

zimbatm commented Nov 28, 2020

zimbatm commented Nov 28, 2020

zimbatm commented Nov 28, 2020

zimbatm commented Nov 28, 2020

zimbatm commented Nov 28, 2020

grahamc commented Nov 29, 2020

happysalada commented May 5, 2021

zimbatm commented May 6, 2021

happysalada commented May 14, 2021

zimbatm commented May 14, 2021

happysalada commented May 14, 2021

happysalada commented Jun 22, 2021

happysalada commented Jun 22, 2021

stale bot commented Jan 8, 2022

happysalada commented Jan 9, 2022

adamcstephens commented Mar 17, 2023

happysalada commented Mar 17, 2023

adamcstephens commented Mar 17, 2023

happysalada commented Mar 17, 2023

zimbatm commented May 6, 2023

zimbatm commented Nov 28, 2020 •

edited