Skip to content

Instantly share code, notes, and snippets.

@dch

dch/jr.fish

Last active September 27, 2023 17:44
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dch/f7ba04a57f3bfca68e77e37e81bff1d1 to your computer and use it in GitHub Desktop.
Save dch/f7ba04a57f3bfca68e77e37e81bff1d1 to your computer and use it in GitHub Desktop.
Download ZIP
see https://git.sr.ht/~dch/diy-jails/ for some background
Raw
jr.fish
function jr
set local
set JAIL $argv[1]
set INSTANCE (jls -j $JAIL path)
pkill -TERM -j $JAIL
sleep 3
doas jail -rv $JAIL
doas umount -f $INSTANCE/tmp
doas umount -f $INSTANCE/dev
end
Raw
jz.fish
function jz
# see https://git.io/JfHOu
set local
set MACH (sysctl -n hw.machine)
set ARCH (sysctl -n hw.machine_arch)
set DOMAIN skunkwerks.at
set RELEASE 13.2-RELEASE
set RELDATE 1302001
set FLAVOUR $RELEASE-$MACH-$ARCH
# grab the name
set JAIL $argv[1]
# the container
set TEMPLATE /jails/templates/$FLAVOUR
set INSTANCE /jails/instances/$FLAVOUR/$JAIL
# find the zpool for jails and jailed datasets
set ZPOOL (zfs list -Ho name /jails | cut -f 1 -d /)
# create a /jailed dataset to contain optional per-jail jailable datasets
# this can be located on an alternative zpool but must not be mountable
zfs list -Ho name -d1 |grep /jailed\$ > /dev/null 2>&1 \
|| doas zfs create -o mountpoint=none -o canmount=off $ZPOOL/jailed
# re-fetch after possible creation
set JAILED (zfs list -Ho name -d1 | grep /jailed\$)
# create mountpoint if missing
zfs list -Ho name $ZPOOL/jails > /dev/null 2>&1 \
|| doas zfs create -o mountpoint=/jails -p $ZPOOL/jails
# clone a template if missing
test -d $INSTANCE \
|| doas zfs clone $ZPOOL$TEMPLATE@ready $ZPOOL$INSTANCE
# add a softlink if missing
test -L /jails/tags/$JAIL || ln -s $INSTANCE /jails/tags/$JAIL
# derive a hopefully non-clashing IP address
set IPA (echo -n $JAIL | sha512 | cut -c 1,2)
set IPB (echo -n $JAIL | sha512 | cut -c 3,4)
set IP6 $IPA$IPB
set IP (printf "%d.%d" 0x$IPA 0x$IPB)
# get the zt 6plane
set IFACE (ifconfig | sort -r | egrep -o '^zt[a-z0-9]+' | head -1)
set SUBNET (ifconfig $IFACE inet6 | rg -Po 'fc\S+::' | uniq)
set ALIAS "$IFACE|$SUBNET$IP6"
# display config and try to start it
echo name=$JAIL ip=$IP ip6=$IP6 iface=$IFACE subnet=$SUBNET alias=$ALIAS
jls -j $JAIL > /dev/null 2>&1
if test $status = 0
doas jexec $JAIL tmux -u new-session -DAs default
else
doas jail -vcmr \
name=$JAIL \
path=$INSTANCE \
exec.created="zfs jail $JAIL $JAILED/$JAIL || true" \
exec.created="test -c $INSTANCE/dev/null || mount -t devfs devfs $INSTANCE/dev" \
exec.created="test -c $INSTANCE/tmp || mount -t tmpfs tmpfs $INSTANCE/tmp" \
exec.start="/bin/sh /etc/rc" \
exec.poststop="umount $INSTANCE/tmp" \
exec.poststop="umount $INSTANCE/dev" \
exec.poststop="logger -p info -t jz removed $JAIL" \
host.domainname=$DOMAIN \
host.hostid=0 \
host.hostname=$JAIL.$DOMAIN \
host.hostuuid=(uuidgen) \
linux.osname="tag=www" \
ip4.addr="lo1|100.64.$IP" \
ip6.addr="$ALIAS" \
allow.mount.nodevfs \
allow.mount.nofdescfs \
allow.mount.nolinprocfs \
allow.mount.nonullfs \
allow.mount.noprocfs \
allow.mount.notmpfs \
allow.nochflags \
allow.nomlock \
allow.noquotas \
allow.noread_msgbuf \
allow.nosocket_af \
allow.nosysvipc \
allow.raw_sockets \
allow.reserved_ports \
allow.sysvipc=1 \
allow.noset_hostname \
children.max=0 \
osreldate=$RELDATE \
osrelease=$RELEASE \
persist \
sysvmsg=disable \
sysvsem=disable \
devfs_ruleset=4 \
enforce_statfs=1 \
securelevel=2 \
allow.mount.zfs \
allow.mount \
command=/bin/sh -c "env -i TERM=$TERM tmux -u new-session -DAs default"
end
# command=/bin/sh
# allow.nounprivileged_proc_debug \
# allow.novmm \
end
@dch
Copy link
Author

dch commented Feb 3, 2022 •
edited

Ideas for improved FreeBSD kernel jail support

  • how to pass env vars through via jexec? we need an allowlist
  • single user id in jail, no exec of root at all aallowed
  • way to "map" a uuid from a zfs dataset to a randomly assigned one in a jail
  • uuids for jails please, or at least names, and not specify jail id
  • create jails by non-privileged user (means we inherit pf rules, zfs delegation, predefined)
  • need to mount both devfs & tmpfs in prior to starting jail app
  • attach random info to each jail, optionally to be made available inside the jail
  • a way to extract current state of all jails (or a subset thereof, more efficiently) to use for administration purposes
  • when jails get wedged during upgrades and restarts, most operators end up rebooting to cure it. This shouldn't need manual recovery each time it happens. Whether this is implementing a statemachine representation for jails, or providing a way to resume wedged operations after fixing this, something needs to be improved. Usually when this happens, its not a good time to get diagnostic information, or wait for a response on a mailing list to elucidate the breakage. Perhaps a coredump would be sufficient.

@dch
Copy link
Author

dch commented Nov 8, 2022

Quick intros

  • discuss some current issues with jails from each other
  • discuss next steps

My 2c

Infra today

  • private dns services & some FreeBSD CI
  • anycast + BGP ECMP to haproxy
  • plan in 2023 to commercialise (ie finish) both of these
  • jails are not dynamic enough for my usage

Wish list

  • looking to hook haproxy routing across multiple jail hosts and have dynamic state of active jails available to haproxy for routing decisions

  • need metadata to attach to jails

  • jail lifecycle mgmt support (jail post scripts don’t get run in some circumstances)

  • treat jails as simple packages (with optional zfs support)

  • jail JSON API for integration via unix socket

  • user space jails please

  • kernel enforced single uid jails (non root)

  • kernel-enforced UUIDs like flake

  • inbuilt lua scripting for state changes

@dch
Copy link
Author

dch commented Mar 8, 2023 •
edited

post jails call

  • libxo support needed
  • improved docs around jail / jail.conf needed
  • lot of shared logic (or even code) out there, albeit not intended
  • jail tools but not in shell
  • what could we add to base (or userland)
  • /etc/start_if_interfacename for simplifying config
  • rootless jails
  • mac framework
  • user jails
  • attach misc config data via https://man.freebsd.org/osd
  • ucl config + live json state

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment