appimage-run: unify appimageTools and appimage-run #81833

bignaux · 2020-03-05T17:54:28Z

Motivation for this change

appimage-run :

doesn't patchelf the appimage packer (?) so not using it using appimageTools method would prevent issue.
share same unpack/autopatchelf/... needs that appimageTools.
separate shell script let me fix with linter ( for example fi missing ), add custom hooks etc.

/home/genesis/tmp/''/home/genesis/.cache/appimage-run/37e15e46f3c5fb6958c22c752050f4effc096745afb3c4f6a97a5c98607bfbe6/squashfs-root/AppRun: line 101: 10002 Segmentation fault (core dumped) pidof appimaged 2> /dev/null

avoiding AppRun in both would avoid such useless issue, as issue on packer. I'm working on it too.

one goal here is provide an improved appimage-run closer to appimageTools so the first really help to package with the second.

Things done

unify both using a common shell script
rewrite a clean? cmdline option parsing via getopts
unify output of unpacking job
simplify appimage-run that it could now be merge in appimageTools (some comment?)
shellcheck checked!

cc @tilpner
cc @timokau
cc @jtojnar
cc @volth

probonopd · 2020-03-07T19:53:08Z

Just for the avoidance of doubt, the payload applications inside different AppImages may require the AppRun scripts do do various things, such as setting certain environment variables, in order for them to work. So I don't think there is a "one size fits all" solution to run all the payload applications in all the AppImages out there while ignoring their AppRun scripts.

bignaux · 2020-03-07T19:56:10Z

To help people figure out what AppRun setup in the env, i start write some appimage-run hook, here an example :

    [genesis@genlaptop:~/tmp]$ appimage-run pcloud
    sha256 = "37e15e46f3c5fb6958c22c752050f4effc096745afb3c4f6a97a5c98607bfbe6";
    pcloud installed in /home/genesis/.cache/appimage-run/37e15e46f3c5fb6958c22c752050f4effc096745afb3c4f6a97a5c98607bfbe6/squashfs-root
    ./AppRun: ligne 141: grep : commande introuvable
    ./AppRun: ligne 142: grep : commande introuvable
    exec=/home/genesis/.cache/appimage-run/37e15e46f3c5fb6958c22c752050f4effc096745afb3c4f6a97a5c98607bfbe6/squashfs-root/pcloud
    Down: Everything Downloaded| Up: Everything Uploaded, status is LOGIN_REQUIRED
    Update for linux-x64-prod-v1.7.2 is not available
    diff /tmp/tmp.lBqONdMRR5 /tmp/tmp.YxPdJZftPm
     
    [genesis@genlaptop:~/tmp]$ diff /tmp/tmp.lBqONdMRR5 /tmp/tmp.YxPdJZftPm
    18a19
    > GSETTINGS_SCHEMA_DIR=/home/genesis/.cache/appimage-run/37e15e46f3c5fb6958c22c752050f4effc096745afb3c4f6a97a5c98607bfbe6/squashfs-root/usr/share/glib-2.0/schemas:
    32c33
    < LD_LIBRARY_PATH=/run/opengl-driver/lib:/run/opengl-driver-32/lib:/usr/lib:/usr/lib32:/run/opengl-driver/lib:/run/opengl-driver-32/lib
    ---
    > LD_LIBRARY_PATH=/home/genesis/.cache/appimage-run/37e15e46f3c5fb6958c22c752050f4effc096745afb3c4f6a97a5c98607bfbe6/squashfs-root/usr/lib:/run/opengl-driver/lib:/run/opengl-driver-32/lib:/usr/lib:/usr/lib32:/run/opengl-driver/lib:/run/opengl-driver-32/lib
    53c54
    < PATH=/nix/store/wicywbih6vklw00gzrmyw5m8gvwhm6rn-diffutils-3.7/bin:/nix/store/l69pbxhsy8z3z7gymlhm88bsx5h53938-strace-5.5/bin:/nix/store/cipina59rj22f8rv8lv1311hnc1d64xv-ripgrep-11.0.2/bin:/nix/store/d7wqn6pn23m6ijj4qxmnyya7313vx4lc-file-5.38/bin:/nix/store/n3kmvnsy2yd4m347mynk35mvy8mfjf8d-radare2-4.2.1/bin:/nix/store/dv602dl9y4aks8w6w2dvfm13pzxrh7v2-libarchive-3.4.2/bin:/nix/store/2bdlpp1xvl971l25h8l9d4664jd8z8ad-jq-1.6-bin/bin:/nix/store/ix2bqs08pknlarlqbdpm7mcfdxk01xy1-squashfs-4.4/bin:/nix/store/xmdf059nf5a1763w21gpc7ifizwk4b3p-coreutils-8.31/bin:/nix/store/8gy2bmpz3qawxr3g8hqhfgkf32wb0wfl-bash-4.4-p23/bin:/home/genesis/.cache/appimage-run/37e15e46f3c5fb6958c22c752050f4effc096745afb3c4f6a97a5c98607bfbe6/squashfs-root/usr/bin
    ---
    > PATH=/home/genesis/.cache/appimage-run/37e15e46f3c5fb6958c22c752050f4effc096745afb3c4f6a97a5c98607bfbe6/squashfs-root:/home/genesis/.cache/appimage-run/37e15e46f3c5fb6958c22c752050f4effc096745afb3c4f6a97a5c98607bfbe6/squashfs-root/usr/sbin:/nix/store/wicywbih6vklw00gzrmyw5m8gvwhm6rn-diffutils-3.7/bin:/nix/store/l69pbxhsy8z3z7gymlhm88bsx5h53938-strace-5.5/bin:/nix/store/cipina59rj22f8rv8lv1311hnc1d64xv-ripgrep-11.0.2/bin:/nix/store/d7wqn6pn23m6ijj4qxmnyya7313vx4lc-file-5.38/bin:/nix/store/n3kmvnsy2yd4m347mynk35mvy8mfjf8d-radare2-4.2.1/bin:/nix/store/dv602dl9y4aks8w6w2dvfm13pzxrh7v2-libarchive-3.4.2/bin:/nix/store/2bdlpp1xvl971l25h8l9d4664jd8z8ad-jq-1.6-bin/bin:/nix/store/ix2bqs08pknlarlqbdpm7mcfdxk01xy1-squashfs-4.4/bin:/nix/store/xmdf059nf5a1763w21gpc7ifizwk4b3p-coreutils-8.31/bin:/nix/store/8gy2bmpz3qawxr3g8hqhfgkf32wb0wfl-bash-4.4-p23/bin:/home/genesis/.cache/appimage-run/37e15e46f3c5fb6958c22c752050f4effc096745afb3c4f6a97a5c98607bfbe6/squashfs-root/usr/bin
    63c64
    < SHLVL=2
    ---
    > SHLVL=3
    78c79
    < XDG_DATA_DIRS=/nix/store/kiwkgczbi8c2v9i5gy9x8vrrcmsvn3xd-qterminal-0.14.1/share:/nix/store/vxdlw8naw28f0df414vkadyd8g35vqs0-lxqt-runner-0.14.1/share:/nix/store/q1yjvfwy959jgajs0ijib3ndk3w5khds-lxqt-session-0.14.1/share:/home/genesis/.nix-profile/share:/etc/profiles/per-user/genesis/share:/nix/var/nix/profiles/default/share:/run/current-system/sw/share:/nix/store/q1yjvfwy959jgajs0ijib3ndk3w5khds-lxqt-session-0.14.1/share
    ---
    > XDG_DATA_DIRS=/home/genesis/.cache/appimage-run/37e15e46f3c5fb6958c22c752050f4effc096745afb3c4f6a97a5c98607bfbe6/squashfs-root/usr/share/:./share/:/usr/share/gnome:/usr/local/share/:/usr/share/:/nix/store/kiwkgczbi8c2v9i5gy9x8vrrcmsvn3xd-qterminal-0.14.1/share:/nix/store/vxdlw8naw28f0df414vkadyd8g35vqs0-lxqt-runner-0.14.1/share:/nix/store/q1yjvfwy959jgajs0ijib3ndk3w5khds-lxqt-session-0.14.1/share:/home/genesis/.nix-profile/share:/etc/profiles/per-user/genesis/share:/nix/var/nix/profiles/default/share:/run/current-system/sw/share:/nix/store/q1yjvfwy959jgajs0ijib3ndk3w5khds-lxqt-session-0.14.1/share:/usr/share/gnome/:/usr/local/share/:/usr/share/

probonopd · 2020-03-07T19:58:39Z

I don't quite understand what you are doing there but you seem to have a plan ;-)

bignaux · 2020-03-07T20:08:43Z

@probonopd : in order to make binaries working on an appimage, we have often to patchelf/patchshebang them with our nixos tools. To use the autopatch* , we have to move them in the right directories so we change the structure of the appimage anyway. I first remove the need to use the packer in a previous PR, then in this, i turn some nix to make easily the fixing of an appimage.

Edit : here i don't yet make any change to how the stuff runs currently. I try to keep really mirrored features exept removing /squashfs-root/ in appimage-run cache directories allowing by unsquashfs instead of packer extract feature option.

timokau · 2020-03-08T13:46:37Z

This sounds good to me in principle, but I don't know enough about appimage to actually review.

bignaux · 2020-03-08T15:31:49Z

@timokau : could it be you missed add "$@" to 19e075b#diff-ae84e7b07459c504b804db2a1f54636dR51 ?
and that @tilpner count on exec to not wrapping the next exec in an else case ? I'd like provide a better way to override the exec argument, perhaps letting people provide rarun2 options on cmd options and add some option in wrapAppImage() to override the runScript ?

timokau · 2020-03-08T16:42:38Z

Yeah, looks like I missed that. Good catch!

bignaux · 2020-03-08T17:09:52Z

Thanks @timokau , i don't touch that immediatly since it's the current behavior. i really think we would enjoy something like i said, but i'd like to listen people feeling.

wrap() {

  rarun2 setenv=APPIMAGE_SILENT_INSTALL=1 chdir="$APPDIR" program=./AppRun "$@"
}

bignaux · 2020-03-08T19:57:35Z

with this last commit, appimage-run doesn't need special (and hidden) option to run the script, so default behavior is how appimage-run users use it, so now it's easier to understand and hack.

Edit, so now, you can even do that :
appimage-run -w /home/genesis/.cache/appimage-run/0447c38529b8b443e93a1724c370b944dbaf7490c8a42495d2ac2d41ee76f7a3/

This illustrate that we can now improve the wrap() for both in same time.

Mic92 · 2020-03-09T10:47:37Z

pkgs/build-support/appimage/appimage-exec.sh

+  local appimageType=0
+
+  # https://github.com/AppImage/libappimage/blob/ca8d4b53bed5cbc0f3d0398e30806e0d3adeaaab/src/libappimage/utils/MagicBytesChecker.cpp#L45-L63
+  eval "$(r2 "$src" -nn -Nqc "p8j 3 @ 8" |


Does this just extract 3 bytes at file offset 8?

yes and it could be done with hexdump or xdd or dd. but here there is a homogeneity with r2+jq for carving elf file by design choice.

Mic92 · 2020-03-09T10:51:37Z

pkgs/build-support/appimage/appimage-exec.sh

+
+        # multiarch offset one-liner using same method as AppImage
+        # see https://gist.github.com/probonopd/a490ba3401b5ef7b881d5e603fa20c93
+        offset=$(r2 "$src" -nn -Nqc "pfj.elf_header @ 0" |\


Also not having a radare2 as an dependency would be nice, it might be good enough for now, given that getting this sort of information might be hard to extract using other tools.

This was already discussed and merge on previous PR.

tilpner

Still haven't done any actual testing

tilpner · 2020-03-09T11:03:42Z

pkgs/build-support/appimage/appimage-exec.sh

+
+apprun() {
+
+  eval "$(rahash2 "$APPIMAGE" -j | jq -r '.[] | @sh "SHA256=\(.hash)"')"


What's the difference/advantage to sha256="$(sha256sum "$APPIMAGE")"?

I try to do homogeneity here around r2+jq, so despite the way that you can easily change the hash algo, not an immediate interest. But people reading the script to modify would perhaps be more interesting to use rarun2 and others tools in radare2 if they see some subliminal examples around the code ....

tilpner · 2020-03-09T11:04:57Z

pkgs/build-support/appimage/appimage-exec.sh

+  else echo "$(basename "$APPIMAGE")" installed in "$APPDIR"
+  fi


Suggested change

else echo "$(basename "$APPIMAGE")" installed in "$APPDIR"

fi

else

echo "$(basename "$APPIMAGE")" installed in "$APPDIR"

fi

tilpner · 2020-03-09T11:08:21Z

pkgs/build-support/appimage/appimage-exec.sh

+fi
+
+PATH="@path@:$PATH"
+apprun_opt=true


This is never changed from true, and could be removed.

yes, but i thought to remove the -w option and rely on $1 type, then appimage-run <appimage|appimage directory> , but i'd more interested to do that after discussion how to write a useful wrap() function. i make test with rarun2 and will come with proposal. For now thanks to close the PR here, look if you would merge appimage-run in appimageTools, and finish wrapTypeX removing .

tilpner · 2020-03-09T11:08:44Z

pkgs/build-support/appimage/appimage-exec.sh

+PATH="@path@:$PATH"
+apprun_opt=true
+
+#DEBUG=0


This is probably a left-over?

more or less, the work is not yet finish, but i hope the step by step would help people understand the way i try to refactoring this stuff.

tilpner · 2020-03-09T11:09:56Z

pkgs/build-support/appimage/appimage-exec.sh

+#DEBUG=0
+
+# src : AppImage
+# dest : let's unpack() create the directory


Suggested change

# dest : let's unpack() create the directory

# out : target directory, assumed to not exist yet

tilpner · 2020-03-09T11:11:20Z

pkgs/build-support/appimage/appimage-exec.sh

+
+  case "$appimageType" in
+    1 ) echo "Uncompress $(basename "$src") of type $appimageType."
+        mkdir "$out"


Perhaps pass -p to handle $out existing already

i could not do that for unsquashfs, so i prefer not. the latest directory should not exist, that's a limitation of unsquashfs -d .

tilpner · 2020-03-09T11:13:27Z

pkgs/build-support/appimage/default.nix

+    src = ./appimage-exec.sh;
+    isExecutable = true;
+    dir = "bin";
+    path = with pkgs; lib.makeBinPath [ pv ripgrep file radare2 libarchive jq squashfsTools coreutils bash ];


Suggested change

path = with pkgs; lib.makeBinPath [ pv ripgrep file radare2 libarchive jq squashfsTools coreutils bash ];

path = lib.makeBinPath [ pv ripgrep file radare2 libarchive jq squashfsTools coreutils bash ];

tilpner · 2020-03-09T11:13:34Z

pkgs/build-support/appimage/default.nix

-        *) echo "Unsupported AppImage Type: $appimageType";;
-      esac
-    '';
+  appimage-exec = pkgs.substituteAll {


Suggested change

appimage-exec = pkgs.substituteAll {

appimage-exec = substituteAll {

tilpner · 2020-03-09T11:14:23Z

pkgs/build-support/appimage/appimage-exec.sh

@@ -0,0 +1,142 @@
+#!@shell@


Should this have a .sh suffix?

i think it's better to set in nixpkgs with the prefix, then it has no prefix when we use it through the wrapper.

tilpner · 2020-03-09T11:17:54Z

pkgs/build-support/appimage/appimage-exec.sh

+wrap() {
+
+  cd "$APPDIR" || exit
+  # quite same in appimageTools


This comment loses its meaning after this PR is merged

Mic92 · 2020-03-09T11:19:05Z

@tilpner sorry. I saw your review too late after I already tested and merged it.

appimage-run: unify appimageTools and appimage-run (cherry picked from commit 115a8e8)

andersk · 2021-07-19T23:19:28Z

pkgs/build-support/appimage/default.nix

-      cd $APPDIR
-      exec ./AppRun "$@"
-    '';
+    runScript = "appimage-exec.sh -w ${src}";


This needs a -- to prevent appimage-exec.sh from stealing further hyphenated arguments intended for the real executable: #128927

appimage-run: replace writeScript to a standalone appimage-exec.sh

349e923

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Mar 5, 2020

bignaux force-pushed the appimage-run branch from 9110ec2 to 5858a74 Compare March 6, 2020 04:10

ofborg bot added 10.rebuild-linux: 11-100 and removed 10.rebuild-linux: 1-10 labels Mar 6, 2020

bignaux force-pushed the appimage-run branch from 5858a74 to 8f8effa Compare March 6, 2020 05:58

bignaux changed the title ~~[WIP] : unify appimagetool and appimage-run~~ [WIP] : unify appimageTools and appimage-run Mar 6, 2020

bignaux force-pushed the appimage-run branch from 8f8effa to 436f6d6 Compare March 6, 2020 17:16

appimage: replace writeScript to a standalone appimage-exec.sh

b36d3f2

bignaux force-pushed the appimage-run branch from 436f6d6 to b36d3f2 Compare March 7, 2020 19:24

bignaux changed the title ~~[WIP] : unify appimageTools and appimage-run~~ appimage-run: unify appimageTools and appimage-run Mar 7, 2020

Bignaux Ronan added 2 commits March 8, 2020 13:38

appimageTools: fix appimage-exec.sh according to shellcheck

2a6cd0e

appimageTools: fix appimage-exec.sh passing arguments

95dc1ef

appimageTools: wrapType1 = wrapType2;

49e7871

bignaux force-pushed the appimage-run branch from edfc001 to 49e7871 Compare March 8, 2020 17:07

appimage-exec.sh: simplify and improve getopts options

a3df64c

Mic92 reviewed Mar 9, 2020

View reviewed changes

tilpner suggested changes Mar 9, 2020

View reviewed changes

Mic92 merged commit 115a8e8 into NixOS:master Mar 9, 2020

dtzWill pushed a commit to dtzWill/nixpkgs that referenced this pull request Mar 9, 2020

Merge pull request NixOS#81833 from bignaux/appimage-run

4aac91e

appimage-run: unify appimageTools and appimage-run (cherry picked from commit 115a8e8)

bignaux deleted the appimage-run branch March 10, 2020 17:47

andersk mentioned this pull request Jul 2, 2021

appimageTools.wrapAppImage: Fix passing arguments to wrapped executable #128927

Merged

11 tasks

andersk reviewed Jul 19, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

appimage-run: unify appimageTools and appimage-run #81833

appimage-run: unify appimageTools and appimage-run #81833

bignaux commented Mar 5, 2020 •

edited

probonopd commented Mar 7, 2020

bignaux commented Mar 7, 2020

probonopd commented Mar 7, 2020

bignaux commented Mar 7, 2020 •

edited

timokau commented Mar 8, 2020

bignaux commented Mar 8, 2020 •

edited

timokau commented Mar 8, 2020

bignaux commented Mar 8, 2020

bignaux commented Mar 8, 2020 •

edited

Mic92 Mar 9, 2020

bignaux Mar 9, 2020

Mic92 Mar 9, 2020 •

edited

bignaux Mar 9, 2020 •

edited

tilpner left a comment

tilpner Mar 9, 2020

bignaux Mar 9, 2020

tilpner Mar 9, 2020

bignaux Mar 9, 2020

tilpner Mar 9, 2020

bignaux Mar 9, 2020

tilpner Mar 9, 2020

bignaux Mar 9, 2020

tilpner Mar 9, 2020

tilpner Mar 9, 2020

bignaux Mar 9, 2020

tilpner Mar 9, 2020

tilpner Mar 9, 2020

tilpner Mar 9, 2020

bignaux Mar 9, 2020

tilpner Mar 9, 2020

Mic92 commented Mar 9, 2020

andersk Jul 19, 2021


		apprun() {

		eval "$(rahash2 "$APPIMAGE" -j \| jq -r '.[] \| @sh "SHA256=\(.hash)"')"

		else echo "$(basename "$APPIMAGE")" installed in "$APPDIR"
		fi

	# dest : let's unpack() create the directory
	# out : target directory, assumed to not exist yet

	path = with pkgs; lib.makeBinPath [ pv ripgrep file radare2 libarchive jq squashfsTools coreutils bash ];
	path = lib.makeBinPath [ pv ripgrep file radare2 libarchive jq squashfsTools coreutils bash ];

	appimage-exec = pkgs.substituteAll {
	appimage-exec = substituteAll {

appimage-run: unify appimageTools and appimage-run #81833

appimage-run: unify appimageTools and appimage-run #81833

Conversation

bignaux commented Mar 5, 2020 • edited

Motivation for this change

Things done

probonopd commented Mar 7, 2020

bignaux commented Mar 7, 2020

probonopd commented Mar 7, 2020

bignaux commented Mar 7, 2020 • edited

timokau commented Mar 8, 2020

bignaux commented Mar 8, 2020 • edited

timokau commented Mar 8, 2020

bignaux commented Mar 8, 2020

bignaux commented Mar 8, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 Mar 9, 2020 • edited

Choose a reason for hiding this comment

bignaux Mar 9, 2020 • edited

Choose a reason for hiding this comment

tilpner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 commented Mar 9, 2020

Choose a reason for hiding this comment

bignaux commented Mar 5, 2020 •

edited

bignaux commented Mar 7, 2020 •

edited

bignaux commented Mar 8, 2020 •

edited

bignaux commented Mar 8, 2020 •

edited

Mic92 Mar 9, 2020 •

edited

bignaux Mar 9, 2020 •

edited