Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve perl shebang lines by switching to use lib ...; #55786

Closed
wants to merge 1 commit into from

Conversation

cleverca22
Copy link
Contributor

@cleverca22 cleverca22 commented Feb 14, 2019

Motivation for this change

this stops the perl shebang lines from reaching 30kb in size

Things done

confirmed that nix-build -A biber builds and the existing tests in the perl stack still pass

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.

# TODO, handle this better?
# `perl5.28.1-Test-Simple-1.302141` bakes line numbers into the expected errors in tests
# inserting a "use lib ...;\n" into the file offsets everything and makes the tests fail
if [[ "$name" == *"Test-Simpl"* ]]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we're going to have to fix TestSimple before merging this :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we can just disable it's tests until we have a better solution.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better solution, just patch things after testing!

Copy link
Member

@matthewbauer matthewbauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! If we do decide to go this route, I would recommend adding some test or a setup-hook to make sure these lines don't exceed the 128 limit. I seem to think python had a similar issue?

@grahamc
Copy link
Member

grahamc commented Feb 14, 2019

fwiw @matthewbauer upstream is reverting or fixing the commit to retain the previous behavior, so it won't be such a hard blocker again

@grahamc
Copy link
Member

grahamc commented Feb 14, 2019

Only 10kb =) (wrt https://gist.githubusercontent.com/dtzWill/5d4324eb3ef9d3c7dfbb0f690768dc91/raw/217d62d8ff88a2579eaf79e5a7c35445044f0be3/biber)

[grahamc@Morbo:~]$ xclip -o | tr ' ' $'\n' | sort | uniq -c
      3 -I/nix/store/0cngi2knrii3l5a1lqwzcv2lm109zzgc-perl5.28.1-Class-Accessor-0.51/lib/perl5/site_perl
      3 -I/nix/store/0jl7i750haazdync46n2kgi3ka670fa1-perl5.28.1-Dist-CheckConflicts-0.11/lib/perl5/site_perl
      3 -I/nix/store/18p5vrk8hh2xisgnk7ay2s3y9cnpyj0y-perl5.28.1-XML-SAX-1.00/lib/perl5/site_perl
      3 -I/nix/store/1gh0gavqbnq7w5ylg66sm3khd61y7m35-perl5.28.1-List-UtilsBy-0.11/lib/perl5/site_perl
      3 -I/nix/store/1i72pvxdlz83szcnzwg3ls50209dmazw-perl5.28.1-Class-Factory-Util-1.7/lib/perl5/site_perl
      3 -I/nix/store/1ih3252a6f67rfspll2y8am1agpdb9z4-perl5.28.1-namespace-clean-0.27/lib/perl5/site_perl
      3 -I/nix/store/2y0gmw5pkr9fvwps94fk15k4mp4d337g-perl5.28.1-Test-Differences-0.64/lib/perl5/site_perl
      3 -I/nix/store/34m77dx60as5dml994cakadr4h44ayrx-perl5.28.1-Encode-Locale-1.05/lib/perl5/site_perl
      3 -I/nix/store/39zl4ly6418py3i92gwwyiisqb0qrjvf-perl5.28.1-Text-Glob-0.11/lib/perl5/site_perl
      3 -I/nix/store/3j73n8x0ypdb5424ja6fjfs1sh5ckjjx-perl5.28.1-DateTime-Format-Builder-0.81/lib/perl5/site_perl
      3 -I/nix/store/3x4kb5bhaww4wcsbp7zkmm91mlvsj0vn-perl5.28.1-Package-DeprecationManager-0.17/lib/perl5/site_perl
      3 -I/nix/store/42057h1j66iiyfn1z9y7lwngblnqqlrc-perl-5.28.1/lib/perl5/site_perl
      3 -I/nix/store/46fc5lam6i62fpp8r0phgfi2fn4gx1j0-perl5.28.1-Mozilla-CA-20180117/lib/perl5/site_perl
      3 -I/nix/store/48mgaxywv80gj8rr1cwijpcip16nga7y-perl5.28.1-Net-HTTP-6.18/lib/perl5/site_perl
      3 -I/nix/store/4bjclk09vy770ipy3apacbahpvhg8gxw-perl5.28.1-Readonly-2.05/lib/perl5/site_perl
      3 -I/nix/store/4jza2znyvh9dld95f8vgp9vz06nhm6b9-perl5.28.1-XML-LibXML-Simple-0.99/lib/perl5/site_perl
      3 -I/nix/store/57sl88xy8mk3wvnm916nqdfazw4nb0s5-perl5.28.1-Class-Data-Inheritable-0.08/lib/perl5/site_perl
      3 -I/nix/store/59kpppzfhzgnyavs8gsl76qzarwh45pb-perl5.28.1-IO-HTML-1.001/lib/perl5/site_perl
      3 -I/nix/store/5ch541w76scm4336ya3xi11nwwp7a1gv-perl5.28.1-Readonly-XS-1.05/lib/perl5/site_perl
      3 -I/nix/store/5ibgbxyymqmbxl8gnhqmf6vgk4sn0im4-perl5.28.1-File-Find-Rule-0.34/lib/perl5/site_perl
      3 -I/nix/store/5mg4rw61k3jhy2pb37lckgskhimcc1r0-perl5.28.1-List-MoreUtils-0.428/lib/perl5/site_perl
      3 -I/nix/store/639iwgfrdii3ra8vpwhzhij5yvya8lhw-perl5.28.1-HTTP-Message-6.18/lib/perl5/site_perl
      3 -I/nix/store/6cnm72nr82sakwz327s57yngw2h65vli-perl5.28.1-HTML-Tagset-3.20/lib/perl5/site_perl
      3 -I/nix/store/6h2ivl3fm6q8x2ijjkv7l1wcsji6jdy5-perl5.28.1-Tie-Cycle-1.225/lib/perl5/site_perl
      3 -I/nix/store/6nmr4f51lkr8akl43v3zqdirgdw4vl6g-perl5.28.1-Lingua-Translit-0.28/lib/perl5/site_perl
      3 -I/nix/store/7khf20ffgkgcldb9fgmx8rz8apc9hqh4-perl5.28.1-XML-LibXSLT-1.96/lib/perl5/site_perl
      3 -I/nix/store/7l4hb3r1mv52k3r6x5j7lidiqxjd8sh5-perl5.28.1-Params-ValidationCompiler-0.30/lib/perl5/site_perl
      3 -I/nix/store/88833ksa3z9s634ddwqh3hfii24yq6ad-perl5.28.1-HTML-Parser-3.72/lib/perl5/site_perl
      3 -I/nix/store/8fzxqdd0a3iy6kiy7kyidq38sb8xii84-perl5.28.1-Exception-Class-1.44/lib/perl5/site_perl
      3 -I/nix/store/8raz4cnrg1pimkzc7nm0ghh5n027nc1v-perl5.28.1-Sub-Name-0.21/lib/perl5/site_perl
      3 -I/nix/store/8xzb3qgddpjsfrb24209ia8rkvsn3yvi-perl5.28.1-Business-ISMN-1.201/lib/perl5/site_perl
      3 -I/nix/store/8z0805fr57kjf1yc8dikk7mkfwy1yya7-perl5.28.1-DateTime-Locale-1.23/lib/perl5/site_perl
      3 -I/nix/store/94fsaw2wy9d1x24zagzal7ygrvxrs9v4-perl5.28.1-Class-Singleton-1.5/lib/perl5/site_perl
      3 -I/nix/store/97gqg65gifi8sgn41b9p4v2kqn29fmlc-perl5.28.1-Class-Inspector-1.32/lib/perl5/site_perl
      3 -I/nix/store/97n9mgc4jhhs0zwdg86195lbblyblipv-perl5.28.1-Sub-Exporter-Progressive-0.001013/lib/perl5/site_perl
      3 -I/nix/store/9mn55sfk0ss6bx74bq1gnmnf3y6h2h40-perl5.28.1-Business-ISBN-Data-20140910.003/lib/perl5/site_perl
      3 -I/nix/store/9q36fwi1sicfzlwx9gz23h7x4xjwmf68-perl5.28.1-MRO-Compat-0.13/lib/perl5/site_perl
      3 -I/nix/store/9qwl1971q3pq93sy05dckxhg40vnzb6s-perl5.28.1-Exporter-Tiny-1.002001/lib/perl5/site_perl
      3 -I/nix/store/a2f3rrmfyfq8dbz8s4m6g5iif5q9h4bd-perl5.28.1-Capture-Tiny-0.48/lib/perl5/site_perl
      3 -I/nix/store/akdl5a2dd1zcw4ynczvilcxzhbcy4lgf-perl5.28.1-XML-NamespaceSupport-1.12/lib/perl5/site_perl
      3 -I/nix/store/b17pzjangcrfskn5yw5phjj73c7s54cz-perl5.28.1-List-AllUtils-0.15/lib/perl5/site_perl
      3 -I/nix/store/b2dpbig158wj9bw0pivqkl8899r3zcrf-perl5.28.1-Data-Dump-1.23/lib/perl5/site_perl
      3 -I/nix/store/b9jqldymii16356jzwbs0knq5g14cnhp-perl5.28.1-Business-ISSN-1.003/lib/perl5/site_perl
      3 -I/nix/store/bn4l2rjffbi0zmr2f95fpv011vjqm17s-perl5.28.1-Devel-StackTrace-2.03/lib/perl5/site_perl
      3 -I/nix/store/bn7l0z9rhwfkr5s60jg741w41g3s3px4-perl5.28.1-Business-ISBN-3.004/lib/perl5/site_perl
      3 -I/nix/store/c192qb0id1lz05gv622gs00kl98np36w-perl5.28.1-XML-LibXML-2.0132/lib/perl5/site_perl
      3 -I/nix/store/caivqiqxqhspkhgjlm5hzdd0qqkzwpfk-perl5.28.1-Text-CSV_XS-1.37/lib/perl5/site_perl
      3 -I/nix/store/d529gplavyz1glwiwhi771sqsn9cyndq-perl5.28.1-File-Which-1.22/lib/perl5/site_perl
      3 -I/nix/store/dggpizl8h4j3n7iqqpyvvcpb9m1sirxv-perl5.28.1-Module-Runtime-0.016/lib/perl5/site_perl
      3 -I/nix/store/fbjyl18jwrpy707i9djh05zfxg7bv7rx-perl5.28.1-LWP-Protocol-https-6.07/lib/perl5/site_perl
      3 -I/nix/store/fh7brbs6higgcr83r81baxs9f5x18v3v-perl5.28.1-libwww-perl-6.36/lib/perl5/site_perl
      3 -I/nix/store/fln0p97jjarbhzqn8x1brd7p8bv5c67r-perl5.28.1-Regexp-Common-2017060201/lib/perl5/site_perl
      3 -I/nix/store/gdmbihp48srcmbpgkz182l1x8r6ihvy3-perl5.28.1-Algorithm-Diff-1.1903/lib/perl5/site_perl
      3 -I/nix/store/gz02mz3mqipm9n34vf9hr40bjmhww1wm-perl5.28.1-Sub-Install-0.928/lib/perl5/site_perl
      3 -I/nix/store/h46144whkn0p9p86mgnvhn7j9ykbgpw3-perl5.28.1-DateTime-Calendar-Julian-0.100/lib/perl5/site_perl
      3 -I/nix/store/h53k6jskgrsz56lvgdaahypc5rwp9iv8-perl5.28.1-XML-Writer-0.625/lib/perl5/site_perl
      3 -I/nix/store/h8jhbm9jvrzai1jvq4137qkjwlpkgw3m-perl5.28.1-Module-Implementation-0.09/lib/perl5/site_perl
      3 -I/nix/store/hgpmaihx8bhwd0fh4rmipwf41gw4nnal-perl5.28.1-Net-SSLeay-1.85/lib/perl5/site_perl
      3 -I/nix/store/ia0bdnsa7vi438i782hgwiikd9zars08-perl5.28.1-List-MoreUtils-XS-0.428/lib/perl5/site_perl
      3 -I/nix/store/is0qn981pyms5kd46hddj7m27k8sj52v-perl5.28.1-File-ShareDir-1.116/lib/perl5/site_perl
      3 -I/nix/store/iy5y8651flgfdrdvkl99vgp9fvfvgfnq-perl5.28.1-Data-Uniqid-0.12/lib/perl5/site_perl
      3 -I/nix/store/j56gc2y034n58252mpvz06kna8dyzj1k-perl5.28.1-HTTP-Date-6.02/lib/perl5/site_perl
      3 -I/nix/store/j5ybf0xz8qaahfb44jn149v6r68czyq5-perl5.28.1-DateTime-Format-Strptime-1.75/lib/perl5/site_perl
      3 -I/nix/store/jxfzjqbw4mij4hbrn7rg34ylz0ssgvhi-perl5.28.1-Params-Util-1.07/lib/perl5/site_perl
      3 -I/nix/store/k5s2wlbw9dkp9z24c8sc14jpzlpnwgd9-perl5.28.1-DateTime-TimeZone-2.21/lib/perl5/site_perl
      3 -I/nix/store/kav2k56sb95fasn6178nzmb5g463fhz0-perl5.28.1-Config-AutoConf-0.317/lib/perl5/site_perl
      3 -I/nix/store/kciggm1j7h48xv8flr7q4m5w56scwczh-perl5.28.1-IO-Socket-SSL-2.060/lib/perl5/site_perl
      3 -I/nix/store/kj5mlrhrcca62a3xzsdxyw7l1bghfqrm-perl5.28.1-DateTime-1.50/lib/perl5/site_perl
      3 -I/nix/store/kkh0ssaw2imr3xpfdw7384fakbrjqhc3-perl5.28.1-Sub-Identify-0.14/lib/perl5/site_perl
      3 -I/nix/store/knrw7kbxp7y554v6jlm3adfy8fillhx0-perl5.28.1-IPC-Run3-0.048/lib/perl5/site_perl
      3 -I/nix/store/kw3xqd8dfg0vj8kmb2wa2a5g4c4lrl22-perl5.28.1-Data-Compare-1.25/lib/perl5/site_perl
      3 -I/nix/store/l35cdn48a9247qd4w05hy3dp4c2x19b8-perl5.28.1-Text-BibTeX-0.85/lib/perl5/site_perl
      3 -I/nix/store/lc183859hb59v662wb001i6g3fm767vz-perl5.28.1-Eval-Closure-0.14/lib/perl5/site_perl
      3 -I/nix/store/lkh27fmhmbbs1w22ff4r0rsqq7znj7q3-perl5.28.1-autovivification-0.18/lib/perl5/site_perl
      3 -I/nix/store/mk8h7n1q7gkabay141rjarzqdwp9qyhy-perl5.28.1-Encode-HanExtra-0.23/lib/perl5/site_perl
      3 -I/nix/store/nh0dw7rbinp2npbwx1fh4c4zj2dfmpww-perl5.28.1-Encode-EUCJPASCII-0.03/lib/perl5/site_perl
      3 -I/nix/store/nqjsmprs39fgg9q43c1nins8qlbpyln7-perl5.28.1-Text-CSV-1.97/lib/perl5/site_perl
      1 -I/nix/store/nwzpj1a1vaair5zpgx0sjaqr4jrr3r3g-perl5.28.1-biber-2.11/lib/perl5/site_perl
      3 -I/nix/store/p2cx98ykpsamqcd413l8adi19n0racn8-perl5.28.1-URI-1.74/lib/perl5/site_perl
      3 -I/nix/store/p2wl5n7fh4xpcz985l10r571935agj28-perl5.28.1-Params-Validate-1.29/lib/perl5/site_perl
      3 -I/nix/store/p7038y4s32nhzcx4qqzq4lviscc5p1mw-perl5.28.1-Encode-JIS2K-0.03/lib/perl5/site_perl
      3 -I/nix/store/pasgsdipxybgzwgf34jy5g0pygfn7hz9-perl5.28.1-Unicode-LineBreak-2018.003/lib/perl5/site_perl
      3 -I/nix/store/pfw7i1cibj35frj9x85cpg46is65fjqh-perl5.28.1-HTTP-Daemon-6.01/lib/perl5/site_perl
      3 -I/nix/store/pn6cg4fm25pw1aj6p3301zipg6bi1dxi-perl5.28.1-namespace-autoclean-0.28/lib/perl5/site_perl
      3 -I/nix/store/pv4i9hwqw0kvpkpzg82nikf61ciaywcb-perl5.28.1-Text-Roman-3.5/lib/perl5/site_perl
      3 -I/nix/store/q28c562rydjh0nax4s2xjcjls23dip91-perl5.28.1-MIME-Charset-1.012.2/lib/perl5/site_perl
      3 -I/nix/store/qkd7ap3bq46092r0np3cjz3jz378i3qk-perl5.28.1-Text-Diff-1.45/lib/perl5/site_perl
      3 -I/nix/store/qkr0yzkcwzczhcfqf8cjblvgh231lk2h-perl5.28.1-ExtUtils-LibBuilder-0.08/lib/perl5/site_perl
      3 -I/nix/store/s4z863pgaabx567z2wvfvjiyips57f5l-perl5.28.1-Sort-Key-1.33/lib/perl5/site_perl
      3 -I/nix/store/sh6d21fvngh2jsqz8np0m426dd6l573w-perl5.28.1-Sub-Quote-2.005001/lib/perl5/site_perl
      3 -I/nix/store/si3660igxgh2sh5qlp411qnyzh9fadlr-perl5.28.1-Specio-0.43/lib/perl5/site_perl
      3 -I/nix/store/sn85d7rlgj768xiza16ngsa91v0r4hd2-perl5.28.1-File-Slurper-0.012/lib/perl5/site_perl
      3 -I/nix/store/sviz38gkdfkw047v4q2r5r3c3sdvqrdq-perl5.28.1-Number-Compare-0.03/lib/perl5/site_perl
      3 -I/nix/store/v78v72waa0w2jgg2ia08k0832yv0m07w-perl5.28.1-Module-Build-0.4224/lib/perl5/site_perl
      3 -I/nix/store/v9lx9nxs0lf9m8arnkz1m3brjlmmr30j-perl5.28.1-XML-SAX-Base-1.09/lib/perl5/site_perl
      3 -I/nix/store/vq0jzjc8cvy41xfawhq2gk1hqrdfssdf-perl5.28.1-Try-Tiny-0.30/lib/perl5/site_perl
      3 -I/nix/store/w0rrbh2gzpaqlhihgg8kpm4zw3mc5nnp-perl5.28.1-WWW-RobotRules-6.02/lib/perl5/site_perl
      3 -I/nix/store/w17gh8kgpmcr7z52ark6nc9ldjdjk8m6-perl5.28.1-LWP-MediaTypes-6.02/lib/perl5/site_perl
      3 -I/nix/store/wa50d8anmn1wf3a11fja3zxlzb77m4k0-perl5.28.1-HTTP-Cookies-6.04/lib/perl5/site_perl
      3 -I/nix/store/wfjhm3r9v1cs2g8cjispbvp580x278iy-perl5.28.1-File-Listing-6.04/lib/perl5/site_perl
      3 -I/nix/store/x56ghwzcfn6m4qj254xmj14641s7kwrl-perl5.28.1-List-SomeUtils-0.56/lib/perl5/site_perl
      3 -I/nix/store/xlgdxwbvyl35rn1zf0g5rmmagns1dlml-perl5.28.1-Date-Simple-3.03/lib/perl5/site_perl
      3 -I/nix/store/y6nf7n9z97bff94l9zzawsbjxydnfs7l-perl5.28.1-Package-Stash-0.37/lib/perl5/site_perl
      3 -I/nix/store/yvx132r77hlnmx2v589laagdrywfy2dz-perl5.28.1-HTTP-Negotiate-6.01/lib/perl5/site_perl
      3 -I/nix/store/z16kzm50a6vb1n4122fl3h2xmvqfrn7f-perl5.28.1-B-Hooks-EndOfScope-0.24/lib/perl5/site_perl
      3 -I/nix/store/zirmaczcsahiajsnph06k70v27cbfysl-perl5.28.1-Role-Tiny-2.000006/lib/perl5/site_perl
      3 -I/nix/store/zqdx954x8dm2842pim7pmjy8agv1hh0p-perl5.28.1-Log-Log4perl-1.49/lib/perl5/site_perl
      1 #!/nix/store/42057h1j66iiyfn1z9y7lwngblnqqlrc-perl-5.28.1/bin/perl

@FRidh
Copy link
Member

FRidh commented Feb 15, 2019

Somewhat off-topic. How about the latency due to stat calls with the old and new method? I know in case of Python we should avoid many different site-packages folders because it's really bad for performance when doing imports.

@cleverca22
Copy link
Contributor Author

yeah, i was initially going to merge everything together with lndir to reduce the size of the search path, but that depends on way too many things!

@vcunat
Copy link
Member

vcunat commented Feb 15, 2019

Are you sure that searching one lndir would be noticeably more efficient that searching multiple directories?

@cleverca22
Copy link
Contributor Author

the merged directory will basically act as an index, the same as what ldconfig does for other distro's
rather then searching 20 different places, it checks the merged result, then follows the symlink to one other dir

@vcunat
Copy link
Member

vcunat commented Feb 15, 2019

But... following the symlink is almost like the original search, isn't it? EDIT: I mean, it feels difficult to be sure without measuring it, as it might be a negligible difference.

@cleverca22
Copy link
Contributor Author

@vcunat following the symlink to a single dir is still cheaper then checking 20 different dirs

@vcunat
Copy link
Member

vcunat commented Feb 15, 2019

It is cheaper, but you end up checking (usually) most of them through those symlinks anyway. Without symlinks you'll do it more times than with them (over the same set), but I'd suspect that kernel dentry caching might make the difference negligible. Still, I do not know for sure.

@7c6f434c
Copy link
Member

7c6f434c commented Feb 15, 2019 via email

@vcunat
Copy link
Member

vcunat commented Feb 15, 2019

Right, certainly significantly cheaper in terms of context switches.

@aanderse
Copy link
Member

aanderse commented Apr 7, 2019

What are the chances someone in here is going to implement a .withPackages for perl? This would be really useful. I would love to implement this, but am not yet skilled enough with nix to do so... :-(

@7c6f434c 7c6f434c assigned 7c6f434c and unassigned 7c6f434c Apr 8, 2019
@7c6f434c
Copy link
Member

7c6f434c commented Apr 8, 2019

Apparently I cannot assign to non-team-members.

@aanderse
Copy link
Member

aanderse commented Apr 8, 2019

@volth thank you very much! I'm also not able to assign the issue to you...

@lheckemann
Copy link
Member

This should probably be based on staging due to the amount of rebuilds.

@FRidh
Copy link
Member

FRidh commented Jul 31, 2019

There is a .withPackages function nowadays. Maybe this is no longer needed?

@lheckemann lheckemann closed this Aug 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

10 participants