Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

texlive: 2018 -> 2019, poppler: 0.74.0 -> 0.81.0 #71419

Merged
merged 4 commits into from Nov 8, 2019

Conversation

veprbl
Copy link
Member

@veprbl veprbl commented Oct 20, 2019

Motivation for this change

Resolves: #60890
Fixes crash in #54286 (does not fix texdoc)

The original PR was done by @dtzWill in #61469. I redid the update from scratch to ensure that I understand all parts of it.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

Closes: #61469

@veprbl
Copy link
Member Author

veprbl commented Oct 20, 2019

@GrahamcOfBorg build texlive.bin

@FRidh FRidh added this to the 20.03 milestone Oct 24, 2019
@veprbl veprbl changed the title texlive: 2018 -> 2019 texlive: 2018 -> 2019, poppler: 0.74.0 -> 0.81.0 Oct 24, 2019
@veprbl
Copy link
Member Author

veprbl commented Oct 24, 2019

@GrahamcOfBorg build texlive.combined.scheme-small poppler cups-filters

@dtzWill
Copy link
Member

dtzWill commented Oct 24, 2019

So what really held back the other update (AFAIK) was our ongoing problem regarding hosting a snapshot of CTAN and our reliance on folks to provide hosting or no updates :(.

Anyway there might be hope! Not to oversell but about a month ago this was sent to the list:

https://tug.org/pipermail/tex-live/2019-September/044086.html

I flagged it for next time I went back to texlive packaging, but I think this might allow us to do two huge improvements:

  1. Don't require adhoc hosting to provide texlive to NixOS users
  2. Update more frequently, largely as a consequence of (1).

Apologies if I misjudged the implications, hope this helps and starts a new chapter of NixOS texlive goodness? 😁

@veprbl
Copy link
Member Author

veprbl commented Oct 24, 2019

@dtzWill This sounds good, however, upon reading into the messages [1][2], it doesn't seem like anywhere they imply that the old versions within a current release will be preserved. In any case this would be a step in the right direction.

[1] https://tug.org/pipermail/tex-live/2019-August/044032.html
[2] https://tug.org/pipermail/tex-live/2019-September/044086.html

@dtzWill
Copy link
Member

dtzWill commented Oct 24, 2019

@dtzWill This sounds good, however, upon reading into the messages [1][2], it doesn't seem like anywhere they imply that the old versions within a current release will be preserved. In any case this would be a step in the right direction.

Aww, was afraid it was something like that. Thanks for taking a look anyway :).

@veprbl
Copy link
Member Author

veprbl commented Oct 24, 2019

Aww, was afraid it was something like that. Thanks for taking a look anyway :).

Well, thank you for sharing, @dtzWill. I will certainly follow how this goes and see if in the end we can nudge them towards preserving the tarballs.

@veprbl
Copy link
Member Author

veprbl commented Oct 27, 2019

nix-review on x86_64-linux (non-NixOS, no sandboxing)

18 package are marked as broken and were skipped:
gazebo gazebo-headless gazeboSimulator.gazebo6 gazeboSimulator.gazebo6-headless gazeboSimulator.gazebo7 gazeboSimulator.gazebo7-headless gimpPlugins.focusblur gimpPlugins.texturize jbrout libsForQt511.khotkeys natron openspace pdf2htmlEX python27Packages.pypandoc python37Packages.pypandoc python38Packages.pypandoc qgis qgis-unwrapped

28 package failed to build:
apt autofs5 deepin.dde-file-manager deepin.deepin-mutter deepin.deepin-wm gmic-qt-krita gscan2pdf krita libreoffice libreoffice-fresh libreoffice-fresh-unwrapped libreoffice-unwrapped modem-manager-gui mysql-workbench odpdown perl528Packages.Po4a perl530Packages.Po4a python27Packages.cartopy python27Packages.geopandas python27Packages.worldengine python38Packages.fpylll python38Packages.paperwork-backend qpdfview sage sageWithDoc spark sssd unoconv

262 package were build:
R adapta-gtk-theme aesop apvlv asciidoc-full asciidoc-full-with-plugins asymptote auctex biber bookworm btrbk calibre cddlib chrome-gnome-shell cups-filters dblatex dblatexFull deepin.dde-api deepin.dde-control-center deepin.dde-daemon deepin.dde-dock deepin.dde-launcher deepin.dde-session-ui deepin.deepin-desktop-base deepin.deepin-desktop-schemas deepin.deepin-metacity deepin.deepin-wallpapers deja-dup denemo diff-pdf diffpdf digikam disorderfs dolphin dragon dropbox-cli dwarf-fortress-packages.dwarf-fortress-full dwarf-therapist dwarf-fortress-packages.dwarf-therapist-original eaglemode eclib elisa empathy enblend-enfuse enlightenment.econnman enlightenment.efl enlightenment.enlightenment enlightenment.ephoto enlightenment.rage enlightenment.terminology epdfview eukleides evince fbida fgallery frescobaldi gImageReader gdal gdal_2 geeqie gfan giac giac-with-xcas gimp gimp-with-plugins gimpPlugins.exposureBlend gimpPlugins.fourier gimpPlugins.gap gimpPlugins.gimplensfun gimpPlugins.gmic gimpPlugins.lightning gimpPlugins.lqrPlugin gimpPlugins.resynthesizer gimpPlugins.ufraw gimpPlugins.waveletSharpen glom gnome-photos gnome3.cheese gnome3.file-roller gnome3.gnome-books gnome3.gnome-boxes gnome3.gnome-contacts gnome3.gnome-control-center gnome3.gnome-documents gnome3.gnome-session gnome3.gnome-shell gnome3.gnome-terminal gnome3.gnome-tweaks gnome3.gnome-user-share gnome3.nautilus gnome3.nautilus-python gnome3.pomodoro gnome3.sushi gnome3.totem tracker-miners gnomeExtensions.gsconnect gp2c grass green-pdfviewer gummi gwenview haskellPackages.H haskellPackages.inline-r herwig hugin ipe k3b kde-cli-tools kdeApplications.baloo-widgets kdeApplications.dolphin-plugins kdenlive kdeApplications.kdepim-addons kfind kitinerary okular libsForQt5.baloo libsForQt5.kfilemetadata kdeplasma-addons kdev-php kdev-python kdevelop kdevelop-unwrapped kile kmenuedit krop kwin-tiling lcalc leela libLAS plasma5.khotkeys libsForQt5.poppler libsForQt511.baloo libsForQt511.kfilemetadata libsForQt511.poppler lilypond lilypond-unstable lilypond-with-fonts lumina.lumina-pdf mapnik mate.atril mcgrid merkaartor mitscheme mitschemeX11 ne nixnote2 nuweb openmolcas openorienteering-mapper pantheon.elementary-session-settings pantheon.extra-elementary-contracts paperless paperwork pari pdf-quench pdf2djvu pdf2svg pdfarranger pdfgrep pdfmod pdfpc pdfsandwich pdftag peruse photoqt plasma-desktop plasma-workspace powerdevil systemsettings polymake poppler poppler_gi poppler_min poppler_utils pqiv pynac python27Packages.cypari2 python27Packages.cysignals python27Packages.fiona python27Packages.fpylll python27Packages.neurotools python27Packages.pdf2image python27Packages.pdftotext python27Packages.poppler-qt5 python27Packages.pplpy python27Packages.pypoppler python27Packages.python-mapnik python27Packages.pythonefl python27Packages.rasterio rivet python27Packages.rpy2 python27Packages.tilestache python37Packages.cartopy python37Packages.cypari2 python37Packages.cysignals python37Packages.fiona python37Packages.fpylll python37Packages.gdal python37Packages.geopandas python37Packages.osmnx python37Packages.paperwork-backend python37Packages.pdf2image python37Packages.pdftotext python37Packages.poppler-qt5 python37Packages.pplpy python37Packages.python-mapnik python37Packages.pythonefl python37Packages.rasterio python37Packages.rpy2 worldengine-cli python38Packages.cartopy python38Packages.cypari2 python38Packages.cysignals python38Packages.fiona python38Packages.gdal python38Packages.pdf2image python38Packages.pdftotext python38Packages.poppler-qt5 python38Packages.pplpy python38Packages.python-mapnik python38Packages.pythonefl python38Packages.rasterio python38Packages.rpy2 python38Packages.worldengine qcomicbook qlandkartegt qmapshack qtikz rWrapper racket racket-minimal recoll ripgrep-all rstudio rstudioWrapper saga scribusUnstable sherpa singular skribilo solfege splix sympow system-config-printer texlive.combined.scheme-basic texlive.combined.scheme-small texmacs texmaker texstudio texworks thepeg tikzit udiskie ufraw valentina wacomtablet xfce.tumbler xfce

@veprbl
Copy link
Member Author

veprbl commented Oct 27, 2019

nix-review on x86_64-darwin (no sandboxing)

41 package are marked as broken and were skipped:
asciidoc-full asciidoc-full-with-plugins asymptote btrbk fgallery frescobaldi gazebo gazebo-headless gazeboSimulator.gazebo6 gazeboSimulator.gazebo6-headless gazeboSimulator.gazebo7 gazeboSimulator.gazebo7-headless gimpPlugins.gmic gimpPlugins.ufraw gmic-qt-krita gnome3.nautilus-python grass green-pdfviewer gscan2pdf lilypond lilypond-unstable lilypond-with-fonts mate.atril odpdown pdfmod perl528Packages.Po4a perl530Packages.Po4a photoqt python27Packages.cartopy python27Packages.pypandoc python37Packages.paperwork-backend python37Packages.pypandoc python38Packages.paperwork-backend python38Packages.pypandoc racket recoll saga spark udiskie ufraw zathura

29 package failed to build:
dwarf-fortress-packages.dwarf-fortress-full giac-with-xcas haskellPackages.H haskellPackages.inline-r libsForQt512.poppler python27Packages.fiona python27Packages.geopandas python27Packages.neurotools python27Packages.poppler-qt5 python27Packages.pplpy python27Packages.rasterio python27Packages.rpy2 python27Packages.worldengine python37Packages.cartopy python37Packages.fiona python37Packages.geopandas python37Packages.osmnx python37Packages.poppler-qt5 python37Packages.python-mapnik python37Packages.rasterio python37Packages.rpy2 python38Packages.cartopy python38Packages.fiona python38Packages.fpylll python38Packages.poppler-qt5 python38Packages.pplpy python38Packages.python-mapnik python38Packages.rasterio python38Packages.rpy2

64 package were build:
auctex biber cddlib dblatex dblatexFull diff-pdf dwarf-fortress-packages.dwarf-therapist-original eclib gdal gdal_2 gfan giac gimp herwig lcalc libLAS libsForQt5.poppler lumina.lumina-pdf mapnik mcgrid ne openorienteering-mapper pari pdf2svg pdfgrep pdfpc pdftag poppler poppler_gi poppler_min poppler_utils pynac python27Packages.cypari2 python27Packages.cysignals python27Packages.fpylll python27Packages.pdf2image python27Packages.pdftotext python27Packages.pypoppler python27Packages.python-mapnik rivet python27Packages.tilestache python37Packages.cypari2 python37Packages.cysignals python37Packages.fpylll python37Packages.gdal python37Packages.pdf2image python37Packages.pdftotext python37Packages.pplpy worldengine-cli python38Packages.cypari2 python38Packages.cysignals python38Packages.gdal python38Packages.pdf2image python38Packages.pdftotext python38Packages.worldengine racket-minimal ripgrep-all sherpa singular texlive.combined.scheme-basic texlive.combined.scheme-small thepeg tikzit xournal

@veprbl
Copy link
Member Author

veprbl commented Oct 27, 2019

Failures appear to be not relevant. I believe this PR should be okay to merge.

@7c6f434c
Copy link
Member

python37Packages.pdftotext might be possibly relevant.

@veprbl
Copy link
Member Author

veprbl commented Oct 28, 2019

python37Packages.pdftotext might be possibly relevant.

This package is in the list of those that build successfully.

@7c6f434c
Copy link
Member

Ah right sorry. I am a bit surprised that LibreOffice fails, though…

@dtzWill
Copy link
Member

dtzWill commented Nov 4, 2019

Been using this for a while, at least the update portion! Sorry for slow review.

Had trouble connecting to mirror address over the past 24 hours or so-- is the host still okay?

Is there a not-the-worst way to share your snapshot such that one or more of us could try mirroring as well? Thanks for offering at all, I'll see what I can put together here too... :). But ideally we'd host the same content... :)

@veprbl
Copy link
Member Author

veprbl commented Nov 4, 2019

Had trouble connecting to mirror address over the past 24 hours or so-- is the host still okay?

I was messing with the server configuration last night, this is now fixed. Thanks.

Is there a not-the-worst way to share your snapshot such that one or more of us could try mirroring as well?

I tried adding IPFS for texlive-2017, but AFAIK nobody joined that. I'm open to proposals, though.

Ah right sorry. I am a bit surprised that LibreOffice fails, though…

@7c6f434c It failed on my machine during the test with IO exception while trying to write to the $TEMP. It builds just fine on NixOS with sandboxing.

@matthuszagh
Copy link
Contributor

If it's in any way useful I've been using this as well and haven't encountered any issues so far.

@dtzWill dtzWill mentioned this pull request Nov 4, 2019
10 tasks
biber: 2.12 -> 2.13

Vendored potrace was moved from libs/potrace to
texk/dvisvgm/dvisvgm-src/libs/potrace. Unfortunately, it is not trivial
to remove it from there, so for now dvisvgm seems to be using its
vendored dependencies.
@veprbl
Copy link
Member Author

veprbl commented Nov 6, 2019

Rebased on master to fix breakage in inkscape that happened after #72491

@GrahamcOfBorg build texlive.combined.scheme-small poppler cups-filters inkscape

@veprbl veprbl merged commit 6fa62db into NixOS:master Nov 8, 2019
dtzWill pushed a commit to dtzWill/nixpkgs that referenced this pull request Nov 10, 2019
texlive: 2018 -> 2019, poppler: 0.74.0 -> 0.81.0
(cherry picked from commit 6fa62db)
dtzWill added a commit to dtzWill/nixpkgs that referenced this pull request Dec 6, 2019
For some context, this was previously and recently discussed:

NixOS#61469 (comment)
NixOS#71419 (comment)

Fortunately it appears, due to developments since those discussion,
that there is now a versioned mirror for ctan contents!!

Naturally this commit is about moving to that mirror:
updating the package lists and regenerating the hashes.

This has two huge advantages:

* Updating the pinned ctan packages can be done more frequently
  than the current "once a year".
* We don't have to rely on community members to create a snapshot
  and mirror it indefinitely and at their own cost.

Minor "advantage":

* I should be writing but instead I was working on this :P

It appears new, I stumbled upon a mention of it here[1]:

> Native TL updates ultimately come from a directory named tlnet (on CTAN).
> An archive for tlnet is available, with daily updates since
> September 2019, and the final yearly release going back about ten years.

[1] https://www.tug.org/texlive/pkginstall.html

----

Concerns and theirs status:

* Package integrity:
There may be concerns about ensuring the package hashes are obtained
from a trusted source-- I haven't looked into this at all "yet".

* Mirror reliability and not-abusing-nice-things-from-good-people
If the mirror is a small operation it may be hostile to point
the network request fury of the Nixpkgs texlive community ;).
This doesn't appear to be the case and there's no mention of it,
and presumably wouldn't be linked from tug.org if wasn't to be used?
Would be good to chase down an announcement or otherwise ensure
this is expected usage.
Time will tell re:mirror reliability (or speed?),
so far it seems like a solid service meant to be used
and unlikely to be problematic.

* Is the archive only updated once a day?
The archive is organized by date, which is wonderful (!)
but might mean the contents may see changes during the day.
I suspect this is not the case but just a thought.
Regardless, any problems seem easily avoided by simply
not updating to the current day and use yesterday instead.

----

If this is the way we move forward,
there are changes/fixes/improvements to be made
in terms of documentation as well as (possibly)
putting together some update automation tooling.

---

I have only tested this with a single project so far,
and it doesn't have any especially esoteric or demanding
needs.  Help testing by checking your projects to see
that they build and produce expected results.
Building against the before/after set and using `diffpdf`
is how I usually vet these updates, if that's useful.
The `diffpdf` especially helps identify the minor
appearance changes I usually miss visually-- tweaks
to microtype behavior or slightly different layout/placement
of figures or handling of beamer slides, etc.
dtzWill added a commit to dtzWill/nixpkgs that referenced this pull request Dec 6, 2019
For some context, this was previously and recently discussed:

NixOS#61469 (comment)
NixOS#71419 (comment)

Fortunately it appears, due to developments since those discussion,
that there is now a versioned mirror for ctan contents!!

Naturally this commit is about moving to that mirror:
updating the package lists and regenerating the hashes.

This has two huge advantages:

* Updating the pinned ctan packages can be done more frequently
  than the current "once a year".
* We don't have to rely on community members to create a snapshot
  and mirror it indefinitely and at their own cost.

Minor "advantage":

* I should be writing but instead I was working on this :P

It appears new, I stumbled upon a mention of it here[1]:

> Native TL updates ultimately come from a directory named tlnet (on CTAN).
> An archive for tlnet is available, with daily updates since
> September 2019, and the final yearly release going back about ten years.

[1] https://www.tug.org/texlive/pkginstall.html

----

Concerns and theirs status:

* Package integrity:
There may be concerns about ensuring the package hashes are obtained
from a trusted source-- I haven't looked into this at all "yet".

* Mirror reliability and not-abusing-nice-things-from-good-people
If the mirror is a small operation it may be hostile to point
the network request fury of the Nixpkgs texlive community ;).
This doesn't appear to be the case and there's no mention of it,
and presumably wouldn't be linked from tug.org if wasn't to be used?
Would be good to chase down an announcement or otherwise ensure
this is expected usage.
Time will tell re:mirror reliability (or speed?),
so far it seems like a solid service meant to be used
and unlikely to be problematic.

* Is the archive only updated once a day?
The archive is organized by date, which is wonderful (!)
but might mean the contents may see changes during the day.
I suspect this is not the case but just a thought.
Regardless, any problems seem easily avoided by simply
not updating to the current day and use yesterday instead.

----

If this is the way we move forward,
there are changes/fixes/improvements to be made
in terms of documentation as well as (possibly)
putting together some update automation tooling.

---

I have only tested this with a single project so far,
and it doesn't have any especially esoteric or demanding
needs.  Help testing by checking your projects to see
that they build and produce expected results.
Building against the before/after set and using `diffpdf`
is how I usually vet these updates, if that's useful.
The `diffpdf` especially helps identify the minor
appearance changes I usually miss visually-- tweaks
to microtype behavior or slightly different layout/placement
of figures or handling of beamer slides, etc.

(cherry picked from commit 9c75eec)
dtzWill added a commit to dtzWill/nixpkgs that referenced this pull request Jan 13, 2020
For some context, this was previously and recently discussed:

NixOS#61469 (comment)
NixOS#71419 (comment)

Fortunately it appears, due to developments since those discussion,
that there is now a versioned mirror for ctan contents!!

Naturally this commit is about moving to that mirror:
updating the package lists and regenerating the hashes.

This has two huge advantages:

* Updating the pinned ctan packages can be done more frequently
  than the current "once a year".
* We don't have to rely on community members to create a snapshot
  and mirror it indefinitely and at their own cost.

Minor "advantage":

* I should be writing but instead I was working on this :P

It appears new, I stumbled upon a mention of it here[1]:

> Native TL updates ultimately come from a directory named tlnet (on CTAN).
> An archive for tlnet is available, with daily updates since
> September 2019, and the final yearly release going back about ten years.

[1] https://www.tug.org/texlive/pkginstall.html

----

Concerns and theirs status:

* Package integrity:
There may be concerns about ensuring the package hashes are obtained
from a trusted source-- I haven't looked into this at all "yet".

* Mirror reliability and not-abusing-nice-things-from-good-people
If the mirror is a small operation it may be hostile to point
the network request fury of the Nixpkgs texlive community ;).
This doesn't appear to be the case and there's no mention of it,
and presumably wouldn't be linked from tug.org if wasn't to be used?
Would be good to chase down an announcement or otherwise ensure
this is expected usage.
Time will tell re:mirror reliability (or speed?),
so far it seems like a solid service meant to be used
and unlikely to be problematic.

* Is the archive only updated once a day?
The archive is organized by date, which is wonderful (!)
but might mean the contents may see changes during the day.
I suspect this is not the case but just a thought.
Regardless, any problems seem easily avoided by simply
not updating to the current day and use yesterday instead.

----

If this is the way we move forward,
there are changes/fixes/improvements to be made
in terms of documentation as well as (possibly)
putting together some update automation tooling.

---

I have only tested this with a single project so far,
and it doesn't have any especially esoteric or demanding
needs.  Help testing by checking your projects to see
that they build and produce expected results.
Building against the before/after set and using `diffpdf`
is how I usually vet these updates, if that's useful.
The `diffpdf` especially helps identify the minor
appearance changes I usually miss visually-- tweaks
to microtype behavior or slightly different layout/placement
of figures or handling of beamer slides, etc.
@veprbl
Copy link
Member Author

veprbl commented Mar 12, 2020

@alexvorobiev This should have been addressed by d6fdab5.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

request: texlive 2019! \o/
7 participants