Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

texlive: 2019 (wip) #61469

Closed
wants to merge 18 commits into from
Closed

Conversation

dtzWill
Copy link
Member

@dtzWill dtzWill commented May 13, 2019

Motivation for this change

Another year, another update!

This uses snapshot hosted on wdtz.org,
which may not be up to the task long-term.

Not sure how hard folks hit our TL mirrors...

Commit history is messy, showing path taken so far.
(Will clean up if/when reach something ready to merge)

Help testing and suggestions for cleanup appreciated!

For example, dvisvgm was having troubles I ultimately resolved
by setting LDFLAGS but likely has a better solution.

Fixes #60890.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@ofborg ofborg bot added the 6.topic: TeX Issues regarding texlive and TeX in general label May 13, 2019
@ofborg ofborg bot requested review from veprbl, viric and vcunat May 13, 2019 21:29
@dtzWill dtzWill requested review from jtojnar, samueldr and xeji May 15, 2019 21:45
@dtzWill dtzWill marked this pull request as ready for review May 25, 2019 20:59
@FRidh FRidh added this to the 19.09 milestone May 28, 2019
@FRidh
Copy link
Member

FRidh commented May 28, 2019

Status?

@dtzWill
Copy link
Member Author

dtzWill commented Jun 4, 2019

Seems to work well for my needs? And I'm not confident about the hosting situation, but mostly I have no idea what to expect from NixOS'ers tex usage ^_^.

And apparently this latest conflict which I'll address tomorrow / next push, time for bed for me O:).

@sondr3
Copy link
Contributor

sondr3 commented Sep 20, 2019

Minor ping, this would be great to have in 19.09 😄

@veprbl veprbl self-assigned this Oct 20, 2019
@veprbl veprbl removed their assignment Oct 20, 2019
@flokli
Copy link
Contributor

flokli commented Nov 1, 2019

@dtzWill what's missing from https://www.tug.org/historic/ , that needs to be hosted at wdtz.org? It seems like the 2019 texlive sources are there, too…

@dtzWill
Copy link
Member Author

dtzWill commented Nov 4, 2019

@dtzWill what's missing from https://www.tug.org/historic/ , that needs to be hosted at wdtz.org? It seems like the 2019 texlive sources are there, too…

The ctan sources-- aka everything in texlive that's not the core binaries :(. Nothing new on this front, just we need to play another round of "host the tarballs" for 2019 :).

@dtzWill
Copy link
Member Author

dtzWill commented Nov 4, 2019

Closing in favor of #71419! \o/

If there's anything useful here not over there, LMK O:).

@dtzWill dtzWill closed this Nov 4, 2019
@veprbl
Copy link
Member

veprbl commented Nov 4, 2019

The ctan sources-- aka everything in texlive that's not the core binaries :(

Actually, the "-texmf" tarball contains exactly those packages from CTAN. And, in theory, it should be possible to build our texlive using that 2.6 GB tarball. The problem is that we would have to duplicate their build system. This is not impossible to do, and, in fact, guix did this [1] (with mere 7k lines of handwritten Guile code). They overcome the problem of the big tarball by fetching from svn [2], but this probably would not work for us since we are looking to provide 3.5k individual packages unless we are going to break compatibility. Right now, I don't see any shortcuts on this path.

[1] https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/tex.scm?h=bf7b08c4fe7d14c25a83bde99f19eca1119d88ff
[2] https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build-system/texlive.scm?h=bf7b08c4fe7d14c25a83bde99f19eca1119d88ff#n66

dtzWill added a commit to dtzWill/nixpkgs that referenced this pull request Dec 6, 2019
For some context, this was previously and recently discussed:

NixOS#61469 (comment)
NixOS#71419 (comment)

Fortunately it appears, due to developments since those discussion,
that there is now a versioned mirror for ctan contents!!

Naturally this commit is about moving to that mirror:
updating the package lists and regenerating the hashes.

This has two huge advantages:

* Updating the pinned ctan packages can be done more frequently
  than the current "once a year".
* We don't have to rely on community members to create a snapshot
  and mirror it indefinitely and at their own cost.

Minor "advantage":

* I should be writing but instead I was working on this :P

It appears new, I stumbled upon a mention of it here[1]:

> Native TL updates ultimately come from a directory named tlnet (on CTAN).
> An archive for tlnet is available, with daily updates since
> September 2019, and the final yearly release going back about ten years.

[1] https://www.tug.org/texlive/pkginstall.html

----

Concerns and theirs status:

* Package integrity:
There may be concerns about ensuring the package hashes are obtained
from a trusted source-- I haven't looked into this at all "yet".

* Mirror reliability and not-abusing-nice-things-from-good-people
If the mirror is a small operation it may be hostile to point
the network request fury of the Nixpkgs texlive community ;).
This doesn't appear to be the case and there's no mention of it,
and presumably wouldn't be linked from tug.org if wasn't to be used?
Would be good to chase down an announcement or otherwise ensure
this is expected usage.
Time will tell re:mirror reliability (or speed?),
so far it seems like a solid service meant to be used
and unlikely to be problematic.

* Is the archive only updated once a day?
The archive is organized by date, which is wonderful (!)
but might mean the contents may see changes during the day.
I suspect this is not the case but just a thought.
Regardless, any problems seem easily avoided by simply
not updating to the current day and use yesterday instead.

----

If this is the way we move forward,
there are changes/fixes/improvements to be made
in terms of documentation as well as (possibly)
putting together some update automation tooling.

---

I have only tested this with a single project so far,
and it doesn't have any especially esoteric or demanding
needs.  Help testing by checking your projects to see
that they build and produce expected results.
Building against the before/after set and using `diffpdf`
is how I usually vet these updates, if that's useful.
The `diffpdf` especially helps identify the minor
appearance changes I usually miss visually-- tweaks
to microtype behavior or slightly different layout/placement
of figures or handling of beamer slides, etc.
dtzWill added a commit to dtzWill/nixpkgs that referenced this pull request Dec 6, 2019
For some context, this was previously and recently discussed:

NixOS#61469 (comment)
NixOS#71419 (comment)

Fortunately it appears, due to developments since those discussion,
that there is now a versioned mirror for ctan contents!!

Naturally this commit is about moving to that mirror:
updating the package lists and regenerating the hashes.

This has two huge advantages:

* Updating the pinned ctan packages can be done more frequently
  than the current "once a year".
* We don't have to rely on community members to create a snapshot
  and mirror it indefinitely and at their own cost.

Minor "advantage":

* I should be writing but instead I was working on this :P

It appears new, I stumbled upon a mention of it here[1]:

> Native TL updates ultimately come from a directory named tlnet (on CTAN).
> An archive for tlnet is available, with daily updates since
> September 2019, and the final yearly release going back about ten years.

[1] https://www.tug.org/texlive/pkginstall.html

----

Concerns and theirs status:

* Package integrity:
There may be concerns about ensuring the package hashes are obtained
from a trusted source-- I haven't looked into this at all "yet".

* Mirror reliability and not-abusing-nice-things-from-good-people
If the mirror is a small operation it may be hostile to point
the network request fury of the Nixpkgs texlive community ;).
This doesn't appear to be the case and there's no mention of it,
and presumably wouldn't be linked from tug.org if wasn't to be used?
Would be good to chase down an announcement or otherwise ensure
this is expected usage.
Time will tell re:mirror reliability (or speed?),
so far it seems like a solid service meant to be used
and unlikely to be problematic.

* Is the archive only updated once a day?
The archive is organized by date, which is wonderful (!)
but might mean the contents may see changes during the day.
I suspect this is not the case but just a thought.
Regardless, any problems seem easily avoided by simply
not updating to the current day and use yesterday instead.

----

If this is the way we move forward,
there are changes/fixes/improvements to be made
in terms of documentation as well as (possibly)
putting together some update automation tooling.

---

I have only tested this with a single project so far,
and it doesn't have any especially esoteric or demanding
needs.  Help testing by checking your projects to see
that they build and produce expected results.
Building against the before/after set and using `diffpdf`
is how I usually vet these updates, if that's useful.
The `diffpdf` especially helps identify the minor
appearance changes I usually miss visually-- tweaks
to microtype behavior or slightly different layout/placement
of figures or handling of beamer slides, etc.

(cherry picked from commit 9c75eec)
dtzWill added a commit to dtzWill/nixpkgs that referenced this pull request Jan 13, 2020
For some context, this was previously and recently discussed:

NixOS#61469 (comment)
NixOS#71419 (comment)

Fortunately it appears, due to developments since those discussion,
that there is now a versioned mirror for ctan contents!!

Naturally this commit is about moving to that mirror:
updating the package lists and regenerating the hashes.

This has two huge advantages:

* Updating the pinned ctan packages can be done more frequently
  than the current "once a year".
* We don't have to rely on community members to create a snapshot
  and mirror it indefinitely and at their own cost.

Minor "advantage":

* I should be writing but instead I was working on this :P

It appears new, I stumbled upon a mention of it here[1]:

> Native TL updates ultimately come from a directory named tlnet (on CTAN).
> An archive for tlnet is available, with daily updates since
> September 2019, and the final yearly release going back about ten years.

[1] https://www.tug.org/texlive/pkginstall.html

----

Concerns and theirs status:

* Package integrity:
There may be concerns about ensuring the package hashes are obtained
from a trusted source-- I haven't looked into this at all "yet".

* Mirror reliability and not-abusing-nice-things-from-good-people
If the mirror is a small operation it may be hostile to point
the network request fury of the Nixpkgs texlive community ;).
This doesn't appear to be the case and there's no mention of it,
and presumably wouldn't be linked from tug.org if wasn't to be used?
Would be good to chase down an announcement or otherwise ensure
this is expected usage.
Time will tell re:mirror reliability (or speed?),
so far it seems like a solid service meant to be used
and unlikely to be problematic.

* Is the archive only updated once a day?
The archive is organized by date, which is wonderful (!)
but might mean the contents may see changes during the day.
I suspect this is not the case but just a thought.
Regardless, any problems seem easily avoided by simply
not updating to the current day and use yesterday instead.

----

If this is the way we move forward,
there are changes/fixes/improvements to be made
in terms of documentation as well as (possibly)
putting together some update automation tooling.

---

I have only tested this with a single project so far,
and it doesn't have any especially esoteric or demanding
needs.  Help testing by checking your projects to see
that they build and produce expected results.
Building against the before/after set and using `diffpdf`
is how I usually vet these updates, if that's useful.
The `diffpdf` especially helps identify the minor
appearance changes I usually miss visually-- tweaks
to microtype behavior or slightly different layout/placement
of figures or handling of beamer slides, etc.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

request: texlive 2019! \o/
6 participants