Vacation PR: multi-paged, styled, searchable docs with wrapper tooling and epubs for NixOS and Nixpkgs #68020

grahamc · 2019-09-03T17:41:47Z

TLDR:

New nixpkgs manual: http://gsc.io/70266391-48a6-49be-ab5d-acb5d7f17e76-nixpkgs/doc/nixpkgs-manual/html/index.html
Nixpkgs manual, epub: http://gsc.io/70266391-48a6-49be-ab5d-acb5d7f17e76-nixpkgs/doc/nixpkgs-manual/nixpkgs-manual.epub
New NixOS manual: http://gsc.io/70266391-48a6-49be-ab5d-acb5d7f17e76-nixos/doc/nixos-manual/html/index.html
NixOS manual, epub: http://gsc.io/70266391-48a6-49be-ab5d-acb5d7f17e76-nixos/doc/nixos-manual/nixos-manual.epub

Motivation for this change

The generated doc pages are huge and don't get indexed.
Ctrl-F is a decent search tool, but a proper search engine with stemming is better.
Smaller pages are easier to navigate.
The output of nix-build was boring.

Things done

Introduce a docbook-index tool for indexing DocBook against the generated HTML.
Split up the outputted documentation in to many smaller pages.
Added a whole new layout and theme to the docs which is consistent across NixOS and Nixpkgs.
Added a local, in-browser search engine (Elasticlunr, what Rust uses.)
Way reduced the complexity of generating the Nixpkgs and NixOS docs.
Made the Nixpkgs and NixOS docs build the same way.
Created nix-doc-tools, a metapackage used for building our documentation.
Updated the how to write docs for nixos docs to account for the nix-doc-tools tooling updates.

BTW: nixpkgs and nixos docs are built the same way, so the docs apply to both.

Major thanks to @samueldr for the work on this.

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Ensured that relevant documentation is up to date
Fits CONTRIBUTING.md.

NOT DONE:

Determined the impact on package closure size (by running nix path-info -S before and after)
Need to determine the impact of building these docs on the minimal ISO.

I'm going on vacation, so I'll be back on the 16th or so to see what y'all think.

btw, samueldr can make fixes to this branch for style/js things :)

jonringer · 2019-09-03T21:39:29Z

manpages no longer exists on manual attr set:

nixpkgs/nixos/modules/misc/documentation.nix

Line 194 in be8bd43

++ optional cfg.man.enable manual.manpages

I only see manual, manualHTML, manualHTMLShell

nixpkgs/nixos/doc/manual/module.nix

Line 41 in be8bd43

in rec {

veprbl · 2019-09-03T22:46:01Z

Will this break all of the current URLs that refer to the anchors on the single page?

samueldr · 2019-09-04T00:55:00Z

Will this break all of the current URLs that refer to the anchors on the single page?

Yes, as it is. I think we could work out a solution, literally thinking of it as I type.

Prepare an index of anchors (they should still be named the same)
On index.html, at load, cross-reference anchor with index, redirect.

The main issue there could be is history hijacking. It may be possible to do something tricky with html5 history, and window.reload, to overwrite the one entry.

Tasks

Implement JS for redirection on index
Implement anchors index

davidak · 2019-09-04T16:40:30Z

Great work!

Can we have the search box in the header like this?

samueldr · 2019-09-04T16:54:49Z

It might be possible, but it will cause issues with narrower layouts, especially with sections having longer titles.

There is another issue with the UX where the current implementation presents a clear separation of the search mode and the documentation consultation mode.

And additionally, while it looks like there is only a search box, there are elements in the search overlay that are currently unlikely to be seen. Among those is the loading state of the index. If we want to add more options to the search engine we would be furthermore constrained in space.

So, the short of it is that horizontal space is a premium, and it can't easily be spared for the search box. I did initially try it in the header, but it didn't feel right.

grahamc · 2019-09-09T19:14:35Z

materializes from vacation for a few minutes hmm not a lot of feedback ❤️ . I hope this can get in for 19.09! :) disappears back to vacation

toonn · 2019-09-09T23:01:36Z

Some feedback from someone who uses a narrow (for today's standards) browser. The index takes up a lot of horizontal space when you're not using it. The python docs handle this really well by allowing you to fold the index.

Another thing that has always kinda annoyed me, the toc is super long (multiple accelerated scrolls long), and beginners have no need for it. The main page imo should either be the first section or a short explanation of the important bits of the manual, describing what's in the nixpkgs manual and that other manuals exist, e.g., nix and nixos manuals respectively and why they might want to read those instead (similar treatment for the other manuals).

samueldr · 2019-09-09T23:24:18Z

Some feedback from someone who uses a narrow (for today's standards) browser. The index takes up a lot of horizontal space when you're not using it. The python docs handle this really well by allowing you to fold the index.

I guess you're not going narrow enough :)

I do use a narrow column, often, and have chosen what felt like an appropriate breakpoint to put the menu away. Though, I do agree that (maybe as a later step) a control could be added to toggle the menu at any levels.

Tasks

Index collapse/open control

jtojnar

Big 👍 for fixing #68199 even before I opened it.

The tools look great, the only pain point is that docs-build is somewhat slower than make out/html/index.html. Maybe it would make sense to split the HTML and EPUB builds. I dream of fast rebuilds with live reloading the HTML page but I doubt there are free XSLT processors that support incremental builds.

jtojnar · 2019-09-10T00:44:10Z

doc/multiple-output.xml

@@ -189,7 +187,7 @@
    desired outputs or it moves the files during the fixup phase. Each group of
    file types has an <varname>outputFoo</varname> variable specifying the
    output name where they should go. If that variable isn't defined by the
-    derivation writer, it is guessed &ndash; a default output name is defined,
+    derivation writer, it is guessed &#x2013; a default output name is defined,


I like to use it too. That’s why I have Compose key enabled:

Suggested change

derivation writer, it is guessed – a default output name is defined,

derivation writer, it is guessed – a default output name is defined,

jtojnar · 2019-09-10T00:47:19Z

pkgs/tools/misc/nix-doc-tools/parameters.xsl

+ xmlns:d="http://docbook.org/ns/docbook"
+ version="1.0">
+ <!--
+  `ul` option puts the `ul` underneat the parent's `li`, whereas `dd` and `dt` (default)


Suggested change

`ul` option puts the `ul` underneat the parent's `li`, whereas `dd` and `dt` (default)

`ul` option puts the `ul` underneath the parent’s `li`, whereas `dd` and `dt` (default)

pkgs/tools/misc/nix-doc-tools/default.nix

pkgs/tools/misc/nix-doc-tools/lib.nix

pkgs/tools/misc/nix-doc-tools/search.js

pkgs/tools/misc/nix-doc-tools/parameters.xsl

pkgs/tools/misc/nix-doc-tools/default.nix

jtojnar · 2019-09-10T02:15:07Z

pkgs/tools/misc/nix-doc-tools/default.nix

@@ -16,6 +17,14 @@ let
    paths = [ doclib.standard-tools ] ++ generated-files;
  };

+  # a `docs-generate` command special for a nix-shell, since it
+  # uses `nix-build` instead of hard-coding a path.


What does this mean? Where does it use nix-build?

jtojnar · 2019-09-10T02:26:17Z

nixos/doc/manual/module.nix

+  sources = lib.sourceFilesBySuffices ./. [".nix" ".xml"];
+
+  modulesDoc = builtins.toFile "modules.xml" ''
+    <section xmlns:xi="http://www.w3.org/2001/XInclude" id="modules">


Suggested change

<section xmlns:xi="http://www.w3.org/2001/XInclude" id="modules">

<section xmlns="http://docbook.org/ns/docbook" xmlns:xi="http://www.w3.org/2001/XInclude" id="modules">

jtojnar · 2019-09-10T02:42:19Z

nixos/doc/manual/development/writing-documentation.xml

-   got generated. The HTML will be accessible through the
-   <filename>result</filename> symlink at
+   This build will produce a styled and indexed HTML document, and an epub. The
+   HTML will be accessible through the <filename>result</filename> symlink at
   <filename>./result/share/doc/nixos/index.html</filename>.


Suggested change

<filename>./result/share/doc/nixos/index.html</filename>.

<filename>./result/share/doc/nixos-manual/html/index.html</filename>.

toonn · 2019-09-10T09:03:02Z

I guess you're not going narrow enough : )

@samueldr apparently it's about a cm's worth of difference. But when you go narrower you lose the index completely? The nice thing about the python docs is that you can choose whether you want the index visible or not and that it's never inaccessible.

samueldr · 2019-09-14T21:43:01Z

I have marked a couple of @jtojnar's suggestions as resolved, they were trivial changes or changes I approve, in this WIP branch.

grahamc/nixpkgs@samuel-docs/searchable-2...samueldr:feature/docs/searchable-3

There are additional changes in there that I would prefer graham would take a look first :)

samueldr · 2019-09-15T00:17:11Z

Answering @toonn's question:

But when you go narrower you lose the index completely?

Not at all! There are three ways the narrow view allows the user to navigate:

The navigation icons at the top, to move to siblings or parent nodes.
The local TOC gets expanded at the top of the content, for easy visualization of the parent nodes, and the document's nodes.
The full TOC is present at the end of the document.

In fact, the narrow view shows more to the user, compared to the full view. Though, in reality the additional navigation at the end of the page is not as trivial to use.

The nice thing about the python docs is that you can choose whether you want the index visible or not and that it's never inaccessible.

And you're right, it's nice to have the option to toggle it :).

samueldr · 2019-09-15T21:32:14Z

cc: @toonn

→ https://stuff.samueldr.com/nixpkgs/manual/

✔️ That version has the togglable TOC.
✔️ That version has redirection.

To test the redirection:

Both index.html and index.html-less versions should redirect to the right location. That is because both index.html and index.html-less versions of the page are served by the nixos.org server.

I made sure to host the page at a location mimicking the URL structure from nixos.org.

toonn · 2019-09-15T22:03:39Z

@samueldr, very nice. All I'm missing is a friendlier "main" page than a detailed TOC like I mentioned earlier and possibly having toggles in the sidebar to open up items and dig down to sub sections. But this is already a major improvement imo. 👍

samueldr · 2019-09-15T22:45:24Z

All I'm missing is a friendlier "main" page

This is out of scope of the PR; contents (except contribution to the documentation) is not part of this PR, considering this reviews the way docs function, both building and using documentation. Though I agree that this is a desirable improvement, to have something more on the main page.

[...] having toggles in the sidebar to open up items and dig down to sub sections.

It was tried, but does not work well. Every level of the documentation has a page it links to. This meant that a control only used to toggle the open state of a section in the TOC would have to be implemented. Instead, I deemed it was more user-friendly to simply have one action for the options in the menu "follow and expand". With a local file, the navigation is instant, meaning there is no drawback. With hosted files (e.g. nixos.org) the navigation is near instant, due to the major decrease in page size. The cost in maintaining an additional non-native feature to the documentation is imho not worth the ability the collapse/expand TOC sections at will. That is especially true considering that following a link loses the expanded/collapsed state.

toonn · 2019-09-16T10:37:53Z

Then perhaps a static extra level to the TOC? The current sections are too broad imo, making the sidebar less useful.

jtojnar · 2019-09-18T21:39:42Z

doc/Makefile

-
-functions/library/generated: doc-support/result
-	ln -rfs ./doc-support/result/function-docs functions/library/generated
+all: ${MD_TARGETS}

 %.section.xml: %.section.md
 	pandoc $^ -w docbook+smart \


According to the manual pandoc does not support smart extension for docbook.

https://pandoc.org/MANUAL#typography

What's the deal with the `cd`?? The filename used in the index are relative to cwd/the given path. This means that to produce the expected filename associations, we need to be in the right directory when producing the index. Alternatively, the indexing tool could have been made to strip the leading path from the filenames.

They were always a separate build anyway, but this way it is easier to tell.

Allows for cross-referencing documentation. The way this is built, nixpkgs can depend on nixos, nixos can depend on nixpkgs, without an infinite loop.

grahamc · 2019-09-19T11:05:59Z

<xsl:template name="make.toc"> is way too slow right now. Disabled for development, but needs to be re-enabled so the ToCs work properly.

Mic92 · 2020-01-13T10:46:22Z

@grahamc is it known why xslt is too slow. Could be there a switch that only generates the expensive variant for nixos.org and not for local use?

nixos-discourse · 2020-03-24T15:29:13Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/documentation-presentation-navigation/6401/3

Mic92 · 2020-05-28T06:45:43Z

Should we close this PR?

jonringer · 2020-05-28T22:16:42Z

It does have a large number of conflicts. And not sure what direction the docs are going after the recent rfc and marketing team changes

cc @garbas

jonringer · 2020-05-28T22:31:33Z

@grahamc your time and effort is still appreciated. Thank you for your hard work :)

grahamc · 2020-05-29T01:37:30Z

:) it was useful to me even if for no other reason than to convince me docbook isn’t worth the fight :).

…

On Thu, May 28, 2020, at 6:31 PM, Jon wrote: @grahamc <https://github.com/grahamc> your time and effort is still appreciated. Thank you for your hard work :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#68020 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAASXLFH6WIZDEZAX2MXHJDRT3Q5FANCNFSM4ITJCMNQ>.

nixos-discourse · 2021-10-18T18:15:12Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/any-consensus-on-documentation-generation-tool-for-nixpkgs-manual/15550/2

ofborg bot added 6.topic: nixos 8.has: documentation 8.has: module (update) labels Sep 3, 2019

jtojnar reviewed Sep 10, 2019

View reviewed changes

lheckemann added this to the 20.03 milestone Sep 10, 2019

This was referenced Sep 15, 2019

Adds anchors index grahamc/docbook-index#4

Merged

Misc. fixes for searchable docs grahamc/nixpkgs#8

Merged

ofborg bot added the 2.status: merge conflict label Sep 17, 2019

jtojnar reviewed Sep 18, 2019

View reviewed changes

grahamc requested a review from nbp as a code owner September 18, 2019 23:56

grahamc added 5 commits September 18, 2019 21:21

docs: initial search hacks

0fbeb7d

I think the parameters now chunk with full tocs?

d7cfda0

params: use UTF-8 content type

dad366a

params: go back to the standard toc-context

0028967

docs: switch to toc.xml for a page-per-toc-enttry

4de2dba

samueldr and others added 9 commits September 18, 2019 21:27

docs: The TOC is togglable and hidden by default on narrow widths

6915c7e

nix-doc-tools: build man pages

202d2e5

nix-doc-tools: add timing information to each xsltproc step

b7b3efe

nixos docs: split man pages in to a separate directory

8925014

They were always a separate build anyway, but this way it is easier to tell.

nix-doc-tools Document what the make.toc override does differently

38207fb

nix-doc-tools: Disable universal toc override since it is too slow.

72d2e0f

nix-doc-tools: allow passing in olinkdb entries

a154cf6

Allows for cross-referencing documentation. The way this is built, nixpkgs can depend on nixos, nixos can depend on nixpkgs, without an infinite loop.

nixos docs: Use olinkdbs for the manual in manpages and manual

edd9da1

grahamc force-pushed the samuel-docs/searchable-2 branch from 7bc1393 to edd9da1 Compare September 19, 2019 01:29

ofborg bot removed the 2.status: merge conflict label Sep 19, 2019

nixos: fixup manualHTMLIndex

62e64c1

grahamc force-pushed the samuel-docs/searchable-2 branch 2 times, most recently from 62e64c1 to edd9da1 Compare September 19, 2019 01:48

ofborg bot added 8.has: package (new) 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Sep 19, 2019

Ericson2314 mentioned this pull request Jan 7, 2020

[RFC 0064] New Documentation Format for nixpkgs and NixOS NixOS/rfcs#64

Closed

disassembler modified the milestones: 20.03, 20.09 Feb 5, 2020

grahamc closed this May 28, 2020

jtojnar mentioned this pull request Jul 4, 2020

nixos manuals are way too big and hard to navigate #92272

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vacation PR: multi-paged, styled, searchable docs with wrapper tooling and epubs for NixOS and Nixpkgs #68020

Vacation PR: multi-paged, styled, searchable docs with wrapper tooling and epubs for NixOS and Nixpkgs #68020

grahamc commented Sep 3, 2019 •

edited by dywedir

jonringer commented Sep 3, 2019

veprbl commented Sep 3, 2019

samueldr commented Sep 4, 2019 •

edited

davidak commented Sep 4, 2019

samueldr commented Sep 4, 2019

grahamc commented Sep 9, 2019 •

edited

toonn commented Sep 9, 2019

samueldr commented Sep 9, 2019 •

edited

jtojnar left a comment

jtojnar Sep 10, 2019

jtojnar Sep 10, 2019

jtojnar Sep 10, 2019

jtojnar Sep 10, 2019

jtojnar Sep 10, 2019

toonn commented Sep 10, 2019 •

edited

samueldr commented Sep 14, 2019 •

edited

samueldr commented Sep 15, 2019

samueldr commented Sep 15, 2019 •

edited

toonn commented Sep 15, 2019

samueldr commented Sep 15, 2019

toonn commented Sep 16, 2019

jtojnar Sep 18, 2019

grahamc commented Sep 19, 2019

Mic92 commented Jan 13, 2020

nixos-discourse commented Mar 24, 2020

Mic92 commented May 28, 2020

jonringer commented May 28, 2020

jonringer commented May 28, 2020

grahamc commented May 29, 2020 via email

nixos-discourse commented Oct 18, 2021

	derivation writer, it is guessed – a default output name is defined,
	derivation writer, it is guessed – a default output name is defined,

	`ul` option puts the `ul` underneat the parent's `li`, whereas `dd` and `dt` (default)
	`ul` option puts the `ul` underneath the parent’s `li`, whereas `dd` and `dt` (default)

	<section xmlns:xi="http://www.w3.org/2001/XInclude" id="modules">
	<section xmlns="http://docbook.org/ns/docbook" xmlns:xi="http://www.w3.org/2001/XInclude" id="modules">

	<filename>./result/share/doc/nixos/index.html</filename>.
	<filename>./result/share/doc/nixos-manual/html/index.html</filename>.

Vacation PR: multi-paged, styled, searchable docs with wrapper tooling and epubs for NixOS and Nixpkgs #68020

Vacation PR: multi-paged, styled, searchable docs with wrapper tooling and epubs for NixOS and Nixpkgs #68020

Conversation

grahamc commented Sep 3, 2019 • edited by dywedir

Motivation for this change

Things done

jonringer commented Sep 3, 2019

veprbl commented Sep 3, 2019

samueldr commented Sep 4, 2019 • edited

Tasks

davidak commented Sep 4, 2019

samueldr commented Sep 4, 2019

grahamc commented Sep 9, 2019 • edited

toonn commented Sep 9, 2019

samueldr commented Sep 9, 2019 • edited

Tasks

jtojnar left a comment

Choose a reason for hiding this comment

jtojnar Sep 10, 2019

Choose a reason for hiding this comment

jtojnar Sep 10, 2019

Choose a reason for hiding this comment

jtojnar Sep 10, 2019

Choose a reason for hiding this comment

jtojnar Sep 10, 2019

Choose a reason for hiding this comment

jtojnar Sep 10, 2019

Choose a reason for hiding this comment

toonn commented Sep 10, 2019 • edited

samueldr commented Sep 14, 2019 • edited

samueldr commented Sep 15, 2019

samueldr commented Sep 15, 2019 • edited

→ https://stuff.samueldr.com/nixpkgs/manual/

toonn commented Sep 15, 2019

samueldr commented Sep 15, 2019

toonn commented Sep 16, 2019

jtojnar Sep 18, 2019

Choose a reason for hiding this comment

grahamc commented Sep 19, 2019

Mic92 commented Jan 13, 2020

nixos-discourse commented Mar 24, 2020

Mic92 commented May 28, 2020

jonringer commented May 28, 2020

jonringer commented May 28, 2020

grahamc commented May 29, 2020 via email

nixos-discourse commented Oct 18, 2021

grahamc commented Sep 3, 2019 •

edited by dywedir

samueldr commented Sep 4, 2019 •

edited

grahamc commented Sep 9, 2019 •

edited

samueldr commented Sep 9, 2019 •

edited

toonn commented Sep 10, 2019 •

edited

samueldr commented Sep 14, 2019 •

edited

samueldr commented Sep 15, 2019 •

edited