Improved search feature for API docs #5229

straight-shoota · 2017-11-01T16:53:39Z

This PR greatly improves client-side search on generated HTML pages of the API docs.
It uses the program description available as JSON data from #4746 and implements a custom search algorithm in JavaScript.
The search results include types, constants, methods, macros matching the query string in one or more of the fields name, namespace, description, arguments or type.
A custom ranking algorithm sorts results based on matching quality and relevance.

The search-algorithm is actually quite simple, it uses no search index or advanced information retrieval functions. It simply traverses through all the types and tries to find matching features. This is really just fast enough.
I considered using something like lunr.js but this would have been complicated to setup to create a domain-specific search experience. With the custom algorithm it is very easy to fine tune matching and ranking behaviour to allow developers to find what they need as fast as possible.

This approach has some drawbacks with regards to fulltext search capabilities (especially concerning docstrings), but this can certainly be improved upon.

Search features:

prefix filter: # matches instance methods, . matches class methods and macros, :/:: matches namespace
search is generally case-insensitive but results with case-equality areranked higher.
search results list can be navigated by key (when in focus):
- i, c, ArrowUp: move upwards
- j, h, ArrowDown: move downwards
- Enter: load result
- Escape: cancel search
- arrows, Enter and Escape also work when the search-input is in focus
support for URL fragment-based search query: /#q=mysearch will initialize a search at page load
page-wide shortkeys to start search: s or /

Preview

Online Preview

This is intended to be alongside #4755 which makes it look even better (Online Preview) though it can be merged separately.

Closes #1792

makenowjust · 2017-11-02T11:38:54Z

src/compiler/crystal/tools/doc/generator.cr

@@ -78,6 +78,7 @@ class Crystal::Doc::Generator

    main_index = Main.new(raw_body, Type.new(self, @program), repository_name)
    File.write File.join(@output_dir, "index.json"), main_index
+    File.write File.join(@output_dir, "index.jsonp"), main_index.to_jsonp


I think search-index.js (from Rust doc) is better than index.jsonp. The extension jsonp is irregular.

Yeah, why JSONP? Still, there might a problem with loading json files over file:///.

search-index.js seems like a good idea 👍

straight-shoota · 2017-11-02T12:20:31Z

@Sija Yes, browsers can't simply load arbitrary files over file protocol. That's why we need to load the search index as a javascript file to enable search when reading the docs locally.

We could also encourage people to load the docs through a local HTTP server like serve (or at best include one directly into the docs generator) but this is always one step more complicated and the workaround with JSONP is super easy.
search-index.js can obviously be ignored when deploying to a HTTP server.

RX14 · 2017-11-03T14:18:36Z

Why isn't it j/k for up/down? That's the standard vim controls...

Would be fantastic if the search handled typos by giving some weight to keys that are close physically to the one actually pressed, but thats really way too much to ask especially with the number of keyboard layouts we have.

RX14 · 2017-11-03T15:16:31Z

Bug reports:

pressing / only works the first time after page load on firefox, because the second time it brings up quick search.
If i type too quickly that search doesn't complete before i press enter, (i.e. pressing string<enter> with no delay) it doesn't wait until the search is complete before jumping.
pressing enter to confirm a search doesn't clear the search box and reset the element focus.
somehow typing .open gets me .dlopen? that's unexpected.

There should also probably be a (x) button to clear the search box.

The small side bar is also too small (for search) on vertical monitors. I'm not sure exactly how to solve that. Perhaps a popout search, or expand the sidebar on search below a window size.

Also, it's not a true fuzzy search, i'd expect htp to get me to http, or cread to get me to Char::Reader.

straight-shoota · 2017-11-03T16:01:57Z

Thanks for the testing.

I copies the key bindings from api.rubyonrails.org - seems like they had it wrong and I didn't notice. Will change to j/k
I've only done rudimentary tests in firefox. Seems a bit odd. I noticed that it might sometimes be a focus issue.
Fast jumping will be fixed.
This is supposed to be a feature. Additionally, also the sidebar scroll viewport stays the same. I find it useful this way, so you can easily look at a different search result instead of having to start over.

RX14 · 2017-11-03T17:11:53Z

The solution is to define a peek button p which does the same as enter but focusses the search bar back again. You could even add ctrl-p so you don't need to press the down arrow

straight-shoota · 2017-11-04T17:48:33Z

@RX14 I'm not sure, I can follow.
In which state should that peek button p be available? It can't be while you're typing. Needs a modifier like Ctrl. What's the problem with the current behaviour?

RX14 · 2017-11-05T00:06:52Z

@straight-shoota the problem is that it's unexpected. When i'm at the docs my muscle memory is to do s<query>. If I have to evaluate the page to see exactly what element im focussed on it's just going to make me a new hundred milliseconds longer because I have to change how I search. (yes I do care about it that much)

The rule should be that after I press enter I should be exactly the same node of the search state machine as I am when the page loads. And the starting node shouldn't depend on anything, a refresh should be a completely blank slate.

For people who want your behaviour of keeping the search results to browse through them, they should have a "peek" shortcut (ctrl-p) to peek the results of a search result, but they're still at the search list so they can find another result if it's incorrect. I'll probably never use peek. 99% of the time i've arrived in the right file, or at the least i'll need to correct my search term not just move to the second result. In fact i'll rarely be moving to the second result, largely my intended result will be rank 1, or i'll be refining my search term.

RX14 · 2017-11-05T00:10:03Z

Another thing which would greatly help for muscle memory would be making it so that pressing s highlights the results of the search box. This is a natural way to make it so that using s<query> never appends to an existing search query, it's always a brand new search. If you want to edit your existing query, use right arrow to get to the end of the query, or use left arrow to add a prefix. This is much more natural and much easier to get to the end/start of the search query than just changing the focus but not selecting the whole contents.

straight-shoota · 2017-11-05T09:24:43Z

If I understand you correctly, with "Highlight the results of the search box" you mean "select the text in the query input field"?

RX14 · 2017-11-05T11:45:59Z

@straight-shoota yep.

RX14 · 2017-11-12T13:27:47Z

@straight-shoota have you had any more time to work on this?

straight-shoota · 2017-11-15T16:35:19Z

@RX14 I've implemented that the results list is not automatically reloaded on a new page and when you focus the search field, the content gets selected.

I haven't done the peek feature yet. Strg + P won't work because in a browser this is the shortcut for printing. I'm not sure if it is worth to implement at all and if so, which shortcut would be suited for this.
Maybe Shift + Enter... but probably it's not necessary at all.

Preview link has been updated.

RX14 · 2017-11-16T13:20:41Z

@straight-shoota bug: typing result types re selects the whole entry field, then overwrites it with ult. I'm pretty sure that's unintentional.

straight-shoota · 2017-11-16T13:40:07Z

Woah, yep. That event shouldn't trigger when the input is selected.

straight-shoota · 2017-11-16T21:20:09Z

I've fixed this and added a modal window listing shortkeys (can be opened with ?).

faustinoaq · 2017-11-16T21:39:19Z

Awesome improvement to documentation 🎉

Would be possible to hide sidebar on small screens and add a hamburger button?

straight-shoota · 2017-11-16T22:54:46Z

@faustinoaq Yes, this is the next step after #4755

RX14 · 2017-11-18T17:46:11Z

I think that the input box should be explicitly cleared and unfocussed at onpageshow. Firefox seems to restore the textbox content and focus when the page is refreshed.

straight-shoota · 2017-11-18T18:40:22Z

@RX14 This is actually done on purpose. Each search query is stored in session storage and the last value is retrieved on page load. This is intended to help refine a query so you don't have to type again.
Autofocus on page load is also a nice feature, because when you open a docs page, the only thing you'd want to type is a search query. This is a common feature, witnessed for example on https://api.rubyonrails.com

But I see that the combination of both can be confusing, and maybe we could remove the auto-filling but keep the focus.

RX14 · 2017-11-18T19:22:21Z

@straight-shoota autofilling is actually OK, since s selects what's filled anyway. Changing focus on pageload isnt. It means that up/down arrows or even pgup/pgdown to scroll the page don't work after a reload, which is terrible UX.

I tend to use the keyboard much more than mouse to scroll webpages.

straight-shoota · 2017-11-18T20:20:42Z

Being mostly a mouse user I didn't put that into consideration.
I guess it comes down to mouse users having to press s or keyboard users having to press Esc 🤣 But since the former is just a nice shortcut the latter is truly a huge restriction for usability, it's certainly better not to have autofocus.
Plus, this makes it possible to keep the autofill.

RX14 · 2017-11-21T17:50:18Z

@straight-shoota have you updated the preview site? I still get the focus on the search box.

straight-shoota · 2017-11-21T18:23:56Z

Oh, sry. ~~Now it is updated.~~ It seems I uploaded a wrong version, will resolve it.

straight-shoota · 2017-11-21T19:19:39Z

Fixed it, preview is now correct.

When a search result is opened, the new page does not load the results list and the query field is not focused (`leaveSearchScope`) To refine or start a new search, when the query field is focused (`q` or `/` shortcut) it's content is selected.

`scrollIntoView({behaviour: smooth, block: center})` is only an experimental DOM feature and not supported by many browsers.

RX14 · 2018-01-02T14:09:16Z

🎉 🎉 🎉 I've been waiting for this forever.

Can't wait for the first official crystal release with these docs on crystal-lang.org.

straight-shoota · 2018-01-02T21:15:53Z

@RX14 The current docs could already be upgraded to a version built by a compiler including this PR.

straight-shoota · 2018-01-02T21:20:50Z

In fact, I would really appreciate it if it could be updated soon. That would allow us to catch and fix errors (specifically improvements for the result ranking) which will only emerge if it is broadly used. This way we could get fixes before the next release.

RX14 · 2018-01-02T22:45:57Z

@straight-shoota I'd be okay with this, but the rest of the core team should weigh in.

asterite · 2018-01-02T23:03:00Z

Using better docs for the current version sounds good to me.

* Implement advanced search for API docs This greatly improves client-side search on generated HTML pages of the API docs. It uses the program description available as JSON data and implements a custom search algorithm. The search results include types, constants, methods, macros matching the query string in one or more of the fields name, namespace, description, arguments or type. A custom ranking algorithm sorts results based on matching quality and relevance. Search features: * prefix filter: `#` matches instance methods, `.` matches class methods and macros, `:` matches namespace * search is generally case-insensitive but results with case-equality are ranked higher. * search results list can be navigated by key (when in focus): * `i`, `c`, `ArrowUp`: move upwards * `j`, `h`, `ArrowDown`: move downwards * `Enter`: load result * `Escape`: cancel search * arrows, `Enter` and `Escape` also work when the search-input is in focus * support for URL fragment-based search query: `/#q=mysearch` will initialize a search at page load * page-wide shortkeys to start search: `s` or `/` * Add jsonp support to allow search when API docs are loaded from local file system * Add jsonp support to allow search when API docs are loaded from local file system * Navigator: Fix keybindings, don't jump to result while searching * Improve result ranking based on in-string match position * Don't keep results list when clicking on result When a search result is opened, the new page does not load the results list and the query field is not focused (`leaveSearchScope`) To refine or start a new search, when the query field is focused (`q` or `/` shortcut) it's content is selected. * Fix: don't trigger global shortkeys when input has focus * Add usage modal * remove autofocus of search input * Delay navigator event after search has been performed * Keep current selected result centered on screen * Remove scroll position persistance (interferes with results list) * Delay execution of search until index is loaded * Add custom implemention for centered scrolling position `scrollIntoView({behaviour: smooth, block: center})` is only an experimental DOM feature and not supported by many browsers.

makenowjust reviewed Nov 2, 2017

View reviewed changes

straight-shoota force-pushed the jm-docs-search branch from 34a51af to 4a2845e Compare November 2, 2017 12:22

straight-shoota force-pushed the jm-docs-search branch 2 times, most recently from 23cb560 to 7d23a89 Compare November 4, 2017 19:14

straight-shoota closed this Nov 16, 2017

straight-shoota reopened this Nov 16, 2017

straight-shoota added 11 commits December 22, 2017 13:59

Navigator: Fix keybindings, don't jump to result while searching

4167699

Improve result ranking based on in-string match position

3c1fe5b

Fix: don't trigger global shortkeys when input has focus

b6de03a

Add usage modal

a4dae37

remove autofocus of search input

49cf59c

Delay navigator event after search has been performed

257b0b3

Keep current selected result centered on screen

20f868f

Remove scroll position persistance (interferes with results list)

1bfa4bd

Delay execution of search until index is loaded

c402827

Add custom implemention for centered scrolling position

9b83d32

`scrollIntoView({behaviour: smooth, block: center})` is only an experimental DOM feature and not supported by many browsers.

straight-shoota force-pushed the jm-docs-search branch from 69ff27c to 9b83d32 Compare December 22, 2017 13:08

asterite approved these changes Jan 2, 2018

View reviewed changes

RX14 added kind:feature topic:compiler kind:docs labels Jan 2, 2018

RX14 approved these changes Jan 2, 2018

View reviewed changes

RX14 added this to the Next milestone Jan 2, 2018

RX14 merged commit 1eeb5c0 into crystal-lang:master Jan 2, 2018

straight-shoota deleted the jm-docs-search branch January 2, 2018 21:15

straight-shoota mentioned this pull request Jan 23, 2018

[API doc browser] Search should include nested types #1486

Closed

straight-shoota mentioned this pull request Feb 10, 2018

[API doc browser] Implement fuzzy search #1791

Closed

faustinoaq mentioned this pull request Jun 19, 2018

Create Documentation, Guides and API amberframework/amberframework.org#3

Closed

3 tasks

soanvig mentioned this pull request Jul 20, 2018

Fix docs navigator not scrolling into open type on page load #6420

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improved search feature for API docs #5229

Improved search feature for API docs #5229

straight-shoota commented Nov 1, 2017 •

edited

makenowjust Nov 2, 2017 •

edited

Sija Nov 2, 2017

straight-shoota Nov 2, 2017

straight-shoota commented Nov 2, 2017

RX14 commented Nov 3, 2017 •

edited

RX14 commented Nov 3, 2017 •

edited

straight-shoota commented Nov 3, 2017

RX14 commented Nov 3, 2017

straight-shoota commented Nov 4, 2017

RX14 commented Nov 5, 2017

RX14 commented Nov 5, 2017

straight-shoota commented Nov 5, 2017

RX14 commented Nov 5, 2017

RX14 commented Nov 12, 2017

straight-shoota commented Nov 15, 2017 •

edited

RX14 commented Nov 16, 2017

straight-shoota commented Nov 16, 2017

straight-shoota commented Nov 16, 2017

faustinoaq commented Nov 16, 2017

straight-shoota commented Nov 16, 2017

RX14 commented Nov 18, 2017 •

edited

straight-shoota commented Nov 18, 2017

RX14 commented Nov 18, 2017 •

edited

straight-shoota commented Nov 18, 2017

RX14 commented Nov 21, 2017

straight-shoota commented Nov 21, 2017 •

edited

straight-shoota commented Nov 21, 2017

RX14 commented Jan 2, 2018 •

edited

straight-shoota commented Jan 2, 2018

straight-shoota commented Jan 2, 2018

RX14 commented Jan 2, 2018 •

edited

asterite commented Jan 2, 2018

Improved search feature for API docs #5229

Improved search feature for API docs #5229

Conversation

straight-shoota commented Nov 1, 2017 • edited

Search features:

Preview

makenowjust Nov 2, 2017 • edited

Choose a reason for hiding this comment

Sija Nov 2, 2017

Choose a reason for hiding this comment

straight-shoota Nov 2, 2017

Choose a reason for hiding this comment

straight-shoota commented Nov 2, 2017

RX14 commented Nov 3, 2017 • edited

RX14 commented Nov 3, 2017 • edited

straight-shoota commented Nov 3, 2017

RX14 commented Nov 3, 2017

straight-shoota commented Nov 4, 2017

RX14 commented Nov 5, 2017

RX14 commented Nov 5, 2017

straight-shoota commented Nov 5, 2017

RX14 commented Nov 5, 2017

RX14 commented Nov 12, 2017

straight-shoota commented Nov 15, 2017 • edited

RX14 commented Nov 16, 2017

straight-shoota commented Nov 16, 2017

straight-shoota commented Nov 16, 2017

faustinoaq commented Nov 16, 2017

straight-shoota commented Nov 16, 2017

RX14 commented Nov 18, 2017 • edited

straight-shoota commented Nov 18, 2017

RX14 commented Nov 18, 2017 • edited

straight-shoota commented Nov 18, 2017

RX14 commented Nov 21, 2017

straight-shoota commented Nov 21, 2017 • edited

straight-shoota commented Nov 21, 2017

RX14 commented Jan 2, 2018 • edited

straight-shoota commented Jan 2, 2018

straight-shoota commented Jan 2, 2018

RX14 commented Jan 2, 2018 • edited

asterite commented Jan 2, 2018

straight-shoota commented Nov 1, 2017 •

edited

makenowjust Nov 2, 2017 •

edited

RX14 commented Nov 3, 2017 •

edited

RX14 commented Nov 3, 2017 •

edited

straight-shoota commented Nov 15, 2017 •

edited

RX14 commented Nov 18, 2017 •

edited

RX14 commented Nov 18, 2017 •

edited

straight-shoota commented Nov 21, 2017 •

edited

RX14 commented Jan 2, 2018 •

edited

RX14 commented Jan 2, 2018 •

edited