HTML report (--log-html) shows incorrect summary #18408
Labels
Comments
Hexcles
changed the title
|
This is a general issue related to the HTML log handler (or its interaction with wptrunner). I've tried Firefox & Chrome and the numbers are also wrong in the HTML file. AFAIK, no one really uses the html report and it must have bit-rotten somewhere... What do you need it for? The JSON format ( |
Hexcles
added
infra
priority:backlog
wptrunner
Hexcles
changed the title
|
Hmm... Ideally speaking, I need the results that can provide me the
following:
1) check the overview: I can have an overview at different levels (top
level, directory level, test file level)
2) get to the failure stack traces when I want to.
3) access partial results before tests get finished: it takes too long to
see any partial results in the form of reporting
log-wptreport seems to provide 1) and 2), but it's a bit inconvenient to
use a separate tool to browser JSON file.
Also, it seems to provide only two levels of details.
Is there any way to compile everything into HTML page as tests progress, in
the format of http://wpt.fyi/results?
…On Wed, Aug 14, 2019 at 10:58 AM Robert Ma ***@***.***> wrote:
This is a general issue related to the HTML log handler (or its
interaction with wptrunner). I've tried Firefox & Chrome and the numbers
are also wrong in the HTML file.
AFAIK, no one really uses the html report and it must have bit-rotten
somewhere...
What do you need it for? The JSON format (--log-wptreport) is much easier
to consume.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#18408?email_source=notifications&email_token=ABS7TR3M3YOA77BW7HUFRKTQERBVJA5CNFSM4ILOBVEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4JTKNQ#issuecomment-521352502>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABS7TR3KUAABQOM5PL3WZZTQERBVJANCNFSM4ILOBVEA>
.
|
|
Are your needs ranked by importance? I'm afraid we don't have a logger that can satisfy all the needs at the moment. The default
Are you saying the JSON does not have stack trace? |
|
Robert, thanks for the answers.
I might be missing something, but when I try locally
--log-grouped seems to produce some errors:
$ ./wpt run android_webview dom/ --log-grouped a.txt
[0/322] /dom/abort/event.any.worker.html
mozlog.structuredlog: Failure calling log handler:
Traceback (most recent call last):
File
"/usr/local/google/code/newdrive/code/wpt/_venv/lib/python2.7/site-packages/mozlog/structuredlog.py",
line 273, in _handle_log
handler(data)
File
"/usr/local/google/code/newdrive/code/wpt/_venv/lib/python2.7/site-packages/mozlog/handlers/base.py",
line 74, in __call__
formatted = self.formatter(data)
File
"/usr/local/google/code/newdrive/code/wpt/_venv/lib/python2.7/site-packages/mozlog/handlers/base.py",
line 42, in __call__
return self.inner(item)
File
"/usr/local/google/code/newdrive/code/wpt/_venv/lib/python2.7/site-packages/mozlog/reader.py",
line 74, in __call__
return handler(data)
File
"/usr/local/google/code/newdrive/code/wpt/_venv/lib/python2.7/site-packages/mozlog/formatters/grouping.py",
line 235, in test_end
del self.running_tests[data['thread']]
KeyError: 'TestRunnerManager-1'
> Also, it seems to provide only two levels of details.
Are you saying the JSON does not have stack trace?
No, I'm just saying that JSON is just a bunch of nodes data and the top
level is Object. Here's an example JSON result:
object {4}
run_info {14}
product : chrome_android
bits : 64
has_sandbox : true
linux_distro : Debian
verify : false
automation : false
os_version : rodete
version : Debian rodete
wasm : false
webrender : false
debug : false
os : linux
processor : x86_64
revision : 82629f3a4950e6c2072a3aa993f8d5b52c661c78
time_start : 15599573441272019-06-08T01:29:04.127Z
time_end : 15599574519952019-06-08T01:30:51.995Z
results [14]
0 {5}
1 {6}
2 {5}
3 {6}
It only has top-level detail and actual test results. It does not seem to
produce any subdirectory-level summary, for example.
…On Wed, Aug 14, 2019 at 2:31 PM Robert Ma ***@***.***> wrote:
Are your needs ranked by importance? I'm afraid we don't have a logger
that can satisfy all the needs at the moment.
The default --log-grouped is a human-friendly, streaming log that tells
you both the progress and the detail stack trace (2 & 3).
Also, it seems to provide only two levels of details.
Are you saying the JSON does not have stack trace?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#18408?email_source=notifications&email_token=ABS7TRZ7TYVIL7F4WWYXKADQER2UHA5CNFSM4ILOBVEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4KFN2Y#issuecomment-521426667>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABS7TRYJ6YG33JL6Y2K66GLQER2UHANCNFSM4ILOBVEA>
.
|
|
Hmm, the As for the JSON report, yes the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Commandline tried:
./wpt run android_webview dom/ --log-html a.html --log-mach -
(chrome_android shows a similar result)
a.html (attached) has the following summary:
322 tests ran in 534.9 seconds.
0 passed, 0 skipped, 0 failed, 10 errors.
0 expected failures, 0 unexpected passes, 0 known intermittent results.
This doesn't seem quite right because the number of passed, skipped, failed, and errors don't sum up to the total number of tests ran (322).
I couldn't attach the file, but here's the beginning of the rest of the HTML file:
EXPECTED_OK | /dom/abort/event.any.worker.html | 1.28 |
EXPECTED_OK | /dom/abort/event.any.html | 0.37 |
EXPECTED_OK | /dom/collections/HTMLCollection-as-prototype.html | 0.37 |
EXPECTED_OK | /dom/collections/HTMLCollection-delete.html | 1.18 |
EXPECTED_OK | /dom/collections/HTMLCollection-empty-name.html | 1.02 |
EXPECTED_OK | /dom/collections/HTMLCollection-supported-property-indices.html | 0.37 |
EXPECTED_OK | /dom/collections/HTMLCollection-supported-property-names.html | 1.08 |
EXPECTED_OK | /dom/collections/domstringmap-supported-property-names.html | 0.87 |
EXPECTED_OK | /dom/collections/namednodemap-supported-property-names.html | 0.42 |
EXPECTED_OK | /dom/events/AddEventListenerOptions-once.html | 0.42 |
EXPECTED_OK | /dom/events/AddEventListenerOptions-passive.html | 0.32 |
EXPECTED_OK | /dom/events/CustomEvent.html | 0.32 |
EXPECTED_OK | /dom/events/Event-cancelBubble.html | 0.37 |
EXPECTED_OK | /dom/events/Event-constants.html | 0.47 |
EXPECTED_OK | /dom/events/Event-constructors.html | 0.52 |
EXPECTED_OK | /dom/events/Event-defaultPrevented-after-dispatch.html | 0.37 |
EXPECTED_OK | /dom/events/Event-defaultPrevented.html | 0.42 |
EXPECTED_OK | /dom/events/Event-dispatch-bubble-canceled.html | 0.37 |
EXPECTED_OK | /dom/events/Event-dispatch-bubbles-false.html | 0.37 |
EXPECTED_OK | /dom/events/Event-dispatch-bubbles-true.html | 0.42 |
EXPECTED_OK | /dom/events/Event-dispatch-click.html | 0.42 |
EXPECTED_OK | /dom/events/Event-dispatch-detached-click.html | 0.42 |
EXPECTED_OK | /dom/events/Event-dispatch-handlers-changed.html | 0.47 |
EXPECTED_OK | /dom/events/Event-dispatch-listener-order.window.html | 0.87 |
EXPECTED_OK | /dom/events/Event-dispatch-multiple-cancelBubble.html | 1.02 |
EXPECTED_OK | /dom/events/Event-dispatch-multiple-stopPropagation.html | 0.37 |
EXPECTED_OK | /dom/events/Event-dispatch-omitted-capture.html | 0.47 |
UNEXPECTED_TIMEOUT | /dom/events/Event-dispatch-on-disabled-elements.html | 60.38 |
EXPECTED_OK | /dom/events/Event-dispatch-order.html | 0.87 |
EXPECTED_OK | /dom/events/Event-dispatch-other-document.html | 0.42 |
EXPECTED_OK | /dom/events/Event-dispatch-propagation-stopped.html | 0.37
...
The text was updated successfully, but these errors were encountered: