nixos: tests: firefox: make more comprehensive #23924

7c6f434c · 2017-03-15T15:52:04Z

Run Firefox inside an XTerm, it doesn't crash mysteriously this way.
Also try opening developer tools and checking that Firefox doesn't
crash in the process.

Motivation for this change

In #23917 there is a Firefox crash that happens all the time but cannot be reproduced manually. The workaround for this crash makes the build that crashes on every keypress pass the test, so I added a check that developer tools can be opened.

Things done

Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- Linux
Opened the resulting screenshot

mention-bot · 2017-03-15T15:52:07Z

@7c6f434c, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @peti and @chaoflow to be potential reviewers.

vcunat · 2017-03-15T16:00:50Z

Certainly 👍 for the xterm change to unblock channel for now.

It would be nice if we could somehow detect the process dying instead of blocking on waitForWindow indefinitely, but I consider that future work.

The space key is meant to test scrolling, or am I missing something? It seems this small page is likely to fit into the window.

7c6f434c · 2017-03-15T16:09:33Z

No. The browser asks whether to make it the default browser with probability around 90%, the default option — we do not care which one — should not crash, and we need to close this window to be able to send events to the real browser window. If by any chance the browser does not ask to make itself the default — then space is probably interpreted as scrolling, no harm done. When we see the situation where both guesses are wrong and something bad happens, we investigate it…

vcunat · 2017-03-15T16:14:06Z

Oh, I forgot that thing :-) I didn't get what "default browser window" meant (parenthesized the other way).

7c6f434c · 2017-03-15T16:14:49Z

@vcunat by the way, the channel is blocked until you merge at least the xterm hack. On the other hand, it doesn't matter until the fixed Firefox is actually built, but that build is hopefully less than an hour.

7c6f434c · 2017-03-15T16:19:11Z

Added a more detailed comment. The test is a scarily good model of user behaviour in this place!

vcunat · 2017-03-15T16:23:34Z

I know that, but just patching ICU is a nontrivial rebuild by itself AFAIK.

Run Firefox inside an XTerm, it doesn't crash mysteriously this way. Also try opening developer tools and checking that Firefox doesn't crash in the process.

7c6f434c · 2017-03-15T16:29:30Z

Ah you are right, the test is fast to run and there will be a lot of other contenders for build slots.

vcunat · 2017-03-15T16:31:38Z

Oh, no. trunk-combined eval is repeatedly throwing

hydra-eval-jobs returned signal 6:
Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS

and consequently hasn't had a successful evaluation for two days. EDIT: at least plain trunk jobset evaluated now, after a few attempts...

7c6f434c · 2017-03-15T16:51:17Z

Oh, ICU update means Freetype rebuild, right.

vcunat · 2017-03-15T16:52:23Z

No, freetype has the same derivation hash.

vcunat · 2017-03-15T16:52:36Z

It seems especially big on Darwin.

vcunat · 2017-03-15T16:56:59Z

... which is because their stdenv depends on it (news for me).

Mic92 · 2017-03-16T17:19:45Z

nixos/tests/firefox.nix

      $machine->waitForWindow(qr/Valgrind/);
      $machine->sleep(40); # wait until Firefox has finished loading the page
+      $machine->execute("xdotool key space"); # do I want to make Firefox the
+                             # default browser? I just want to close the dialog


Just an off-topic question, how do you debug the test driver? I tried to unbrick gdm test, but I have no idea in what state the desktop is.

7c6f434c · 2017-03-16T17:45:04Z

Just an off-topic question, how do you debug the test driver?

By randomly issuing arbitrary commands, of course! Well, you set HOME to something you don't care much about and run the script `nixos-test-driver` and then feed it the contents of `test-script` line-by-line manually. If the test-script is too long, you can probably use `expect`. I don't use the mainline NixOS, and I don't usually touch its tests, it's just this time that a test was relevant to a problem I also had, so I am probably missing a much better solution somewhere.

7c6f434c · 2017-03-17T12:00:15Z

OK, no negative feedback and a hope to unblock the channel. I merge.

vcunat · 2017-03-17T12:08:08Z

trunk-combined still can't be evaluated by Hydra, so this won't be enough, I'm afraid.

nixos: tests: firefox: make more comprehensive (cherry picked from commit f9fb38f) The test will block the channel without the xterm change.

7c6f434c · 2017-03-17T12:42:31Z

`trunk-combined` still can't be evaluated by Hydra, so this won't be enough, I'm afraid.

Well, you are right, of course. I think this change does nothing for evaluation but increases the chances of build success after the evaluation is fixed. As I don't use channels I won't notice the moment when this PR becomes immediately relevant, so I merge it now.

globin · 2017-03-22T14:23:10Z

The test is still failing:
https://hydra.nixos.org/build/50586188
Log: http://cache.nixos.org/log/cna2imysh97liqsp8l0fnkymv2jd4r57-vm-test-run-firefox.drv

vcunat · 2017-03-22T14:38:14Z

The log... I wondered why logs disappeared from hydra.nixos.org - but I can't determine how to read the log URL you provided. It's not plaintext and I can't detect any common compression.

7c6f434c · 2017-03-22T14:38:43Z

And content type is text/plain — that's just not true…

vcunat · 2017-03-22T14:50:47Z

That's some other problem:

machine# [    5.085679] display-manager[812]: (WW) Falling back to old probe method for modesetting
machine# [    5.086332] display-manager[812]: (EE) open /dev/dri/card0: No such file or directory
machine# [    5.086982] display-manager[812]: (EE) Screen 0 deleted because of no matching config section.
machine# [    5.087667] display-manager[812]: (II) UnloadModule: "modesetting"
machine# [    5.088207] display-manager[812]: (EE) Device(s) detected, but none match those in the config file.
machine# [    5.088881] display-manager[812]: (EE)
machine# [    5.089278] display-manager[812]: Fatal server error:
machine# [    5.089747] display-manager[812]: (EE) no screens found(EE)
machine# [    5.090261] display-manager[812]: (EE)
machine# [    5.090672] display-manager[812]: Please consult the The X.Org Foundation support
machine# [    5.091238] display-manager[812]:    at http://wiki.x.org
machine# [    5.091743] display-manager[812]:  for help.
machine# [    5.092154] display-manager[812]: (EE) Please also check the log file at "/dev/null" for additional information.
machine# [    5.092876] display-manager[812]: (EE)
machine# [    5.093309] display-manager[812]: (EE) Server terminated with error (1). Closing log file.

globin · 2017-03-22T14:56:02Z

@vcunat breaks in firefox, works in chromium 🤷‍♂️

vcunat · 2017-03-22T15:02:08Z

It also works in firefox on 17.03 (it includes this PR).

7c6f434c · 2017-03-22T15:33:48Z

Fails in chromium on master for me. i3wm.nix test is also unhappy. Seems to be a non-specific problem…

Edited to add: gnome3.nix — which is likely fresher — also fails.

vcunat · 2017-03-22T19:28:05Z

OK, let my try to bisect this on i3wm (it's lighter). It will take some time, as Hydra evaluations were blocked for several days and the potential interval is long.

vcunat · 2017-03-22T20:29:55Z

The culprit is 0c262a6; discussion: #23890 (comment)

7c6f434c force-pushed the improve-firefox-test branch from 6a62966 to 14a3412 Compare March 15, 2017 16:18

nixos: tests: firefox: make more comprehensive

14a3412

Run Firefox inside an XTerm, it doesn't crash mysteriously this way. Also try opening developer tools and checking that Firefox doesn't crash in the process.

Mic92 reviewed Mar 16, 2017

View reviewed changes

vcunat added the 1.severity: blocker label Mar 17, 2017

7c6f434c merged commit f9fb38f into NixOS:master Mar 17, 2017

vcunat pushed a commit that referenced this pull request Mar 17, 2017

Merge pull request #23924 from 7c6f434c/improve-firefox-test

df5a471

nixos: tests: firefox: make more comprehensive (cherry picked from commit f9fb38f) The test will block the channel without the xterm change.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nixos: tests: firefox: make more comprehensive #23924

nixos: tests: firefox: make more comprehensive #23924

7c6f434c commented Mar 15, 2017

mention-bot commented Mar 15, 2017

vcunat commented Mar 15, 2017

7c6f434c commented Mar 15, 2017

vcunat commented Mar 15, 2017 •

edited

7c6f434c commented Mar 15, 2017

7c6f434c commented Mar 15, 2017

vcunat commented Mar 15, 2017

7c6f434c commented Mar 15, 2017

vcunat commented Mar 15, 2017 •

edited

7c6f434c commented Mar 15, 2017

vcunat commented Mar 15, 2017

vcunat commented Mar 15, 2017

vcunat commented Mar 15, 2017

Mic92 Mar 16, 2017 •

edited

7c6f434c commented Mar 16, 2017 via email

7c6f434c commented Mar 17, 2017

vcunat commented Mar 17, 2017

7c6f434c commented Mar 17, 2017 via email

globin commented Mar 22, 2017

vcunat commented Mar 22, 2017

7c6f434c commented Mar 22, 2017

vcunat commented Mar 22, 2017

globin commented Mar 22, 2017

vcunat commented Mar 22, 2017

7c6f434c commented Mar 22, 2017 •

edited

vcunat commented Mar 22, 2017

vcunat commented Mar 22, 2017

nixos: tests: firefox: make more comprehensive #23924

nixos: tests: firefox: make more comprehensive #23924

Conversation

7c6f434c commented Mar 15, 2017

Motivation for this change

Things done

mention-bot commented Mar 15, 2017

vcunat commented Mar 15, 2017

7c6f434c commented Mar 15, 2017

vcunat commented Mar 15, 2017 • edited

7c6f434c commented Mar 15, 2017

7c6f434c commented Mar 15, 2017

vcunat commented Mar 15, 2017

7c6f434c commented Mar 15, 2017

vcunat commented Mar 15, 2017 • edited

7c6f434c commented Mar 15, 2017

vcunat commented Mar 15, 2017

vcunat commented Mar 15, 2017

vcunat commented Mar 15, 2017

Mic92 Mar 16, 2017 • edited

Choose a reason for hiding this comment

7c6f434c commented Mar 16, 2017 via email

7c6f434c commented Mar 17, 2017

vcunat commented Mar 17, 2017

7c6f434c commented Mar 17, 2017 via email

globin commented Mar 22, 2017

vcunat commented Mar 22, 2017

7c6f434c commented Mar 22, 2017

vcunat commented Mar 22, 2017

globin commented Mar 22, 2017

vcunat commented Mar 22, 2017

7c6f434c commented Mar 22, 2017 • edited

vcunat commented Mar 22, 2017

vcunat commented Mar 22, 2017

vcunat commented Mar 15, 2017 •

edited

vcunat commented Mar 15, 2017 •

edited

Mic92 Mar 16, 2017 •

edited

7c6f434c commented Mar 22, 2017 •

edited