Right now this is a module that encodes every frame delta from QEMU to
an intermediate format so that we can use FFmpeg to encode it into a
proper video.

In the end, the goal is to have videos of the graphical machine output
for the NixOS tests, which is especially useful in tests involving X
where we're basically blind when we run into a race condition or a
loaded Hydra node causes the test to fail.

The reason why I picked the approach to use an intermediate format and
encode it properly later is based on my benchmarks of some seemingly
simple approaches I tried before.

First of all I tried to search the web for solutions that already
existed and found a few, but they weren't really suitable:

  * Use the screendump QMP command to collect frames from the VM, which
    works to some degree but it misses frames.
  * Enable SPICE[1] and capture video from the server, which I actually
    tried to implement before the next option. However, existing
    solutions for capturing video off a SPICE server are rare and when
    testing with my own PoC implementation, I got frame drops as well
    and I didn't manage to capture early boot.
  * Try a patch[2] from the QEMU development mailing list, which adds a
    HMP command to capture and encode it directly to a video. This was
    the slowest option of all and it even lead to test failures because
    we got a timeout during VM startup.
  * Similar results to SPICE I had when capturing video using VNC and
    VncProxy[3].

So I dug through the QEMU code base and found out that UI modules get
frame deltas from Pixman, which is perfect for us, because we're not
losing frames and it also allows direct access to pixel data. It also is
fast and I couldn't even properly benchmark the overhead properly as
tests usually tend to vary in speed for a few seconds.

Before actually writing our own intermediate format, I tried to use an
existing format that would be suitable for us. The requirements for this
format would be to support different frame sizes and variable
framerates, plus it needs to be very fast to encode.

While asking in the #ffmpeg channel on freenode, the best format for
these requirements would be using the NUT[4] format (thanks to "furq"
for the suggestion). However while reading the format specification I
came to the realisation that our requirements are so simple that even
NUT is complicated in comparison, which is why I written our own format.

The specification is as follows:

  The first byte (the opcode) is either an 'S' (0x53, for "switch") or
  an 'U' (0x55, for "update") and determines the format of the following
  data.

  A "switch" is a surface change, like eg. a resize of the display and
  the data following the opcode are the dimensions (width and height,
  both are unsigned 32 bit integers), format (unsigned 8 bit integer)
  and bytes per pixel (unsigned 8 bit integer, currently either 2 or 4)
  of the surface.

  An "update" is a portion of the region that has changed since the last
  update and it's followed by X, Y, width and height (all 32 bit
  unsigned integers) coordinates of the updated region, the absolute
  time (64 bit unsigned integer) and the raw frame data afterwards. Note
  that we don't provide a length here, because we can infer that from
  the bytes per pixel of the last "switch" packet and the coordinates.

  All of the data is in the native endian format of the host processor
  architecture, which is not a problem, because encoding of the final
  video will take place on the same processor architecture.

All of the data is also gzip compressed, so that we don't accumulate
gigabytes of frame data during test runs.

I also moved the qemu_test expression out of the default.nix of the main
qemu expression, so that when we improve this we don't accidentally
break stuff for users of the normal QEMU.

[1]: https://www.spice-space.org/
[2]: https://lists.gnu.org/archive/html/qemu-devel/2017-05/msg00865.html
[3]: https://github.com/amitbet/vncproxy
[4]: https://ffmpeg.org/~michael/nut.txt

Signed-off-by: aszlig <aszlig@nix.build>

This tool converts the intermediate format that we now get from QEMU
when running with --nixos-test outfile into any video format that can be
played by most video players, particularly web browsers when watching
the videos on Hydra.

The video format to be used is determined by the output file name, so
it's easy to change it to a different format rather than WebM (which is
the format this was designed for) by simply changing the file name.

I also have to give credit to "kepstin" from the #ffmpeg channel on
freenode for helping with the rescaling of the presentation timestamp.

Signed-off-by: aszlig <aszlig@nix.build>

This now encodes the raw video frames for every single VM test in our
internal format. Due to compression the average video file size is about
~3M so I think it's a good idea to enable it by default so we can see
post-mortem what went wrong.

Instead of killing the machines with the SIGKILL signal during cleanup,
we now kill it with SIGTERM because otherwise the atexit() handler
doesn't run and we get a video file without the gzip buffer flushed.

I'm also adding a helper attribute called 'videos' to every test, so
that all these videos can be encoded on Hydra. The derivation propagates
the build products from the actual test runner and also adds videos to
the Hydra build products.

Signed-off-by: aszlig <aszlig@nix.build>

This is mainly for Hydra so that the videos show up in the build
products. Encoding this to WebM takes a while especially for long test
runs, but it also helps debugging tests without the need to manually run
the encoding process on the actual output path of the test runner's
videos.

Signed-off-by: aszlig <aszlig@nix.build>

When writing with header_size - 1 we shouldn't check whether header_size
has been written. I also added an exit(1) to make sure this is fatal,
because otherwise things such as the issue here go completely unnoticed.

Signed-off-by: aszlig <aszlig@nix.build>

For some tests such as runInMachine, there is neither a .videos
attribute nor does it actually need to record videos.

However, right now it still does record videos so we need to fix that
soon.

Signed-off-by: aszlig <aszlig@nix.build>

This is actually quite common whenever video frames are written that a
signal can interrupt a call to write(). The first option that came into
my mind was to use sigprocmask() to make sure that we don't get
interrupted while writing the frame data. However this would also
introduce some overhead.

So instead, we're now just ignoring a frame update/switch whenever we
reach the end of the file, so we don't have that overhead and also only
loose one frame to the end of the video stream because we do actually
flush the gzip buffer in the atexit() handler.

Signed-off-by: aszlig <aszlig@nix.build>

When building with Nix there is a NIX_BUILD_CORES environment variable
set to either the default, which should be equal to av_cpu_count() or a
user-set value, so let's respect that.

If the value is 0 or unset (not inside a Nix build), we use
av_cpu_count() instead.

Signed-off-by: aszlig <aszlig@nix.build>

If we only want to run tests without actually encoding the videos, it
doesn't really make sense to pull in the FFmpeg dependency.

When debugging tests we can still either use the encoder manually or
simply append the .videos attribute in order to get the encoded videos
(which then of course will pull in the FFmpeg dependency).

Signed-off-by: aszlig <aszlig@nix.build>