mbedtls: cmake&ninja, threading, clean-up; hiawatha: options, Nix mbedtls #41722

Anton-Latukha · 2018-06-08T18:34:01Z

Motivation for this change

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

Anton-Latukha · 2018-06-08T18:37:11Z

Let's discuss what to do more.

> cmake -LH
...
// Choose the type of build: None Debug Release Coverage ASan ASanDbg MemSan MemSanDbg Check CheckFull
CMAKE_BUILD_TYPE:STRING=

// Install path prefix, prepended onto install directories.
CMAKE_INSTALL_PREFIX:PATH=/var/empty/local

// Build mbed TLS programs.
ENABLE_PROGRAMS:BOOL=ON

// Build mbed TLS tests.
ENABLE_TESTING:BOOL=ON

// Build mbed TLS with zlib library.
ENABLE_ZLIB_SUPPORT:BOOL=OFF

// Install mbed TLS headers.
INSTALL_MBEDTLS_HEADERS:BOOL=ON

// Explicitly link mbed TLS library to pthread.
LINK_WITH_PTHREAD:BOOL=OFF

// Allow unsafe builds. These builds ARE NOT SECURE.
UNSAFE_BUILD:BOOL=OFF

// Build mbed TLS with the pkcs11-helper library.
USE_PKCS11_HELPER_LIBRARY:BOOL=OFF

// Build mbed TLS shared library.
USE_SHARED_MBEDTLS_LIBRARY:BOOL=OFF

// Build mbed TLS static library.
USE_STATIC_MBEDTLS_LIBRARY:BOOL=ON

To configure CMake for building shared libraries, use:

cmake -DUSE_SHARED_MBEDTLS_LIBRARY=On /path/to/mbedtls_source

There are many different build modes available within the CMake buildsystem. Most of them are available for gcc and clang, though some are compiler-specific:

`Release` - This generates the default code without any unnecessary information in the binary files.
`Debug` - This generates debug information and disables optimization of the code.
`Coverage` - This generates code coverage information in addition to debug information.
`ASan` - This instruments the code with AddressSanitizer to check for memory errors. (This includes LeakSanitizer, with recent version of gcc and clang.) (With recent version of clang, this mode also instruments the code with UndefinedSanitizer to check for undefined behaviour.)
`ASanDbg` - Same as ASan but slower, with debug information and better stack traces.
`MemSan` - This instruments the code with MemorySanitizer to check for uninitialised memory reads. Experimental, needs recent clang on Linux/x86_64.
`MemSanDbg` - Same as MemSan but slower, with debug information, better stack traces and origin tracking.
`Check` - This activates the compiler warnings that depend on optimization and treats all warnings as errors.

Anton-Latukha · 2018-06-08T18:39:32Z

Tests

Mbed TLS includes an elaborate test suite in tests/ that initially requires Perl to generate the tests files (e.g. test_suite_mpi.c). These files are generated from a function file (e.g. suites/test_suite_mpi.function) and a data file (e.g. suites/test_suite_mpi.data). The function file contains the test functions. The data file contains the test cases, specified as parameters that will be passed to the test function.

For machines with a Unix shell and OpenSSL (and optionally GnuTLS) installed, additional test scripts are available:

`tests/ssl-opt.sh` runs integration tests for various TLS options (renegotiation, resumption, etc.) and tests interoperability of these options with other implementations.
`tests/compat.sh` tests interoperability of every ciphersuite with other implementations.
`tests/scripts/test-ref-configs.pl` test builds in various reduced configurations.
`tests/scripts/key-exchanges.pl` test builds in configurations with a single key exchange enabled
`tests/scripts/all.sh` runs a combination of the above tests, plus some more, with various build options (such as ASan, full config.h, etc).

Anton-Latukha · 2018-06-08T18:43:39Z

Both gcc and clang are supported.

Anton-Latukha · 2018-06-08T21:10:49Z

https://github.com/ARMmbed/mbedtls/blob/c041435fcf9e1fd55f9e4ea15c6f6e4107a4c97e/include/mbedtls/config.h#L2612-L2629

And all other options are there.

Anton-Latukha · 2018-06-09T01:58:45Z

Configure script keys:
https://github.com/ARMmbed/mbedtls/blob/master/scripts/config.pl

Anton-Latukha · 2018-06-09T08:57:27Z

/tests/scripts/all.sh has many completely destructive scripts, and also relies heavily on automake and everything.

# Warning: the test is destructive. It includes various build modes and
# configurations, and can and will arbitrarily change the current CMake
# configuration. The following files must be committed into git:
#    * include/mbedtls/config.h
#    * Makefile, library/Makefile, programs/Makefile, tests/Makefile
# After running this script, the CMake cache will be lost and CMake
# will no longer be initialised.
#
# The script assumes the presence of a number of tools:
#   * Basic Unix tools (Windows users note: a Unix-style find must be before
#     the Windows find in the PATH)
#   * Perl
#   * GNU Make
#   * CMake
#   * GCC and Clang (recent enough for using ASan with gcc and MemSan with clang, or valgrind)
#   * arm-gcc and mingw-gcc
#   * ArmCC 5 and ArmCC 6, unless invoked with --no-armcc
#   * Yotta build dependencies, unless invoked with --no-yotta
#   * OpenSSL and GnuTLS command line tools, recent enough for the
#     interoperability tests. If they don't support SSLv3 then a legacy
#     version of these tools must be present as well (search for LEGACY
#     below).

Anton-Latukha · 2018-06-09T10:55:06Z

Looks something like this.

CMake build works great on Linux.
I removed hacks related to autotools on macOS.

Please launch macOS build.

Anton-Latukha · 2018-06-09T17:24:07Z

Calling maintainer @fpletz. Hello!

Anton-Latukha · 2018-06-09T17:25:24Z

@jtojnar.
Please, can you say your word on this one.

jtojnar · 2018-06-09T17:27:56Z

@GrahamcOfBorg build mbedtls

GrahamcOfBorg · 2018-06-09T17:28:38Z

Success on aarch64-linux (full log)

Attempted: mbedtls

Partial log (click to expand)

shrinking /nix/store/f74df1im3bjgidd99cpkjbm6am0c30lz-mbedtls-2.10.0/bin/crypt_and_hash
shrinking /nix/store/f74df1im3bjgidd99cpkjbm6am0c30lz-mbedtls-2.10.0/bin/aescrypt2
shrinking /nix/store/f74df1im3bjgidd99cpkjbm6am0c30lz-mbedtls-2.10.0/lib/libmbedcrypto.so.2.10.0
shrinking /nix/store/f74df1im3bjgidd99cpkjbm6am0c30lz-mbedtls-2.10.0/lib/libmbedx509.so.2.10.0
shrinking /nix/store/f74df1im3bjgidd99cpkjbm6am0c30lz-mbedtls-2.10.0/lib/libmbedtls.so.2.10.0
strip is /nix/store/qg2agrqkf240s656d207zqhipl0bc2id-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/f74df1im3bjgidd99cpkjbm6am0c30lz-mbedtls-2.10.0/lib  /nix/store/f74df1im3bjgidd99cpkjbm6am0c30lz-mbedtls-2.10.0/bin
patching script interpreter paths in /nix/store/f74df1im3bjgidd99cpkjbm6am0c30lz-mbedtls-2.10.0
checking for references to /build in /nix/store/f74df1im3bjgidd99cpkjbm6am0c30lz-mbedtls-2.10.0...
/nix/store/f74df1im3bjgidd99cpkjbm6am0c30lz-mbedtls-2.10.0

GrahamcOfBorg · 2018-06-09T17:28:43Z

Success on x86_64-linux (full log)

Attempted: mbedtls

Partial log (click to expand)

shrinking /nix/store/34x4h1y0cdax58vyv3inj1hr6vag5iqk-mbedtls-2.10.0/bin/crl_app
shrinking /nix/store/34x4h1y0cdax58vyv3inj1hr6vag5iqk-mbedtls-2.10.0/bin/rsa_sign
shrinking /nix/store/34x4h1y0cdax58vyv3inj1hr6vag5iqk-mbedtls-2.10.0/bin/dtls_server
shrinking /nix/store/34x4h1y0cdax58vyv3inj1hr6vag5iqk-mbedtls-2.10.0/lib/libmbedcrypto.so.2.10.0
shrinking /nix/store/34x4h1y0cdax58vyv3inj1hr6vag5iqk-mbedtls-2.10.0/lib/libmbedx509.so.2.10.0
shrinking /nix/store/34x4h1y0cdax58vyv3inj1hr6vag5iqk-mbedtls-2.10.0/lib/libmbedtls.so.2.10.0
strip is /nix/store/21ymadblbmsbb2bk4q7gl4kjasp8zmgd-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/34x4h1y0cdax58vyv3inj1hr6vag5iqk-mbedtls-2.10.0/lib  /nix/store/34x4h1y0cdax58vyv3inj1hr6vag5iqk-mbedtls-2.10.0/bin
patching script interpreter paths in /nix/store/34x4h1y0cdax58vyv3inj1hr6vag5iqk-mbedtls-2.10.0
checking for references to /build in /nix/store/34x4h1y0cdax58vyv3inj1hr6vag5iqk-mbedtls-2.10.0...

GrahamcOfBorg · 2018-06-09T17:29:03Z

Success on x86_64-darwin (full log)

Attempted: mbedtls

Partial log (click to expand)

-- Installing: /nix/store/csyz7nid4zkf766z9djsxspx4b6cafnd-mbedtls-2.10.0/bin/req_app
-- Installing: /nix/store/csyz7nid4zkf766z9djsxspx4b6cafnd-mbedtls-2.10.0/bin/cert_req
-- Installing: /nix/store/csyz7nid4zkf766z9djsxspx4b6cafnd-mbedtls-2.10.0/bin/cert_write
-- Installing: /nix/store/csyz7nid4zkf766z9djsxspx4b6cafnd-mbedtls-2.10.0/bin/strerror
-- Installing: /nix/store/csyz7nid4zkf766z9djsxspx4b6cafnd-mbedtls-2.10.0/bin/pem2der
post-installation fixup
strip is /nix/store/yyak5sjv68n9vdgnkd2cb5djk1l9sqj5-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/csyz7nid4zkf766z9djsxspx4b6cafnd-mbedtls-2.10.0/lib  /nix/store/csyz7nid4zkf766z9djsxspx4b6cafnd-mbedtls-2.10.0/bin
patching script interpreter paths in /nix/store/csyz7nid4zkf766z9djsxspx4b6cafnd-mbedtls-2.10.0
/nix/store/csyz7nid4zkf766z9djsxspx4b6cafnd-mbedtls-2.10.0

jtojnar · 2018-06-09T17:29:18Z

pkgs/servers/http/hiawatha/default.nix

@@ -7,7 +7,7 @@
 , libxslt
 , libxml2

-, enableSSL ? true
+, enableSystemTls ? true, mbedtls ? null


We should always use system TLS, if we build with TLS support.

System-wide thread-able dynamic library - makes it a less secure variant to bundled library.
Through threading process in some cases some information can be shared, or gathered.
https://tls.mbed.org/kb/development/thread-safety-and-multi-threading

That is why I made this switch. For truly paranoid people to crank-up the security more.

Sigh. Of course, everything always has some security explanation.

P.S.

As this is secure server - I enabled TLS by default.
Case when TLS would not be needed, but secure server needed - is possible, but probably not a spread case.

I can imagine encryption to be disabled on embedded devices or local server in protected network.

Using bundled libraries is inherently less secure than using the system one. Nixpkgs maintainers can apply security fixes and update the library globally, long before hiawatha or other consumer updates. As for the link, I think it refers to memory safety (as usual with shared memory and concurrency), different applications would use different memory ranges.

Yes. I also understood words as they are about parallel threads in one application can indirectly disclose some information to one another.

I would change comment notes.

As they are elementary, I implemented all Hiawatha build switches at this point. That looks like not too superfluous.

If this is the case, it really would not matter when system library is used. The threads will use the same library in either case.

jtojnar · 2018-06-09T17:32:03Z

pkgs/development/libraries/mbedtls/default.nix

-  ];
-
-  postInstall = stdenv.lib.optionalString stdenv.isDarwin ''
-    install_name_tool -change libmbedcrypto.dylib $out/lib/libmbedcrypto.dylib $out/lib/libmbedtls.dylib


Possibly ask the author of these lines.

@Mic92 Good day!

Long story short.
I migrate mbed TLS to officially supported CMake.

Mostly Linux and Mac builds seems to be clean.

Please say your toughs on these changes that undo also positInstalls from commit.

I can not find way to text macOS, or get a bundle of builded by bot /nix/store/csyz7nid4zkf766z9djsxspx4b6cafnd-mbedtls-2.10.0 to investigate.

So, please say your thoughts on these changes.

I think the fixes are there primarily for the consumers of the library

I did not add these lines.
Are those required? @mnacamura

My addition was the DLEXT=dylib part only. If extension is .so, some programs cannot find library on darwin. So I added it.

The contents of postInstall was I think added in #20940.

But the build by cmake seems to do it correctly without these lines.

Summary: We decided that this clean-up of code is legitimate, so it stays.

jtojnar · 2018-06-19T18:37:21Z

pkgs/servers/http/hiawatha/default.nix

+    ( if enableCache     then "-DENABLE_CACHE=on"       else "-DENABLE_CACHE=off"       )
+    ( if enableIpV6      then "-DENABLE_IPV6=on"        else "-DENABLE_IPV6=off"        )
+    ( if enableTls       then "-DENABLE_TLS=on"         else "-DENABLE_TLS=off"         )
+    ( if enableSystemTls then "-DUSE_SYSTEM_MBEDTLS=on" else "-DUSE_SYSTEM_MBEDTLS=off" )


Adding it again because it was collapsed by GitHub, we should always build with system libraries.

Mic92 · 2018-06-28T09:33:02Z

@GrahamcOfBorg build mbedtls

GrahamcOfBorg · 2018-06-28T09:33:14Z

Failure on x86_64-linux (full log)

Partial log (click to expand)

Merge failed

Mic92 · 2018-06-28T09:33:20Z

@GrahamcOfBorg build hiawatha

GrahamcOfBorg · 2018-06-28T09:33:29Z

Failure on x86_64-linux (full log)

Partial log (click to expand)

Merge failed

GrahamcOfBorg · 2018-06-28T09:34:47Z

Failure on x86_64-darwin (full log)

Partial log (click to expand)

Merge failed

GrahamcOfBorg · 2018-06-28T09:34:53Z

Failure on x86_64-darwin (full log)

Partial log (click to expand)

Merge failed

GrahamcOfBorg · 2018-06-28T09:35:42Z

Failure on aarch64-linux (full log)

Partial log (click to expand)

Merge failed

GrahamcOfBorg · 2018-06-28T09:35:50Z

Failure on aarch64-linux (full log)

Partial log (click to expand)

Merge failed

Anton-Latukha · 2018-06-28T10:24:16Z

@Mic92
Synced with upstream. Please run bot gain.

Mic92 · 2018-06-28T20:21:30Z

@GrahamcOfBorg build hiawatha

Mic92 · 2018-06-28T20:22:24Z

@Anton-Latukha you can also ask for ofborg access here: NixOS/ofborg#173

GrahamcOfBorg · 2018-06-28T20:22:37Z

Success on aarch64-linux (full log)

Attempted: hiawatha

Partial log (click to expand)

shrinking /nix/store/7d8my7pmgqx8dqjxsxp5nnzfccakpib6-hiawatha-10.8.1/sbin/cgi-wrapper
shrinking /nix/store/7d8my7pmgqx8dqjxsxp5nnzfccakpib6-hiawatha-10.8.1/sbin/wigwam
shrinking /nix/store/7d8my7pmgqx8dqjxsxp5nnzfccakpib6-hiawatha-10.8.1/sbin/hiawatha
gzipping man pages under /nix/store/7d8my7pmgqx8dqjxsxp5nnzfccakpib6-hiawatha-10.8.1/share/man/
strip is /nix/store/0pjsgkxz0rp5baycq5sp2s72lrr5q9sg-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/7d8my7pmgqx8dqjxsxp5nnzfccakpib6-hiawatha-10.8.1/bin  /nix/store/7d8my7pmgqx8dqjxsxp5nnzfccakpib6-hiawatha-10.8.1/sbin
patching script interpreter paths in /nix/store/7d8my7pmgqx8dqjxsxp5nnzfccakpib6-hiawatha-10.8.1
checking for references to /build in /nix/store/7d8my7pmgqx8dqjxsxp5nnzfccakpib6-hiawatha-10.8.1...
moving /nix/store/7d8my7pmgqx8dqjxsxp5nnzfccakpib6-hiawatha-10.8.1/sbin/* to /nix/store/7d8my7pmgqx8dqjxsxp5nnzfccakpib6-hiawatha-10.8.1/bin
/nix/store/7d8my7pmgqx8dqjxsxp5nnzfccakpib6-hiawatha-10.8.1

GrahamcOfBorg · 2018-06-28T20:23:37Z

Success on x86_64-darwin (full log)

Attempted: hiawatha

Partial log (click to expand)

-- Installing: /nix/store/sxvapa8vysai6gakq74zfr1x7y1layb7-hiawatha-10.8.1/var/log/hiawatha
-- Installing: /nix/store/sxvapa8vysai6gakq74zfr1x7y1layb7-hiawatha-10.8.1/var/run
-- Installing: /nix/store/sxvapa8vysai6gakq74zfr1x7y1layb7-hiawatha-10.8.1/var/lib/hiawatha
post-installation fixup
gzipping man pages under /nix/store/sxvapa8vysai6gakq74zfr1x7y1layb7-hiawatha-10.8.1/share/man/
strip is /nix/store/7ddbq63v97nk8gkbf7gcsfmby37h6gbl-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/sxvapa8vysai6gakq74zfr1x7y1layb7-hiawatha-10.8.1/bin  /nix/store/sxvapa8vysai6gakq74zfr1x7y1layb7-hiawatha-10.8.1/sbin
patching script interpreter paths in /nix/store/sxvapa8vysai6gakq74zfr1x7y1layb7-hiawatha-10.8.1
moving /nix/store/sxvapa8vysai6gakq74zfr1x7y1layb7-hiawatha-10.8.1/sbin/* to /nix/store/sxvapa8vysai6gakq74zfr1x7y1layb7-hiawatha-10.8.1/bin
/nix/store/sxvapa8vysai6gakq74zfr1x7y1layb7-hiawatha-10.8.1

GrahamcOfBorg · 2018-06-28T20:25:00Z

Success on x86_64-linux (full log)

Attempted: hiawatha

Partial log (click to expand)

shrinking /nix/store/0dvfknysdyam8jyvwfxh335g9fg22n64-hiawatha-10.8.1/sbin/wigwam
shrinking /nix/store/0dvfknysdyam8jyvwfxh335g9fg22n64-hiawatha-10.8.1/sbin/cgi-wrapper
shrinking /nix/store/0dvfknysdyam8jyvwfxh335g9fg22n64-hiawatha-10.8.1/bin/ssi-cgi
gzipping man pages under /nix/store/0dvfknysdyam8jyvwfxh335g9fg22n64-hiawatha-10.8.1/share/man/
strip is /nix/store/4qvrxzxa535y8304mk195x50b6p9607d-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/0dvfknysdyam8jyvwfxh335g9fg22n64-hiawatha-10.8.1/bin  /nix/store/0dvfknysdyam8jyvwfxh335g9fg22n64-hiawatha-10.8.1/sbin
patching script interpreter paths in /nix/store/0dvfknysdyam8jyvwfxh335g9fg22n64-hiawatha-10.8.1
checking for references to /build in /nix/store/0dvfknysdyam8jyvwfxh335g9fg22n64-hiawatha-10.8.1...
moving /nix/store/0dvfknysdyam8jyvwfxh335g9fg22n64-hiawatha-10.8.1/sbin/* to /nix/store/0dvfknysdyam8jyvwfxh335g9fg22n64-hiawatha-10.8.1/bin
/nix/store/0dvfknysdyam8jyvwfxh335g9fg22n64-hiawatha-10.8.1

…dtls (NixOS#41722) * mbedtls: build with cmake&ninja, clean-up * mbedtls: cmake ninja Darwin build clean-up * hiawatha: add build options, use system mbedTLS, platforms -> unix

GrahamcOfBorg added 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels Jun 8, 2018

Anton-Latukha force-pushed the mbedtls branch from 506909d to c2cf9a0 Compare June 9, 2018 10:51

jtojnar reviewed Jun 9, 2018

View reviewed changes

Anton-Latukha force-pushed the mbedtls branch from c2cf9a0 to 0e00c7d Compare June 10, 2018 11:15

Anton-Latukha changed the title ~~[WIP] mbedtls: build with cmake&ninja~~ [WIP] mbedtls: cmake&ninja build, clean-up; hiawatha: add options, use Nix mbedtls Jun 10, 2018

Anton-Latukha changed the title ~~[WIP] mbedtls: cmake&ninja build, clean-up; hiawatha: add options, use Nix mbedtls~~ [WIP] mbedtls: cmake&ninja, threading, clean-up; hiawatha: options, Nix mbedtls Jun 10, 2018

GrahamcOfBorg added the 8.has: package (new) label Jun 10, 2018

Anton-Latukha changed the title ~~[WIP] mbedtls: cmake&ninja, threading, clean-up; hiawatha: options, Nix mbedtls~~ mbedtls: cmake&ninja, threading, clean-up; hiawatha: options, Nix mbedtls Jun 10, 2018

jtojnar added the 2.status: work-in-progress label Jun 11, 2018

jtojnar requested changes Jun 19, 2018

View reviewed changes

Anton-Latukha force-pushed the mbedtls branch from 0e00c7d to 3eb0626 Compare June 24, 2018 18:12

Anton-Latukha added 2 commits June 28, 2018 13:05

mbedtls: build with cmake&ninja, clean-up

a3e1a5e

mbedtls: cmake ninja Darwin build clean-up

1e6096e

Anton-Latukha force-pushed the mbedtls branch 2 times, most recently from f23a7a2 to f241a5f Compare June 28, 2018 10:19

hiawatha: add build options, use system mbedTLS, platforms -> unix

ea083f2

Anton-Latukha force-pushed the mbedtls branch from f241a5f to ea083f2 Compare June 28, 2018 10:21

Mic92 merged commit c876db6 into NixOS:master Jun 28, 2018

mbedtls: cmake&ninja, threading, clean-up; hiawatha: options, Nix mbedtls #41722

mbedtls: cmake&ninja, threading, clean-up; hiawatha: options, Nix mbedtls #41722

Conversation

Anton-Latukha commented Jun 8, 2018 • edited

Motivation for this change

Things done

Anton-Latukha commented Jun 8, 2018 • edited

Anton-Latukha commented Jun 8, 2018

Anton-Latukha commented Jun 8, 2018

Anton-Latukha commented Jun 8, 2018 • edited

Anton-Latukha commented Jun 9, 2018

Anton-Latukha commented Jun 9, 2018

Anton-Latukha commented Jun 9, 2018

Anton-Latukha commented Jun 9, 2018

Anton-Latukha commented Jun 9, 2018

jtojnar commented Jun 9, 2018

GrahamcOfBorg commented Jun 9, 2018

GrahamcOfBorg commented Jun 9, 2018

GrahamcOfBorg commented Jun 9, 2018

Choose a reason for hiding this comment

Anton-Latukha Jun 10, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Anton-Latukha Jun 10, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mic92 commented Jun 28, 2018

GrahamcOfBorg commented Jun 28, 2018

Mic92 commented Jun 28, 2018

GrahamcOfBorg commented Jun 28, 2018

GrahamcOfBorg commented Jun 28, 2018

GrahamcOfBorg commented Jun 28, 2018

GrahamcOfBorg commented Jun 28, 2018

GrahamcOfBorg commented Jun 28, 2018

Anton-Latukha commented Jun 28, 2018

Mic92 commented Jun 28, 2018

Mic92 commented Jun 28, 2018

GrahamcOfBorg commented Jun 28, 2018

GrahamcOfBorg commented Jun 28, 2018

GrahamcOfBorg commented Jun 28, 2018

Anton-Latukha commented Jun 8, 2018 •

edited

Anton-Latukha commented Jun 8, 2018 •

edited

Anton-Latukha commented Jun 8, 2018 •

edited

Anton-Latukha Jun 10, 2018 •

edited

Anton-Latukha Jun 10, 2018 •

edited