Adds support for OpenBSD #3326

wmoxam · 2016-09-18T00:19:36Z

There is enough here to cross-compile an object file which can be used to compile a working compiler. The ported compiler can then fully compile itself.

Some notes:

[ThreadLocal] directives are not supported on OpenBSD
Specs pass except ThreadLocal specs
Building Shards fails, possibly due to a problem in LibC::Stat
libevent prints warnings (ex: https://gist.github.com/wmoxam/46f0ec35f17d265ebac6e169e50ff774)
crystal docs command is broken
crystal deps is broken

ysbaddaden

Thanks for porting Crystal to OpenBSD! This looks great overall. I just wrote a few notes here and there.

Could you explain how you generated the LibC bindings? Did you use the posix generator, or did you copy and adapted the FreeBSD bindings?

Could you explain how to configure an OpenBSD installation, or better provide an openbsd target to the Vagrantfile? I tried once to port to OpenBSD but failed at installing dependencies (I believe). This way I could test a few things, like why is crystal docs segfaulting, why is Shards not compiling, ...

ysbaddaden · 2016-09-18T12:42:26Z

src/openssl/hmac.cr

@@ -19,7 +21,7 @@ class OpenSSL::HMAC
    key_slice = key.to_slice
    data_slice = data.to_slice
    buffer = Slice(UInt8).new(128)
-    LibCrypto.hmac(evp, key_slice, key_slice.size, data_slice, data_slice.size, buffer, out buffer_len)
+    LibCrypto.hmac(evp.not_nil!, key_slice, key_slice.size, data_slice, data_slice.size, buffer, out buffer_len)


evp should never be nil, because it was either assigned a value or an exception was raised.

It shouldn't be, but I was getting a compiler error after adding the conditional macro code on line 11. I'll investigate further to confirm that this is a reproducable problem, if so it may indicate a bug in the compiler.

ysbaddaden · 2016-09-18T12:56:24Z

src/openssl/hmac.cr

@@ -8,7 +8,9 @@ class OpenSSL::HMAC
          when :md4       then LibCrypto.evp_md4
          when :md5       then LibCrypto.evp_md5
          when :ripemd160 then LibCrypto.evp_ripemd160
+          {% if !flag?(:openbsd) %}
          when :sha       then LibCrypto.evp_sha


EVP_sha was removed in OpenSSL 1.1.0. Let's drop it altogether.

ysbaddaden · 2016-09-18T12:59:22Z

src/thread.cr

@@ -50,7 +50,9 @@ class Thread
  # All threads, so the GC can see them (GC doesn't scan thread locals)
  @@threads = Set(Thread).new

+  {% if !flag?(:openbsd) %}
  @[ThreadLocal]


How to access Thread.current and be sure to always memoize the current Thread without @[ThreadLocal]? This i the only issue with ThreadLocal not being supported on OpenBSD.

Maybe not memoizing @@current on OpenBSD, then iterating @@threads to find the current thread that matches the id returned by pthread_self?

That sounds like a viable idea. I'll give it a try

ysbaddaden · 2016-09-18T13:00:38Z

src/lib_c/amd64-unknown-openbsd/c/termios.cr

+  ONOCR     = 0x00000040
+  ONLRET    = 0x00000080
+
+  TABDLY    = 0x00000004


TABDLY is defined here, but not in the OutputMode enum. Is this intended?

No, not intended

ysbaddaden · 2016-09-18T13:11:17Z

building Shards fails: I'll have a look when I have a working OpenBSD installation.
crystal deps is broken: this merely calls shards, so that's expected.
libevent prints warnings: that one is weird. Is it possible that OpenBSD doesn't like we call random()?

An improvement (can be postponed):

use OpenBSD arc4random instead of urandom in SecureRandom (see libsodium).

ysbaddaden · 2016-09-18T17:03:50Z

Current status: I got an initial OpenBSD setup. I'm just missing an up to date boehm-gc (the very recent 7.6 release).

wmoxam · 2016-09-19T18:03:55Z

@ysbaddaden Thanks for taking the time to look at this!

Regarding your initial questions:

I generated the LibC bindings by adapting the FreeBSD bindings. I wasn't aware of your posix generator, I will take a look at it soon.

As for configuring OpenBSD, I installed the following packages: llvm boehm-gc libevent pcre gmake bash

I listed the steps I took to build on a fresh OpenBSD install here:

https://gist.github.com/wmoxam/d06f9ccc82e03f6fea64747d82caa421

ysbaddaden · 2016-09-20T21:32:56Z

Thanks! I cross-compiled a Crystal binary for OpenBSD which was indeed capable to build itself :-)

I generated the LibC bindings, and here is the patch against this PR (mostly LibC::Stat fix, some types, removed constants);
I built Shards, and just pushed a fix to support OpenBSD (so it works);
crystal docs generated docs for Shards (so it seems it works) and gmake docs generated crystal's docs (so it works, too).
I can't build Crystal itself in release mode though; the compiler eventually segfaults. I can build Shards in release mode, thought.
EDIT: I can't cross compile crystal on OpenBSD itself too. Again I can cross compile Shards.

~~I have a feeling release mode and cross-compilation may be related to my VM being "only" granted 4GB of RAM.~~ EDIT: nope, I upgraded to 8GB and it still crashes.

Compilation prints some warnings, that appear to be OpenBSD specific. For example libyaml prints "use snprintf instead of often misused sprintf" or "libevent: random() may return deterministic values". I believe they're harmless warnings.

ysbaddaden · 2016-09-21T12:05:57Z

I adapted the pkg.sh script I use for building FreeBSD packages, and here comes an alpha OpenBSD tarball of 0.19.2 for testing: https://dl.dropboxusercontent.com/u/53345358/openbsd/index.html

wmoxam · 2016-09-26T01:44:40Z

I was able to build crystal in release mode. I didn't do anything special to do so.

@ysbaddaden What steps did you take to cross-compile crystal on OpenBSD itself? I would like to try to reproduce the problem.

There is enough here to cross-compile an object file which can be used to complile a working compiler The ported compiler can then fully compile itself.

wmoxam · 2016-09-29T14:04:53Z

@ysbaddaden I believe all of the issues have been addressed. Let me know if you think there is anything else that should be addressed.

ysbaddaden · 2016-09-29T14:37:42Z

src/thread.cr

  {% end %}
  @@current = new

  def self.current
+    {% if flag?(:openbsd) %}
+      pthread_self_id = LibC.pthread_self().as(UInt64*).address


Wouldn't pthread_equal be safer?

class Thread {% if flag?(:openbsd) %} def self.current current_thread_id = LibC.pthread_self @@threads.find { |thread| LibC.pthread_equal(thread.id, current_thread_id) != 0 }.not_nil! end {% else %} @[ThreadLocal] def self.current @@current end {% end %} def id @th end end

Oh, I didn't know about pthread_equal, that seems much better

asterite · 2016-09-29T16:16:10Z

@ysbaddaden Feel free to merge this whenever you think this is ready. I looked at the changed and they look fine :) (once travis passes, of course)

ysbaddaden · 2016-09-30T15:47:41Z

I still can't build in release mode, Crystal crashes during "codegen (bc+obj)" with the following. It looks like an exception fails to raise in LLVM. Maybe compiling LLVM in debug mode (with assertions enabled) could help catch the issue.

(gdb) run build --release --stats  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
Starting program: /home/vagrant/crystal/.build/crystal build --release --stats  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
Parse:                             00:00:00.0006841 (   0.19MB)
Semantic (top level):             [New process 75459]
 00:00:01.0705360 (  67.32MB)
Semantic (new):                    00:00:00.0066914 (  67.32MB)
Semantic (type declarations):      00:00:00.1122292 (  75.32MB)
Semantic (abstract def check):     00:00:00.0017784 (  75.32MB)
Semantic (cvars initializers):     00:00:00.0276357 (  75.32MB)
Semantic (ivars initializers):     00:00:00.0494701 (  83.32MB)
Semantic (main):                  /usr/local/lib/libevent_core.so.1.1: warning: warning: random() may return deterministic values, is that what you want?
 00:00:14.8357150 ( 743.26MB)
Semantic (cleanup):                00:00:00.0026391 ( 743.26MB)
Semantic (recursive struct check): 00:00:00.0031308 ( 751.26MB)
Codegen (crystal):                 00:00:08.7591427 ( 867.26MB)
Codegen (bc+obj):                 
Program received signal SIGABRT, Aborted.
0x00000fb36d2ab78a in thrkill () at <stdin>:2
2       <stdin>: No such file or directory.
        in <stdin>
Current language:  auto; currently asm
(gdb) bt
#0  0x00000fb36d2ab78a in thrkill () at <stdin>:2
#1  0x00000fb36d276f99 in *_libc_abort () at /usr/src/lib/libc/stdlib/abort.c:52
#2  0x00000fb1569158de in _Unwind_SetGR (context=0x0, index=Unhandled dwarf expression opcode 0xf3
) at /usr/obj/ports/gcc-4.9.3/gcc-4.9.3/libgcc/unwind-dw2.c:275
#3  0x00000fb154ca9250 in __crystal_personality () at raise.cr:69
#4  0x00000fb3f62299e3 in _Unwind_RaiseException_Phase2 (exc=0xfb43971f160, context=0x7f7ffffec6c0) at unwind.inc:62
#5  0x00000fb3f622a09a in _Unwind_RaiseException (exc=0xfb43971f160) at unwind.inc:131
#6  0x00007f7ffffec938 in ?? ()
#7  0x00007f7ffffec940 in ?? ()
#8  0x0000000000000000 in ?? ()

ysbaddaden · 2016-09-30T16:38:04Z

I fixed a few issues. OpenBSD doesn't support keepalive configuration on a per socket basis. I also changed the Thread.current implementation a bit. I'll push the commits later.

wmoxam · 2016-10-01T04:26:43Z

@ysbaddaden I was able to build in release mode, the problem was in linking. See: https://gist.github.com/wmoxam/93120d51e3be3af2479f3842f75ebb49#file-success-txt

ysbaddaden · 2016-10-01T11:46:44Z

I was able to build in release mode

That's good news! There must be an issue in my Vagrant box (kaorimatz/openbsd-6.0-amd64) or VirtualBox or something.

The problem was in linking

Yes, the LLVM package must have been compiled with clang++ and trying to link to LLVM libraries using gcc just fails. I had the same issue compiling the LLVM ABI specs. Using clang just works. I have no idea why. Maybe it's the C++ thing?

wmoxam · 2016-10-06T12:02:27Z

Closed in favour of #3369

ysbaddaden requested changes Sep 18, 2016

View reviewed changes

wmoxam force-pushed the openbsd branch from bc76127 to b77e504 Compare September 25, 2016 23:04

wmoxam added 7 commits September 26, 2016 02:52

Adds support for OpenBSD

0f5c8c0

There is enough here to cross-compile an object file which can be used to complile a working compiler The ported compiler can then fully compile itself.

Disable local thread specs on OpenBSD as they are not yet implemented

c7921aa

Fix LibC bindings using generated bindings (thanks @ysbaddaden)

b77e504

Dropping EVP_sha as it was removed in OpenSSL 1.1.0

36648a4

Drop EVP_sha spec as method has been removed

8cb44d6

Fixes Thread.current on OpenBSD

40510fb

Fixes spacing

e913b04

wmoxam changed the title ~~[WIP] Adds support for OpenBSD~~ Adds support for OpenBSD Sep 29, 2016

ysbaddaden reviewed Sep 29, 2016

View reviewed changes

ysbaddaden mentioned this pull request Oct 1, 2016

Openbsd support #3369

Merged

wmoxam added 2 commits October 2, 2016 01:26

Merge branch 'master' into openbsd

bf5f544

Fixes formatting

4715534

wmoxam closed this Oct 6, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds support for OpenBSD #3326

Adds support for OpenBSD #3326

wmoxam commented Sep 18, 2016

ysbaddaden left a comment

ysbaddaden Sep 18, 2016

wmoxam Sep 19, 2016

ysbaddaden Sep 18, 2016

wmoxam Sep 19, 2016

ysbaddaden Sep 18, 2016

ysbaddaden Sep 18, 2016

wmoxam Sep 19, 2016

ysbaddaden Sep 18, 2016

wmoxam Sep 19, 2016

ysbaddaden commented Sep 18, 2016

ysbaddaden commented Sep 18, 2016 •

edited

Loading

wmoxam commented Sep 19, 2016

ysbaddaden commented Sep 20, 2016 •

edited

Loading

ysbaddaden commented Sep 21, 2016

wmoxam commented Sep 26, 2016 •

edited

Loading

wmoxam commented Sep 29, 2016

ysbaddaden Sep 29, 2016

wmoxam Sep 29, 2016

asterite commented Sep 29, 2016

ysbaddaden commented Sep 30, 2016

ysbaddaden commented Sep 30, 2016

wmoxam commented Oct 1, 2016 •

edited

Loading

ysbaddaden commented Oct 1, 2016

wmoxam commented Oct 6, 2016

Adds support for OpenBSD #3326

Adds support for OpenBSD #3326

Conversation

wmoxam commented Sep 18, 2016

ysbaddaden left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ysbaddaden commented Sep 18, 2016

ysbaddaden commented Sep 18, 2016 • edited Loading

wmoxam commented Sep 19, 2016

ysbaddaden commented Sep 20, 2016 • edited Loading

ysbaddaden commented Sep 21, 2016

wmoxam commented Sep 26, 2016 • edited Loading

wmoxam commented Sep 29, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asterite commented Sep 29, 2016

ysbaddaden commented Sep 30, 2016

ysbaddaden commented Sep 30, 2016

wmoxam commented Oct 1, 2016 • edited Loading

ysbaddaden commented Oct 1, 2016

wmoxam commented Oct 6, 2016

ysbaddaden commented Sep 18, 2016 •

edited

Loading

ysbaddaden commented Sep 20, 2016 •

edited

Loading

wmoxam commented Sep 26, 2016 •

edited

Loading

wmoxam commented Oct 1, 2016 •

edited

Loading