Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Windows #3582

Closed
wants to merge 46 commits into from
Closed

[WIP] Windows #3582

wants to merge 46 commits into from

Conversation

lbguilherme
Copy link
Contributor

@lbguilherme lbguilherme commented Nov 26, 2016

Warning: This is very very initial.

Fixes #26

Roadmap:

  • Make it compile code
  • Add {% if flag?(:windows) %} all over the core library and make the prelude portable
    • Basic IO
    • Fibers
    • Concurrent IO
    • Exceptions (backtraces)
    • Threads
    • Mutex
    • Process
    • Regex (should work out of the box, but needs to compile libpcre)
    • GC
    • Encodings
    • ...
  • Port rest of std library
    • DNS
    • Sockets
    • FileUtils and Dir
    • ...
  • Remove dependency on wine clang.exe hack
  • Compile crystal.exe and self-host it
  • Pass all specs
  • Figure out how to do CI testing for Windows
  • Add support for 32-bit Windows as well
  • Release Crystal for Windows!

How to use it: (from Linux)

  1. Get the branch win from this fork:

    git clone https://github.com/lbguilherme/crystal.git
    git checkout win
    
  2. Install Wine: https://www.winehq.org/

  3. Download Clang 3.8.1 for Windows x64. Install it somewhere with Wine. Make sure this works:

    $ wine bin/clang.exe --version
    clang version 3.8.1 (branches/release_38)
    Target: x86_64-pc-windows-msvc
    
  4. Install MXE from my fork (I'll get this patch upstream at some point Update gc to 7.6.0 (and add libatomic_ops because of it) mxe/mxe#1539)

    git clone https://github.com/lbguilherme/mxe.git
    git checkout updated-gc
    # Check http://mxe.cc/#requirements for a list of dependencies
    # (or just run the comand below and get an error listing what is missing)
    make MXE_TARGETS=x86_64-w64-mingw32.static gcc gc libevent libiconv # This will take some considerable time
    
  5. Build the crystal compiler having at least LLVM 3.8 in your system. Or maybe download some older clang.exe, compactible with the LLVM in your Crystal. If needed, build it: make

  6. Write some simple Crystal:

    # foo.cr
    ch = Channel(String).new
    
    spawn do
      3.times { puts ch.receive }
    end
    
    ch.send "Hello Crystal"
    ch.send "  running on Windows!"
    sleep 1
    ch.send "\nYEAH!!"
    
  7. And finally: Build code!

./bin/crystal build foo.cr --ll --single-module --cross-compile --target x86_64-windows-gnu --prelude=windows-prelude
wine /clang-path/bin/clang.exe -g3 foo.ll -c -o foo.o
/mxe-path/usr/bin/x86_64-w64-mingw32.static-gcc -g3 foo.o -o foo.exe -liconv -lkernel32 -lgc -levent -lws2_32
  1. And run:

    $ wine foo.exe
    Hello Windows!

FAQ

  • When will it be ready? I'm doing it in my free time, so I really have no idea.

  • Why not use CYGWIN? I really believe Windows should be a first-class target and that this is fundamental for Crystal's future as a language. The more native the better. I would ditch even glibc if it were possible (at some point it will be).

  • How can I help? Pick a very small Crystal code. Try to compile it. See a lot of errors and crashes. Do whatever it takes to make them go away. Send a patch with your changes.

(almost all stdlib doesn't work right now)
@asterite
Copy link
Member

Nice!

I think we should try, for one release, to push this forward until we have at least a bootstrappable compiler to work with. Then things should be much easier.

I noticed some files are completely hidden behind a {% if flag?(:windows) %}. Maybe we can split these files into three. For example: termios.cr, termios_posix.cr, termios_windows.cr and then termios.cr would do the switch over the flag.

Another thing probably worth doing (on my side) is to mark a whole lib with a CallConvention, so all funs inside it will inherit that call convention. It looks like all windows functions use the X86_StdCall convention, right?

@lbguilherme
Copy link
Contributor Author

I noticed some files are completely hidden behind a {% if flag?(:windows) %}. Maybe we can split these files into three. For example: termios.cr, termios_posix.cr, termios_windows.cr and then termios.cr would do the switch over the flag.

This is similar to what Go does, right? Some files will probably need a full rewrite, like scheduler.cr. Splitting into multiple files is the way to go.

Some compiler support would help here, as it wouldn't be possible anymore to require *. For example: splitting io/file_descriptor.cr into io/file_descriptor.posix.cr and io/file_descriptor.windows.cr fails because io.cr has require "./io/*" in it.

Another thing probably worth doing (on my side) is to mark a whole lib with a CallConvention, so all funs inside it will inherit that call convention. It looks like all windows functions use the X86_StdCall convention, right?

That would be awesome! (That's correct, all WinAPI follows this convention)


@asterite
Copy link
Member

Oh, the thing about require is true. I guess for now we'll have to break those into individual requires. I think there aren't many cases like that, maybe io or socket are affected, but there aren't many files under those directories. We could try to let the compiler know how to handle this, but I don't know how would that work... maybe check if other filenames exist and see if, splitting by dots, choose the one for which all of the flags meet.

@RX14
Copy link
Contributor

RX14 commented Nov 26, 2016

How about being able to mark a file as ignored or not at the top of a file, similar to surrounding the whole file in a macro if flags, then simply require every platform's files? Maybe make it so any macro expression works.

@asterite
Copy link
Member

@RX14 That's a good idea. When I read the {% if flag? %} at the top I thought maybe we could have return inside a macro, something like:

# file_windows.cr
{% return unless flag?(:windows) %}

# ...

So code below the {% return ... %} will only be analyzed if the condition is met. The "problem" is that this would be a special rule in the macro language, but maybe it's not that bad.

@RX14
Copy link
Contributor

RX14 commented Nov 26, 2016

@asterite Honestly I wouldn't use return, instead something like ignore_file. And only allow it as the first statement in a file, otherwise it'll get messy.

src/fiber.cr Outdated
sleep(0)
{% else %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stray else

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, messed up selecting which lines to commit. Fixed.

@ysbaddaden
Copy link
Contributor

Eventually, maybe not fully duplicate implementations, but extract the OS specific calls, and limit full extracts for cases where POSIX and Windows have strictly nothing in common.

Some methods may require a specific implementation, but IO:: Buffered#buffered_close, for example, only differs on a single platform specific call. We could keep a single implementation that would call a #close_os_impl method for example.

Same for Scheduler: the overall implementation remains identical, only some helpers are POSIX specific, and some others will be Windows specific.

@shayneoAtNorwood
Copy link

I know this isn't a useful comment, but I just want to chip in and say this is wonderful stuff guys. Windows support=closer to world domination. mwahaha

@jsaak
Copy link

jsaak commented Dec 4, 2016

You may be familiar with libuv. I do not know why, but crystal uses different library. Maybe you can use libuv in windows port?

@lbguilherme
Copy link
Contributor Author

lbguilherme commented Dec 4, 2016 via email

@canyonblue77
Copy link

canyonblue77 commented Dec 22, 2016

Novice Coder and wanted some feedback. Since this thread is about Windows and there is a 1% chance I could benefit the effort I wanted to share my attempt at porting code for starting a new process from kernel32/Windows library.(linux fork?) No idea what I'm actually doing but hey, fake it till ya make it... Is this close too accurate? Would additional ports of other functions be beneficial? Or just annoying background noise, ya won't hurt my feelings. :)

@[CallConvention("X86_StdCall")]
fun winfork = CreateProcessA(
        lpApplicationName : Char*,
        lpCommandLine : Char*,
        lpProcessAttributes : SECURITY_ATTRIBUTES*,
        lpThreadAttributes : SECURITY_ATTRIBUTES*,
        bInheritHandles : Bool,
        dwCreationFlags : Int,
        lpEnvironment : DWord*,
        lpCurrentDirectory : Char*,
        lpStartupInfo : STARTUPINFO,
        lpProcessInformation : PROCESS_INFORMATION
) : Bool`
struct SECURITY_ATTRIBUTES
        length : Dword
        security_descriptors : Void*
        inherit_handle : Bool
end
struct STARTUPINFO
        cb : Dword
        Reserved : Char*
        Desktop : Char*
        Title : Char*
        dwX : Dword
        dwY : Dword
        dwXSize : Dword
        dwYSize : Dword
        dwXCountChars : Dword
        dwYCountChars : Dword
        dwFillAttribute : Dword
        dwFlags : Dword
        ShowWindow : UInt16
        cbReserved2 : UInt16
        Reserved2 : Void*
        StdInput : Handle
        StdOutput : Handle
        StdError : Handle
end
struct PROCESS_INFORMATION
        Process : Handle
        Thread : Handle
        ProcessId : Dword
        ThreadId : Dword
end

Brian J. Cardiff added 2 commits December 29, 2016 11:24
Conflicts:
	src/fiber.cr
	src/io/file_descriptor.cr
@bcardiff
Copy link
Member

@lbguilherme which version of wine are you using here?

Sorry about that push I messed with the remote name (I didn't expect github will allow me to push on you fork :-S). I wanted to bring the to the latest crystal version and see if there is anything I can contribute with.

@txe
Copy link
Contributor

txe commented Jul 10, 2017

The branch doesn't contain the latest changes from crystal repository, so this is why there is no failures

@bararchy
Copy link
Contributor

@txe How hard would it be to rebase it ?

@txe
Copy link
Contributor

txe commented Jul 10, 2017

Even if it's hard, I'm ready to rebase it and add Crystal::System. I just want to know if it will be enough to merge it finally,

@mverzilli
Copy link

Yes. If it doesn't break tests for already supported platforms we can make it a priority to merge this, aiming at having it in 0.24.

@txe
Copy link
Contributor

txe commented Jul 10, 2017

Good, then I will try to prepare the branch this week.

@RX14
Copy link
Contributor

RX14 commented Jul 10, 2017

My concern is that if nobody is willing to port this work over to Crystal::System now why would they be willing to do so after this PR is merged. If that work doesn't happen we're left with an ugly duality of files which have to keep the exact same interface over 2 platforms with nothing enforcing that. Imagine PRing a fix to the posix version of the stdlib which tweaks a method signature a little, what is there to enforce that that change also takes place on windows? That's part of the reason why we have Crystal::System: to create a clean, minimal interface to platform-specifics that changes at a rate much slower than the rest of the standard library.

It also invalidates a lot of the open PRs by renaming the posix-supporting files. Even if this PR doesn't break anything visibly outside the standard library, it does break the flow for everyone already working on crystal by renaming everything platform specific.

Please, let's be patient and do windows support properly.

@akzhan
Copy link
Contributor

akzhan commented Jul 10, 2017

Will CI allowed for Windows builds?

@bararchy
Copy link
Contributor

@RX14 It seems though that @txe is willing to do the System change.

@mverzilli
Copy link

@RX14, sure. Making this a priority means we should prioritize everything that stands on the way to it as well :). Maybe I rushed a bit to say it was for 0.24, but I don't see it that far.

@akzhan we can start with experimental support (which is to say no-CI). I wouldn't tie this to having a CI on Windows. Otherwise we'll deadlock progress on both.

@RX14
Copy link
Contributor

RX14 commented Jul 10, 2017

@bararchy I was more responding to @oprypin's suggestion to merge this essentially as-is.

Rebasing this to use Crystal::System is a lot of hard work and design decisions which affect linux as well (but I think it's well worth it). Plus rebasing to use Crystal::System naturally splits up this PR into separately mergeable sections, so at that point you might as well PR each bit seperately to speed up the process.

AFAIK we still have the exception handling issue on windows as well, is that still an issue? That's a compiler change that can be PRed seperately right?

@txe
Copy link
Contributor

txe commented Jul 10, 2017

As I see, @RX14 is going to create a Crystal::System version of IO::FileDescriptor, so I can wait for it and then use it as an example of Crystal::System to do PR for: process.cr, scheduler.cr, file.cr, file\stat.cr, thread.cr, dir.cr.
What do you think?

@hanyuone
Copy link

Any news on this?

@RX14
Copy link
Contributor

RX14 commented Jul 29, 2017

@Qwerp-Derp The next step is to get #4707 merged.

@bararchy
Copy link
Contributor

What's holding #4707 back ?

@sdogruyol
Copy link
Member

Let's rock now that #4707 is merged 👍

@sdogruyol
Copy link
Member

Update #4832 is also merged 👍

@piedoom
Copy link

piedoom commented Oct 12, 2017

Is there any news on this? I've been checking here every few weeks to see progress.

@Yardanico
Copy link

There is AppVeyor for CI testing on Windows

@ylluminate
Copy link

Just a heads up, interesting discussion from Nanobox guys with some interest in using Crystal instead of Go (see #crystal) if there was native Windows support. While Windows doesn't personally mean much to me, as a platform and for growing the exposure, Crystal could have some really great wins publicly if this were gnawed on to completion.

@ilog2000
Copy link

Congratulations on one year anniversary of this pull request!

@RX14
Copy link
Contributor

RX14 commented Nov 29, 2017

If we get #5333 merged, there's a relatively tiny patch (that already works) to getting hello world working on windows. I think that hello world is working as long as it isn't ifdef hell. And after that we can finally get to merging in @txe and @lbguilherme's work. It's been a year but I think we can do it this time.

@RX14 RX14 mentioned this pull request Dec 1, 2017
@RX14
Copy link
Contributor

RX14 commented Dec 1, 2017

That patch has now become #5339.

I'll probably close this issue once it's merged. Although full or even basic windows support won't have been added by that PR, it'll be a slow slog of dissecting this current PR and putting it's bits in Crystal::System, which will take a while. So the most logical time to close this PR will be then, since it's obvious this isn't going to get merged as-is.

@ylluminate
Copy link

Thanks @RX14. I know the Nanobox guys were thrilled to hear the progress and many are excited to be able to have this opportunity to start using Crystal broadly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Windows
  
Done
Development

Successfully merging this pull request may close these issues.

None yet