-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DNS: threaded resolver [WIP] #2829
Conversation
|
||
def initialize | ||
@que = Deque(T).new(16) | ||
@mutex = Thread::Mutex.new |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using a mutex is easy, but maybe not optimal in terms of speed. Using http://llvm.org/releases/3.3/docs/LangRef.html#cmpxchg-instruction and http://llvm.org/releases/3.3/docs/LangRef.html#atomicrmw-instruction for acquiring/releasing a lock may lead to better performance.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For example try to lock with cmpxchg(counter, 1, 0)
and yielding the fiber until it succeeds, then release the lock with modifyrmw(counter, sub, counter, 1)
. We'd need to differentiate between push/pop/clear from the event-loop thread (there is a fiber) and calling from a thread (no fiber) and act accordingly; maybe using a semaphore from the thread.
This may be too complex, thought, and we should just not care until we tackle parallelism.
Thanks! I think this will be part of the solution that eventually will remain in the standard library (that is, firing a thread and invoking |
loop do | ||
@mutex.synchronize do | ||
if @que.empty? | ||
raise Error.new("queue is empty") unless blocking |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When called from the event-loop, this should always non blocking, otherwise the next call will block the event loop until a result a available (bad).
It could also merely return nil, allowing the calling Fiber to yield itself, instead of spending resources raising an exception, catching it and then yielding. Thought there won't be any mean to differentiate between a "would block" and a "popping nil" scenarios.
Memory usage is stable over time (on linux glibc) when resolving |
be332b4
to
5064997
Compare
I made some modifications (with new commits):
Still a work in progress. I must generate POSIX bindings for the remaining supported targets, and squash things a bit. |
5064997
to
41be0e6
Compare
Bump on this 👍 |
Bump again. |
41be0e6
to
5f4004e
Compare
@ysbaddaden @RX14 @straight-shoota |
No, the ability to configure the resolver in #4236 is better. |
@ysbaddaden makes sense. |
This Pull Request proposes to call
getaddrinfo
within threads and to synchronize with the main thread to receive the request and send back the response, thus allowing to not stop-the-world whenever a Fiber needs to resolve a Domain Name. The threadpool size can be adjusted withSocket::Addrinfo.threadpool_size = 4
for example before a Domain Name is first resolved, or even disabled withSocket::Addrinfo.threadpool_size = 0
.getaddrinfo
in threads;getaddrinfo
from the main thread if the threadpool is disabled (ie. to set0
);IPSocket
,TCPSocket
,TCPServer
andUDPSocket
to useDNS.getaddrinfo
;IPSocket.getaddrinfo
orand avoid aDNS
namespace to useSocket::Addrinfo
instead;consider reducing duplication ofthe Addrinfo wrapper struct was dropped, because of IP4/IP6 issues and it's not particularly useful, unless we want to open a public API (let's see that later);DNS::Addrinfo
withSocket::IPAddress
if any (?)specs !this is a private API, socket specs already test that it works;AI_*
andEAI_*
LibC constants for each supported target.I was wondering about having different resolvers under
DNS
that let applications use the strategy they want, since none is perfect. For example a blocking resolver that merely callsgetaddrinfo
; a threaded resolver that uses threads to callgetaddrinfo
(ie. this PR, should be the default); a resolver that would link against c-ares for applications that don't care about libc extensions and want a real async DNS resolver (eg: servers).Note that this solution may be dropped and no longer be the default when the event-loop will become multithreaded. Limiting the number of concurrent calls to
getaddrinfo
in order to not block all threads (like go does) may be preferable.refs #2745 #2660