Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: m-labs/artiq
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 364e3d48ab6a
Choose a base ref
...
head repository: m-labs/artiq
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: eb68ab9ad0d3
Choose a head ref
  • 2 commits
  • 4 files changed
  • 1 contributor

Commits on Apr 6, 2016

  1. Copy the full SHA
    221d1bd View commit details
  2. doc: remote execution

    sbourdeauducq committed Apr 6, 2016
    Copy the full SHA
    eb68ab9 View commit details
Showing with 85 additions and 4 deletions.
  1. +69 −0 artiq/protocols/remote_exec.py
  2. +8 −3 doc/manual/developing_a_ndsp.rst
  3. +1 −1 doc/manual/installing.rst
  4. +7 −0 doc/manual/protocols_reference.rst
69 changes: 69 additions & 0 deletions artiq/protocols/remote_exec.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,38 @@
"""
This module provides facilities for experiment to execute code remotely on
controllers.
The remotely executed code has direct access to the driver, so it can transfer
large amounts of data with it, and only exchange higher-level, processed data
with the experiment (and over the network).
Controllers with support for remote execution contain an additional target
that gives RPC access to instances of ``RemoteExecServer``. One such instance
is created per client (experiment) connection and manages one Python namespace
in which the experiment can execute arbitrary code by calling the methods of
``RemoteExecServer``.
The namespaces are initialized with the following global values:
* ``controller_driver`` - the driver instance of the controller.
* ``controller_initial_namespace`` - a controller-wide dictionary copied
when initializing a new namespace.
* all values from ``controller_initial_namespace``.
Access to a controller with support for remote execution is done through an
additional device database entry of this form: ::
"$REXEC_DEVICE_NAME": {
"type": "controller_aux_target",
"controller": "$CONTROLLER_DEVICE_NAME",
"target_name": "$TARGET_NAME_FOR_REXEC"
}
Specifying ``target_name`` is mandatory in all device database entries for all
controllers with remote execution support.
"""

from functools import partial
import inspect

@@ -8,20 +43,37 @@


class RemoteExecServer:
"""RPC target created at each connection by controllers with remote
execution support. Manages one Python namespace and provides RPCs
for code execution.
"""
def __init__(self, initial_namespace):
self.namespace = dict(initial_namespace)
# The module actually has to exist, otherwise it breaks e.g. Numba
self.namespace["__name__"] = "artiq.protocols.remote_exec"

def add_code(self, code):
"""Executes the specified code in the namespace.
:param code: a string containing valid Python code
"""
exec(code, self.namespace)

def call(self, function, *args, **kwargs):
"""Calls a function in the namespace, passing it positional and
keyword arguments, and returns its value.
:param function: a string containing the name of the function to
execute.
"""
return self.namespace[function](*args, **kwargs)


def simple_rexec_server_loop(target_name, target, host, port,
description=None):
"""Runs a server with remote execution support, until an exception is
raised (e.g. the user hits Ctrl-C) or termination is requested by a client.
"""
initial_namespace = {"controller_driver": target}
initial_namespace["controller_initial_namespace"] = initial_namespace
targets = {
@@ -33,6 +85,23 @@ def simple_rexec_server_loop(target_name, target, host, port,

def connect_global_rpc(controller_rexec, host=None, port=3251,
target="master_dataset_db", name="dataset_db"):
"""Creates a global RPC client in a controller that is used across
all remote execution connections. With the default parameters, it connects
to the dataset database (i.e. gives direct dataset access to experiment
code remotely executing in controllers).
If a global object with the same name already exists, the function does
nothing.
:param controller_rexec: the RPC client connected to the controller's
remote execution interface.
:param host: the host name to connect the RPC client to. Default is the
local end of the remote execution interface (typically, the ARTIQ
master).
:param port: TCP port to connect the RPC client to.
:param target: name of the RPC target.
:param name: name of the object to insert into the global namespace.
"""
if host is None:
host = controller_rexec.get_local_host()
code = """
11 changes: 8 additions & 3 deletions doc/manual/developing_a_ndsp.rst
Original file line number Diff line number Diff line change
@@ -110,9 +110,9 @@ When using the driver in an experiment, the ``Client`` instance can be returned
Command-line arguments
----------------------

Use the Python ``argparse`` module to make the bind address and port configurable on the controller, and the server address, port and message configurable on the client.
Use the Python ``argparse`` module to make the bind address(es) and port configurable on the controller, and the server address, port and message configurable on the client.

We suggest naming the controller parameters ``--bind`` and ``--port`` so that those parameters stay consistent across controller, and use ``-s/--server`` and ``--port`` on the client. The ``artiq.tools.simple_network_args`` library function adds such arguments for the controller.
We suggest naming the controller parameters ``--bind`` (which adds a bind address in addition to a default binding to localhost), ``--no-bind-localhost`` (which disables the default binding to localhost), and ``--port``, so that those parameters stay consistent across controllers. Use ``-s/--server`` and ``--port`` on the client. The ``artiq.tools.simple_network_args`` library function adds such arguments for the controller, and the ``artiq.tools.bind_address_from_args`` function processes them.

The controller's code would contain something similar to this: ::

@@ -125,7 +125,7 @@ The controller's code would contain something similar to this: ::

def main():
args = get_argparser().parse_args()
simple_server_loop(Hello(), args.bind, args.port)
simple_server_loop(Hello(), bind_address_from_args(args), args.port)

We suggest that you define a function ``get_argparser`` that returns the argument parser, so that it can be used to document the command line parameters using sphinx-argparse.

@@ -169,6 +169,11 @@ The program below exemplifies how to use logging: ::
main()


Remote execution support
------------------------

If you wish to support remote execution in your controller, you may do so by simply replacing ``simple_server_loop`` with :class:`artiq.protocols.remote_exec.simple_rexec_server_loop`.

General guidelines
------------------

2 changes: 1 addition & 1 deletion doc/manual/installing.rst
Original file line number Diff line number Diff line change
@@ -192,7 +192,7 @@ and the ARTIQ kernels.

$ mkdir build
$ cd build
$ cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/llvm-or1k -DLLVM_TARGETS_TO_BUILD="OR1K;X86" -DCMAKE_BUILD_TYPE=Rel -DLLVM_ENABLE_ASSERTIONS=ON
$ cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/llvm-or1k -DLLVM_TARGETS_TO_BUILD="OR1K;X86" -DLLVM_ENABLE_ASSERTIONS=ON
$ make -j4
$ sudo make install

7 changes: 7 additions & 0 deletions doc/manual/protocols_reference.rst
Original file line number Diff line number Diff line change
@@ -34,3 +34,10 @@ Protocols reference

.. automodule:: artiq.protocols.sync_struct
:members:


:mod:`artiq.protocols.remote_exec` module
-----------------------------------------

.. automodule:: artiq.protocols.remote_exec
:members: