Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Communication with core device fails #631

Closed
r-srinivas opened this issue Nov 29, 2016 · 24 comments
Closed

Communication with core device fails #631

r-srinivas opened this issue Nov 29, 2016 · 24 comments
Assignees
Milestone

Comments

@r-srinivas
Copy link

Using 3.0 on windows, I've been running some basic experiments like the one below,

from artiq.experiment import *
import numpy as np

class RPC_Test(EnvExperiment):
    
    def build(self):
        self.setattr_device("core")
        self.setattr_device("ttl0")
        self.setattr_argument("points", NumberValue(10, scale=1,
                                                    ndecimals=0, step=1))
        self.setattr_argument("time", NumberValue(20e-6, scale=1e-6,
                                                  ndecimals=0, step=1e-6,
                                                  unit='us'))
        
    def prepare(self):
        self.set_dataset("data.y", np.full(self.points, np.nan),
                         broadcast=True, save=False)
    
    @kernel
    def run(self):
        self.core.reset()
        for i in range(self.points):
            delay(self.time)
            self.ttl0.pulse(self.time)
            self.async_rpc_print(i)
#            self.mutate_dataset("data.y", i, i)
            
    @rpc(flags={"async"})
    def async_rpc_print(self, x):
        print(x)

Twice today I've ran into the situation where communication with the core device just fails altogether. I try running the experiment above and nothing seems to happen. On the gui it just lists the experiment as running.

I can't reproduce this reliably. The second time it happened I deleted the experiment and tried running another experiment and got the following error message,

ERROR:worker(20059,asynchronous_rpc_test.py):root:Terminating with exception (Co
nnectionResetError: [WinError 10054] An existing connection was forcibly closed
by the remote host)
Traceback (most recent call last):
  File "C:\Anaconda3\envs\artiq-2016-11-25\lib\site-packages\artiq\master\worker
_impl.py", line 240, in main
    exp_inst.run()
  File "C:\Anaconda3\envs\artiq-2016-11-25\lib\site-packages\artiq\language\core
.py", line 53, in run_on_core
    return getattr(self, arg).run(run_on_core, ((self,) + k_args), k_kwargs)
  File "C:\Anaconda3\envs\artiq-2016-11-25\lib\site-packages\artiq\coredevice\co
re.py", line 118, in run
    self.comm.check_ident()
  File "C:\Anaconda3\envs\artiq-2016-11-25\lib\site-packages\artiq\coredevice\co
mm_generic.py", line 203, in check_ident
    self._read_header()
  File "C:\Anaconda3\envs\artiq-2016-11-25\lib\site-packages\artiq\coredevice\co
mm_generic.py", line 103, in _read_header
    (sync_byte, ) = struct.unpack("B", self.read(1))
  File "C:\Anaconda3\envs\artiq-2016-11-25\lib\site-packages\artiq\coredevice\co
mm_tcp.py", line 58, in read
    rn = self.socket.recv(min(8192, length - len(r)))
ConnectionResetError: [WinError 10054] An existing connection was forcibly close
d by the remote host

Running another experiment again after this produced a different error message,

Error: Incorrect reply from device: _D2HMsgType.RPC_REQUEST (expected _D2HMsgTyp
e.IDENT_REPLY))
Traceback (most recent call last):
  File "C:\Anaconda3\envs\artiq-2016-11-25\lib\site-packages\artiq\master\worker
_impl.py", line 240, in main
    exp_inst.run()
  File "C:\Anaconda3\envs\artiq-2016-11-25\lib\site-packages\artiq\language\core
.py", line 53, in run_on_core
    return getattr(self, arg).run(run_on_core, ((self,) + k_args), k_kwargs)
  File "C:\Anaconda3\envs\artiq-2016-11-25\lib\site-packages\artiq\coredevice\co
re.py", line 118, in run
    self.comm.check_ident()
  File "C:\Anaconda3\envs\artiq-2016-11-25\lib\site-packages\artiq\coredevice\co
mm_generic.py", line 204, in check_ident
    self._read_expect(_D2HMsgType.IDENT_REPLY)
  File "C:\Anaconda3\envs\artiq-2016-11-25\lib\site-packages\artiq\coredevice\co
mm_generic.py", line 119, in _read_expect
    format(self._read_type, ty))
OSError: Incorrect reply from device: _D2HMsgType.RPC_REQUEST (expected _D2HMsgT
ype.IDENT_REPLY)
ERROR:master:artiq.master.scheduler:got worker exception in run stage, deleting
RID 20060

After this it reached the situation I described where an experiment runs perpetually without actually doing anything. I'm not sure if this is a separate issue from #599, but I'm using artiq 3.0.dev py_245+git95c885b which has that fix presumably.

The solution seems to be to reset the core device and then things work again.

@whitequark
Copy link
Contributor

When such an error message fails, please run artiq_corelog and post the output also. (@jordens, do we have a "writing bug reports" doc page somewhere?)

@r-srinivas
Copy link
Author

I managed to reproduce this by running

from artiq.experiment import *

class Core_Test(EnvExperiment):
    def build(self):
        self.setattr_device("core")

    @kernel
    def run(self):
        self.core.reset()
        print('test')

And just deleting it while running. That seems to prevent it from reconnecting to the core device. I tried running artiq_corelog and it doesn't produce anything until I reset the core device. Then the error message is,

(artiq-2016-11-25) C:\artiq-3-test>artiq_corelog
Traceback (most recent call last):
  File "C:\Anaconda3\envs\artiq-2016-11-25\Scripts\artiq_corelog-script.py", lin
e 10, in <module>
    load_entry_point('artiq==3.0.dev0+245.g95c885b', 'console_scripts', 'artiq_c
orelog')()
  File "C:\Anaconda3\envs\artiq-2016-11-25\lib\site-packages\artiq\frontend\arti
q_corelog.py", line 25, in main
    comm.check_ident()
  File "C:\Anaconda3\envs\artiq-2016-11-25\lib\site-packages\artiq\coredevice\co
mm_generic.py", line 203, in check_ident
    self._read_header()
  File "C:\Anaconda3\envs\artiq-2016-11-25\lib\site-packages\artiq\coredevice\co
mm_generic.py", line 103, in _read_header
    (sync_byte, ) = struct.unpack("B", self.read(1))
  File "C:\Anaconda3\envs\artiq-2016-11-25\lib\site-packages\artiq\coredevice\co
mm_tcp.py", line 58, in read
    rn = self.socket.recv(min(8192, length - len(r)))
ConnectionResetError: [WinError 10054] An existing connection was forcibly close
d by the remote host

@whitequark
Copy link
Contributor

Can you connect to the serial port and post the output?

@r-srinivas
Copy link
Author

By serial port do you mean the UART? That is connected, as is the JTAG. Should it produce the output automatically?

@whitequark
Copy link
Contributor

Yes. In case where the core device crashes, there will almost certainly be some useful output.

@r-srinivas
Copy link
Author

r-srinivas commented Nov 29, 2016

While the core device is working, I get the following output

(artiq-2016-11-25) C:\artiq-3-test>artiq_corelog
[           9us]  INFO(runtime): booting ARTIQ
[         197us]  INFO(runtime): software version 3.0.dev+245.g95c885b
[         410us]  INFO(runtime): gateware version 3.0.dev+245.g95c885b
[        4131us]  INFO(runtime): press 'e' to erase startup and idle kernels...
[     1004007us]  INFO(runtime): continuing boot
[     1004211us]  INFO(runtime::rtio_crg): startup RTIO clock: internal
[     1005194us]  INFO(runtime::session): running startup kernel
[     1007941us]  INFO(runtime::session): no startup kernel found
[     1013613us]  INFO(runtime::session): accepting network sessions in Rust
[     1020256us]  INFO(runtime::session): no connection, starting idle kernel
[     1027202us]  INFO(runtime::session): no idle kernel found
[    73021722us]  INFO(runtime::session): new connection from *:0
[    73021967us]  INFO(runtime::session): terminating running kernel
[    75474741us]  INFO(runtime::session): connection closed
[    75475083us]  INFO(runtime::session): no connection, starting idle kernel
[    75475585us]  INFO(runtime::session): no idle kernel found
[   369576451us]  INFO(runtime::session): new connection from *:0
[   369576694us]  INFO(runtime::session): terminating running kernel
[   372098949us]  INFO(runtime::session): connection closed
[   372099302us]  INFO(runtime::session): no connection, starting idle kernel
[   372099806us]  INFO(runtime::session): no idle kernel found
[   376227766us]  INFO(runtime::session): new connection from *:0
[   376228008us]  INFO(runtime::session): terminating running kernel
[   378710227us]  INFO(runtime::session): connection closed
[   378710553us]  INFO(runtime::session): no connection, starting idle kernel
[   378711055us]  INFO(runtime::session): no idle kernel found
[   381579001us]  INFO(runtime::session): new connection from *:0
[   381579243us]  INFO(runtime::session): terminating running kernel

But if I disconnect the ethernet cable but have the UART connected it doesn't do anything but produce a socket time out error. Is there something I should be doing to produce an output through the UART?

@r-srinivas
Copy link
Author

I tried flterm just to check the UART connection, it connects but the output is different to what it is usually,

C:\artiq-3-test>python -m misoc.tools.flterm COM2

MiSoC BIOS
(c) Copyright 2007-2016 M-Labs Limited
Built Nov 26 2016 10:17:05

BIOS CRC passed (408b1ded)
Initializing SDRAM...
Write leveling: 14* 15* 14  14   9   8   5   6  completed
Read bitslip: 7 6 5 4 3 2
Read delays: 7:00-11  6:01-12  5:03-14  4:03-15  3:11-21  2:10-20  1:00-09  0:00
-09  completed
Memtest OK
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
Booting from flash...
Loading 427496 bytes from flash...

MiSoC BIOS
(c) Copyright 2007-2016 M-Labs Limited
Built Nov 26 2016 10:17:05

BIOS CRC passed (408b1ded)
Initializing SDRAM...
Write leveling: 14* 15* 14  14   9   8   5   6  completed
Read bitslip: 7 6 5 4 3 2
Read delays: 7:00-11  6:01-12  5:03-14  4:03-15  3:11-21  2:10-20  1:00-09  0:00
-09  completed
Memtest OK
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
Booting from flash...
Loading 427496 bytes from flash...
Executing booted program.
[         112us]  INFO(runtime): booting ARTIQ
[         295us]  INFO(runtime): software version 3.0.dev+245.g95c885b
[         503us]  INFO(runtime): gateware version 3.0.dev+245.g95c885b
[        4227us]  INFO(runtime): press 'e' to erase startup and idle kernels...
[     1004081us]  INFO(runtime): continuing boot
[     1004305us]  INFO(runtime::rtio_crg): startup RTIO clock: internal
[     1005281us]  INFO(runtime::session): running startup kernel
[     1008033us]  INFO(runtime::session): no startup kernel found
[     1013714us]  INFO(runtime::session): accepting network sessions in Rust
[     1020353us]  INFO(runtime::session): no connection, starting idle kernel
[     1027295us]  INFO(runtime::session): no idle kernel found

and I can't seem to get to the test mode.

@whitequark
Copy link
Contributor

@r-srinivas This is normal output from the new (Rust) runtime. It does not have the test mode anymore. If you crash the core device now, the flterm output should contain useful information.

@r-srinivas
Copy link
Author

I tried that but it produces the same output. I need to reset the core device after initialising flterm which I guess destroys any of the useful information?

@whitequark
Copy link
Contributor

@r-srinivas You can just leave flterm running.

Aynway, I'll try reproducing this locally later today.

Sorry, something went wrong.

@sbourdeauducq
Copy link
Member

do we have a "writing bug reports" doc page somewhere

https://github.com/m-labs/artiq/blob/master/CONTRIBUTING.rst#reporting-issuesbugs

Sorry, something went wrong.

@r-srinivas
Copy link
Author

So it's fine to leave flterm on while the master is running? Then if I can reproduce the problem, the console with flterm should hopefully have the useful information?

Sorry, something went wrong.

@sbourdeauducq
Copy link
Member

Yes.

Sorry, something went wrong.

@whitequark
Copy link
Contributor

I've reproduced the bug. Here is the relevant core log section:

[  1171154620us]  INFO(runtime::session): new connection from *:0
[  1171154860us]  INFO(runtime::session): terminating running kernel
[  1171155413us] TRACE(runtime::session): comm<-host Ident
[  1171155533us] TRACE(runtime::session): comm->host Ident("3.0.dev+250.gc5b55c1")
[  1171440562us] TRACE(runtime::session): comm<-host SwitchClock(0)
[  1171440845us] TRACE(runtime::session): comm->host ClockSwitchCompleted
[  1172027711us] TRACE(runtime::session): comm<-host LoadLibrary(...)
[  1172043139us] TRACE(runtime::session): comm->kern LoadRequest(...)
[  1172043716us] TRACE(runtime::session): comm<-kern LoadReply(Ok(()))
[  1172043870us] TRACE(runtime::session): comm->host LoadCompleted
[  1172361271us] TRACE(runtime::session): comm<-host RunKernel
[  1172361568us] TRACE(runtime::session): comm<-kern NowInitRequest
[  1172361678us] TRACE(runtime::session): comm->kern NowInitReply(1169005461416)
[  1172362161us] TRACE(runtime::session): comm<-kern RpcSend { async: false, service: 2, tag: [115, 58, 110], data: 0x4ffffee8 }
[  1172362437us] TRACE(runtime::session): comm->host RpcRequest { async: false }
[  1172362583us] TRACE(runtime::rpc_proto): send<2>(String)->None
[  1172949752us] TRACE(runtime::session): comm<-host RpcReply { tag: [110] }
[  1172950023us] TRACE(runtime::session): comm<-kern RpcRecvRequest(0x4ffffeec)
[  1172950165us] TRACE(runtime::rpc_proto): recv ...->None
[  1172950272us] TRACE(runtime::session): comm->kern RpcRecvReply(Ok(0))
[  1172950706us] TRACE(runtime::session): comm<-kern NowSave(1171357811296)
[  1172950949us] TRACE(runtime::session): comm<-kern (async RPC)
[  1172951052us] TRACE(runtime::session): comm->host RpcRequest { async: true }
[  1173225238us]  INFO(runtime::session): new connection from *:0
[  1173225480us]  INFO(runtime::session): terminating running kernel
[  1173225866us] ERROR(runtime::session): session aborted: not connected
[  1173231246us] TRACE(runtime::session): comm<-kern (async RPC)
[  1173231358us] TRACE(runtime::session): comm->host RpcRequest { async: true }
[  1173237256us] TRACE(runtime::session): [0, 0, 0, 1, 110, 0, 0, 0, 0, 1, 110]
[  1173239340us] TRACE(runtime::session): comm<-host Ident
[  1173239463us] TRACE(runtime::session): comm->host Ident("3.0.dev+250.gc5b55c1")
[  1173510730us]  INFO(runtime::session): connection closed

@r-srinivas
Copy link
Author

r-srinivas commented Nov 29, 2016

Thanks for fixing it. I tried again on flterm to see if I could get the log but it seems to get stuck when the core device fails.

(C:\Anaconda3) C:\Users\rabi>python -m misoc.tools.flterm COM2
[ 62078313171us]  INFO(runtime::session): new connection from *:0
[ 62078313422us]  INFO(runtime::session): terminating running kernel
[ 62083026459us]  INFO(runtime::session): connection closed
[ 62083026839us]  INFO(runtime::session): no connection, starting idle kernel
[ 62083027384us]  INFO(runtime::session): no idle kernel found
[ 62089709467us]  INFO(runtime::session): new connection from *:0
[ 62089709718us]  INFO(runtime::session): terminating running kernel
[ 62097584533us]  INFO(runtime::session): new connection from *:0
[ 62097584788us]  INFO(runtime::session): terminating running kernel

Is the output on the UART. Did that work for you to get the core log?

Edit: I have misoc 0.1 installed, would that be an issue?

@whitequark
Copy link
Contributor

Is the output on the UART. Did that work for you to get the core log?

Yes, this is the correct output (and the same as I get). I thought the core device crashed; it isn't, that was just a logic bug. I had to enable debug output and run artiq_corelog locally to diagnose.

@r-srinivas
Copy link
Author

Okay, thanks. Has flterm's test mode been replaced in someway? We found it pretty useful for debugging, dds' especially.

@r-srinivas
Copy link
Author

r-srinivas commented Nov 29, 2016

Seems like this problem still happens even with the latest commit. It doesn't happen as easily as before: instead of deleting the experiment 1 or 2 times I had to try maybe 4-5 times but the symptoms seem the same.

Looking at flterm,

(artiq-2016-11-29) c:\artiq-3-test>python -m misoc.tools.flterm COM2
[    43202821us]  INFO(runtime::session): new connection from *:0
[    43203046us]  INFO(runtime::session): terminating running kernel
[    47961377us]  INFO(runtime::session): connection closed
[    47961721us]  INFO(runtime::session): no connection, starting idle kernel
[    47962207us]  INFO(runtime::session): no idle kernel found
[    59038786us]  INFO(runtime::session): new connection from *:0
[    59039011us]  INFO(runtime::session): terminating running kernel
[    61615353us] ERROR(runtime::session): session aborted: connection reset
[    61615660us]  INFO(runtime::session): no connection, starting idle kernel
[    61616621us]  INFO(runtime::session): no idle kernel found
[    70356141us]  INFO(runtime::session): new connection from *:0
[    70356367us]  INFO(runtime::session): terminating running kernel
[    75087666us]  INFO(runtime::session): connection closed
[    75088012us]  INFO(runtime::session): no connection, starting idle kernel
[    75088513us]  INFO(runtime::session): no idle kernel found
[    79885585us]  INFO(runtime::session): new connection from *:0
[    79885810us]  INFO(runtime::session): terminating running kernel
[    83001643us] ERROR(runtime::session): session aborted: connection reset
[    83001951us]  INFO(runtime::session): no connection, starting idle kernel
[    83002917us]  INFO(runtime::session): no idle kernel found
[    84995782us]  INFO(runtime::session): new connection from *:0
[    84996007us]  INFO(runtime::session): terminating running kernel
[    89751302us]  INFO(runtime::session): connection closed
[    89751647us]  INFO(runtime::session): no connection, starting idle kernel
[    89752054us]  INFO(runtime::session): no idle kernel found
[    91645454us]  INFO(runtime::session): new connection from *:0
[    91645679us]  INFO(runtime::session): terminating running kernel
[    96359475us]  INFO(runtime::session): connection closed
[    96359821us]  INFO(runtime::session): no connection, starting idle kernel
[    96360306us]  INFO(runtime::session): no idle kernel found
[    98447373us]  INFO(runtime::session): new connection from *:0
[    98447598us]  INFO(runtime::session): terminating running kernel
[   103218677us]  INFO(runtime::session): connection closed
[   103219025us]  INFO(runtime::session): no connection, starting idle kernel
[   103219538us]  INFO(runtime::session): no idle kernel found
[   104953722us]  INFO(runtime::session): new connection from *:0
[   104953949us]  INFO(runtime::session): terminating running kernel
[   108135387us] ERROR(runtime::session): session aborted: connection reset
[   108135696us]  INFO(runtime::session): no connection, starting idle kernel
[   108136663us]  INFO(runtime::session): no idle kernel found
[   110388709us]  INFO(runtime::session): new connection from *:0
[   110388936us]  INFO(runtime::session): terminating running kernel
[   115197765us]  INFO(runtime::session): connection closed
[   115198112us]  INFO(runtime::session): no connection, starting idle kernel
[   115198613us]  INFO(runtime::session): no idle kernel found
[   142679456us]  INFO(runtime::session): new connection from *:0
[   142679683us]  INFO(runtime::session): terminating running kernel
[   148494767us]  INFO(runtime::session): new connection from *:0
[   148494998us]  INFO(runtime::session): terminating running kernel

MiSoC BIOS
(c) Copyright 2007-2016 M-Labs Limited
Built Nov 30 2016 03:23:16

BIOS CRC passed (06d5ed0e)
Initializing SDRAM...
Write leveling: 14* 15* 14  14   9   8   5   6  completed
Read bitslip: 7 6 5 4 3 2
Read delays: 7:00-11  6:01-12  5:03-14  4:03-15  3:11-21  2:10-20  1:00-09  0:00
-09  completed
Memtest OK
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
Booting from flash...
Loading 428792 bytes from flash...
Executing booted program.
[         112us]  INFO(runtime): booting ARTIQ
[         300us]  INFO(runtime): software version 3.0.dev+252.g5b7e068
[         510us]  INFO(runtime): gateware version 3.0.dev+252.g5b7e068
[        4227us]  INFO(runtime): press 'e' to erase startup and idle kernels...
[     1004080us]  INFO(runtime): continuing boot
[     1004303us]  INFO(runtime::rtio_crg): startup RTIO clock: internal
[     1005314us]  INFO(runtime::session): running startup kernel
[     1008033us]  INFO(runtime::session): no startup kernel found
[     1013710us]  INFO(runtime::session): accepting network sessions in Rust
[     1020349us]  INFO(runtime::session): no connection, starting idle kernel
[     1027300us]  INFO(runtime::session): no idle kernel found

There ERRORs are when I deleted the experiment and at some point the kernel stopped communicating and I had to reset it. It looks like it's using the version with the fix as well.

Edit: The experiment I'm running is the same as before,

from artiq.experiment import *

class Core_Test(EnvExperiment):
    def build(self):
        self.setattr_device("core")

    @kernel
    def run(self):
        self.core.reset()
        print('test')

And I'm just deleting it before it completes.

@r-srinivas
Copy link
Author

r-srinivas commented Nov 29, 2016

It seems to matter when you delete the experiment. What reproduces it quite reliably is when you delete the experiment such that it still prints test. If test is printed and the experiment was deleted it results in not being able to communicate with the core device anymore.

Edit: The log on the gui looks like something like this

artiq.dashboard.experiments:Submitted 'repo:test/Core_Test', RID is 20123
artiq.dashboard.schedule:Deleted RID 20123
print:test

@whitequark whitequark reopened this Nov 29, 2016
@r-srinivas
Copy link
Author

Strangely if I try running this from the virtual machine, the experiment never works and always triggers this problem.

On flterm, I can see that it connects,

[    10689657us]  INFO(runtime::session): new connection from *:0
[    10689867us]  INFO(runtime::session): terminating running kernel

But then gets stuck. Curiously, I can still ping the device while it's stuck.

(artiq-2016-11-29)rabi@68810MAGTRAPVM:~$ ping 132.163.182.89
PING 132.163.182.89 (132.163.182.89) 56(84) bytes of data.
64 bytes from 132.163.182.89: icmp_seq=1 ttl=255 time=0.289 ms
64 bytes from 132.163.182.89: icmp_seq=2 ttl=255 time=0.307 ms
64 bytes from 132.163.182.89: icmp_seq=3 ttl=255 time=0.303 ms
64 bytes from 132.163.182.89: icmp_seq=4 ttl=255 time=0.646 ms
^C
--- 132.163.182.89 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2998ms
rtt min/avg/max/mdev = 0.289/0.386/0.646/0.150 ms

@r-srinivas
Copy link
Author

I had to enable debug output and run artiq_corelog locally to diagnose.

Would it help if I tried to do this to get the log? How do I do that exactly?

@whitequark
Copy link
Contributor

@sbourdeauducq This is not a runtime bug but rather another lwip bug:

Assertion "tcp_input: pcb->next != pcb (before cache)" failed at line 227 in /home/whitequark/Work/artiq-dev/artiq/artiq/runtime/liblwip/../lwip/src/core/tcp_in.c
Assertion "tcp_input: pcb->next != pcb (after cache)" failed at line 235 in /home/whitequark/Work/artiq-dev/artiq/artiq/runtime/liblwip/../lwip/src/core/tcp_in.c

@sbourdeauducq
Copy link
Member

Either way this needs fixing, as it makes ARTIQ unusable.

jordens added a commit that referenced this issue Dec 2, 2016
* master:
  dashboard: mention disable in CCB policies
  runtime: clear async RPC queue when kernel stops (fixes #631).
  artiq_devtool: fix incorrect use of nargs in argparse.
@whitequark
Copy link
Contributor

The remaining bug is a manifestation of #637, let's discuss it there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants