Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

experiments submitted with Flush=True cannot interrupt other experiments #640

Closed
dleibrandt opened this issue Dec 11, 2016 · 7 comments
Closed
Assignees
Milestone

Comments

@dleibrandt
Copy link
Contributor

Running 2.0.

I'd like to schedule some calibration experiments that run every x seconds such that they interrupt the main experiment. The calibration experiments need to use the newest values of the various parameters (that are being computed by other calibration experiments), so I'd like to run them with Flush=True. However, when my main experiment calls scheduler.check_pause(), it always returns False because the calibration experiments were submitted with Flush=True, so the calibration experiments never run.

I actually think that the behavior of scheduler.check_pause() makes sense, but can we add another function to the scheduler that returns true if there is a higher priority experiment in the schedule that was submitted with Flush=True? Then when this new function returns True I can end the current experiment and reschedule it.

@sbourdeauducq
Copy link
Member

Are you sure you want to do that, and not simply set the lowest priority on your long experiment that calls pause in a loop (so if there are any other experiments in the pipeline, they would always get executed when there's a pause, and it would finally reach your flush=True experiment)? Note that ARTIQ supports an arbitrary number of priority levels.

@sbourdeauducq
Copy link
Member

And if you really want that, you can write this function yourself and put it into the experiment. This will come in handy: https://m-labs.hk/artiq/manual-release-2/management_system.html#artiq.master.scheduler.Scheduler.get_status

@dleibrandt
Copy link
Contributor Author

I have set the lowest priority on my long (main) experiment, and the calibration experiments still never run. I believe this is because scheduler.check_pause() only looks for experiments that have finished prepare and doesn't look for experiments that are waiting for the queue to flush. I assume the same is true for scheduler.pause(). However, my calibration experiments cannot finish prepare until the main experiment ends, because I've submitted them with Flush=True, so they get stuck in Status=Flushing and never run.

I've also already written this function myself, but I figure that other groups will also want this functionality, so it probably makes sense to build it into artiq.

@sbourdeauducq
Copy link
Member

I have set the lowest priority on my long (main) experiment, and the calibration experiments still never run.

Please post a screenshot of the scheduler showing the problem.

@dleibrandt
Copy link
Contributor Author

In the example below, ExpA is the "main" experiment, and ExpB is the "calibration" experiment. ExpA is submitted first with Priority=0. ExpB is submitted second with Priority=1 and Flushing=True. Exp never runs.

from time import sleep
from artiq.experiment import *

class ExpA(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("scheduler")
        
    def run(self):
        while True:
            self.run_kernel()
            
            self.core.comm.close()
            self.scheduler.pause()
    
    @kernel
    def run_kernel(self):
        while True:
            print("ExpA run")
            if self.scheduler.check_pause():
                break
            sleep(1)
            
class ExpB(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        
    @kernel
    def run(self):
        print("ExpB run")

Screenshot:
screenshot from 2016-12-13 14 37 04

If you look at

prepared_runs = filter(lambda r: r.status == RunStatus.prepare_done,

you'll see that check_pause ignores experiments that have Status=flushing.

@r-srinivas
Copy link

I ran into this problem as well. It would be useful if experiments set to flush at a higher priority would run ahead of experiments running in the background at a lower priority.

@sbourdeauducq
Copy link
Member

This is not a bug, "flushed" previously meant "nothing else ahead of us", which is a perfectly valid definition. It now means "nothing else ahead of us at our and higher priority levels".

Without complexity explosion, we can have either behavior selectable at experiment submission time, if that's relevant. Open a new issue if you want that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants