What's the goal? Isn't ofborg sufficient? I'd hate to add yet another CI system that can break.

What's the goal? Isn't ofborg sufficient? I'd hate to add yet another CI system that can break.

It's to migrate responsibilities from ofborg. Ofborg can be only maintained right now by a 1-2 people, because there is not testing setup that allows to reproduce the environment. The idea is to combined this with a binary cache so that builds can be re-used between runs (important for staging) and also that reviewers can download them without re-compiling. ofborg also does not build macOS for all users since builds run without sandbox. Users needs to be explicitly trusted/whitelisted. Github actions solve this problem.

@grahamc also has expressed that we wants to move ofborg to a different infrastructure (i.e. buildkite). Self-hosted github actions would be one solution to run evaluations that requires a lot of RAM.

It's to migrate responsibilities from ofborg.

Do you mean that this is intended to replace ofborg?

The idea is to combined this with a binary cache

Who has write access to this binary cache?

It's to migrate responsibilities from ofborg.

Do you mean that this is intended to replace ofborg?

Ideally yes. It's less maintenance effort in long term if we offload the github api integration to github itself. There have been some problems with webhooks in the past if I remember correctly. Also the current UI for logs is not as good as having the log output directly in github itself. But I also would like to hear @grahamc opinion on that.

The idea is to combined this with a binary cache

Who has write access to this binary cache?

The Github CI and repo admins. The signing key would be stored in github itself. @domenkozar offered to host it. In any case it would be kept separate from hydra.

We may want to see about getting the required status checks working if we're going to start adding more actions.

We may want to see about getting the required status checks working if we're going to start adding more actions.

I don't think with the current CI infrastructure we will get required status checks, @grahamc was concerned about the availability of ofborg. The Wait for ofborg action improves the situation because if ofborg is down users will get an indication for that. This can be further improved if ofborg would create one succeeding check even before the evaluation finishes and also create the evaluation checks before it actually start the evaluation. The current check api is buggy with w.r.t. external checks and only exposes finished checks and not pending one, I also tried web scraping however github only exposes CI status to logged in users...

To further improve the situation I created NixOS/ofborg#527
It adds a check that gets triggered earlier so that "Waiting for ofborg" are less likely to timeout.

It would be good to write out the intentions on say the mailing list (discourse) so it does not come as a surprise. I was already surprised to see red crosses due to some linting.

I see one problem with the integration of cachix. The signing key would be only available to builds in master, not every PR. Also every nixpkgs member could potentially export the key by pushing a commit to master. @domenkozar do you have scope how this could be limited?

What's the goal? Isn't ofborg sufficient? I'd hate to add yet another CI system that can break.

The bus factor of our current CI is also too low: NixOS/ofborg#527 (comment)

I see one problem with the integration of cachix. The signing key would be only available to builds in master, not every PR.

That depends on the goal. If the goal is to reduce build times, then it doesn't matter if we trust committers as long as people are not using the binary cache to trust the binaries, but only in a sandbox environment such as hosted runners.

If the goal is to trust the binaries without exposing the signing key to write users, then github model doesn't allow that - we have to wait for them to add a way to limit access further by locking down to a branch or a whitelist of users.

This can be somewhat limited by introducing IP whitelist on Cachix side, but an attacker could leak on github actions and push binaries via github actions.

We don't have a macOS builder running at the moment. Graham won't be able to apply fixes to ofborg due to personal reasons that would make them work better together with github actions. Hence my proposal is to start those actions manually:

on: 
  issue_comment:
    types: [created]
name: Build packages mentioned in commits
jobs:
  build:
    name: build
    if: github.event.issue.pull_request != '' && contains(github.event.comment.body, '/build')
    runs-on: ubuntu-latest # and macOS?
    steps:
    - name: Checkout the latest code
      uses: actions/checkout@v2
    # ...

That should be a good compromise for people that want to test packages on different platforms, no?
Also it would gives us the opportunity to test new actions before enabling them by default.

I see one problem with the integration of cachix. The signing key would be only available to builds in master, not every PR.

That depends on the goal. If the goal is to reduce build times, then it doesn't matter if we trust committers as long as people are not using the binary cache to trust the binaries, but only in a sandbox environment such as hosted runners.

If the goal is to trust the binaries without exposing the signing key to write users, then github model doesn't allow that - we have to wait for them to add a way to limit access further by locking down to a branch or a whitelist of users.

This can be somewhat limited by introducing IP whitelist on Cachix side, but an attacker could leak on github actions and push binaries via github actions.

Sounds good to me, if you are happy to implement this in cachix. Things would be a lot easier if we would have more control on what actions are executed, i.e. if actions would always come from the master branch.

We don't have a macOS builder running at the moment.

https://monitoring.nix.ci/d/000000002/ofborg?orgId=1&refresh=10s

Looks like it's back?

We don't have a macOS builder running at the moment.

monitoring.nix.ci/d/000000002/ofborg?orgId=1&refresh=10s

Looks like it's back?

Still it's not available for general use, whereas github is. And since we only got one builder its availability is potentially also not so great.

See #99722

I marked this as stale due to inactivity. → More info