NixOS · May 20, 2020 · May 20, 2020 · May 20, 2020
Showing with 156 additions and 528 deletions.

+153 −187 README.md

+0 −82 config.extra-known-users.json

+0 −193 config.known-users.json

+1 −16 ofborg/src/acl.rs

+0 −5 ofborg/src/config.rs

+2 −5 ofborg/src/tasks/evaluationfilter.rs

+0 −40 scripts/update-known-users.sh
diff --git a/README.md b/README.md
@@ -1,21 +1,19 @@
-# grahamcofborg
+# ofborg
 
 ## Guidelines
 
-1. make sure you've reviewed the code before you trigger it on a PR
-   that isn't your own
-2. be gentle, preferably don't run mass rebuilds / massive builds like
-   chromium on it
+1. Review the code of all PRs before triggering the bot on them.
+2. Be gentle; try not to run mass rebuilds or massive builds (like Chromium) on
+   it.
 
 ## Automatic Building
 
-Users who are _trusted_ or _known_ (see: Trusted Users vs Known Users)
-will have their PRs automatically trigger builds if their commits
-follow the well-defined format of Nixpkgs. Specifically: prefixing the
-commit title with the package attribute. This includes package bumps
-as well as other changes.
+All users will have their PRs automatically trigger builds if their commits
+follow the well-defined format of Nixpkgs. Specifically: prefixing the commit
+title with the package attribute. This includes package bumps as well as other
+changes.
 
-Example messages and the builds:
+Example commit titles and the builds they will start:
 
 | Message                                                               | Automatic Build                                          |
 |-----------------------------------------------------------------------|----------------------------------------------------------|
@@ -24,245 +22,213 @@ Example messages and the builds:
 | `python36Packages.requests,python27Packages.requests: 1.0.0 -> 2.0.0` | `python36Packages.requests`, `python27Packages.requests` |
 | `python{2,3}Packages.requests: 1.0.0 -> 2.0.0`                        | _nothing_                                                |
 
-If a PR is opened with many commits, it will create a single build job
-for all of the detected packages. If a PR is opened and many commits
-are pushed one by one to the open PR, many build jobs will be created.
+When opening a PR with multiple commits, ofborg creates a single build job for
+all detected packages. If multiple commits get pushed to a PR one-by-one, each
+detected package will get a separate build job.
 
-To disable automatic building of packages on a PR, add `[WIP]` to the
-PR's title, or the `2.status: work-in-progress` label.
+If the title of a PR begins with `WIP:`, contains `[WIP]` anywhere, or has the
+`2.status: work-in-progress` label, its packages are not built automatically.
+**Note**: Marking a PR as a draft does not prevent automatic builds.
 
 ## Commands
 
-The comment parser is line-based, so comments can be interleaved with
-instructions.
+The comment parser is line-based, so commentary can be interwoven with
+instructions for ofborg.
 
-1. To trigger the bot, the line _must_ start with a case
-   insensitive version of `@GrahamcOfBorg`.
-2. To use multiple commands, insert a bit of whitespace and then your
-   new command.
+1. To trigger the bot, the line _must_ start with `@ofborg` (case insensitive).
+   * **Note**: GitHub will not suggest `@ofborg` to you, but it will work all
+     the same. When in doubt, preview your comment and verify that `@ofborg`
+     links to https://github.com/ofborg/.
+2. To use multiple commands, separate them with whitespace. For examples, see
+   the "[Multiple Commands](#multiple-commands)" section.
 
-Commands:
-
-### test (added: 2017-11-24)
+### test
 
 ```
-@grahamcofborg test list of tests
+@ofborg test list of tests
 ```
 
-This will run `nix-build ./nixos/release.nix -A tests.list -A tests.of -A tests.tests` in
-the nixpkgs checkout. Note: this will only run on x86_64-linux machines.
+This will run `nix-build ./nixos/release.nix -A tests.list -A tests.of -A
+tests.tests` from the root of the Nixpkgs checkout.
+
+Tests will run on all allowed machines. For more information, see the "[Trusted
+Users](#trusted-users)" section.
 
 ### eval
 
 ```
-@grahamcofborg eval
+@ofborg eval
 ```
 
-Note: Every PR automatically evaluates when it is opened and when the
-commits change. There is no reason to run eval on a PR unless the
-evaluation has failed for weird reasons, or because master was broken
-before.
+See "[How does ofborg call
+`nix-instantiate`?](#how-does-ofborg-call-nix-instantiate)" for what command(s)
+this will run.
+
+**Note**: Every PR automatically evaluates both upon creation and when the
+commits change. There is no reason to run eval on a PR unless the evaluation
+failed for weird reasons or master was previously broken.
 
 ### build
 
 ```
-@grahamcofborg build list of attrs
+@ofborg build list of attrs
 ```
 
-This will run `nix-build ./default.nix -A list -A of -A attrs` in
-the nixpkgs checkout.
+This will run `nix-build ./default.nix -A list -A of -A attrs` from the root of
+the Nixpkgs checkout (see also "[How does ofborg call
+`nix-build`?](#how-does-ofborg-call-nix-build)").
 
----
+Builds will run on all allowed machines. For more information, see the "[Trusted
+Users](#trusted-users)" section.
 
+## Multiple Commands
 
-Multiple Commands:
+You can use multiple commands in a variety ways. Here are some valid
+combinations:
 
-```
-@grahamcofborg build list of attrs
-@grahamcofborg eval
-```
-
-or even:
+*
+    ```
+    @ofborg build list of attrs
+    @ofborg eval
+    ```
 
-```
-@grahamcofborg build list of attrs @grahamcofborg eval
-```
+*
+    ```
+    @ofborg build list of attrs @ofborg eval
+    ```
 
-This will also work:
+*
+    ```
+    looks good to me!
+    @ofborg eval
+    @ofborg build list of attrs
+    ```
 
-```
-looks good to me!
-@grahamcofborg build list of attrs
-```
+*
+    ```
+    @ofborg eval
+    @ofborg build list of attrs
+    looks good to me!
+    ```
 
-And this is fine:
+*
+    ```
+    @ofborg build list of attrs
+    @ofborg test list of attrs
+    ```
 
-```
-@grahamcofborg build list of attrs
-looks good to me!
-```
+* This will build `list`, `of`, `attrs`, `looks`, `good`, `to`, and `me!` (which is probably not what you want):
+    ```
+    @ofborg build list of attrs looks good to me!
+    ```
 
-This is will build `list`, `of`, `attrs`, `looks`, `good`, `to`, `me!`:
+## Trusted Users
 
-```
-@grahamcofborg build list of attrs looks good to me!
-```
+Trusted users have their builds and tests executed on _all_ available platforms,
+including those without good sandboxing. Because this exposes the host to a
+higher risk of security issues, the trusted users list consists of only
+well-known, trusted members of the community.
 
-## Trusted Users vs Known Users
-
-Known users have their builds executed on platforms with working
-sandboxing. At the time of writing, that means:
+At the time of writing, trusted users have their builds and tests run on these
+platforms:
 
  - `x86_64-linux`
- - `aarch64_linux`
-
-Trusted users have their builds executed on _all_ platforms, even if
-they don't have good sandboxing. This opens the host up to a higher
-risk of security issues, so only well known, trusted member of the
-community should be added to the trusted user list.
-
-At the time of writing, trusted users have their builds run on the
-following platforms:
-
- - `x86_64-linux`
- - `aarch64_linux`
+ - `aarch64-linux`
  - `x86_64-darwin`
 
-See ./config.public.json and ./config.known-users.json for a list of
-all the trusted and known users.
-
-# How does OfBorg call nix-build?
+See [`config.public.json`](./config.public.json) for a list of all trusted users.
 
-Builds are run like:
+# How does ofborg call `nix-build`?
 
-> HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-build ./default.nix
-> --no-out-link --keep-going -A hello
-> --option restrict-eval true
-> --option build-timeout 1800
-> --argstr system thesystem
-> --show-trace
+ofborg runs builds with a command similar to the following:
 
-# How does OfBorg call nix-instantiate?
+```shell
+$ HOME=/homeless-shelter NIX_PATH=ofborg-nixpkgs-pr=$(pwd) nix-build ./default.nix \
+    -A hello \
+    --no-out-link \
+    --keep-going \
+    --option restrict-eval true \ 
+    --option build-timeout 1800 \ 
+    --argstr system thesystem \
+    --show-trace
+```
 
-NixOS evals are run like:
+# How does ofborg call `nix-instantiate`?
 
-> HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate ./nixos/release.nix
-> -A manual
-> --option restrict-eval true
-> --option build-timeout 1800
-> --argstr system thesystem
-> --show-trace
+ofborg runs NixOS evals with a command similar to the following:
 
-Nixpkgs evals are run like:
+```shell
+$ HOME=/homeless-shelter NIX_PATH=ofborg-nixpkgs-pr=$(pwd) nix-instantiate ./nixos/release.nix \
+    -A manual \
+    --option restrict-eval true \
+    --option build-timeout 1800 \
+    --argstr system thesystem \
+    --show-trace
+```
 
-> HOME=/homeless-shelter NIX_PATH=nixpkgs=$(pwd) nix-instantiate ./pkgs/top-level/release.nix
-> -A manual
-> --option restrict-eval true
-> --option build-timeout 1800
-> --argstr system thesystem
-> --show-trace
+ofborg runs Nixpkgs evals with a command similar to the following:
 
+```shell
+$ HOME=/homeless-shelter NIX_PATH=ofborg-nixpkgs-pr=$(pwd) nix-instantiate ./pkgs/top-level/release.nix \
+    -A manual \
+    --option restrict-eval true \
+    --option build-timeout 1800 \
+    --argstr system thesystem \
+    --show-trace
+```
 
 # Running meta checks locally
 
-```
+To run the meta checks, you will need the
+[`outpaths.nix`](./ofborg/src/outpaths.nix) file. You can acquire this file and
+run the checks themselves like so:
+
+```shell
 $ curl -o outpaths.nix https://raw.githubusercontent.com/NixOS/ofborg/released/ofborg/src/outpaths.nix
 $ GC_INITIAL_HEAP_SIZE=4g nix-env -f ./outpaths.nix -qaP --no-name --out-path --arg checkMeta true > out-paths
 ```
 
----
-
-
+# Hacking
 
-
-# Running a builder
-
-It is recommended to create a special user for the ofBorg operation.
-This user should have git configuration for default username/email for
-non-interactive merges. For example:
-```
-git config --global user.email "graham+cofborg@example.com"
-git config --global user.name "GrahamCOfBorg"
-```
-
-
-```
-nix-shell ./shell.nix
+```shell
+$ git clone https://github.com/NixOS/ofborg/
 $ cd ofborg
+$ nix-shell ./shell.nix
+$ cd ofborg # enter the subdirectory with Rust code
+# make your changes
 $ cargo build
+$ cargo check
+$ cargo test
 ```
 
-```
-cargo build
-```
-
-then copy example.config.json to config.json and edit its vars. Set
-`nix.remote` to an empty string if you're not using the daemon.
-
-Run
+To test whether or not Continuous Integration will pass with your changes, you
+can run the following commands from the root of your checkout:
 
-```
-./target/debug/builder ./config.json
+```shell
+$ nix-shell --run checkPhase -A mozilla-rust-overlay # checks rustfmt and clippy
+$ nix-shell --run checkPhase # runs the test suite
+$ nix-build -A ofborg.rs # build ofborg
 ```
 
+Currently there is no easy way to set up a test instance of ofborg. If `cargo
+check` and `cargo test` both succeed, feel free to Pull Request your changes.
+Make sure to format your code with `cargo fmt` and check for additional warnings
+with `cargo clippy`. If you added, removed, or updated the dependencies, also be
+sure to update Carnix by running
+[`./nix/update-carnix.sh`](./nix/update-carnix.sh).
 
-Note the config.public.json for the public pieces of how I run ofborg,
-which is merged with config.known-users.json and a third private
-config file of credentials. These files contain some special keys like
+To disable warnings as errors, run your command with an empty `RUSTFLAGS`. For
+example:
 
- - known users
- - authorized users
- - log storage
-
-they are only used in the backend processing tasks, and there is no
-need for them on builders. However, to update the list in
-config.known-users.json, run `./scripts/update-known-users.sh`.
-
-In case you have a non-trivial setup on Linux, make sure that the ofborg
-user has access to `dev/kvm`, as it is needed for running tests.
-
-If you want to run multiple builder instances on the same physical
-machine please make sure they use different configs with different
-instance identity (same username/password is OK) and different
-repository paths. Running two builders with the same config risks data
-corruption.
-
-See also: https://github.com/NixOS/ofborg/wiki/Operating-a-Builder
-
-## old php stuff...
-
-Only Graham needs to do this, since I run the only remaining PHP
-components.
-
-```php
-<?php
-
-require_once __DIR__ . '/vendor/autoload.php';
-use PhpAmqpLib\Connection\AMQPSSLConnection;
-use PhpAmqpLib\Message\AMQPMessage;
-
-function rabbitmq_conn($timeout = 3) {
-    $host = 'events.nix.gsc.io';
-    $connection = new AMQPSSLConnection(
-        $host, 5671,
-        'eventsuser, eventspassword, '/',
-        array(
-            'verify_peer' => true,
-            'verify_peer_name' => true,
-            'peer_name' => $host,
-            'verify_depth' => 10,
-            'ca_file' => '/etc/ssl/certs/ca-certificates.crt',
-        ), array(
-            'connection_timeout' => $timeout,
-        )
-    );
-
-    return $connection;
-}
+```shell
+$ RUSTFLAGS= cargo clippy
+```
 
-function gh_secret() {
-    return "github webhook secret";
-}
+This will override the default of `-D warnings` set in
+[`shell.nix`](./shell.nix), which tells Rust to error if it detects any
+warnings.
 
+# Running a builder
 
-```
+If you want to run a builder of your own, check out the [wiki page on operating
+a builder](https://github.com/NixOS/ofborg/wiki/Operating-a-Builder/).
diff --git a/config.extra-known-users.json b/config.extra-known-users.json
diff --git a/config.known-users.json b/config.known-users.json
diff --git a/ofborg/src/acl.rs b/ofborg/src/acl.rs
@@ -2,25 +2,18 @@ use crate::systems::System;
 
 pub struct ACL {
     trusted_users: Vec<String>,
-    known_users: Vec<String>,
     repos: Vec<String>,
 }
 
 impl ACL {
-    pub fn new(
-        repos: Vec<String>,
-        mut trusted_users: Vec<String>,
-        mut known_users: Vec<String>,
-    ) -> ACL {
+    pub fn new(repos: Vec<String>, mut trusted_users: Vec<String>) -> ACL {
         trusted_users
             .iter_mut()
             .map(|x| *x = x.to_lowercase())
             .last();
-        known_users.iter_mut().map(|x| *x = x.to_lowercase()).last();
 
         ACL {
             trusted_users,
-            known_users,
             repos,
         }
     }
@@ -53,14 +46,6 @@ impl ACL {
             .collect()
     }
 
-    pub fn can_build_restricted(&self, user: &str, repo: &str) -> bool {
-        if repo.to_lowercase() != "nixos/nixpkgs" {
-            return false;
-        }
-
-        self.known_users.contains(&user.to_lowercase())
-    }
-
     pub fn can_build_unrestricted(&self, user: &str, repo: &str) -> bool {
         if repo.to_lowercase() == "nixos/nixpkgs" {
             self.trusted_users.contains(&user.to_lowercase())

diff --git a/ofborg/src/config.rs b/ofborg/src/config.rs
@@ -69,7 +69,6 @@ pub struct RunnerConfig {
     pub identity: String,
     pub repos: Option<Vec<String>>,
     pub trusted_users: Option<Vec<String>>,
-    pub known_users: Option<Vec<String>>,
 
     /// If true, will create its own queue attached to the build job
     /// exchange. This means that builders with this enabled will
@@ -100,10 +99,6 @@ impl Config {
                 .trusted_users
                 .clone()
                 .expect("fetching config's runner.trusted_users"),
-            self.runner
-                .known_users
-                .clone()
-                .expect("fetching config's runner.known_users"),
         )
     }
 

diff --git a/ofborg/src/tasks/evaluationfilter.rs b/ofborg/src/tasks/evaluationfilter.rs
@@ -107,11 +107,8 @@ mod tests {
         let job: ghevent::PullRequestEvent =
             serde_json::from_str(&data.to_string()).expect("Should properly deserialize");
 
-        let mut worker = EvaluationFilterWorker::new(acl::ACL::new(
-            vec!["nixos/nixpkgs".to_owned()],
-            vec![],
-            vec![],
-        ));
+        let mut worker =
+            EvaluationFilterWorker::new(acl::ACL::new(vec!["nixos/nixpkgs".to_owned()], vec![]));
 
         assert_eq!(
             worker.consumer(&job),

diff --git a/scripts/update-known-users.sh b/scripts/update-known-users.sh