rustPlatform.fetchCargo: handle custom Cargo.lock patchfiles with validation #80262

bhipple · 2020-02-16T10:07:14Z

Previously, we would asssert that the lockfiles are consistent during the
unpackPhase, but if the pkg has a patch for the lockfile itself then we must
wait until the patchPhase is complete to check.

This also removes an implicity dependency on the src attribute coming from
fetchzip / fetchFromGitHub, which happens to name the source directory
"source". Now we glob for it, so different fetchers will work consistently.

Motivation for this change

Things done

bhipple

CC @zimbatm I found an issue with the pre-existing validation logic: it ran prior to the patch phase, so if the patch phase were going to fix a deficient cargo.lock it would've checked too early.

This isn't causing any build failures, but prevents us from upgrading the ~20 packages that need it to the newer infra. I've done 3 packages in staging here to demonstrate this works.

bhipple · 2020-02-16T10:10:55Z

pkgs/applications/science/misc/rink/default.nix

-  # Delete this on next update; see #79975 for details
-  legacyCargoFetcher = true;
+  # Upstreamed in https://github.com/tiffany352/rink-rs/pull/53
+  cargoPatches = [ ./cargo-lock.patch ];


This one has been merged in tiffany352/rink-rs#53, so we'll be able to drop this patch once the next release comes out.

bhipple · 2020-02-16T10:11:45Z

pkgs/applications/virtualization/railcar/default.nix

  # Submitted upstream https://github.com/oracle/railcar/pull/44
  cargoPatches = [ ./cargo-lock.patch ];
+  cargoSha256 = "10qxkxpdprl2rcgy52s3q5gyg3i75qmx68rpl7cx1bgjzppfn9c3";


This one was upstreamed months ago, but it looks like Oracle has killed the Railcar project, so I suppose we'll carry this patch around for as long as we keep it packaged.

bhipple · 2020-02-16T10:12:31Z

pkgs/development/tools/rust/cargo-make/default.nix

-      cp -R ${source} $out
-      chmod +w $out
-      cp ${./Cargo.lock} $out/Cargo.lock
-    '';


This was really clever . . . I've now replaced it with the cookie cutter implementation though :)

Since it was effectively spoofing the src attr as-if it had a Cargo.lock, it would have worked.

bhipple · 2020-02-16T10:14:44Z

pkgs/build-support/rust/default.nix

  '' + ''
    unset cargoDepsCopy
-  '' + (args.postUnpack or "");
+  '';


There are ~20 Rust applications that need this before they can use the newer fetcher. They're working just fine on the old fetcher for the moment, since they were never validating anything (the old validate flag defaulted to false and only a few packages had opted in).

Of course, it's debatable how valuable validation is when we are the ones providing the Cargo.lock in the first place, but this mainly serves to keep the branching/conditionals in the code limited.

bhipple · 2020-02-16T10:16:07Z

doc/languages-frameworks/rust.section.md

+file in git to ensure a reproducible build. However, a few packages do not, and
+Nix depends on this file, so if it missing you can use `cargoPatches` to apply
+it in the `patchPhase`. Consider sending a PR upstream with a note to the
+maintainer describing why it's important to include in the application.


Aside from helping us out by letting us delete custom patches, it's also in the best interests of users/maintainers to provide a reproducible binary build and explicitly called out in the Rust/Cargo guides, so I'd imagine most maintainers would be receptive to a PR on this one. Always worth a shot at least, particularly since cargoPatches already requires us to express it in patch form!

pkgs/build-support/rust/default.nix

ghost · 2020-02-17T16:20:39Z

Tested, fixes a problem with src = ./.; for me 👍

zimbatm

LGTM minus comment

bhipple · 2020-02-17T22:10:41Z

@GrahamcOfBorg build railcar rink cargo-make

pkgs/build-support/rust/default.nix

bhipple · 2020-02-21T22:40:23Z

Updated with a fix for sourceRoots that are more than 1 dir deep; other packages appear to still build just fine:

@GrahamcOfBorg build railcar rink cargo-make crosvm

pkgs/build-support/rust/default.nix

bhipple · 2020-02-23T21:22:05Z

@GrahamcOfBorg build crosvm railcar ripgrep

ghost · 2020-02-28T01:43:14Z

I would love to see this merged, are there any open concerns?

alyssais

LGTM once conflicts are fixed.

jonringer

I noticed a regression in the vector package

builder for '/nix/store/xc4fh32s5cwy3z0mf6ibd8kwcfccbx3l-vector-0.8.0.drv' failed with exit code 101; last 10 log lines:
    |
  4 | use snafu::{futures01::FutureExt, ResultExt};
    |                                   ^^^^^^^^^

  error: could not compile `vector`.

  Caused by:
    process didn't exit successfully: `rustc --edition=2018 --crate-name vector src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --emit=dep-info,link -C debuginfo=2 --test --cfg 'feature="disable-resolv-conf"' --cfg 'feature="jemallocator"' --cfg 'feature="leveldb"' --cfg 'feature="rdkafka"' -C metadata=00ec2b320bc02b7b -C extra-filename=-00ec2b320bc02b7b --out-dir /build/source/target/debug/deps -C linker=/nix/store/gxxi2f1i7vfi4mpn0gg6w03ijsn4vnqc-gcc-wrapper-9.2.0/bin/cc -C incremental=/build/source/target/debug/incremental -L dependency=/build/source/target/debug/deps --extern approx=/build/source/target/debug/deps/libapprox-e293ba9ee6a5075f.rlib --extern atty=/build/source/target/debug/deps/libatty-09e24eb4cb9f7ffd.rlib --extern base64=/build/source/target/debug/deps/libbase64-14efae3f1664713b.rlib --extern bytes=/build/source/target/debug/deps/libbytes-34eb0b7eff501add.rlib --extern bytesize=/build/source/target/debug/deps/libbytesize-c55e7622cc3bb2f7.rlib --extern chrono=/build/source/target/debug/deps/libchrono-3eee3bb73425a3df.rlib --extern codec=/build/source/target/debug/deps/libcodec-9b64a7eb56f2103c.rlib --extern colored=/build/source/target/debug/deps/libcolored-95c9a1afbc5632d6.rlib --extern criterion=/build/source/target/debug/deps/libcriterion-c692f4af58126557.rlib --extern db_key=/build/source/target/debug/deps/libdb_key-fe0ea33ce4b53cbb.rlib --extern derivative=/build/source/target/debug/deps/libderivative-9915bd352602dc62.so --extern derive_is_enum_variant=/build/source/target/debug/deps/libderive_is_enum_variant-31379608365948b1.so --extern dirs=/build/source/target/debug/deps/libdirs-ca7908c20b9cef50.rlib --extern elastic_responses=/build/source/target/debug/deps/libelastic_responses-4482f0d45979f0ef.rlib --extern evmap=/build/source/target/debug/deps/libevmap-ff35f921e2931b12.rlib --extern exitcode=/build/source/target/debug/deps/libexitcode-570db69bab1f4547.rlib --extern file_source=/build/source/target/debug/deps/libfile_source-a70b9a2454f89f68.rlib --extern flate2=/build/source/target/debug/deps/libflate2-212ac50e61dd7662.rlib --extern futures=/build/source/target/debug/deps/libfutures-6da1e5f1d651a2b9.rlib --extern futures03=/build/source/target/debug/deps/libfutures-e5ee77f6fe0c9a60.rlib --extern getset=/build/source/target/debug/deps/libgetset-93450ebd8c10b4b4.so --extern glob=/build/source/target/debug/deps/libglob-48e6309e8ceff1bc.rlib --extern goauth=/build/source/target/debug/deps/libgoauth-14765b927b66af86.rlib --extern grok=/build/source/target/debug/deps/libgrok-cf7b999c64a8d408.rlib --extern headers=/build/source/target/debug/deps/libheaders-ae9f5550f351344e.rlib --extern hostname=/build/source/target/debug/deps/libhostname-fd4f83d96ae1d48e.rlib --extern hotmic=/build/source/target/debug/deps/libhotmic-9f161deed9292ef3.rlib --extern http=/build/source/target/debug/deps/libhttp-2f1524dc4584d9a4.rlib --extern hyper=/build/source/target/debug/deps/libhyper-c56b26f286df6781.rlib --extern hyper_openssl=/build/source/target/debug/deps/libhyper_openssl-64480c2c491c2ad9.rlib --extern hyper_tls=/build/source/target/debug/deps/libhyper_tls-88d829cc83f0a3a6.rlib --extern indexmap=/build/source/target/debug/deps/libindexmap-5cbcc103d3d35b78.rlib --extern inventory=/build/source/target/debug/deps/libinventory-1824b349268b9c14.rlib --extern jemallocator=/build/source/target/debug/deps/libjemallocator-d9b1c47ba41ffa24.rlib --extern k8s_openapi=/build/source/target/debug/deps/libk8s_openapi-02686e266045eed5.rlib --extern kube=/build/source/target/debug/deps/libkube-87b143b9a54cd593.rlib --extern lazy_static=/build/source/target/debug/deps/liblazy_static-1bed6942fb4f0acf.rlib --extern leveldb=/build/source/target/debug/deps/libleveldb-f63fc2121c25ceff.rlib --extern libc=/build/source/target/debug/deps/liblibc-710086ccac216f22.rlib --extern libz_sys=/build/source/target/debug/deps/liblibz_sys-fc9b4f50bba1c4ea.rlib --extern listenfd=/build/source/target/debug/deps/liblistenfd-4fffc1c59b31bef7.rlib --extern logfmt=/build/source/target/debug/deps/liblogfmt-2fd46c22fe740b96.rlib --extern matches=/build/source/target/debug/deps/libmatches-4b6e1baaba28ddd9.rlib --extern maxminddb=/build/source/target/debug/deps/libmaxminddb-bb60060197692465.rlib --extern maxminddb=/build/source/target/debug/deps/libmaxminddb-bb60060197692465.so --extern native_tls=/build/source/target/debug/deps/libnative_tls-d0bf8a4f0d5c5c06.rlib --extern nix=/build/source/target/debug/deps/libnix-a70c5a5cefc111c4.rlib --extern nom=/build/source/target/debug/deps/libnom-bb94d8ff12eee5fa.rlib --extern notify=/build/source/target/debug/deps/libnotify-e1354aaf0b1c6ff7.rlib --extern num_cpus=/build/source/target/debug/deps/libnum_cpus-bb59305567d9083c.rlib --extern once_cell=/build/source/target/debug/deps/libonce_cell-08001497092a95df.rlib --extern openssl=/build/source/target/debug/deps/libopenssl-f49c303bd150a142.rlib --extern openssl_probe=/build/source/target/debug/deps/libopenssl_probe-af69212d422118dc.rlib --extern owning_ref=/build/source/target/debug/deps/libowning_ref-d3072eb69a31b59f.rlib --extern pretty_assertions=/build/source/target/debug/deps/libpretty_assertions-2bcab11e6b81ef8f.rlib --extern proptest=/build/source/target/debug/deps/libproptest-63e24fbe3424be5c.rlib --extern prost=/build/source/target/debug/deps/libprost-a0fbce34b394cac8.rlib --extern prost_derive=/build/source/target/debug/deps/libprost_derive-83ef40d366f40c7e.so --extern prost_types=/build/source/target/debug/deps/libprost_types-2fdb2f8db3fe53f7.rlib --extern rand=/build/source/target/debug/deps/librand-d4572f13a49729a8.rlib --extern rdkafka=/build/source/target/debug/deps/librdkafka-1e0f0d2c608c9bb2.rlib --extern regex=/build/source/target/debug/deps/libregex-63448cc751d319a7.rlib --extern reqwest=/build/source/target/debug/deps/libreqwest-79b216e3fe484ffa.rlib --extern rlua=/build/source/target/debug/deps/librlua-2070655a4738c73e.rlib --extern rusoto_cloudwatch=/build/source/target/debug/deps/librusoto_cloudwatch-3fc2136f39e2c960.rlib --extern rusoto_core=/build/source/target/debug/deps/librusoto_core-5050ce6afd126795.rlib --extern rusoto_credential=/build/source/target/debug/deps/librusoto_credential-da00a095e1abca7b.rlib --extern rusoto_firehose=/build/source/target/debug/deps/librusoto_firehose-c01dce28ff5e5509.rlib --extern rusoto_kinesis=/build/source/target/debug/deps/librusoto_kinesis-0dfedd4983ec98d9.rlib --extern rusoto_logs=/build/source/target/debug/deps/librusoto_logs-84b592d9e83af308.rlib --extern rusoto_s3=/build/source/target/debug/deps/librusoto_s3-4807ede6859f98b3.rlib --extern rusoto_sts=/build/source/target/debug/deps/librusoto_sts-01498d3f0dd82d85.rlib --extern seahash=/build/source/target/debug/deps/libseahash-d6245ac2f506d324.rlib --extern serde=/build/source/target/debug/deps/libserde-eba788f8c6826262.rlib --extern serde_json=/build/source/target/debug/deps/libserde_json-fa3dc00badcd4c87.rlib --extern serde_yaml=/build/source/target/debug/deps/libserde_yaml-0524c3dfa0fe9c24.rlib --extern shiplift=/build/source/target/debug/deps/libshiplift-d5accd30a7940ea7.rlib --extern smpl_jwt=/build/source/target/debug/deps/libsmpl_jwt-36af848f39e134c7.rlib --extern snafu=/build/source/target/debug/deps/libsnafu-9aee70e0d95811b0.rlib --extern stream_cancel=/build/source/target/debug/deps/libstream_cancel-704809aa16c2af5f.rlib --extern string_cache=/build/source/target/debug/deps/libstring_cache-1a053ebac340f3ab.rlib --extern strip_ansi_escapes=/build/source/target/debug/deps/libstrip_ansi_escapes-a901c20744b68908.rlib --extern structopt=/build/source/target/debug/deps/libstructopt-ebda4ba8c219d00b.rlib --extern syslog_loose=/build/source/target/debug/deps/libsyslog_loose-8b11b21def3d7e43.rlib --extern tempfile=/build/source/target/debug/deps/libtempfile-7df8d5ae81f06e65.rlib --extern tokio=/build/source/target/debug/deps/libtokio-6eac61df695512d0.rlib --extern tokio_retry=/build/source/target/debug/deps/libtokio_retry-40314c671b7b8b12.rlib --extern tokio_signal=/build/source/target/debug/deps/libtokio_signal-021c70d24cf32b8d.rlib --extern tokio_threadpool=/build/source/target/debug/deps/libtokio_threadpool-cdd27ce0afaf6160.rlib --extern tokio_tls=/build/source/target/debug/deps/libtokio_tls-5e39b7feaff1ebe5.rlib --extern tokio_uds=/build/source/target/debug/deps/libtokio_uds-556e572499febac0.rlib --extern tokio01_test=/build/source/target/debug/deps/libtokio01_test-be62e645a21b453f.rlib --extern toml=/build/source/target/debug/deps/libtoml-c10b33d0b30fa9dd.rlib --extern tower=/build/source/target/debug/deps/libtower-e1f93a35eed3f2a8.rlib --extern tower_hyper=/build/source/target/debug/deps/libtower_hyper-7efcf78985b813d0.rlib --extern tower_test=/build/source/target/debug/deps/libtower_test-0456c318aef4aeb4.rlib --extern tracing=/build/source/target/debug/deps/libtracing-cfcc1e7b733c370c.rlib --extern tracing_futures=/build/source/target/debug/deps/libtracing_futures-dd97e53947f6600c.rlib --extern tracing_limit=/build/source/target/debug/deps/libtracing_limit-de7441f6e5d0f8d8.rlib --extern tracing_log=/build/source/target/debug/deps/libtracing_log-82833101b1d99bc3.rlib --extern tracing_metrics=/build/source/target/debug/deps/libtracing_metrics-f9e297be27b621d6.rlib --extern tracing_subscriber=/build/source/target/debug/deps/libtracing_subscriber-bfba1a7ed6299e90.rlib --extern tracing_tower=/build/source/target/debug/deps/libtracing_tower-54d3933826ccf527.rlib --extern trust_dns=/build/source/target/debug/deps/libtrust_dns-bf40ab15c98d296d.rlib --extern trust_dns_proto=/build/source/target/debug/deps/libtrust_dns_proto-0ac17d5a836b8a31.rlib --extern trust_dns_resolver=/build/source/target/debug/deps/libtrust_dns_resolver-2d80fd5cbeec54d8.rlib --extern trust_dns_server=/build/source/target/debug/deps/libtrust_dns_server-f039b5790ddeac7b.rlib --extern typetag=/build/source/target/debug/deps/libtypetag-d9da2af541db0861.rlib --extern url=/build/source/target/debug/deps/liburl-816fa43d9dceefb9.rlib --extern uuid=/build/source/target/debug/deps/libuuid-7ba863a65afa6f07.rlib --extern walkdir=/build/source/target/debug/deps/libwalkdir-5c465c487411b778.rlib --extern warp=/build/source/target/debug/deps/libwarp-c488a131748f288d.rlib -L native=/build/source/target/debug/build/backtrace-sys-1dd4b70950dadd7e/out -L native=/nix/store/hpx4npwq2hmnkyhz74l58g0h7bv0drq5-openssl-1.1.1d/lib -L native=/build/source/target/debug/build/onig_sys-bdb78c865606ad9e/out -L native=/build/source/target/debug/build/jemalloc-sys-83c7e1965cfc46a6/out/build/lib -L native=/build/source/target/debug/build/leveldb-sys-09e266030824dc03/out -L native=/build/source/target/debug/build/libz-sys-92675ccfa7d86713/out/build -L native=/nix/store/vjwd74jcgm9q4kd3ax8zjsymyz655zc0-rdkafka-1.3.0/lib -L native=/build/source/target/debug/build/rlua-87b2f1fd39198e28/out -L native=/build/source/target/debug/build/libsqlite3-sys-0a250cc5f1df27a1/out` (signal: 9, SIGKILL: kill)

jonringer · 2020-02-28T16:04:11Z

other than that, LGTM. Built ~2000/2515 of the packages

…idation Previously, we would asssert that the lockfiles are consistent during the unpackPhase, but if the pkg has a patch for the lockfile itself then we must wait until the patchPhase is complete to check. This also removes an implicity dependency on the src attribute coming from `fetchzip` / `fetchFromGitHub`, which happens to name the source directory "source". Now we glob for it, so different fetchers will work consistently.

bhipple · 2020-02-29T01:22:23Z

Rebased w/ merge conflict fix, no other changes.

@GrahamcOfBorg build railcar ripgrep crosvm

jonringer · 2020-02-29T02:49:44Z

@GrahamcOfBorg build railcar ripgrep crosvm

jonringer · 2020-02-29T02:54:16Z

I feel like outstanding suggestions have been addressed, and this is ready to merged

bhipple · 2020-02-29T03:22:56Z

FWIW I just built vector successfully on this branch, so I suspect your detected regression on staging was either a build flake (OOM killer maybe?) or has been addressed by another merge to staging.

misuzu · 2020-03-14T21:27:08Z

This probably broke rdedup: https://hydra.nixos.org/build/113938557
I tried updating cargoSha256 but it fails with this weird error:

error: failed to run custom build command for `rust-lzma v0.2.1`

Caused by:
  process didn't exit successfully: `/build/source/target/release/build/rust-lzma-b6077171679ca2f8/build-script-build` (exit code: 101)
--- stderr
thread 'main' panicked at 'Could not find liblzma using pkg-config', /build/rdedup-3.1.1-vendor/rust-lzma/build.rs:10:3
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

warning: build failed, waiting for other jobs to finish...
error: build failed
builder for '/nix/store/30ryms2dk7yy9j66b67n74l89n15afg0-rdedup-3.1.1.drv' failed with exit code 101
error: build of '/nix/store/30ryms2dk7yy9j66b67n74l89n15afg0-rdedup-3.1.1.drv' failed

bhipple · 2020-03-14T21:50:48Z

Good catch; see #82616 for a description of the issue and fix!

bhipple requested a review from andir as a code owner February 16, 2020 10:07

bhipple commented Feb 16, 2020

View reviewed changes

bhipple requested review from zimbatm and jonringer February 16, 2020 10:17

ofborg bot added 6.topic: rust 8.has: documentation labels Feb 16, 2020

ofborg bot requested review from Br1ght0ne, sbourdeauducq and spacekookie February 16, 2020 10:39

ofborg bot added 10.rebuild-darwin: 501+ 10.rebuild-darwin: 501-1000 10.rebuild-linux: 501+ 10.rebuild-linux: 1001-2500 labels Feb 16, 2020

bhipple force-pushed the fix/fetchcargo-patch-lockfile-support branch from 2f4f62f to f180adc Compare February 16, 2020 23:31

bhipple commented Feb 16, 2020

View reviewed changes

pkgs/build-support/rust/default.nix Outdated Show resolved Hide resolved

bhipple requested a review from Ma27 February 17, 2020 16:02

bhipple mentioned this pull request Feb 17, 2020

fetchcargo: use flat tar.gz file for vendored src instead of recursive hash dir #78501

Merged

10 tasks

zimbatm reviewed Feb 17, 2020

View reviewed changes

bhipple force-pushed the fix/fetchcargo-patch-lockfile-support branch from f180adc to 5031d96 Compare February 17, 2020 22:09

FRidh added this to WIP in Staging via automation Feb 19, 2020

FRidh moved this from WIP to Needs review in Staging Feb 19, 2020

alyssais requested changes Feb 20, 2020

View reviewed changes

pkgs/build-support/rust/default.nix Outdated Show resolved Hide resolved

bhipple force-pushed the fix/fetchcargo-patch-lockfile-support branch from 5031d96 to c1e263b Compare February 21, 2020 22:39

ofborg bot requested a review from alyssais February 21, 2020 22:50

alyssais reviewed Feb 22, 2020

View reviewed changes

pkgs/build-support/rust/default.nix Outdated Show resolved Hide resolved

bhipple force-pushed the fix/fetchcargo-patch-lockfile-support branch from c1e263b to a0986bb Compare February 23, 2020 21:21

ofborg bot requested a review from alyssais February 23, 2020 21:34

B4dM4n mentioned this pull request Feb 25, 2020

buildRustPackage: do validateCargoDeps after patchPhase #81065

Closed

10 tasks

This was referenced Feb 25, 2020

Rust Application Migration to fetchCargoTarball Implementation #79975

Closed

fetchCargoTarball: backport to 20.03 #79981

Closed

alyssais approved these changes Feb 28, 2020

View reviewed changes

Staging automation moved this from Needs review to Ready Feb 28, 2020

alyssais added the 2.status: merge conflict label Feb 28, 2020

jonringer reviewed Feb 28, 2020

View reviewed changes

bhipple force-pushed the fix/fetchcargo-patch-lockfile-support branch from a0986bb to 89a630e Compare February 29, 2020 01:23

ofborg bot removed the 2.status: merge conflict label Feb 29, 2020

ofborg bot requested a review from alyssais February 29, 2020 01:57

jonringer merged commit ad30a30 into NixOS:staging Feb 29, 2020

Staging automation moved this from Ready to Done Feb 29, 2020

dnaq mentioned this pull request Mar 1, 2020

buildRustPackage: fix erroneous hard-coded source path #81203

Closed

10 tasks

bhipple mentioned this pull request Mar 6, 2020

staging -> staging-next #81863

Merged

drewrisinger mentioned this pull request Jul 21, 2020

python3Packages.retworkx: 0.3.4 -> 0.4.0 drewrisinger/nur-packages#13

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustPlatform.fetchCargo: handle custom Cargo.lock patchfiles with validation #80262

rustPlatform.fetchCargo: handle custom Cargo.lock patchfiles with validation #80262

bhipple commented Feb 16, 2020 •

edited

bhipple left a comment

bhipple Feb 16, 2020

bhipple Feb 16, 2020

bhipple Feb 16, 2020

bhipple Feb 16, 2020

bhipple Feb 16, 2020

ghost commented Feb 17, 2020

zimbatm left a comment

bhipple commented Feb 17, 2020

bhipple commented Feb 21, 2020

bhipple commented Feb 23, 2020

ghost commented Feb 28, 2020

alyssais left a comment

jonringer left a comment

jonringer commented Feb 28, 2020

bhipple commented Feb 29, 2020 •

edited

jonringer commented Feb 29, 2020

jonringer commented Feb 29, 2020

bhipple commented Feb 29, 2020 •

edited

misuzu commented Mar 14, 2020

bhipple commented Mar 14, 2020

rustPlatform.fetchCargo: handle custom Cargo.lock patchfiles with validation #80262

rustPlatform.fetchCargo: handle custom Cargo.lock patchfiles with validation #80262

Conversation

bhipple commented Feb 16, 2020 • edited

Motivation for this change

Things done

bhipple left a comment

Choose a reason for hiding this comment

bhipple Feb 16, 2020

Choose a reason for hiding this comment

bhipple Feb 16, 2020

Choose a reason for hiding this comment

bhipple Feb 16, 2020

Choose a reason for hiding this comment

bhipple Feb 16, 2020

Choose a reason for hiding this comment

bhipple Feb 16, 2020

Choose a reason for hiding this comment

ghost commented Feb 17, 2020

zimbatm left a comment

Choose a reason for hiding this comment

bhipple commented Feb 17, 2020

bhipple commented Feb 21, 2020

bhipple commented Feb 23, 2020

ghost commented Feb 28, 2020

alyssais left a comment

Choose a reason for hiding this comment

jonringer left a comment

Choose a reason for hiding this comment

jonringer commented Feb 28, 2020

bhipple commented Feb 29, 2020 • edited

jonringer commented Feb 29, 2020

jonringer commented Feb 29, 2020

bhipple commented Feb 29, 2020 • edited

misuzu commented Mar 14, 2020

bhipple commented Mar 14, 2020

bhipple commented Feb 16, 2020 •

edited

bhipple commented Feb 29, 2020 •

edited

bhipple commented Feb 29, 2020 •

edited