sssd: init at 1.14.2 #21150

outergod · 2016-12-14T16:39:15Z

Motivation for this change

This pull request is meant to supersede #14697 and fix #11407. I need sssd for work myself.

My additional changes include:

patching glibc to look for NSS modules in /run/nss-modules/$UNIQUE_ID_OF_GLIBC_, first
an nss-modules buildEnv package that includes all existing packages that used to refer to system.nssModules, plus sssd

I've ditched one of the original ideas to also support per-user NSS modules because I'm not sure how to meaningfully support them with the activation script for nss-modules. Additionally, it would require that numeric IDs don't change. Using the username would cause an infinite recursive loop when attempting to resolve the name, instead.

My system is still nixos-rebuild --switching for a full-fledged test, which will probably take really long.

Things done

Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- Linux
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

mention-bot · 2016-12-14T16:39:20Z

@e-user, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @rickynils and @abbradar to be potential reviewers.

abbradar · 2016-12-14T16:47:41Z

Is there any guarantees which glibc gives on ABI stability of NSS modules? If yes, we may want weaken path requirements (ideally it would be just /run/nss-modules/ if ABI hasn't changed and is guaranteed to be forward-compatible, but we can do e.g. /run/nss-modules/${glibc.version}/). This is to ease pain when having packages compiled against different glibc versions in the system (e.g. right after the switch, or from nix-env).

abbradar

Generic packaging nitpicks. I'm not knowledgeable much about NSS in general and SSSD specifically, but overall I seem to understand what's going on and it seems good.

abbradar · 2016-12-14T16:50:12Z

pkgs/os-specific/linux/sssd/default.nix

+  };
+
+  preConfigure = ''
+    cd "$NIX_BUILD_TOP/$name-$version"


This and further those should be unnecesary -- you don't change directory anywhere else.

To be honest, I didn't even look at @benwbooth's original submission properly. I'm using this opportunity to clean up all of it. Do you think it's a good idea to update to the latest SSSD before finalizing the submission? There seem to have been a couple of releases ever since.

yes, please update before.

abbradar · 2016-12-14T16:50:35Z

pkgs/tools/misc/ding-libs/default.nix

+  '';
+
+  buildPhase = ''
+    cd "$NIX_BUILD_TOP/$name-$version"


Same thing about cd.

abbradar · 2016-12-14T16:54:57Z

pkgs/tools/text/sgml/jade/default.nix

+  installPhase = ''
+    cd "$NIX_BUILD_TOP/$name-$version"
+    install -d -m755 "$out"/lib
+    make DESTDIR="$out" install


You can rather do:

makeFlags = [ "DESTDIR=$(out)" ]; preInstall = ''install -d -m755 "$out"/lib''; postInstall = ''mv "$out/bin/sx" "$out/bin/sgml2xml"'';

This way you'd reuse default installPhase which runs Make.

abbradar · 2016-12-14T17:00:59Z

It seems to me that ABI is stable: http://www.gnu.org/software/libc/manual/html_node/NSS-Basics.html . What do you think?

outergod · 2016-12-14T17:03:49Z

Do you suggest dumping X in /run/nss-modules/$X altogether or make it 2 for glibc's major version?

abbradar · 2016-12-14T17:20:25Z

Ideally the first, but I'm not sure at all about compatibility. I can't found any claims or guarantees. On the contrary, I found that nss-myhostname has an update for new glibc NSS interfaces. I think we may want to ask around in libc-help about whether there are any de jure/de facto (historical) guarantees. Can you ask them and cc me (e-mail is in my profile)?

Mic92 · 2016-12-15T00:07:51Z

This is also useful for other nss modules. Currently nss-myhostname is not working

outergod · 2016-12-15T08:58:09Z

Done, email sent.

edolstra · 2016-12-15T13:23:17Z

I'd prefer not to have /run/nss-modules. The way that NixOS supports NSS modules is via nscd. That way, NSS modules only need to be in the LD_LIBRARY_PATH of nscd. This also ensures that they work for 32-bit applications on 64-bit systems.

outergod · 2016-12-15T13:34:31Z

Hi @edolstra, may I point your attention to this bit from the RHEL documentation:

SSSD is not designed to be used with the NSCD daemon. Even though SSSD does not directly conflict 
with NSCD, using both services can result in unexpected behavior, especially with how long entries are 
cached.

nscd's caching actually breaks the way SSSD works, which should always have precedence and does its own caching, AFAIK. From what I can collect on the net, SSSD is also meant to replace nscd, eventually.

Mic92 · 2016-12-15T13:41:07Z

@e-user could SSSD make use of LD_LIBRARY_PATH?

outergod · 2016-12-15T13:51:32Z

I can try to build SSSD so that it is only used through nscd and LD_LIBRARY_PATH. Maybe switching off nscd's caching for everything but host name resolution does the trick.

Mic92 · 2016-12-15T13:57:56Z

@edolstra btw, the libnss modules of systemd are not used by nscd, because LD_LIBRARY_PATH points to the lib output, while the nss modules live in the systemd default out package.

Update fixed in #21175

outergod · 2016-12-15T17:03:02Z

This happens when attempting to start SSSD with nscd present:

sssd[26627]: NSCD socket was detected. NSCD caching capabilities may conflict with SSSD for users and groups. It is recommended not to run NSCD in parallel with SSSD, unless NSCD is configured not to cache the passwd, group, netgroup and services nsswitch maps.
sssd.service: Control process exited, code=exited status=4

Investigating.

abbradar · 2016-12-15T17:07:46Z

If nscd abstracts modules over a socket, that indeed seems to be a superior solution. If it can be configured to not conflict, avoiding all the hassle of ABI compatibility is awesome. Let's hope it works!

outergod · 2016-12-15T17:10:20Z

I agree. It seems like SSSD will stop complaining if nscd is configured not to cache the services it provides. That means nscd's config needs to change when SSSD is enabled. I'm working on it.

outergod · 2016-12-15T17:24:56Z

Great news: I set SSSD's debug_level to 9 and ran a getent passwd query with nscd up - and could see it was attempting to resolve the request.

outergod · 2016-12-16T09:19:59Z

I force-pushed a new branch that is cleansed from the previous mess and updates ding-libs to 0.6.0 and SSSD to the current stable release, 1.14.2. The most severe change is to how the NixOS nscd and SSSD modules are configured and interact, which will probably require a careful review.
What's still left to do is adding an option to supply SSSD with configuration, insert the default configuration without which SSSD will refuse to run and figure out how to enable its DBUS-specific bits. Also, there seems to be a PAM module whose purpose I'm not sure of, yet - but it seems to be required by the default configuration. I'm also not sure how and whether to tackle the Kerberos plugins SSSD provides.

abbradar

Commented on several things.

If the package has dbus .service files you can enable them by adding sssd to services.dbus.packages. But I don't think it needs to have them, daemons can just register themselves in D-Bus when they start (should work out of the box).

Also let's just add PAM modules when SSSD is enabled. Not sure what to do about Kerberos... I don't think we have any module system for it right now (disclaimer: I know barely a thing about Kerberos).

abbradar · 2016-12-16T10:09:48Z

nixos/modules/services/misc/sssd.nix

@@ -28,5 +33,10 @@ in {
    };

    system.nssModules = optional cfg.enable pkgs.sssd;
+    services.nscd = {
+      enable = mkForce true;


I think we shouldn't use mkForce; rather, add an assertion that it's enabled. It is anyway by default but this way if user disables it for some reason they'd know of problems explicitly and can choose what to do.

Good point, I didn't know that.

abbradar · 2016-12-16T10:11:18Z

pkgs/os-specific/linux/sssd/default.nix

-      --with-nscd=${glibc.bin}/sbin/nscd
+      ${lib.optionalString (nscdConfig != null) ''
+        --with-nscd=${glibc.bin}/sbin/nscd
+        --with-nscd-conf=${nscdConfig}


Ugh, why does it need nscd config? We can symlink it into /etc to make it accessible. Otherwise people need to rebuild SSSD each time they change config, which is not nice.

EDIT: but maybe we can just point it to non-existent config? I'm not sure why would it need one.

Ha, I was exactly asking myself what the proper way to do this would be - so I let nscd create a symlink for its current configuration file in /etc and just point SSSD at its absolute path..?

I can elaborate on the SSSD-nscd specific bit. SSSD queries whether nscd is running (--with-nscd) and will additionally inspect its active configuration file (--with-nscd-conf). If it's not running, it won't do anything. If it is running and the configuration file doesn't disable passwd, group, netgroup and services caching, it will print a warning to syslog because it essentially breaks SSSD.

In case you're interested about the details, it's defined in src/monitor.c:

/* Warn if nscd seems to be running */ ret = check_file(NSCD_SOCKET_PATH, -1, -1, S_IFSOCK, S_IFMT, NULL, false); if (ret == EOK) { ret = sss_nscd_parse_conf(NSCD_CONF_PATH); switch (ret) { case ENOENT: sss_log(SSS_LOG_NOTICE, "NSCD socket was detected. NSCD caching capabilities " "may conflict with SSSD for users and groups. It is " "recommended not to run NSCD in parallel with SSSD, " "unless NSCD is configured not to cache the passwd, " "group, netgroup and services nsswitch maps."); break; case EEXIST: sss_log(SSS_LOG_NOTICE, "NSCD socket was detected and seems to be configured " "to cache some of the databases controlled by " "SSSD [passwd,group,netgroup,services]. It is " "recommended not to run NSCD in parallel with SSSD, " "unless NSCD is configured not to cache these."); break; case EOK: DEBUG(SSSDBG_TRACE_FUNC, "NSCD socket was detected and it " "seems to be configured not to interfere with " "SSSD's caching capabilities\n");

abbradar · 2016-12-16T10:35:01Z

Thank you for elaborated response! I see now; well, let's then indeed link it to `/etc`. You can do it with `environment.etc`; let's then point nscd there too, just for consistency. Don't forget `restartTriggers` so that systemd services will be restarted when configuration file changes.

…

On December 16, 2016 1:25:47 PM GMT+03:00, Alexander Kahl ***@***.***> wrote: e-user commented on this pull request. > @@ -39,7 +41,11 @@ stdenv.mkDerivation { --without-semanage --with-xml-catalog-path=''${SGML_CATALOG_FILES%%:*} --with-ldb-lib-dir=$out/modules/ldb - --with-nscd=${glibc.bin}/sbin/nscd + ${lib.optionalString (nscdConfig != null) '' + --with-nscd=${glibc.bin}/sbin/nscd + --with-nscd-conf=${nscdConfig} Ha, I was exactly asking myself what the _proper_ way to do this would be - so I let nscd create a symlink for its current configuration file in `/etc` and just point SSSD at its absolute path..? I can elaborate on the SSSD-nscd specific bit. SSSD queries whether nscd is running (`--with-nscd`) and will additionally inspect its active configuration file (`--with-nscd-conf`). If it's not running, it won't do anything. If it is running and the configuration file doesn't disable passwd, group, netgroup and services caching, it will print a warning to syslog because it essentially breaks SSSD. In case you're interested about the details, it's defined in `src/monitor.c`: ```c /* Warn if nscd seems to be running */ ret = check_file(NSCD_SOCKET_PATH, -1, -1, S_IFSOCK, S_IFMT, NULL, false); if (ret == EOK) { ret = sss_nscd_parse_conf(NSCD_CONF_PATH); switch (ret) { case ENOENT: sss_log(SSS_LOG_NOTICE, "NSCD socket was detected. NSCD caching capabilities " "may conflict with SSSD for users and groups. It is " "recommended not to run NSCD in parallel with SSSD, " "unless NSCD is configured not to cache the passwd, " "group, netgroup and services nsswitch maps."); break; case EEXIST: sss_log(SSS_LOG_NOTICE, "NSCD socket was detected and seems to be configured " "to cache some of the databases controlled by " "SSSD [fpasswd,group,netgroup,services]. It is " "recommended not to run NSCD in parallel with SSSD, " "unless NSCD is configured not to cache these."); break; case EOK: DEBUG(SSSDBG_TRACE_FUNC, "NSCD socket was detected and it " "seems to be configured not to interfere with " "SSSD's caching capabilities\n"); ``` -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #21150

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

cruegge · 2016-12-16T10:41:27Z

Concerning PAM: I think the pam_sss module replaces pam_krb5 and pam_ccred, so all krb5-stuff in pam.nix should probably be conditional on config.krb5.enable && !config.services.sssd.enable. I'm not sure whether enabling sssd should automatically enable krb5, though, i.e. is krb5Full required for sssd?

abbradar

New wave of comments (sorry for being persistent, I think we are getting close!).

Questions about PAM and KRB are out of my league, but I think we can rely on you there given that you actually use it ~_^.

abbradar · 2016-12-16T11:33:20Z

nixos/modules/services/misc/sssd.nix

@@ -16,7 +17,7 @@ in {
    };

    nixpkgs.config.packageOverrides = pkgs: {
-      sssd = pkgs.sssd.override { nscdConfig = nscd.confFile; };
+      sssd = pkgs.sssd.override { nscdConfig = nscdConfig; };


This will keep the problem because source points to the changing path. Instead I'd drop nscdConfig argument and just hardcode it to /etc/nscd.conf in the package.. This way you'd also get rid of an override.

abbradar · 2016-12-16T11:34:07Z

nixos/modules/security/pam.nix

@@ -447,6 +455,7 @@ in
      # Include the PAM modules in the system path mostly for the manpages.
      [ pkgs.pam ]
      ++ optional config.users.ldap.enable pam_ldap
+      ++ optionals config.services.sssd.enable [ pkgs.sssd ]


Nitpick: use optional.

abbradar · 2016-12-16T11:40:33Z

nixos/modules/services/system/nscd.nix


        serviceConfig =
-          { ExecStart = "@${pkgs.glibc.bin}/sbin/nscd nscd -f ${cfg.confFile}";
+          { ExecStart = "@${pkgs.glibc.bin}/sbin/nscd nscd -f ${confFile}";


We can now just omit -f because /etc/nscd.conf is the default path.

abbradar · 2016-12-16T11:41:00Z

nixos/modules/services/system/nscd.nix

@@ -83,7 +83,7 @@ in
        # its pid. So wait until it's ready.
        postStart =
          ''
-            while ! ${pkgs.glibc.bin}/sbin/nscd -g -f ${cfg.confFile} > /dev/null; do
+            while ! ${pkgs.glibc.bin}/sbin/nscd -g -f ${confFile} > /dev/null; do


Same thing about -f.

abbradar

More comments for the whole patch -- sorry, I should have done it in one go.

abbradar · 2016-12-16T11:46:32Z

pkgs/os-specific/linux/sssd/default.nix

+  ];
+
+  installPhase = ''
+    make install \


Why call make by hand? Instead I'd move all those flags to makeFlags and use postInstall -- you can use $out there as $(out).

abbradar · 2016-12-16T11:48:06Z

pkgs/tools/text/sgml/jade/default.nix

+  enableParallelBuilding = true;
+  buildInputs = [ gcc gzip gnum4 ];
+
+  preConfigure = ''


Instead: NIX_CFLAGS_COMPILE = [ "-fpermissive" "-Wno-deprecated" ]; -- let's not use Bash when we can rely on Nix. BTW -fpermissive allows pretty scary things like calling undefined functions IIRC -- why is it needed?

Good question, I adopted it unchanged from the original submitter. ;)

Oh, only now I have noticed that not all changes are yours! Still, are you okay with fixing those?

outergod · 2016-12-16T17:22:17Z

I took over some of your additions, @dezgeg. However, for some weird reason, security.pam.makeHomeDir = true; doesn't actually add the string to the PAM configuration files for me, on two different NixOS installations. Could you please help me find out what's wrong?

dezgeg · 2016-12-16T17:26:22Z

I added this:

security.pam.services.su.makeHomeDir = true;
security.pam.services.sshd.makeHomeDir = true;

I don't know what's the way to change it for all modules, though it would be better obviously.

Mic92 · 2016-12-17T15:25:35Z

@e-user one option I see is to disable nscd, when sssd is enabled. Would that work?

outergod · 2016-12-17T17:57:18Z

@Mic92 unfortunately and quite paradoxically that will break SSSD. @edolstra doesn't like the original approach to load NSS modules from a fixed location to circumvent the circular sssd->glibc->sssd dependency problem, so we're essentially abusing nscd as a no-cache module loader. I'm not 100% happy with it, but it does actually work.

outergod · 2016-12-18T20:03:27Z

I'll rebase once more and squash the commits. @dezgeg, would you like to become co-maintainer of SSSD and its new dependencies?

abbradar · 2016-12-18T20:35:19Z

Given that our current way of supporting NSS modules can't get full SSSD support (nscd only caches particular types of databases, so you don't get for example automount of shared corporate storages if I understand correctly) -- @edolstra, what do you think of moving to /run/nss-modules{,-32}? On one hand, this gives us full support and ABI risks are minimal according to reply in libc-help. On the other hand, the libraries can fail to load because of address space conflicts as evidenced in another reply and crash the binary they are loaded into (but crashing the daemon is arguably not much better) if written badly.

I personally don't have a strong opinion (arguments on both sides are strong and understandable) but we may want to review that once more in light of new problems.

outergod · 2016-12-20T10:31:09Z

I'm not sure whether waiting for @edolstra is worth it. I have to keep rebasing my changes and fixing conflicts while others and myself are waiting for this feature, so it might be easier to merge the changes and let me file a bug right afterwards that explains how supporting external NSS modules through nscd is a hack and lists things that don't work correctly.

abbradar · 2016-12-20T12:14:41Z

Indeed, moving to the new way of handling NSS is of a bigger scope than this PR. I don't have time to run tests for this (even with SSSD disabled, just to make sure nothing is broken) but may have more time at the weekend. If someone beats me to it I don't mind ~_^.

outergod · 2016-12-27T10:02:48Z

Hey @abbradar, any news?

abbradar · 2016-12-27T10:06:57Z

Thanks for the reminder, life caught on. I'll look at this today. -- Sent from my Android device with K-9 Mail. Please excuse my brevity.

abbradar · 2016-12-27T23:12:46Z

Looks like jade fails to build now:

/nix/store/p9g0vm6bj88sdknpjw9gwziyh8hjw71g-binutils-2.27/bin/ld: cannot find -lsp
/nix/store/p9g0vm6bj88sdknpjw9gwziyh8hjw71g-binutils-2.27/bin/ld: cannot find -lspgrove

This is atop of current unstable channel. Any ideas?

dezgeg · 2016-12-28T13:46:22Z

Still on the /run/nss-modules vs nscd discussion: It's not just the libnss ABI stability that needs consideration but also any dependent shared libraries of the NSS module itself. E.g. libnss_hostname.so has this:

	libcap.so.2 => /nix/store/73zh7aw558bdahk27j4l825wiz46p1g8-libcap-2.25-lib/lib/libcap.so.2 (0x00007f6fe145d000)

I don't know if it applies to __libc_dlopen used by glibc when loading the NSS module, but at least in some other cases (#16779) impurely loaded shared library dependencies have been incompatible with a different version of the corresponding library causing really weird failure modes.

Maybe it's worth asking the glibc folks for a better way of doing this non-caching passthru with nscd (and support for automount maps). I'd imagine other distros would find this use case useful as well.

abbradar · 2016-12-28T13:48:28Z

@dezgeg You are correct, dependent libraries do taint address spaces, as was also mentioned in the mailing list discussion. That's the main disadvantage of using /run/nss-modules IMO -- certainly NSCD would be better if it can learn to handle all the databases.

outergod · 2016-12-30T12:00:35Z

Using my own machine with its extensive system package collection is too time-consuming for master build cycles over and over again, so I'll start using some AWS EC2 power on Monday.

@dezgeg @abbradar how are you guys' C programming skills? It might be worth looking into extending nscd. The objective would be to a) add the missing services and b) an additional passthru option for each of the services.

abbradar · 2016-12-30T12:08:51Z

@e-user You can build on the top of latest unstable channel, this should decrease the build times drastically. Testing it on unstable channel is good enough, I believe.

I feel confident to write C but not confident enough to write complex and secure C (given that nscd runs as root and is used to authenticate users!). However it depends on how complex the code to add this to nscd would be. I imagine that if it's written nicely a simple addition of cases/ifs with duplicated code from other tested paths would suffice, but my previous encounter with code in glibc (namely ld.so) wasn't so nice D:

outergod · 2017-01-01T13:54:34Z

@abbradar I just built jade successfully atop 9d8a179, which is latest in channels/nixos-unstable at the time of writing. No changes made. I just pushed the rebased version.

I'll take a look at nscd. Let's see whether its maintainers would want such a change in to begin with.

perlPackages.TextWrapI18N: init at 0.06 perlPackages.Po4a: init at 0.47 jade: init at 1.2.1 ding-libs: init at 0.6.0 Switch nscd to no-caching mode if SSSD is enabled.

outergod · 2017-01-02T13:47:52Z

Good news regarding sudoers and automount in nsswitch.conf: AFAIK from their respective documentation and glibc's source code, glibc doesn't implement queries for those itself but the respective services read nsswitch.conf autonomously and query NSS modules themselves. This also means that LD_LIBRARY_PATH have to be extended for them, correspondingly.

@abbradar, would you like me to also add this for this pull request or create a subsequent ticket that reports the missing behavior as a bug? Also, is there anything else preventing a merge right now? Would you like me to provide documentation? I've never provided NixOS documentation before, are there any pointers how this works?

abbradar · 2017-01-02T13:52:34Z

It still fails for me -- do you have nix.useSandbox = true; in your NixOS configuration?

Apart from that I think nothing holds this from being merged -- you can leave improvements for subsequent PRs.

outergod · 2017-01-03T20:58:05Z

Nope, sorry, still works with sandboxing on. Could you please try in a nixos-container or VM?

abbradar · 2017-01-04T00:07:34Z

After testing on other machines I found the reason -- jade Makefiles are not parallel building safe. If it's disabled it works for me. For reference: I have 8 build cores.

I went ahead and merged it with this change. Thank you!

outergod · 2017-01-04T07:22:57Z

Thank you @abbradar for your patience and persistence!

perlPackages.TextWrapI18N: init at 0.06 perlPackages.Po4a: init at 0.47 jade: init at 1.2.1 ding-libs: init at 0.6.0 Switch nscd to no-caching mode if SSSD is enabled. abbradar: disable jade parallel building. Closes NixOS#21150

abbradar requested changes Dec 14, 2016

View reviewed changes

Mic92 mentioned this pull request Dec 15, 2016

sssd: init at 1.13.3 #14697

Closed

7 tasks

vcunat added the 1.severity: mass-rebuild label Dec 15, 2016

outergod force-pushed the pkgs/sssd-1.13.3 branch from 8b420d0 to 2b8101a Compare December 16, 2016 09:03

outergod changed the title ~~sssd: init at 1.13.3~~ sssd: init at 1.14.2 Dec 16, 2016

abbradar requested changes Dec 16, 2016

View reviewed changes

abbradar approved these changes Dec 16, 2016

View reviewed changes

abbradar added 6.topic: nixos 8.has: module (new) labels Dec 16, 2016

outergod force-pushed the pkgs/sssd-1.13.3 branch from c5609b7 to 10c8036 Compare December 18, 2016 20:38

outergod force-pushed the pkgs/sssd-1.13.3 branch from 10c8036 to 7b38b1a Compare December 30, 2016 12:04

outergod force-pushed the pkgs/sssd-1.13.3 branch from 7b38b1a to 9589ff1 Compare January 1, 2017 13:52

sssd: init at 1.14.2

c438b81

perlPackages.TextWrapI18N: init at 0.06 perlPackages.Po4a: init at 0.47 jade: init at 1.2.1 ding-libs: init at 0.6.0 Switch nscd to no-caching mode if SSSD is enabled.

outergod force-pushed the pkgs/sssd-1.13.3 branch from 9589ff1 to c438b81 Compare January 2, 2017 08:09

abbradar closed this in 61d125b Jan 4, 2017

arianvp mentioned this pull request Sep 22, 2021

Provide NSS modules globally, make nscd unnecessary #138178

Closed

12 tasks

bbenne10 mentioned this pull request Mar 2, 2023

Support sssd to enable nix-channel --update working DeterminateSystems/nix-installer#294

Closed

sssd: init at 1.14.2 #21150

sssd: init at 1.14.2 #21150

Conversation

outergod commented Dec 14, 2016 • edited

Motivation for this change

Things done

mention-bot commented Dec 14, 2016

abbradar commented Dec 14, 2016

abbradar left a comment

Choose a reason for hiding this comment

abbradar Dec 14, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abbradar commented Dec 14, 2016

outergod commented Dec 14, 2016

abbradar commented Dec 14, 2016 • edited

Mic92 commented Dec 15, 2016 • edited

outergod commented Dec 15, 2016

edolstra commented Dec 15, 2016

outergod commented Dec 15, 2016 • edited

Mic92 commented Dec 15, 2016

outergod commented Dec 15, 2016

Mic92 commented Dec 15, 2016 • edited

outergod commented Dec 15, 2016 • edited

abbradar commented Dec 15, 2016

outergod commented Dec 15, 2016

outergod commented Dec 15, 2016

outergod commented Dec 16, 2016

abbradar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abbradar Dec 16, 2016 • edited

Choose a reason for hiding this comment

outergod Dec 16, 2016 • edited

Choose a reason for hiding this comment

abbradar commented Dec 16, 2016 via email

cruegge commented Dec 16, 2016 • edited

abbradar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abbradar left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

outergod commented Dec 16, 2016

dezgeg commented Dec 16, 2016

Mic92 commented Dec 17, 2016 • edited

outergod commented Dec 17, 2016

outergod commented Dec 18, 2016 • edited

abbradar commented Dec 18, 2016

outergod commented Dec 20, 2016

abbradar commented Dec 20, 2016

outergod commented Dec 27, 2016

abbradar commented Dec 27, 2016 via email

abbradar commented Dec 27, 2016

dezgeg commented Dec 28, 2016

abbradar commented Dec 28, 2016

outergod commented Dec 30, 2016

abbradar commented Dec 30, 2016

outergod commented Jan 1, 2017

outergod commented Jan 2, 2017

abbradar commented Jan 2, 2017

outergod commented Jan 3, 2017

abbradar commented Jan 4, 2017

outergod commented Jan 4, 2017

outergod commented Dec 14, 2016 •

edited

abbradar Dec 14, 2016 •

edited

abbradar commented Dec 14, 2016 •

edited

Mic92 commented Dec 15, 2016 •

edited

outergod commented Dec 15, 2016 •

edited

Mic92 commented Dec 15, 2016 •

edited

outergod commented Dec 15, 2016 •

edited

abbradar Dec 16, 2016 •

edited

outergod Dec 16, 2016 •

edited

cruegge commented Dec 16, 2016 •

edited

abbradar left a comment •

edited

Mic92 commented Dec 17, 2016 •

edited

outergod commented Dec 18, 2016 •

edited