Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] staytus, staytus-module: init at 2017-11-08 #31475

Closed
wants to merge 4 commits into from

Conversation

ravloony
Copy link

@ravloony ravloony commented Nov 10, 2017

Motivation for this change

Add Staytus (staytus.co) package and service.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Fits CONTRIBUTING.md.

At the moment, this builds and sets up the database, but does not start the service. I have not yet managed to figure out all the environment stuff needed to get procodile, puma, bundler, rake and rails to work together happily. Any help would be much appreciated!

@ravloony
Copy link
Author

Currently, the systemd unit file that is generated looks ok. But when it runs, it says:

staytus-procodile-start[10962]: No processes to start.

procodile is supposed to start two jobs: web, and worker

Running the worker: /nix/store/vs983zxrjb8xwfdx4z2s6lrq6n18ll9c-staytus-rake/bin/staytus-jobs
works,

but running the web job: /nix/store/vs983zxrjb8xwfdx4z2s6lrq6n18ll9c-staytus-rake/bin/staytus-puma
Puma starting in single mode...

  • Version 3.9.1 (ruby 2.3.5-p376), codename: Private Caller
  • Min threads: 5, max threads: 5
  • Environment: production
    ERROR: No application configured, nothing to run

Still trying to figure this out, but not getting very far.

@ravloony
Copy link
Author

I finally have it working, with themes!

Config value services.staytus.stateDir allows the user to chose a state dir. This keeps the logs around, and also tracks database state.
There is also a run directory. The service empties the run directory, regenerates configs based on the config values, and sets up a symlink to the log directory in the state directory.

There are two systemd services that are started. staytusjobs runs the job worker, and staytus the puma web server. Nginx handles static file accesses and passes off to puma when necessary.

Setting a theme is a matter of setting
services.staytus.themes = { themename = ./path/to/theme/named/themename.zip };
services.staytus.theme = "themename";

@ravloony
Copy link
Author

Should I squash my commits into one?

# type = types.str;
# default = "postgresql";
# description = "Type of database";
#};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can your remove this?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

rm -rf /run/staytus
mkdir -p /run/staytus/pids

cp -Rv ${package}/share/staytus/* /run/staytus
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How much data is stored here? /run is a tmpfs with a limited size, /var/lib might be a better choice.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was not aware of that, will change.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved to /var/run/staytus

config.services.mysql.package
];
preStart = ''
export HOME=$TMP
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be dangerous if the application creates files in /tmp naively (without using mktemp or similar). Other users might trick this service into to different locations by creating symlinks. A dedicated home is recommend.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, makes sense. Maybe it is not needed anyway. I'll see if it can simply be removed.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

stateDir = mkOption {
type = types.str;
default = "/var/staytus";
description = "The state directory, logs and plugins are stored here";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an important use-case for making this configurable? If we advertise this as an API it is harder to change the location in future without breaking compatibility. What does upstream proposes here as an directory? I would use /var/lib/staytus otherwise.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that the users should be able to decide where their logs go. I'll set the default to /var/lib/staytus.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

themes = mkOption {
type = types.attrsOf types.path;
default = {};
description = "Set of themes";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you provide an example value here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

cd $out
unpackFile ${source}
'';
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are staytus themes always zip archives? The following api might be more general:

services.staytus.themes = [
  (fetchzip {
    url = "https://host/theme.zip";
    sha256 = "...";
  })
  (fetchgit {
    url = "https://host/theme.git"
    rev = "some-release";
    sha256 = "...";
  })
];

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be much better, yes. Could you share an example of how to handle that at the receiving end? I just use unpackFile at the moment.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Mic92 actually in this api, would it be possible to use local files as well as remote ones? Is there a local version of fetchzip, or is it compatible with relative paths?

Reason is, I am using this in nixops, and have my company theme in the same dir as the server nixops config, so I just use ./theme.zip.

type = types.str;
default = "";
description = "SMTP user password";
};
Copy link
Member

@Mic92 Mic92 Nov 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

smtpPassword and secretKeyBase will end world-readable up in the nix store. This might be not desired depending on the deployment. Until NixOS/nix#8 is solved we add the option to store secrets in a dedicated file, that is copied/loaded at runtime by the service. As the configuration is created at build time, substitution with sed can be used to replace a password place holder with a concrete value.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll look into it, thanks.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does one handle the password file with NixOps?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might be possible to use unix socket authentication to reset the password to something random every time the service is started.

So the staytus user would connect to mysql via unix socket (along the lines of this: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/databases/mysql.nix#L368), then add an identification using a password that is auto generated, which is then set in an env var, and picked up by rails.

On the face of it, it seems possible, so I'll have a closer look next week.

@Mic92
Copy link
Member

Mic92 commented Nov 14, 2017

@ravloony you can squash the last two commits into a nixos/staytus: add service commit.

@ravloony
Copy link
Author

@Mic92 Thanks for the input. I should have fixed the issues tomorrow.

@florianjacob
Copy link
Contributor

@ravloony regarding declarative mysql users & databases part: I'd consider to use databaseHost = "localhost"; instead of 127.0.0.1 / double-check if 127.0.0.1 really works. piwik uses 127.0.0.1 as default host as well, but socket authentication did not work properly with that, so I patched the default to be localhost. Apart from that I've nothing to add in terms of declarative mysql, looks good!

@ravloony ravloony force-pushed the staytus branch 2 times, most recently from 74173b8 to db9b284 Compare December 4, 2017 14:01
@FRidh
Copy link
Member

FRidh commented Jul 20, 2019

Closing due to inactivity.

@FRidh FRidh closed this Jul 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants