Add flake.nix #327

edolstra · 2020-02-25T14:02:51Z

This adds a flake.nix to the repo, ensuring a reproducible build of the homepage. Previously the homepage used a Makefile that was downloading stuff from the Internet and calling nix-build, so we couldn't build the homepage inside a Nix build.

It also adds a NixOS container configuration, so you run

$ nixos-container create homepage --flake nixos-homepage
$ nixos-container start homepage

to get a running version of the homepage.

Some dynamically fetched files (e.g. blogs.xml) are now part of the repository. We probably should add a script that updates these files and commits them.

For people who don't have a flake-enabled version of Nix, there is a default.nix and shell.nix that uses https://github.com/edolstra/flake-compat.

CC @garbas

You can get a NixOS homepage in a container by running $ nixos-container create homepage --flake nixos-homepage

Flake input changes: * Added 'hydra': 'github:NixOS/hydra/be8eb9d00d6a3fcac520bc7dfd5740c0b3ade746' * Added 'hydra/nix': 'github:NixOS/nix/4ad5826a182f8eeb4cbc61e27d2128b08050b0d4' * Added 'hydra/nix/nixpkgs': 'github:edolstra/nixpkgs/43e283790fbe330b723ac44b584824cf6d85c1ee' * Added 'hydra/nixpkgs': 'github:edolstra/nixpkgs/d959778e0934c4c93896d509a65de798909c7cdd' * Added 'nix': 'github:NixOS/nix/4a4521f46253e87a06af9333381c050e983b3c2f' * Added 'nix/nixpkgs': 'github:edolstra/nixpkgs/43e283790fbe330b723ac44b584824cf6d85c1ee' * Added 'nixpkgsStable': 'github:edolstra/nixpkgs/581caef4f5d58573c96ab6e9dbf1160209372029' * Added 'nixpkgsUnstable': 'github:NixOS/nixpkgs/82cbc6af3cbb77a07e1725554119e0fb032d7cf6'

garbas · 2020-02-25T16:08:41Z

nixos/amis.nix

@@ -0,0 +1,295 @@
+let self = {


couldn't we also get this file from <nixpkgs/nixos/modules/virtualisation/ec2-amis.nix> or ${pkgs.path}/nixos/modules/virtualisation/ec2-amis.nix?

samueldr · 2020-02-25T18:19:10Z

nixos-release.tt

@@ -0,0 +1 @@
+[%- latestNixOSSeries = "19.03"; latestNixOSRelease = "19.03.172883.1601f559e89" -%]


Isn't it currently 19.09? And how does this get updated?

I'm split between manually updating this and getting this from nixpkgs.
Having this generated would of course mean less work for us, just rerunning the build every day.

Having to manually update this value it would allow us organize releases in public better.

Example of this would be: Friday master receives a bump of a version, we have some time to finish release announcements and publish them on (eg.) Monday. I hope that we would already do this for next release (20.9)

samueldr · 2020-02-25T18:23:20Z

Makefile


 # Cute hack, this allows future expansion if desired
 # Mainly, this allows tracking NIXOS_SERIES
 nixpkgs/packages-channels.json: Makefile
-	echo '["nixos-$(NIXOS_SERIES)", "nixos-unstable", "nixpkgs-unstable"]' > $@
+	echo '["nixos-$(NIXOS_SERIES)", "nixpkgs-unstable"]' > $@


Lately, we have had consistently fresher nixos-unstable, compared to nixpkgs-unstable. Should we prefer the channel that updates more regularly?

samueldr · 2020-02-25T18:48:25Z

blogs.xml

This is something that we control. Since we need to make a static and reproducible page, we don't have much options.

Update the data in the repo
Remove the section from the site
Augment the site at run-time

Update the data in the repo

The first option is what is already proposed. Though I feel, and I guess others would tend to agree, that this is a dirty solution. The main advantage here is that it would continue providing the same level of service from the website, that is, the homepage shows the blogs listing even without JavaScript and AJAX.

Remove the section from the site

Removing the whole section from the homepage is an option, though I feel it is a net negative. Replacing with a simple link to the planet's homepage reduces eyeballs on the contents of the community blogs.

Augment the site at run-time

The last option is meeting halfway. The blogs.xml pre-baked rendering is removed. We leave a space with a proper JavaScript-less fallback inviting the user to visit the planet site. On load, the RSS feed from the planet is loaded via AJAX to fill in the contents.

This does mean that we lose the contents listing for users that load the page without JavaScript, but we also remove the need for the (albeit not that much) added complexity to keep blogs.xml updated in the repo. Since the homepage twitter feed already follows that convention, I feel this compromise is sensible.

Though, a real drawback here, this solution does require a bit of work to implement. Luckily, this is simple enough to implement, and does not require a complex setup like the packages listing, this can be done through trivial hand-written old-school enough JavaScript.

samueldr · 2020-02-25T19:07:51Z

nixpkgs-commit*.json

Those are a bit harder than blogs.xml. Two main reasons:

The information is more volatile.
We do not have a local data source to mine.

Really, the fact that we get so much more updates daily, even hourly, makes the option of updating the repo a less palatable one.

Off the top of my head, not much better solutions here. I looked at it quickly for the mobile-nixos site, but had no great solution. (In addition to it being less useful there.)

I think all solutions other than removing the section from the homepage revolve around the GitHub API, and JS augmentation.

GitHub API

https://api.github.com/repos/NixOS/nixpkgs/commits?sha=master

Browser console toy

fetch("https://api.github.com/repos/NixOS/nixpkgs/commits?sha=master").then((response) => response.json()).then(console.log)

This would allow us to get the commits for the master branch, without any infrastructure change. (Dropping ?sha=master would show all new commits, regardless of branch.)

A hefty drawback here is that we're transmitting data to a third-party, and I don't know what kind of issues this could cause.

Another issue I can see, but have not researched, is that this may be eating in that user's unauthenticated requests to the GitHub API, and may fail if the user made unauthenticated queries!

Mirroring the data

This could be solved by having the data mirrored at some location that is not a third-party. The mirror could be refreshed every minute, or even five or ten minutes. It already was refreshing on a timer!

Though this does mean some kind of infrastructure or setup has to handle this. The website-part for fetching and using the data is trivial and a non-issue. The other part is not. I figure a "nixos-homepage-data" repository with the data could be used, with the appropriate cached github raw data file, and some automation.

Side-note: This solution could also be used to mirror the twitter feed, rather than embed a third-party widget.

garbas · 2020-02-25T21:02:16Z

Hi @samueldr,

I'll try to quickly answer few concerns.

blogs.xml

The easier will be to configure to run travis on nightly basis and pull new blogs each time. I will configure it in next PR which will be about pushing to netlify. I think if we would put all this changes in one PR it will be very hard to review.

nixpkgs-commit*.json

This section should be removed from front-page altogether. It doesn't bring any value. Those who know where to look for commits they will look on GitHub.

#327 (comment)

edolstra · 2020-02-26T12:37:19Z

Couple of changes:

I've removed nixpkgs-commit*.json.
AMIs / Azure blobs are now obtained from Nixpkgs directly.
I added an ./update.sh script that runs nix flake update, and updates blogs.xml and nixos-release.tt.

We can get rid of nixos-release.tt once 20.03 is released, because we can get that information directly from the Nixpkgs flake. (Nixpkgs 19.09 does not contain a flake.nix so we can't use it here yet. The 19.09 flake is not up to date and doesn't have the same commit hashes.)

For blogs.xml, I agree it would be nicest to fetch it at runtime in the browser.

garbas

Looks good to me now.

In travis I will just build nix with flake support for now. We might consider adding it to nixpkgs (nixWithFlakes) for the time being so that devs can play with it more easily.

garbas · 2020-02-26T16:12:49Z

Ignore my previous comment. I wasn't aware of nixFlakes.

I'll slowly get up to speed. :)

nixos-discourse · 2020-03-18T15:38:16Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-unstable-disappear/6285/3

edolstra added 19 commits June 18, 2019 17:51

Add flake.nix

a133994

Include the Nix/Nixpkgs/NixOS/NixOps/Hydra manuals

fea5f4e

Generate packages / options

67095a0

Move nixpkgs JSON generation to a separate derivation

330b323

Set commit

00f9740

Merge remote-tracking branch 'origin/master' into flake

3034c92

Update flake.nix

762285d

Update lock file

32c8a74

Add a NixOS container config for testing

506001a

You can get a NixOS homepage in a container by running $ nixos-container create homepage --flake nixos-homepage

Merge remote-tracking branch 'origin/master' into flake

f18a25b

Serve packages.json.gz properly

f55ce7b

Update flake.nix

ec06c04

Merge remote-tracking branch 'origin/master' into flake

07816e4

uri -> url

65595af

Update flake.lock

f6c62ae

Merge remote-tracking branch 'origin/master' into flake

731e4c2

epoch -> edition

5bf073f

Add compatibility default.nix / shell.nix

615f338

garbas self-requested a review February 25, 2020 14:41

garbas reviewed Feb 25, 2020

View reviewed changes

samueldr suggested changes Feb 25, 2020

View reviewed changes

edolstra added 5 commits February 26, 2020 11:39

Fix building a container from a dirty tree

625b99a

Remove Nixpkgs commit stats

d7901b8

#327 (comment)

Get NixOS AMIs from nixpkgs

9ef8b15

Get NixOS Azure blobs from nixpkgs

e46b906

Add update.sh script

586d394

garbas approved these changes Feb 26, 2020

View reviewed changes

samueldr mentioned this pull request Feb 26, 2020

Fetch RSS client-side for the planet #328

Open

samueldr approved these changes Feb 26, 2020

View reviewed changes

garbas merged commit 00ab08e into master Mar 5, 2020

zimbatm deleted the flake branch March 6, 2020 13:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add flake.nix #327

Add flake.nix #327

edolstra commented Feb 25, 2020

garbas Feb 25, 2020

samueldr Feb 25, 2020

garbas Feb 25, 2020 •

edited

samueldr Feb 25, 2020

samueldr commented Feb 25, 2020 •

edited

samueldr commented Feb 25, 2020

garbas commented Feb 25, 2020

edolstra commented Feb 26, 2020

garbas left a comment

garbas commented Feb 26, 2020

nixos-discourse commented Mar 18, 2020

		@@ -0,0 +1 @@
		[%- latestNixOSSeries = "19.03"; latestNixOSRelease = "19.03.172883.1601f559e89" -%]

Add flake.nix #327

Add flake.nix #327

Conversation

edolstra commented Feb 25, 2020

garbas Feb 25, 2020

Choose a reason for hiding this comment

samueldr Feb 25, 2020

Choose a reason for hiding this comment

garbas Feb 25, 2020 • edited

Choose a reason for hiding this comment

samueldr Feb 25, 2020

Choose a reason for hiding this comment

samueldr commented Feb 25, 2020 • edited

blogs.xml

Update the data in the repo

Remove the section from the site

Augment the site at run-time

samueldr commented Feb 25, 2020

nixpkgs-commit*.json

GitHub API

Mirroring the data

garbas commented Feb 25, 2020

edolstra commented Feb 26, 2020

garbas left a comment

Choose a reason for hiding this comment

garbas commented Feb 26, 2020

nixos-discourse commented Mar 18, 2020

garbas Feb 25, 2020 •

edited

samueldr commented Feb 25, 2020 •

edited