ipfs: 0.4.23 -> 0.5.0 #86242

lordcirth · 2020-04-28T18:44:17Z

Motivation for this change

Major go-ipfs update to 0.5.0

Things done

Perhaps some commits should be squashed. I don't know why I the preload.sh script failed, some weird path problem. Seems to work fine without it.

sorki · 2020-04-29T14:35:23Z

Please rebase and squash the commits into one with commit message according to guidelines https://github.com/NixOS/nixpkgs/blob/master/.github/CONTRIBUTING.md#submitting-changes

Feel free to ping me if you need help.

lordcirth · 2020-04-29T17:04:10Z

Ok, I think this is correct?

vojta001 · 2020-04-29T21:18:32Z

According to the release notes on GitHub, the ipfs repo fsck command no longer exists as it is no longer necessary. We should remove it from nixos/modules/services/network-filesystems/ipfs.nix.

nixpkgs/nixos/modules/services/network-filesystems/ipfs.nix

Line 41 in bb5e20e

    
                 ipfs repo fsck # workaround for BUG #4212 (https://github.com/ipfs/go-ipfs/issues/4214)

nixos/modules/services/network-filesystems/ipfs.nix

NixOS#86242 (comment)

pkgs/applications/networking/ipfs/default.nix

Mic92 · 2020-05-08T09:49:55Z

@GrahamcOfBorg test ipfs

mweinelt · 2020-05-08T12:02:06Z

Since this upgrade my local IPFS instance tells me it requires repo migrations and we don't seem to handle that yet.

● ipfs-init.service - IPFS Initializer
     Loaded: loaded (/nix/store/z6bg2vgg8q4qp3j72bw3ykjscmygcpd0-unit-ipfs-init.service/ipfs-init.service; linked; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2020-05-08 13:51:18 CEST; 7min ago
    Process: 19520 ExecStart=/nix/store/hgw4ll6kfpw8cwdxs0flcyd22mars268-unit-script-ipfs-init-start (code=exited, status=1/FAILURE)
   Main PID: 19520 (code=exited, status=1/FAILURE)
         IP: 0B in, 0B out
        CPU: 15ms

May 08 13:51:18 gaia systemd[1]: Starting IPFS Initializer...
May 08 13:51:18 gaia hgw4ll6kfpw8cwdxs0flcyd22mars268-unit-script-ipfs-init-start[19521]: Error: ipfs repo needs migration
May 08 13:51:18 gaia systemd[1]: ipfs-init.service: Main process exited, code=exited, status=1/FAILURE
May 08 13:51:18 gaia systemd[1]: ipfs-init.service: Failed with result 'exit-code'.
May 08 13:51:18 gaia systemd[1]: Failed to start IPFS Initializer.
May 08 13:51:19 gaia systemd[1]: ipfs-init.service: Start request repeated too quickly.
May 08 13:51:19 gaia systemd[1]: ipfs-init.service: Failed with result 'exit-code'.
May 08 13:51:19 gaia systemd[1]: Failed to start IPFS Initializer.

The 0.5.0 release note say the following:

Repo Migrations

IPFS uses repo migrations to make structural changes to the "repo" (the config, data storage, etc.) on upgrade.

This release includes two very simple repo migrations: a config migration to ensure that the config contains working bootstrap nodes and a keystore migration to base32 encode all key filenames.

In general, migrations should not require significant manual intervention. However, you should be aware of migrations and plan for them.

    If you update go-ipfs with ipfs update, ipfs update will run the migration for you. Note: ipfs update will refuse to run the migrations while ipfs itself is running.
    If you start the ipfs daemon with ipfs daemon --migrate, ipfs will migrate your repo for you on start.

Otherwise, if you want more control over the repo migration process, you can manually install and run the repo migration tool.

Also it seems somewhat tricky:

$ ipfs daemon --migrate
Initializing daemon...
go-ipfs version: 0.5.0
Repo version: 9
System version: amd64/linux
Golang version: go1.14.2
Found outdated fs-repo, migrations need to be run.
  => Looking for suitable fs-repo-migrations binary.
  => None found, downloading.
The migrations of fs-repo failed:
  no fs-repo-migration binary found for version 9: failed to check migrations version: fork/exec /tmp/go-ipfs-migrate435794195/fs-repo-migrations: no such file or directory
If you think this is a bug, please file an issue and include this whole log output.
  https://github.com/ipfs/fs-repo-migrations

Error: no fs-repo-migration binary found for version 9: failed to check migrations version: fork/exec /tmp/go-ipfs-migrate435794195/fs-repo-migrations: no such file or directory

The ipfs-migrator package is also not quite intuitive:

$ ipfs-update 
NAME:
   ipfs-update - update ipfs

USAGE:
   ipfs-update [global options] command [command options] [arguments...]
   
VERSION:
   0.1.0
   
COMMANDS:
   versions	print out all available versions
   version	print out currently installed version
   install	install a version of ipfs
   stash	stashes copy of currently installed ipfs binary
   revert	revert to previously installed version of ipfs
   fetch	fetch a given (default: latest) version of ipfs
   help, h	Shows a list of commands or help for one command
   
GLOBAL OPTIONS:
   --verbose		print verbose output
   --help, -h		show help
   --version, -v	print the version

How did y'all solve that?

priegger · 2020-05-08T12:14:08Z

What I did so far:

Bump ipfs-migrator to the latest released version (1.5.1): priegger@ec70d7c
Install it for the ipfs user on the system:

users.users."${config.services.ipfs.user}".packages = let
    rev = "ec70d7cbc3c58c8db8bc45a810968b90a5c4ccc7";
    sha256 = "1qfbjzbdz2idiin8z34gdmac3w65r2641fzgh69q068417smcd1n";
    nixpkgs = import (
      pkgs.fetchzip {
        url = "https://github.com/priegger/nixpkgs/archive/${rev}.tar.gz";
        inherit sha256;
      }) {};
  in
  [ nixpkgs.ipfs-migrator ];

Migrate my repo (in the default path) using

sudo -u ipfs env IPFS_PATH=/var/lib/ipfs /nix/store/b1h2fch7s0xc2mh93ks9zi0rx728zx3y-ipfs-migrator-1.5.1/bin/fs-repo-migrations

I can create a PR but I'm not sure if further steps (like documentation, ipfs service adjustments, genaming of the package to fs-repo-migrations) should be done, too.

Mic92 · 2020-05-08T13:07:31Z

To be honest I have used ipfs not before and just made sure our test was running. Should we revert this until the upgrade path becomes clear?

priegger · 2020-05-08T13:31:02Z

I created #87265 which updates ipfs-migrator to the latest version.

mweinelt · 2020-05-08T22:37:27Z

That did the trick. Thanks!

Mic92 · 2020-05-09T06:10:35Z

Do you think we can add this migrator to our NixOS module somehow? Or should we mention this in the release notes?

priegger · 2020-05-11T07:44:44Z

I don't have any background in IPFS or IPFS on NixOS, I just played around with it a bit in the last weeks.

From what I got from the nixpkgs git log and issues and the IPFS Migration Tool README:

auto migrate was disabled in the IPFS Service in NixOS about 3 years ago.
There seems to have version updated in the NixOS IPFS package during that 3 years. I did not find anything about that in the release notes.

Also there's a ticket that the IPFS test seems to be broken (#74000) which might be nice to have for this kind of feature.

Mic92 · 2020-05-11T07:47:35Z

I have fixed the tests in this pull requests actually.

priegger · 2020-05-11T07:52:52Z

Ah, I missed that. Thanks!

Dietr1ch · 2020-05-13T06:11:28Z

I think I might be in a state close to @Mic92

I just started playing with IPFS, and after my last update it stopped starting saying that I need to migrate.

I just tried to do it manually (running as ipfs), but got some errors when doing it.

It seems that the version I have installed right now should be able to do the migration, thanks to #87265 I guess. The -to <target_version> flag defaults to 9, which is the highest version it supports (It doesn't even support --version).

However, when it runs it fails to find a repo,

sudo su ipfs -s (which bash) -c (which fs-repo-migrations)
ipfs migration:  stat /var/lib/ipfs/.ipfs: no such file or directory

From the --help it doesn't seem I can even point it to somewhere else and there's no manual for the tool either.

According to /var/lib/ipfs/version I'm on version 7, but upgrading to 8 didn't worked either.
I seem to be missing the repo,

sudo ls -lsa /var/lib/ipfs/
total 64
4 drwxr-xr-x   5 ipfs ipfs  4096 May 12 23:02 .
4 drwxr-xr-x  25 root root  4096 May  9 15:15 ..
24 drwxr-xr-x 936 ipfs ipfs 20480 May 12 22:32 blocks
8 -rw-rw----   1 ipfs ipfs  5403 May 12 22:32 config
8 -rw-------   1 ipfs ipfs  5405 May 12 22:32 config-pre-local-discovery-440372136
4 drwxr-xr-x   2 ipfs ipfs  4096 May 12 22:32 datastore
4 -rw-------   1 ipfs ipfs   190 May  9 15:15 datastore_spec
4 drwx------   2 ipfs ipfs  4096 May  9 15:15 keystore
4 -rw-r--r--   1 ipfs ipfs     2 May  9 15:15 version

UPDATE: After adding ipfs-migrator to environment.systemPackages I was able to upgrade my (non-existing?) repo with,

sudo su ipfs -s (which bash) -c "ipfs daemon"

I just killed the daemon after is succeeded migrating and restarted ipfs-initand ipfs, which started normally.

sudo systemctl restart ipfs-init
sudo systemctl restart ipfs

priegger · 2020-05-13T06:39:49Z

You can also override the IPFS_PATH for fs-repo-migrations, see #86242 (comment).

Dietr1ch · 2020-05-13T06:48:12Z

Oh, thanks for the tip. However I think that /var/lib/ipfs was already the default as that's where I saw that I was on v7 and after the upgrade I got to v9.

I think that the issue is that by running the ipfs service we should just have the ipfs-migrator installed just in case we need an upgrade.

priegger · 2020-05-13T07:10:19Z

I think " ${HOME}/.ipfs" is the default for IPFS_PATH, but the NixOS Module uses IPFS_PATH=HOME. So in your error message, the additional ".ipfs" was the problem.

But yes, making ipfs-migrator available might be the right thing to do as long as we make sure, we don't destroy other people's fs repos by auto-upgrading, make auto-upgrade non-default or at least allow people to opt-out. That's the tricky part, I think.

fabianhjr · 2020-10-16T19:29:54Z

Hi @priegger, thanks for the clear instructions on migrating IPFS. Encountered a similar issue when upgrading from NixOS 20.03 to NixOS 20.09beta recently in which the daemon wasn't starting due to the same error.

lordcirth · 2020-10-17T17:59:17Z

I think " ${HOME}/.ipfs" is the default for IPFS_PATH, but the NixOS Module uses IPFS_PATH=HOME. So in your error message, the additional ".ipfs" was the problem.

But yes, making ipfs-migrator available might be the right thing to do as long as we make sure, we don't destroy other people's fs repos by auto-upgrading, make auto-upgrade non-default or at least allow people to opt-out. That's the tricky part, I think.

I created a symlink from /var/lib/ipfs/.ipfs to /var/lib/ipfs to fix the migration. I tried setting IPFS_PATH when calling fs-repo-migration but it didn't work, so I filed a bug: ipfs/fs-repo-migrations#110

lordcirth force-pushed the ipfs05 branch from af8555a to 8106ffb Compare April 29, 2020 16:46

ofborg bot added the 2.status: merge conflict label Apr 29, 2020

ipfs: 0.4.23 -> 0.5.0

445d554

lordcirth force-pushed the ipfs05 branch from 8106ffb to 445d554 Compare April 29, 2020 17:03

ofborg bot removed the 2.status: merge conflict label Apr 29, 2020

ofborg bot requested review from kalbasit and fpletz April 29, 2020 17:10

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels Apr 29, 2020

vojta001 mentioned this pull request Apr 29, 2020

ipfs: workaround for upstream bug; other small fixes #29133

Merged

8 tasks

ofborg bot added 6.topic: nixos 8.has: module (update) labels Apr 30, 2020

This comment has been minimized.

Sign in to view

vojta001 reviewed May 2, 2020

View reviewed changes

nixos/modules/services/network-filesystems/ipfs.nix Outdated Show resolved Hide resolved

HugoReeves reviewed May 5, 2020

View reviewed changes

nixos/modules/services/network-filesystems/ipfs.nix Show resolved Hide resolved

ipfs: remove ipfs repo fsck

4968990

NixOS#86242 (comment)

lordcirth force-pushed the ipfs05 branch from 65fbbcf to 4968990 Compare May 5, 2020 22:19

vojta001 approved these changes May 7, 2020

View reviewed changes

Mic92 changed the title ~~Ipfs05~~ ipfs: 0.4.23 -> 0.5.0 May 7, 2020

Mic92 reviewed May 7, 2020

View reviewed changes

pkgs/applications/networking/ipfs/default.nix Outdated Show resolved Hide resolved

Mic92 added 4 commits May 7, 2020 15:26

ipfs: patch shebang in builder script

e89ceff

ipfs: reference tests

3e6e4ce

nixos/ipfs: convert tests to python driver & simplify

8b5707b

ipfs: reference correct test

11ace01

Mic92 merged commit ddef887 into NixOS:master May 8, 2020

Dietr1ch mentioned this pull request May 13, 2020

IPFS,Fix: Require the ipfs-migrator package for handling upgrades #87723

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ipfs: 0.4.23 -> 0.5.0 #86242

ipfs: 0.4.23 -> 0.5.0 #86242

lordcirth commented Apr 28, 2020 •

edited

sorki commented Apr 29, 2020

lordcirth commented Apr 29, 2020

vojta001 commented Apr 29, 2020

This comment has been minimized.

Mic92 commented May 8, 2020

mweinelt commented May 8, 2020

priegger commented May 8, 2020

Mic92 commented May 8, 2020

priegger commented May 8, 2020

mweinelt commented May 8, 2020

Mic92 commented May 9, 2020

priegger commented May 11, 2020

Mic92 commented May 11, 2020

priegger commented May 11, 2020

Dietr1ch commented May 13, 2020 •

edited

priegger commented May 13, 2020

Dietr1ch commented May 13, 2020

priegger commented May 13, 2020

fabianhjr commented Oct 16, 2020

lordcirth commented Oct 17, 2020

ipfs: 0.4.23 -> 0.5.0 #86242

ipfs: 0.4.23 -> 0.5.0 #86242

Conversation

lordcirth commented Apr 28, 2020 • edited

Motivation for this change

Things done

sorki commented Apr 29, 2020

lordcirth commented Apr 29, 2020

vojta001 commented Apr 29, 2020

This comment has been minimized.

Mic92 commented May 8, 2020

mweinelt commented May 8, 2020

priegger commented May 8, 2020

Mic92 commented May 8, 2020

priegger commented May 8, 2020

mweinelt commented May 8, 2020

Mic92 commented May 9, 2020

priegger commented May 11, 2020

Mic92 commented May 11, 2020

priegger commented May 11, 2020

Dietr1ch commented May 13, 2020 • edited

priegger commented May 13, 2020

Dietr1ch commented May 13, 2020

priegger commented May 13, 2020

fabianhjr commented Oct 16, 2020

lordcirth commented Oct 17, 2020

lordcirth commented Apr 28, 2020 •

edited

Dietr1ch commented May 13, 2020 •

edited