Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trustful IPFS Store #3727

Draft
wants to merge 163 commits into
base: master
Choose a base branch
from

Conversation

Ericson2314
Copy link
Member

@Ericson2314 Ericson2314 commented Jun 21, 2020

This is a rewritten https://github.com/NixIPFS/nix/tree/ipfs_binary_cache_legacy (thanks for pioneering!) that

Needless to say, this is a WIP PR so the the store format is subject to change, so if you try this, and then git pull, then find you can't make use of the data you previous put in IPFS, don't be sad.

mguentner and others added 30 commits January 28, 2017 04:37
either /ipfs or /ipns URIs can be used to address a
store.

To change the default values (127.0.0.1, 5001) for the
local API `host` and `port` are used in the query such
as
/ipfs/QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn?host=192.168.42.1&port=1111

If a gateway should be used instead of a local daemon
`use_gateway` is set to 1.
If the another gateway should be used, `gateway` can
be changed.

In combination:

/ipns/cache.example.com?gateway=https://gw.example.com
Build fails as:

$ make
  CXX    src/libmain/stack.o
src/libmain/stack.cc: In function 'void nix::sigsegvHandler(int, siginfo_t*, void*)':
src/libmain/stack.cc:21:21: error: 'ucontext' was not declared in this scope
     sp = (char *) ((ucontext *) ctx)->uc_mcontext.gregs[REG_RSP];
                     ^~~~~~~~
src/libmain/stack.cc:21:21: note: suggested alternative: 'ucontext_t'
     sp = (char *) ((ucontext *) ctx)->uc_mcontext.gregs[REG_RSP];
                     ^~~~~~~~
                     ucontext_t

It's caused by upstream rename:
https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=251287734e89a52da3db682a8241eb6bccc050c9

which basically changes
    typedef struct ucontext {} ucontext_t;
to
    typedef struct ucontext_t {} ucontext_t;

The change uses ucontext_t.

Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
This adds a post bool that can be used to POST content instead of GET
or PUT it.
need to use a multipart form to correctly insert the data. this
mirrors how curl -Ffile=asdf works.
This makes a number of changes:

- Use ipfs:// and ipns:// for URI, this is a proper URI and now
documented by ipfs
- Remove ipfs.hh, move everything into store
- Store daemonUri, which is the, replacing individual API settings
- Remove support for gateways
- Remove nar info cache - ipfs should do all of the caching for us
- Only retry getFile once - if it doesn’t work the ipfs daemon is
probably down
- Use /object/stat to check if file exists instead of cat
To check if IPFS daemon works, we need to do a test that should alway
work. In the future, we may want to have a min IPFS daemon that we
support, but for now any version will work.
This is okay for /ipns/ because we can always mutate it, but it’s a
hard error for ipfs.
This is really slow, but is useful to test things. It does a few API
calls to work:

- /api/v0/add : to add the new file
- /api/v0/object/patch/add-link : to insert the file into unixfs
- /api/v0/name/publish : to publish the new ipfs object to ipns

The last step is the slow part since we need to wait for it to clear.
unfortunately we don’t have a good way to handle this

for now, avoid rethrowing and just accept 500
this doesn’t wait for the final response, but updates ipns internally.
To avoid race conditions while doing /object/patch/add-link &
/name/publish, we need to lock some state. This adds a bool for
inProgressUpsert so that we can know when other threads are also
trying to publish to IPNS.
this is called at the end of a sequence of addToStores calls to tell
the store that it can now commit the data.
Hopefully we can instead pass arguments like Hydra does, soon.

(cherry picked from commit 4ced63c)
This uses the Sync primitive to store an ipfsPath that is written to
as we add stuff to IPNS. IPNS is still optional. When finished, we do
a publish on the ipnsPath.
It isn't very safe, so best to do it once right after the condition. If
only we had real pattern matching...
…load-sync

Update ipfs binary cache with upload support (sync version)
…load

Update ipfs binary cache with upload support
@Ericson2314 Ericson2314 changed the title WIP: IPFS Store --- contains #3714 WIP: IPFS Store Oct 12, 2020
@Ericson2314 Ericson2314 changed the title WIP: IPFS Store WIP: Trustful IPFS Store Oct 12, 2020
@bqv
Copy link
Contributor

bqv commented Dec 13, 2020

Hello. Has this stalled? I see no new commits on ipfs-develop and these PRs have still not been merged

@Ericson2314
@edolstra

@Ericson2314
Copy link
Member Author

@bqv We finished everything we set out to implement. The larger works had parts that @edolstra was wary of. I suppose I could look into getting this bit merged, I hadn't focused on this as much I think the trustless variant (and our original plan) is much interesting/useful.

@bqv
Copy link
Contributor

bqv commented Dec 13, 2020

That's rather a shame, I was excited for the hype on this to continue.. Is there a place where the final news was announced? I missed it, I think

@Ericson2314
Copy link
Member Author

Sorry we haven't done a final blog or anything like that. Currently working though the details of content-addressed derivations is taking up the majority of my bandwidth with @edolstra. But if January has gone by and there is no further review of IPFS Nix PRs, do make a fuss --- I shouldn't have any excuses by then :).

@Ericson2314
Copy link
Member Author

Ericson2314 commented Dec 14, 2020

@bqv the other thing is that this PR in particular could be turned into a plugin, as the JSON code and whatnot is easy to move from where it is today.

While the IPFS store could be plugin in all cases, the other branches require more meaningful extensions to the rest of the Nix in the new ways to content-address things.

So in some sense this PR is only worth merging if we know the others will be, and I suppose I hadn't yet turned it into a a plugin as that felt like prematurely throwing in the towel, but maybe that is a bad call on my part.

@bqv
Copy link
Contributor

bqv commented Dec 14, 2020

That might be cool. I wasn't singling out this PR though, don't worry, I just don't know where the bulk of the work is and whats been merged

@stale
Copy link

stale bot commented Jun 12, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the stale label Jun 12, 2021
@asymmetric
Copy link
Contributor

But if January has gone by and there is no further review of IPFS Nix PRs, do make a fuss --- I shouldn't have any excuses by then :).

@Ericson2314 here I am, making a fuss :)

What is going on in the IPFS+Nix world?

@stale stale bot removed the stale label Jul 24, 2021
@Ericson2314
Copy link
Member Author

@stale
Copy link

stale bot commented Apr 16, 2022

I marked this as stale due to inactivity. → More info

@stale stale bot added the stale label Apr 16, 2022
@Ericson2314
Copy link
Member Author

Probably best to keep this open until RFC.

@stale stale bot removed the stale label Apr 19, 2022
@stale stale bot added the stale label Oct 29, 2022
@Ericson2314 Ericson2314 changed the title WIP: Trustful IPFS Store Trustful IPFS Store Feb 22, 2023
@Ericson2314 Ericson2314 marked this pull request as draft February 22, 2023 16:05
@stale stale bot removed the stale label Feb 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants