WIP: separate private and public blocks #1538

whyrusleeping · 2015-07-29T15:18:56Z

Use a different datastore prefix to keep private and public blocks separate. Abstract this upwards into a private blockstore and a private dagservice.

License: MIT
Signed-off-by: Jeromy jeromyj@gmail.com

jbenet · 2015-07-29T15:44:29Z

08:19 <•whyrusleeping> jbenet: thats the gist of it
08:19 <•whyrusleeping> doing this also makes me *really* want to work on unifying daemon construction
08:24 <•jbenet> yeah, to be totally blunt... do not want. it's going to be a mess knowing where to put what, and carrying 2x of each everywhere. let alone that this is just binary, it has no way of growing towards the middle gray areas of serving specific things to specific authed groups. i think coloring (marking things that are allowed or denied) is a much simpler
08:24 <•jbenet> approach. it can be made fast in the exact same way (you can use a datastore to get O(1) access to the perms on any given object)
08:25 → pfraze joined (~pfraze@2605:6000:e900:d200:a8eb:d04e:3359:967e)
08:25 <•whyrusleeping> thats going to suck...
08:28 <•jbenet> .... because?
08:29 <•whyrusleeping> then we have to change all the interfaces to support putting permissions with each block stored
08:29 <•whyrusleeping> the possibility of making a mistake goes up much more
08:29 <•whyrusleeping> and things become a lot slower because now you have 2x the disk hits and latency for each block
08:29 <•jbenet> no, not at all, why? permissions are only relevant on a small set of cases: when an object is externalized
08:29 <•whyrusleeping> every write is now two writes
08:30 <•whyrusleeping> every read is now two reads
08:30 → niran joined (sid3808@gateway/web/irccloud.com/x-oexerhbkbrozumye)
08:30 <•jbenet> "every write is now two writes" not at all. writing perms and writing objects is correlated once.
08:31 <•jbenet> caps are small, can likely be kept in memory (or lots of it)
08:32 <•jbenet> "the possibility of making a mistake goes up much more" this applies just as much to jugging two dagservs/blockstores/.... all over the place. these are different approaches with different things to watch out for. 
08:33 <•jbenet> i think instead of being tied to specific solutions, we should be picking based on the problems we need to solve, which include authentication/capability sets.
08:34 <•jbenet> in fact, can just use caps for everything, store sensitive blocks encrypted, whoever presents the right cap can get something, whoever has the right decrypt cap can decrypt.
08:35  → voxelot joined  
08:35 <•jbenet> zero bookkeeping, as the cap itself used to retrieve the object is the "permission".
08:44 <•jbenet> anyway, let's keep discussing on gh. this is not pressing for 0.3.6

whyrusleeping · 2015-07-29T20:21:35Z

So would you want something like this?

dagserv.Add(node, flags.Private)

or what would the interface for what youre thinking look like?

whyrusleeping · 2015-08-04T18:01:08Z

I'm thinking that the actual put call by the blockstore can write a small header before the blocks, including a permissions/capabilities info, as well as a timestamp. Then consumers of the blockstore can get a handle to the blockstore with a given capabilities set, and can only read blocks that match their caps, and all blocks they write would be written with their caps, querying that blockstore handle would only return blocks with matching caps, etc.

The timestamp allows us to ignore recently added blocks during a GC, which is going to be very important to avoid randomly breaking user operations with a rouge GC. The downside here is that each subsequent write of a given block would have to write the entire block again to update the timestamp (since the datastore interface doesnt support partial writes).

whyrusleeping · 2015-08-12T20:12:03Z

@jbenet re: our discussion yesterday, you wanna CR this?

jbenet · 2015-08-13T04:09:40Z

core/core.go

+	Blocks     *bserv.BlockService // the block service, get/add blocks.
+
+	DAG       merkledag.DAGService // the merkle dag service, get/add objects.
+	PrivDAG   merkledag.DAGService // the private merkledag service for local objects


maybe:

NetworkDAG NodeStateDAG

DataDAG StateDAG

jbenet · 2015-08-13T04:14:45Z

I'm not certain that this works for pinning + gc. can you outline the logic flow of how a GC operation would work now, given the pinset and the objects are in two different dagservices with two different GCBlockstores?

whyrusleeping · 2015-08-13T15:03:42Z

We're just going to have to run two separate GC operations, one on the normal blockstore, and one on the private blockstore.

whyrusleeping · 2015-08-13T15:58:35Z

@jbenet i'm trying to determine the best places to take the private blockstores GC locks now.

whyrusleeping · 2015-08-13T23:49:05Z

@jbenet I think the simplest thing to do would be to always lock both blockstores. at which point we might as well just have one lock. This is tricky.

jbenet · 2015-08-23T00:00:30Z

@jbenet I think the simplest thing to do would be to always lock both blockstores. at which point we might as well just have one lock. This is tricky.

Yeah, this is the sort of trickiness i worried about.

jbenet · 2015-08-23T00:02:11Z

blocks/blockstore/blockstore.go

@@ -21,6 +21,8 @@ var log = eventlog.Logger("blockstore")
 // BlockPrefix namespaces blockstore datastores
 var BlockPrefix = ds.NewKey("blocks")

+var PrivateBlockPrefix = ds.NewKey("private")


let's rename these:

StateBlocks -- the state of the node DataBlocks -- the data of the node

(fine to keep DataBlocks at /blocks to avoid a migration there

jbenet · 2015-08-23T00:22:51Z

@whyrusleeping confirm this is how this is supposed to work:

the pinset is stored in the StateDAG, not in the DataDAG.
the pin leaves point to objects either in the StateDAG or in the DataDAG.
the GC function is called for both DAGs
the GC function calls ColoredSet(pinSet, dagToGC) -- https://github.com/ipfs/go-ipfs/blob/private-blocks/pin/gc/gc.go#L35
ColoredSet calls Descendants(dag, keyset, roots) -- https://github.com/ipfs/go-ipfs/blob/private-blocks/pin/gc/gc.go#L97
Descendants adds all the descendants of given roots present in the given dag (i.e. if not present, nothing happens?)
thus, the ColoredSet for either DAG consists only of nodes from that DAG which have roots in the pinset (which may overlap).
things not in the ColoredSet are removed, as usual.

Seems to work. Need to test the hell out of this.

need one test that:

generates a bunch of dags,
puts them in either dagset randomly,
pins some nodes recursively at random (without discriminating where they are)
calls GarbageCollect() (GC both)
ensures the right objects are still in the right DAGs.

(this will likely need to be in Go)

jbenet · 2015-08-23T00:24:03Z

Ok, i think this will work, but

needs confirmation above
needs srs tests
needs rename (state and data)

whyrusleeping · 2015-09-09T04:42:44Z

@jbenet i think this is ready for another look over (when you have time)

This used to lead to large refcount numbers, causing Flush to create a lot of IPFS objects, and merkledag to consume tens of gigabytes of RAM. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

OS X sed is documented as "-i SUFFIX", GNU sed as "-iSUFFIX". The one consistent case seems to be "-iSUFFIX", where suffix cannot empty (or OS X will parse the next argument as the suffix). This used to leave around files named `refsout=` on Linux, and was just confusing. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

Without this, all entries will have nlink==0, which confuses a bunch of tools. Most dramatically, systemd-nspawn enters a busy loop in its lock utility function. License: MIT Signed-off-by: Tommi Virtanen <tv@eagain.net>

This used to cause files e.g. being edited with `vi` to become 0-size. License: MIT Signed-off-by: Tommi Virtanen <tv@eagain.net>

Callers assume this is safe to call whenever, let's make it so. License: MIT Signed-off-by: Tommi Virtanen <tv@eagain.net>

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> run GC over internal blockstore as well License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> rename blockstores to Data and State License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> fix up garbage collection and add a test License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com> separate flatfs License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

whyrusleeping · 2015-09-16T19:21:02Z

@jbenet CR when you can.

jbenet · 2015-09-16T20:31:41Z

at first pass it looks fine. i'll do a deeper look later today.

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

jbenet · 2015-09-18T06:27:49Z

core/corerepo/gc_test.go

+	u "github.com/ipfs/go-ipfs/util"
+)
+
+func TestBasicSeparation(t *testing.T) {


for the sort of guarantees we expect from this separation, we'll probably want some sharness tests too.

theres not really any way of knowing any of the pinners internal hashes through the API, but we can check 'ipfs refs local' to make sure that the only entries there are actual data. sound alright?

some blocks should be internal only, like for example keys. could add the private key or something.

sure that's fine.

when we migrate keys to the blockstore we can have a test like that, but we arent there yet.

jbenet · 2015-09-18T06:37:09Z

needs sharness tests verifying the expected security properties.
also tests with 2 nodes, with one node trying to get private objects from the other.

I don't feel confident in this. I'm not sure that this is actually going to do what we expect it to. It seems right to me, but i'm not sure about all the corner cases in which expectations may be incorrect. Up to @whyrusleeping

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

whyrusleeping · 2015-09-22T01:25:53Z

@jbenet I'm really not sure how I could possibly write a sharness test for this, outside of the code, theres no way to know what the keys of the 'private objects' are, they wont show up in 'ipfs refs local'. The tests I have in core/corerepo/gc_test.go construct a node the same way a daemon would, and they work as expected.

jbenet · 2015-09-22T03:17:31Z

Fine let's wait until keystore is implemented

—
Sent from Mailbox

On Mon, Sep 21, 2015 at 6:25 PM, Jeromy Johnson notifications@github.com
wrote:

@jbenet I'm really not sure how I could possibly write a sharness test for this, outside of the code, theres no way to know what the keys of the 'private objects' are, they wont show up in 'ipfs refs local'. The tests I have in core/corerepo/gc_test.go construct a node the same way a daemon would, and they work as expected.

Reply to this email directly or view it on GitHub:
#1538 (comment)

jbenet added the status/in-progress label Jul 29, 2015

whyrusleeping force-pushed the dev0.4.0 branch from 25fd6b9 to f39597e Compare August 10, 2015 18:47

whyrusleeping force-pushed the private-blocks branch from 609a335 to ec6a2a8 Compare August 12, 2015 19:56

jbenet reviewed Aug 13, 2015
View reviewed changes

jbenet mentioned this pull request Aug 22, 2015

Sprint Aug 17 ipfs/team-mgmt#28

Closed

16 tasks

jbenet reviewed Aug 23, 2015
View reviewed changes

jbenet assigned whyrusleeping Aug 23, 2015

whyrusleeping force-pushed the dev0.4.0 branch from 4a66fb4 to ff3ad99 Compare September 3, 2015 02:16

whyrusleeping force-pushed the private-blocks branch 3 times, most recently from 44e955a to bc2d27d Compare September 8, 2015 22:31

whyrusleeping force-pushed the dev0.4.0 branch from ff3ad99 to 6e5d1df Compare September 10, 2015 04:12

tv42 added 3 commits September 11, 2015 10:09

pin: Guard against callers causing refcount underflow

80bcb16

This used to lead to large refcount numbers, causing Flush to create a lot of IPFS objects, and merkledag to consume tens of gigabytes of RAM. License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

sharness: fusermount -u is the documented way to unmount FUSE on Linux

99ea9d4

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

tv42 added 3 commits September 11, 2015 12:11

fuse/ipns: Only change file size in Setattr if asked to

bcf98bd

This used to cause files e.g. being edited with `vi` to become 0-size. License: MIT Signed-off-by: Tommi Virtanen <tv@eagain.net>

p2p/net/filter: Guard with a mutex

47d6836

Callers assume this is safe to call whenever, let's make it so. License: MIT Signed-off-by: Tommi Virtanen <tv@eagain.net>

whyrusleeping force-pushed the dev0.4.0 branch from 6e5d1df to 47d6836 Compare September 11, 2015 20:11

whyrusleeping force-pushed the private-blocks branch from bc2d27d to 23511bf Compare September 11, 2015 20:53

jbenet mentioned this pull request Sep 14, 2015

Authenticate access to IPFS objects #1665

Open

whyrusleeping assigned jbenet and unassigned whyrusleeping Sep 16, 2015

fixup naming

54bae3e

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

jbenet reviewed Sep 18, 2015
View reviewed changes

comment corerepo gc method better

ffbb648

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

RichardLitt mentioned this pull request Sep 23, 2015

Sprint Sep 21 ipfs/team-mgmt#33

Closed

65 tasks

whyrusleeping force-pushed the dev0.4.0 branch 3 times, most recently from 764bef9 to 1bbc472 Compare November 11, 2015 18:04

ghost added the topic/encryption label Dec 22, 2015

whyrusleeping force-pushed the dev0.4.0 branch 2 times, most recently from 68b9745 to b0a8591 Compare December 28, 2015 14:36

whyrusleeping force-pushed the dev0.4.0 branch from cf514d8 to 3224ae0 Compare January 12, 2016 16:23

Kubuxu closed this May 13, 2016

Kubuxu deleted the private-blocks branch February 27, 2017 20:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: separate private and public blocks #1538

WIP: separate private and public blocks #1538

whyrusleeping commented Jul 29, 2015

jbenet commented Jul 29, 2015

whyrusleeping commented Jul 29, 2015

whyrusleeping commented Aug 4, 2015

whyrusleeping commented Aug 12, 2015

jbenet Aug 13, 2015

jbenet Aug 23, 2015

jbenet commented Aug 13, 2015

whyrusleeping commented Aug 13, 2015

whyrusleeping commented Aug 13, 2015

whyrusleeping commented Aug 13, 2015

jbenet commented Aug 23, 2015

jbenet Aug 23, 2015

jbenet Aug 23, 2015

jbenet commented Aug 23, 2015

jbenet commented Aug 23, 2015

whyrusleeping commented Sep 9, 2015

whyrusleeping commented Sep 16, 2015

jbenet commented Sep 16, 2015

jbenet Sep 18, 2015

whyrusleeping Sep 18, 2015

jbenet Sep 18, 2015

whyrusleeping Sep 18, 2015

jbenet commented Sep 18, 2015

whyrusleeping commented Sep 22, 2015

jbenet commented Sep 22, 2015

WIP: separate private and public blocks #1538

WIP: separate private and public blocks #1538

Conversation

whyrusleeping commented Jul 29, 2015

jbenet commented Jul 29, 2015

whyrusleeping commented Jul 29, 2015

whyrusleeping commented Aug 4, 2015

whyrusleeping commented Aug 12, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbenet commented Aug 13, 2015

whyrusleeping commented Aug 13, 2015

whyrusleeping commented Aug 13, 2015

whyrusleeping commented Aug 13, 2015

jbenet commented Aug 23, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbenet commented Aug 23, 2015

jbenet commented Aug 23, 2015

whyrusleeping commented Sep 9, 2015

whyrusleeping commented Sep 16, 2015

jbenet commented Sep 16, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbenet commented Sep 18, 2015

whyrusleeping commented Sep 22, 2015

jbenet commented Sep 22, 2015