You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: _posts/2014-11-05-Playing-with-invertible-bloom-lookup-tables-and-bitcoin-transactions.md
+30-27
Original file line number
Diff line number
Diff line change
@@ -106,7 +106,8 @@ turns out that fragment 1 is also in that third bucket, so we subtract
106
106
index: 0
107
107
frag: I love
108
108
109
-
Now we can extract id 0x00f, which is "I love Triffids".
109
+
Now we can extract id 0x00f, which is "I love Triffids" (and leaving the IBLT
110
+
empty).
110
111
111
112
<aname="Testing-IBLTs"></a>
112
113
@@ -129,10 +130,10 @@ transaction as we find it may leave other fragments exposed so we can
129
130
make more progress.
130
131
131
132
The result turns out to be a cliff: with 300 transactions we can
132
-
extract all the transactions 100% of the time, with 320 it's down to
133
-
0% (this is only 10 runs though). We only care about full recovery,
134
-
because if we can't get one transaction, we fall back to getting the
135
-
whole block:
133
+
extract all the transactions ~100% of the time, with 320 it's down to
134
+
0% (this is only tested with 10 runs though). We only care about full
135
+
recovery, because if we can't get one transaction, we fall back to
136
+
getting the whole block:
136
137
137
138

138
139
@@ -146,18 +147,19 @@ the block, and also any which have reached us while the block was
146
147
being relayed to us, so we expect "our" transactions to dominate.
147
148
148
149
And it turns out that it's much easier to detect transactions which
149
-
aren't in the block than those that are; we only need to find a single
150
-
standalone fragment with count -1, and we can be fairly confident that
151
-
it means we've found one of our transactions and should remove the
152
-
whole thing. Finding a single exposed fragment for a transaction like
153
-
this is far more likely than finding all fragments of a transaction
154
-
exposed, and indeed our cliff for these is far higher, around 3300:
150
+
aren't in the block than recover those that are; we only need to find
151
+
a single standalone fragment with count -1, and we can be fairly
152
+
confident that it means we've found one of our transactions and should
153
+
remove the whole thing. Finding a single exposed fragment for a
154
+
transaction like this is far more likely than finding all fragments of
155
+
a transaction exposed, and indeed our cliff for these is far higher,
156
+
around 3300:
155
157
156
158

157
159
158
-
Finally, here's a heat map of interaction between the two, as the
159
-
number of "their" transactions we need to recover and the number of
160
-
"our" transactions we should remove varies. Blue is 100% recovered
160
+
Finally, here's a graph of interaction between the two, as the number
161
+
of "their" transactions we need to recover and the number of "our"
162
+
transactions we should remove varies. Blue is 100% recovered
161
163
(ie. always success) red is 0% (ie. never):
162
164
163
165

@@ -184,10 +186,11 @@ as most space is wasted:
184
186
185
187
### Smaller IBLTs ###
186
188
187
-
As expected it's not linear: doubling gives us less than double the
188
-
number of transactions we can recover. But conversely, if we shrink
189
-
down to 1/33 the size, we can still recover 30 transactions (this map
190
-
uses the 64 byte slices from above):
189
+
Using more memory (bandwitdh) for the IBLT is not a linear tradeoff:
190
+
doubling gives us less than double the number of transactions we can
191
+
recover. But conversely, if we shrink down to 1/33 the size, we can
192
+
still recover 30 transactions (this map uses the 64 byte slices from
193
+
above):
191
194
192
195

193
196
@@ -201,23 +204,23 @@ on one run).
201
204
202
205
Once we check the 48-bit id field, which is based on a cryptographic
203
206
hash of the transaction, we're already extremely robust without an
204
-
additional hashSum.
207
+
additional hashSum. In effect, our id field replaces hashSum.
205
208
206
209
## Results and Suggestions ##
207
210
208
-
8 bytes per slice is too small, larger sizes should be explored.
211
+
-8 bytes per slice is too small, larger sizes should be explored.
209
212
210
-
1M is vast overkill for the current network; as well as making block
211
-
propagation slower than the current scheme, it's unnecessary for now.
212
-
30k is probably more than enough for current blocks.
213
+
-1M is vast overkill for the current network; as well as making block
214
+
propagation slower than the current scheme, it's unnecessary for now.
215
+
30k is probably more than enough for current blocks.
213
216
214
-
Nodes may want to include discarded doublespend transactions in their
215
-
IBLT since it's cheaper to include a transaction for consideration
216
-
than to extract one which we didn't consider.
217
+
-Nodes may want to include discarded doublespend transactions in their
218
+
IBLT since it's cheaper to include a transaction for consideration
219
+
than to extract one which we didn't consider.
217
220
218
221
## Flaws and Future Work ##
219
222
220
-
Obviously my code could use optimization.
223
+
Obviously my code could use optimization and more testing.
221
224
222
225
Also, my selection of transactions was "the first N", which means it's
223
226
of limited accuracy. For example, transaction 613 is a huge 50k
0 commit comments