Skip to content
This repository was archived by the owner on Mar 10, 2020. It is now read-only.

WIP: add and cat big file test (failing) #131

Merged
merged 9 commits into from
Nov 21, 2015
Merged

WIP: add and cat big file test (failing) #131

merged 9 commits into from
Nov 21, 2015
+63 −11

Conversation

daviddias
Copy link
Contributor

I was suspecting that something could be weird when adding big files and it seems I was right to have my suspicion (context https://botbot.me/freenode/ipfs/2015-11-21/?msg=54623933&page=6), however, I might be missing something, please review.

It seems that when cat'ing a big file, the parsing chunks one by one breaks the final file:
https://github.com/ipfs/js-ipfs-api/blob/master/src/request-api.js#L35-L46

@daviddias
add and cat big file test (failing)

Verified

This commit was signed with the committer’s verified signature.
renovate-bot Mend Renovate
GPG key ID: 81C63AE1C224182B
Verified
Learn about vigilant mode
ed71dfd
@daviddias
browser screams with large files
bf2d5b3
@dignifiedquire
Copy link
Contributor

Thanks @diasdavid for this, I did test cat on the terminal but with an earlier version. I will take a look at this later today.

@dignifiedquire
Copy link
Contributor

@diasdavid just pushed two commits improving this, but on my machine cating the file still takes way too long and times out, so we should use sth smaller for our tests.

@daviddias
Copy link
Contributor Author

Reduce the file size to 15Mb, before reducing I was getting this error:

Message:
    readStream2.on is not a function
Details:
    domain: [object Object]
    domainThrown: true
Stack:
TypeError: readStream2.on is not a function
    at streamEqual (/Users/david/Documents/code/ipfs/js-ipfs-api/node_modules/stream-equal/lib/index.js:53:15)
    at /Users/david/Documents/code/ipfs/js-ipfs-api/test/tests.js:253:9
    at /Users/david/Documents/code/ipfs/js-ipfs-api/src/request-api.js:47:35
    at finish (/Users/david/Documents/code/ipfs/js-ipfs-api/node_modules/wreck/lib/index.js:146:20)
    at wrapped (/Users/david/Documents/code/ipfs/js-ipfs-api/node_modules/hoek/lib/index.js:867:20)
    at ClientRequest.onResponse (/Users/david/Documents/code/ipfs/js-ipfs-api/node_modules/wreck/lib/index.js:169:20)
    at ClientRequest.g (events.js:260:16)
    at emitOne (events.js:77:13)
    at ClientRequest.emit (events.js:169:7)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:415:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:88:23)
    at Socket.socketOnData (_http_client.js:305:20)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at readableAddChunk (_stream_readable.js:146:16)
    at Socket.Readable.push (_stream_readable.js:110:10)

and now it is just timing out :(

nvm, tests pass again :)

  • make sure I remove from history the 100mb file before merging

Sorry, something went wrong.

@daviddias
Copy link
Contributor Author

@dignifiedquire try again please, should pass now :)

@daviddias
Copy link
Contributor Author

Still intrigued why the API doesn't think a big file is a stream and not chunked

stream: true  chunked: false
      ✓ cat BIG file (146ms)

@daviddias daviddias mentioned this pull request Nov 21, 2015
Closed
53 tasks
daviddias added a commit that referenced this pull request Nov 21, 2015
@daviddias
Merge pull request #131 from ipfs/test/bigfile
594b859 
WIP: add and cat big file test (failing)
@daviddias daviddias merged commit 594b859 into master Nov 21, 2015
@daviddias daviddias deleted the test/bigfile branch November 21, 2015 16:01
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@daviddias @dignifiedquire