Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/gob: Encode gives "encoder: message too big" when given a large map #13850

Closed
mmp opened this issue Jan 7, 2016 · 3 comments
Closed
Milestone

Comments

@mmp
Copy link

mmp commented Jan 7, 2016

OSX / go version go1.5.1 darwin/amd64

The repro case below creates a map that's a few GB big and then attempts to encode it to a file using Encode(). Encode() fails with a "message too big" error. (Presumably because the full size of the map is greater than 1<<30, which is the value of "tooBig" in the gob implementation.

I would expect the gob implementation to be able to handle a map that's larger than the maximum message size, since the individual elements being encoded are much smaller than the limit.

package main

import (
    "encoding/gob"
    "fmt"
    "io/ioutil"
    "log"
)

type File struct {
    A, B string
    V    int
}

func main() {
    f, err := ioutil.TempFile("", "foo")
    if err != nil {
        panic(err)
    }

    e := gob.NewEncoder(f)

    m := make(map[string][]*File)
    for i := 0; i < (1 << 22); i += 1 {
        fl := File{"00000000000000000000000000000000000000000000000",
            "11111111111111111111111111111111111111111", 42}
        var arr []*File
        for j := 0; j < 4; j += 1 {
            arr = append(arr, &fl)
        }
        m[fmt.Sprintf("%d", i)] = arr
        if i%(1<<20) == 0 {
            log.Printf("i = %d / %d", i, 1<<22)
        }
    }

    if err := e.Encode(m); err != nil {
        panic(err)
    }
}
@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Jan 7, 2016
@ianlancetaylor ianlancetaylor changed the title gob Encode() gives "encoder: message too big" when given a large map encoding/gob: Encode gives "encoder: message too big" when given a large map Jan 7, 2016
mmp pushed a commit to google/skicka that referenced this issue Jan 8, 2016
Serialize the file id to File struct map manually, rather than by passing it to gob/Encode() in its entirety, as the Encode() implementation recently changed to issue an error if the map is very large.

(I believe that this will fix skicka issue #98.)
@Wouterbeets
Copy link

I have the same issue, is there a work around?

@bradfitz bradfitz modified the milestones: Go1.9, Unplanned Dec 7, 2016
@bradfitz
Copy link
Contributor

bradfitz commented Dec 7, 2016

/cc @robpike

@robpike
Copy link
Contributor

robpike commented Dec 11, 2016

Sorry but no. The model requires that a data item is transmitted atomically (although types may break the stream into submessages). Too hard to change.

@robpike robpike closed this as completed Dec 11, 2016
@golang golang locked and limited conversation to collaborators Dec 11, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

6 participants