Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kokoro disk is full #1670

Closed
mkurc-ant opened this issue Sep 22, 2020 · 3 comments · Fixed by #1847
Closed

Kokoro disk is full #1670

mkurc-ant opened this issue Sep 22, 2020 · 3 comments · Fixed by #1847

Comments

@mkurc-ant
Copy link
Collaborator

While trying to investigate what I've broken in #1651 I stumbled upon "disk full" messages like:

sqlite3.OperationalError: database or disk is full

capnp.lib.capnp.KjException: kj/io.c++:406: overloaded: ::writev(fd, current, iov.end() - current): No space left on device; fd = 10

Found these in "Xilinx Series 7 - Vendor Tool Tests (Presubmit)" logs.
@litghost
Copy link
Contributor

I've seen this too. I created #1725 to debug this, and it looks like the kokoro working directory is a magic 4 TB mount point. Presumably this underlying magic may fail, which manifests as disk full? Unclear what is actually happening here. Maybe a GCP bug?

This was referenced Oct 29, 2020
Closed
Closed
@litghost
Copy link
Contributor

litghost commented Dec 4, 2020

/home/kbuilder/.pyenv/libexec/pyenv-init: line 131: cannot create temp file for here-document: No space left on device

Looks like HOME might be full?

@litghost
Copy link
Contributor

litghost commented Dec 4, 2020

Example failure:

Filesystem      Size  Used Avail Use% Mounted on
udev             52G     0   52G   0% /dev
tmpfs            11G   17M   11G   1% /run
/dev/sda1        99G   95G     0 100% /
tmpfs            52G     0   52G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            52G     0   52G   0% /sys/fs/cgroup
/dev/sdc1       246G   43G  203G  18% /opt/Xilinx
cgmfs           100K     0  100K   0% /run/cgmanager/fs
tmpfs            11G     0   11G   0% /run/user/1000
/dev/sdd        246G   29G  217G  12% /image

@litghost litghost mentioned this issue Dec 4, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Move HOME to /tmpfs. litghost/symbiflow-arch-defs
2 participants
@litghost @mkurc-ant