wasm32-wasi 3.x buildbot: No space left on device

@brettcannon

wasm32-wasi 3.x: https://buildbot.python.org/#/builders/1046/builds/12537

/usr/bin/ld.bfd: final link failed: No space left on device
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:1791: _bootstrap_python] Error 1

@brettcannon: I'm sorry! Your previous attempt to free enough disk space on wasm32-wasi 3.x buildbot was not enough. The "No space left on device" error is back.

cc @stratakis: We do have frequent "No space left on device" in our Fedora/CentOS/RHEL buildbot workers as well :-(

I'm running out of ideas here.

Filesystem     1K-blocks     Used Available Use% Mounted on
udev             8151240        0   8151240   0% /dev
tmpfs            1633752      808   1632944   1% /run
/dev/sdb1       65742576 10617668  52116244  17% /
tmpfs            8168740      252   8168488   1% /dev/shm
tmpfs               5120        0      5120   0% /run/lock
/dev/sdb15        126678    12064    114614  10% /boot/efi
/dev/sda1       32845584       24  31151560   1% /mnt
tmpfs            1633748        8   1633740   1% /run/user/1000

There's plenty of space on that disk. That df output if after stopping podman running https://github.com/brettcannon/wasi-buildbot/blob/main/run.py and deleting the checkouts which are mounted in ~buildbot/builldarea which were like 23 GB, so still well under 52 GB. I can try checking when it happens again with df to see what's actually running out of disk space.

The buildbot worker is back to green, so I close the issue.

@brettcannon

I reopen the issue, "No space left on device" is back. cc @brettcannon

Build12739 failed with "Bus error": https://buildbot.python.org/#/builders/1046/builds/12739

rm -f libpython3.16d.a
/opt/wasi-sdk-33.0-x86_64-linux/bin/llvm-ar rcs libpython3.16d.a Modules/getbuildinfo.o Parser/token.o  (...)  Modules/_elementtree.o  Modul
s/cjkcodecs/_codecs_cn.o  Modules/cjkcodecs/_codecs_hk.o  Modules/cjkcodecs/_codecs_iso2022.o  (...) Python/frozen.o
/opt/wasi-sdk-33.0-x86_64-linux/bin/clang    -z stack-size=16777216 -Wl,--stack-first -Wl,--initial-memory=41943040   -o python.wasm Programs/python.o (...)  Modules/sha3module.o  Modules/

xpat.a                           -lm                     -lm 
clang: error: unable to execute command: Bus error

The following build 12740 failed with "No space left on device": https://buildbot.python.org/#/builders/1046/builds/12740

ar rcs libpython3.16d.a Modules/getbuildinfo.o (...) Python/frozen.o
ar: unable to copy file 'libpython3.16d.a'; reason: No space left on device
make: *** [Makefile:1164: libpython3.16d.a] Error 1

The checkouts are taking up over 50 GB:

$ du . --max-depth 1
496920	./3.12.bcannon-wasi.wasi.nondebug
10784744	./3.14.bcannon-wasi.wasi.debug
23050496	./3.x.bcannon-wasi.wasi.debug
443112	./pull_request.bcannon-wasi.wasi.debug
7888764	./3.15.bcannon-wasi.wasi.debug
7765536	./3.13.bcannon-wasi.wasi.debug
12	./info
1255504	./3.11.bcannon-wasi.wasi.nondebug
51721164	.

Is that expected to be happening?

$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
udev             8151240        0   8151240   0% /dev
tmpfs            1633752      800   1632952   1% /run
/dev/sdb1       65742576 62615292    118620 100% /
tmpfs            8168740      252   8168488   1% /dev/shm
tmpfs               5120        0      5120   0% /run/lock
/dev/sdb15        126678    12064    114614  10% /boot/efi
/dev/sda1       32845584       24  31151560   1% /mnt
tmpfs            1633748        8   1633740   1% /run/user/1000

https://devguide.python.org/testing/new-buildbot-worker/#required-resources says I should only need about 30 GB of free space.

I've once again deleted the checkouts and the buildbot is starting up again, but is this amount of space usage expected? I thought some change was made a while back to use less space in the Git checkout?

Zach pointed me at https://github.com/python/buildmaster-config/blob/d74e3cbef971ca5e5c1cfeb2a6740343592ccfa1/master/custom/workers.py#L341-L348 as a possible solution as they think this is due to a bug in buildbot itself.

Zach pointed me at https://github.com/python/buildmaster-config/blob/d74e3cbef971ca5e5c1cfeb2a6740343592ccfa1/master/custom/workers.py#L341-L348 as a possible solution as they think this is due to a bug in buildbot itself.

The CPython repo has grown significantly. Waiting for a new release of buildbot for buildbot/buildbot#9058 to hopefully resolve some of the disk space issues.

https://buildbot.python.org/#/builders/1046 is back to green, I close the issue.

Got an email that it's happened again, so when I have time I will clear out the space and then turn on shallow checkouts as the SC has not told me I can just skip having a buildbot yet.

I've opened python/buildmaster-config#788 to use shallow clones to try and work around the disk usage problem.

Navigation Menu

Repository navigation

wasm32-wasi 3.x buildbot: No space left on device #154532

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

wasm32-wasi 3.x buildbot: No space left on device #154532

Description

Activity

brettcannon commented on Jul 23, 2026

vstinner commented on Jul 24, 2026

vstinner commented on Aug 14, 2026

brettcannon commented on Aug 14, 2026

brettcannon commented on Aug 14, 2026

stratakis commented on Aug 15, 2026

vstinner commented on Aug 31, 2026

brettcannon commented on Sep 2, 2026

brettcannon commented on Sep 8, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions