Compile bug: Race condition during compilation, compilation works with -j 1 but not with -j 8

[paul-civitas]

Git commit

ea1431b

Operating systems

Linux

GGML backends

CPU

Problem description & steps to reproduce

I ran podman build -f .devops/cpu.Dockerfile . on my macbook and I got a compile error.

I went into the intermediate image, and I tried instead of running cmake --build build -j $(nproc) running cmake --build build -j 1 and it built successfully.

$(nproc) was giving 8.

The error it gave seems to be getting eaten, here's an example:

[ 42%] Building CXX object src/CMakeFiles/llama.dir/unicode-data.cpp.o
c++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
gmake[2]: *** [src/CMakeFiles/llama.dir/build.make:174: src/CMakeFiles/llama.dir/llama-grammar.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:934: src/CMakeFiles/llama.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2

The extact point of failure seems to shift around, as can be expected for a race condition.

First Bad Commit

unknown

Compile command

Fails:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF -DGGML_CPU_ARM_ARCH=armv8-a;
cmake --build build -j 8


Passes:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF -DGGML_CPU_ARM_ARCH=armv8-a;
cmake --build build -j 1

Relevant log output

[ 32%] Built target llama-gguf-hash
[ 33%] Building CXX object src/CMakeFiles/llama.dir/llama-hparams.cpp.o
[ 33%] Building CXX object src/CMakeFiles/llama.dir/llama-impl.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/llama-io.cpp.o
[ 35%] Building CXX object src/CMakeFiles/llama.dir/llama-kv-cache.cpp.o
[ 35%] Building CXX object src/CMakeFiles/llama.dir/llama-kv-cache-unified.cpp.o
[ 36%] Building CXX object src/CMakeFiles/llama.dir/llama-kv-cache-unified-iswa.cpp.o
[ 36%] Building CXX object src/CMakeFiles/llama.dir/llama-kv-cache-recurrent.cpp.o
[ 37%] Building CXX object src/CMakeFiles/llama.dir/llama-memory.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/llama-mmap.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/llama-model-loader.cpp.o
[ 39%] Building CXX object src/CMakeFiles/llama.dir/llama-model-saver.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/llama-model.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/llama-quant.cpp.o
[ 41%] Building CXX object src/CMakeFiles/llama.dir/llama-sampling.cpp.o
[ 41%] Building CXX object src/CMakeFiles/llama.dir/llama-vocab.cpp.o
[ 42%] Building CXX object src/CMakeFiles/llama.dir/unicode-data.cpp.o
c++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
gmake[2]: *** [src/CMakeFiles/llama.dir/build.make:174: src/CMakeFiles/llama.dir/llama-grammar.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:934: src/CMakeFiles/llama.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2

[paul-civitas]

I dug deep into this and docker + mac is unbelievably broken. There's probably a CVE there even or something I'd bet.

I'm doing to declare the whole thing a superfund site and say that it's not worth it getting the build to work in that sort of environment.

[taronaeo]

When I see this

c++: fatal error: Killed signal terminated program cc1plus

It suggests to me that it might be an out-of-memory error. A quick google search suggests the same as well.

1. c++: fatal error: Killed signal terminated program cc1plus mlpack/mlpack#2775 (comment)
2. https://stackoverflow.com/a/67644572

Can you try and see if this works? I'm unsure about podman but there should be a similar option for it: https://stackoverflow.com/a/39720010

[paul-civitas]

Oh yeah that's probably it. I forgot one of the broken things is you get some tiny amount of RAM by default ugg.