Parallel runtests fails

This is run on Fedora 44.

../scripts/runtests -j 16
bwrap: execvp scripts/runtests: No such file or directory
...
bwrap: execvp scripts/runtests: No such file or directory
Runtest parallel: 0 tests run, 0 successful, 0 failed, 0 skipped, 0 shmem errors
Bug: 353 tests but this does not match 0 successful + 0 failed + 0 skipped

Apparently, the code does not take into account where it is started and blindly tries to run scripts/runtests in the bwrap command. However, it should be running ${RUNTESTS}, which is set rather high at the top of the script. The RUNTESTS variable is also exported because a test script can call runtests recursively and needs to know what to call.

But that is not all. Many tests fail with a variety of errors, including "Internal error" messages because the result/stderr files cannot be found. It seems that the parallel setup is not yet abstracted enough to be portable and leaves quite a few functional holes.

Output of ../scripts/runtests -j 16 when run from the clone directory: runtests-parallel.txt

Indeed, running scripts/runtests was an oversight from my side.

Did you try less jobs? If you fully peg your PC, tests start to fail due to timeouts or even oom kills. Do they also fail with -j 4 or -j 1 ?

Any insight from dmesg? Segfaults or oom kills?

I often used scripts/runtests -j 1 -p tests/mqtt for example to debug a failing test in bwrap but otherwise running equally than normal.

I replaced scripts/runtests with ${RUNTESTS}, executed ../scripts/runtests -j32 all pass on my PC.

If the reason above is not the issue, there is probably something different on Fedora 44 or in your paths than what I expected to be used. I somewhat expected that will happen on some Linux flavors...

What I see from the logs is that you run linuxcnc from /home/lcnc.git. If this folder or an other folder that is used by linuxcnc is neither in $HOME or $TOPDIR, it is readonly due to there is no tmp-overlay. It is possible that you have to add another folder around here:

linuxcnc/scripts/runtests.in

Line 329 in e1c6a55

"--overlay-src" "$HOME" "--tmp-overlay" "$HOME"

Can you send all the logs from one failing test?

I haven't had the time to experiment and look in detail. Will need to do so before I can make a better report. Will look into the number of parallel processes, etc.

BTW, the "/home/lcnc.git/" prefix is just to mask my actual $HOME. This all works as expected.

Meanwhile, I got linuxcnc running in a docker with Fedora 44. After installing what felt like 100 packages to get everything built by hand, the only failing test is tests/halcompile/halname and the qtdragon tests. Looks like a bug in the test or might be I have to install another package.

Most probably, your home mask is the reason for the failing tests. Right now, I run the tests as user tests but from /linuxcnc-src.

My build is purely a RIP build and test. And, no, my post-processing of the posted text file is not the problem. I just don't normally publish my directory hierarchy to a public forum.

As I said, I still need to take a deep dive in the real problems. Docker is probably not the right platform. And my install has an 8+ year history of dist-upgrades behind it ;-)

Running ../scripts/runtests -n -j 1 ../tests/abort/feed-rate/ already fails. The problem seems to be:

+ halcmd loadusr -Wn inihal milltask -ini /home/lcnc.git/tests/abort/feed-rate/test.ini
tool_mmap_creator(): file open fail: Permission denied
<commandline>:0: waitpid failed milltask inihal
<commandline>:0: milltask exited without becoming ready

This issue has been up before in #4656.
Adding touch "$HOME/.tool.mmap" && chmod 600 "$HOME/.tool.mmap" to the top of the linuxcnc script helps. Not sure what the root cause is, but I suspect the flags in the call to open() when creating the file. Blindly creating the file beforehand is not the correct solution. The whole tooldata stuff needs refactoring.

and, creating the $HOME/.tool.mmap file makes the runtests run parallel properly.

Ah, this is post processing only, so yes, should have no influence.

Thanks for fixing my bug. I just wanted to start creating a PR but it's also fine like this.

Strange, on my PC, in podman Fedora 44 and in Debian Trixie, the following works fine:
rm -f ~/.tool.mmap ; scripts/runtests -p -j 1 tests/abort/feed-rate/
It works both on master and on 6c30063 (before #4656 was merged)

#4656 was only an issue with the ui-smoke tests and the linuxcnc gui's in general where the order of creating and opening was reversed due to a race condition.

It might be a security thing from apparmor denying access, an overlay issue or an issue with the open flags. But until now, it does not reproduce on my PC.

Can you verify if this fails only with -j1 or also without? Be aware that if you run runtests single threaded once, ~/.tool.mmap is created and it works fine from then on.

The reason for the failure is the existence of (my real home) $HOME/.tool.mmap with mode 400. A leftover from previous experiments hacking the parallel stuff while it was being developed. The chmod 600 $HOME/.tool.mmap I added did the actual trick.

The creation of file fails with a permission denied because it is read-only and truncation is not allowed in the bwrap container. My normal runs do not seem to care about the permission. Very interesting... It appears that overlaying $HOME causes it to seal the read only file and result in a EPERM error.

There might be a simple solution by forcing a mode:

diff --git a/src/emc/tooldata/tooldata_mmap.cc b/src/emc/tooldata/tooldata_mmap.cc
index e2cb5ad7cf..664e2cda79 100644
--- a/src/emc/tooldata/tooldata_mmap.cc
+++ b/src/emc/tooldata/tooldata_mmap.cc
@@ -151,6 +151,15 @@ int tool_mmap_creator(EMC_TOOL_STAT const * ptr,int random_toolchanger)
         exit(EXIT_FAILURE);
     }
     toolstat = ptr; //note NULL for sai
+
+    // If the file exists, make sure it can be written. Parallel tests will
+    // fail on mode 400 because a read only file is sealed in the container.
+    // We don't really care if the chmod() succeeds. It is just fixing an
+    // annoyance by bypassing an error mode that is hard to detect otherwise.
+    // All relevant errors will be caught by the following open() should that
+    // fail.
+    chmod(tool_mmap_fname(), TOOL_MMAP_MODE);
+
     creator_fd = open(tool_mmap_fname(),
                      TOOL_MMAP_CREATOR_OPEN_FLAGS,TOOL_MMAP_MODE);
     if (creator_fd < 0) {

Ah now it reproduces:
touch "$HOME/.tool.mmap" && chmod 400 "$HOME/.tool.mmap"
But at least with my PC, both normal and parallel fail afterwards as I would expect:
scripts/runtests -p -j 1 tests/abort/feed-rate/
scripts/runtests -p tests/abort/feed-rate/

Does your setup use a different location for .tool.mmap than $HOME/.tool.mmap for whatever reason? Printing out the path could give a clue.
One reason could be secure_getenv("HOME") failing:

linuxcnc/src/emc/tooldata/tooldata_mmap.cc

Line 96 in 67437d8

char* hdir = secure_getenv("HOME");

Otherwise it would be strange... I mean if it is read only, open with O_RDWR should probably fail always...

I can not even start LinuxCNC with axis_mm after this:

scripts/linuxcnc
...
tool_mmap_creator(): file open fail: Keine Berechtigung
...
tool_mmap_user(): tool mmap not available: /home/user/.tool.mmap: open: Permission denied

This could give a hint:

diff --git a/src/emc/tooldata/tooldata_mmap.cc b/src/emc/tooldata/tooldata_mmap.cc
index e2cb5ad7cf..2c2b773a77 100644
--- a/src/emc/tooldata/tooldata_mmap.cc
+++ b/src/emc/tooldata/tooldata_mmap.cc
@@ -154,24 +154,24 @@ int tool_mmap_creator(EMC_TOOL_STAT const * ptr,int random_toolchanger)
     creator_fd = open(tool_mmap_fname(),
                      TOOL_MMAP_CREATOR_OPEN_FLAGS,TOOL_MMAP_MODE);
     if (creator_fd < 0) {
-        perror("tool_mmap_creator(): file open fail");
+        fprintf(stderr, "tool_mmap_creator(): open %s fail: %s\n", tool_mmap_fname(), strerror(errno));
         exit(EXIT_FAILURE);
     }
     if (lseek(creator_fd, TOOL_MMAP_SIZE, SEEK_SET) == -1) {
         close(creator_fd);
-        perror("tool_mmap_creator() lseek fail");
+        fprintf(stderr, "tool_mmap_creator() lseek %s fail: %s\n", tool_mmap_fname(), strerror(errno));
         exit(EXIT_FAILURE);
     }
     if (write(creator_fd, "\0", 1) < 0) {
         close(creator_fd);
-        perror("tool_mmap_creator(): file tail write fail");
+        fprintf(stderr, "tool_mmap_creator(): file tail write %s fail: %s\n", tool_mmap_fname(), strerror(errno));
         exit(EXIT_FAILURE);
     }
     tool_mmap_base = (char*)mmap(NULL, TOOL_MMAP_SIZE, PROT_READ | PROT_WRITE,
                                  MAP_SHARED, creator_fd, 0);
     if (tool_mmap_base == MAP_FAILED) {
         close(creator_fd);
-        perror("tool_mmap_creator(): mmap fail");
+        fprintf(stderr, "tool_mmap_creator(): mmap %s fail: %s\n", tool_mmap_fname(), strerror(errno));
         exit(EXIT_FAILURE);
     }

Hmm, did you set something setuid in your linuxcnc bin folder?
https://man7.org/linux/man-pages/man3/getenv.3.html
The secure_getenv() function is just like getenv() except that it returns NULL in cases where "secure execution" is required.

This could explain why it works sometimes and sometimes not. Inside bwrap, root is mapped to nobody, so a setuid executable will behave differently inside than outside if it uses secure_getenv().

Test:

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>

int main(void)
{
    char* hdir = secure_getenv("HOME");
    printf("HOME: %s\n", hdir ? hdir : "NULL");
    return 0;
}

gcc test.c -o test
sudo chown root:root test
sudo chmod 4755 test
./test
HOME: NULL
bwrap --ro-bind / / --dev /dev --unshare-ipc --unshare-pid --unshare-net --proc /proc --die-with-parent --tmpfs /tmp -- ./test
HOME: /home/user

For starters, never run anything LCNC setuid, never allow to run with sudo.

The $HOME/.tool.mmap is really in my home directory. Otherwise I hadn't been able to test it so easily.

The error always happened on the open() call. The original perror() will automatically append the strerror() of the latest errno, so the "Permission denied" in tool_mmap_creator(): file open fail: Permission denied is the genuine point of failure. If the location would be different, then adding the blind chmod() would have no effect. However, with it, the mode is set at 600 and the parallel run no longer fails.

That all said,... I removed the original $HOME/.tool.mmap file and re-ran all test procedures. I now too have a failure with mode 400 and not running parallel. So I guess this has been a fluke. It may have been caused by a hanging or zombie process that was undetected that was still mmap attached? Not likely, but there are some possible error modes.

What I do suggest is that when open() fails, then we need to test more for a better error. Maybe we should add an extra message like in:

diff --git a/src/emc/tooldata/tooldata_mmap.cc b/src/emc/tooldata/tooldata_mmap.cc
index e2cb5ad7cf..8616e67a19 100644
--- a/src/emc/tooldata/tooldata_mmap.cc
+++ b/src/emc/tooldata/tooldata_mmap.cc
@@ -155,6 +155,13 @@ int tool_mmap_creator(EMC_TOOL_STAT const * ptr,int random_toolchanger)
                      TOOL_MMAP_CREATOR_OPEN_FLAGS,TOOL_MMAP_MODE);
     if (creator_fd < 0) {
         perror("tool_mmap_creator(): file open fail");
+        struct stat sb;
+        if (0 == stat(tool_mmap_fname(), &sb)) {
+            fprintf(stderr, "tool_mmap_creator(): mmap file '%s' exists with mode %03o\n",
+                    tool_mmap_fname(), (unsigned)sb.st_mode);
+        } else if (ENOENT != errno) {
+            perror("tool_mmap_creator(): stat fail after open fail");
+        }
         exit(EXIT_FAILURE);
     }
     if (lseek(creator_fd, TOOL_MMAP_SIZE, SEEK_SET) == -1) {

@grandixximo

Yes, since setcap is available, I switched to use it, no setuid any more. But even with setuid, the behavior was the same on my side. The tooldata is only used in user processes, not in real time, right?

BTW: Is setcap already used in the 2.10 Debian packages? If not, it could be an idea to enable it.

Glad there no different behavior from your side any more. I sometimes had zombies after experimenting with parallel tests, could be a reason.

I find changing the mode from read-only to read-write a bit intrusive. Most probably there is no good reason to set it read-only. Except you for a test and then forget about it, happens... ;-) But still.

Your suggestion to add more information if open fails sounds good to me. It would be worthwhile adding also the filename to all error messages as @grandixximo did in #4658 for tool_mmap_user().

Are you creating a PR or should I do it?

Yes, since setcap is available, I switched to use it, no setuid any more. But even with setuid, the behavior was the same on my side.

Never run setcap either. Only ever test without additional privileges on my dev machine.

The tooldata is only used in user processes, not in real time, right?

Purely user space.

BTW: Is setcap already used in the 2.10 Debian packages? If not, it could be an idea to enable it.

That would be a different discussion.

Your suggestion to add more information if open fails sounds good to me.
Are you creating a PR or should I do it?

Please, you are welcome to send one.

Navigation Menu

Repository navigation

Parallel runtests fails #4738

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Parallel runtests fails #4738

Description

Activity

hdiethelm commented on Oct 29, 2026

BsAtHome commented on Oct 30, 2026

hdiethelm commented on Oct 31, 2026

BsAtHome commented on Oct 31, 2026

BsAtHome commented on Oct 31, 2026

hdiethelm commented on Nov 1, 2026

BsAtHome commented on Nov 1, 2026

hdiethelm commented on Nov 1, 2026

hdiethelm commented on Nov 1, 2026

hdiethelm commented on Nov 1, 2026

BsAtHome commented on Nov 1, 2026

hdiethelm commented on Nov 2, 2026

BsAtHome commented on Nov 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions