Skip to content

Modernize the dbgprint module and make it portable. - #523

Open
daliborfox wants to merge 19 commits into
OpenTS-Developers:mainfrom
daliborfox:portable-dbgprint
Open

daliborfox wants to merge 19 commits into
OpenTS-Developers:mainfrom
daliborfox:portable-dbgprint

Conversation

@daliborfox

@daliborfox daliborfox commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

This pull request updates the dbgprint module so that it makes use of C++20 standard library facilities instead of Windows ones where possible, and allows the module to work on non-Windows systems, having been tested on Linux and Mac:

Screenshot From 2026-11-08 18-38-23 image

On Mac, the operating system version for the log banner is retrieved from the SystemVersion.plist file, on other systems the /etc/os-release or /usr/lib/os-release file is queried. The output of uname -a is provided as a fallback in case those files don't exist.

Reading the .plist file does require an additional dependency of libxml2 on Mac, though it's a part of the macOS SDK, so the user doesn't need to install any additional dependencies themselves when building OpenTS with this module (build system support not yet available).

The default location for the logging directory is as follows:

  • Windows: <dir-with-game-exe>\Debug\
  • Mac: ~/Library/Logs/OpenTS/
  • Others (Linux): $XDG_STATE_HOME/OpenTS/, with a fallback of $HOME/.local/state/OpenTS/
  • Fallback: ./OpenTS-Logs/

A new feature is the ability to specify the desired location of the log file. If the -USERDIR=<path> command-line option is specified, the logs will be located in the following directory:

  • <USERDIR>/Debug/

In order to gracefully use objects that dynamically allocate memory, the module holds its state in a lazy initialization singleton. This singleton contains an m_DebugDestroyed flag that causes all of the module's functions to gracefully fail if they try to make use of the shared state after it's already had its destructor called by the C++ runtime.

The same applies to the thread-local buffers used for holding the vsnprintf() formatting data.

As a consequence, there is a high likelihood that messages logged from the global destructors of other modules in the project will not end up being processed.

Another important thing to keep in mind is that the Debug_Init() function call is now mandatory, as it's used to process the command-line options for whether or not to enable the console, and where to store the logs. It is permissible to call DebugString() before Debug_Init() is invoked, e.g. from global constructors, but the messages will only be displayed and/or written to the log file / sent to the debugger after Debug_Init() had been invoked.

Please let me know if you disagree if some of the things I've done, or see room for improvement! I've tried my best to polish things up, but it's always possible that I've missed something.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Development builds of 79842e6:

The links work without a GitHub account. Artifacts expire after 90 days, and this comment follows the latest successful build.

@ZivDero

ZivDero commented Nov 14, 2026

Copy link
Copy Markdown
Member

Hey! Overall looks solid, but found a few things:

  1. std::chrono::current_zone is probably a no-go because apparently with MSVC it requires icu.dll shipped since Windows 10 1903. And while I personally do not care much for older Windows systems, losing support for them now over this would not fly well with others.
  2. Where you use std::filesystem, please use the std::error_code versions, because the normal versions can throw exceptions.
  3. You replaced the process ID with the thread ID for the file name tiebreaker, but under Linux/MacOS it can actually end up the same between two processes.
  4. "Press any key" at console exit now holds the logger's lock and so any other thread that tries to write to the console on shutdown (e.g. audio) will get stuck waiting on it. You should release the lock before waiting for input.
  5. You can now forget to call Debug_Init() which 5 tests actually do, and they keep buffering their logs indefinitely and unbound. I guess the minimal fix is to make them call it, too. Or dunno.
  6. std::cout may be slower on Windows. Would be nice to measure it, if it actually is much slower then maybe keep WriteConsole on _WIN32.
  7. A few minor things - missing <format> and <sstream> includes, bool requested that is never set to true, in some doc comments <param> is mistakenly inside the description itself.

And we're now onto 0.3.0 so that needs updating :P

@daliborfox

daliborfox commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor Author

Thank you for the inputs! I agree with you on all points, and I've cleaned up the PR.

I've done some measurements on the Windows console output, comparing WriteConsoleA() with std::cout.write(), and somewhat surprisingly, std::cout.write() actually ended up being faster.

Here's the testcase: consoletest.cpp

I've commented out the file write and debugger write statements, and only kept the console write statements, and these are the results I got on Windows Terminal a) with std::cout.write():
Screenshot 2026-11-15 135457
And with ConsoleWriteA():
Screenshot 2026-11-15 135609

Same test with the Windows console host, with std::cout.write():
Screenshot 2026-11-15 135944
And with ConsoleWriteA():
Screenshot 2026-11-15 140102

And just for completeness sake, I also ran it on Linux, on the same computer in the Alacritty terminal (I dual-boot):

Screenshot From 2026-11-15 14-36-51

I think the reason why std::cout.write() ends up being faster is due to buffering, which means that the language runtime is ultimately able to issue fewer ConsoleWrite() calls than when calling it for both the timestamp and every single message fragment.

What surprised me about the results is that I had expected ConsoleWrite() to win in the 3rd test, which contains a lot of newlines in a single call to DebugString(), since I expected the line-terminated buffer in the language runtime to generate a ConsoleWrite() call for each of the lines, but it appears to be smart enough to lump them together into a single call.

And unfortunately, the performance test is marginal on the 32-bit debug build:

2026-11-15T13:50:51.3972307Z [doctest] doctest version is "2.5.3"
2026-11-15T13:50:51.3972599Z [doctest] run with "--help" for options
2026-11-15T13:50:51.3972943Z ===============================================================================
2026-11-15T13:50:51.3973332Z D:\a\OpenTS\OpenTS\tests\logstress\logstress.cpp(80):
2026-11-15T13:50:51.3974113Z TEST CASE:  the logger keeps up with many threads and writes every record whole, after a banner
2026-11-15T13:50:51.3974470Z 
2026-11-15T13:50:51.3975393Z D:\a\OpenTS\OpenTS\tests\logstress\logstress.cpp(120): MESSAGE: uncontended latency: mean 10.23 us, p99.9 42.50 us, worst 223.90 us
2026-11-15T13:50:51.3976801Z 
2026-11-15T13:50:51.3977480Z D:\a\OpenTS\OpenTS\tests\logstress\logstress.cpp(122): ERROR: CHECK( solo_mean_us <= 10.0 ) is NOT correct!
2026-11-15T13:50:51.3978747Z   values: CHECK( 10.2291 <= 10 )
2026-11-15T13:50:51.3979393Z   logged: uncontended mean call latency within 10 us
2026-11-15T13:50:51.3979871Z 
2026-11-15T13:50:51.3981312Z D:\a\OpenTS\OpenTS\tests\logstress\logstress.cpp(167): MESSAGE: 4 threads x 20000 messages in 1721 ms (46485 messages/second); latency: mean 78.83 us, p99.9 824.10 us, worst 1939.20 us (worst is informational)
2026-11-15T13:50:51.3982909Z 
2026-11-15T13:50:51.3983206Z ===============================================================================
2026-11-15T13:50:51.3983970Z [doctest] test cases:  1 |  0 passed | 1 failed | 2 skipped
2026-11-15T13:50:51.3984633Z [doctest] assertions: 14 | 13 passed | 1 failed |
2026-11-15T13:50:51.3985197Z [doctest] Status: FAILURE!
2026-11-15T13:50:51.3985503Z 

It was a lot worse at first (40us on my system, managed to get it down to 5us), I've spent quite some time optimizing it on the previous weekend (the biggest bottleneck was timestamp generation). However, the debug build version of STL stuff is quite a bit slower than the release build.

Would it make sense to have a separate threshold for the debug build, e.g. 15us instead of 10?

@daliborfox

Copy link
Copy Markdown
Contributor Author

Also, I forgot to mention, but re: it being possible to forget Debug_Init() in tests, I've added a warning message into the destructor that warns the user in case they logged messages but didn't initialize the logger, thus didn't actually log the messages:
Screenshot 2026-11-15 123540
I've used this to add the missing Debug_Init() calls to the tests.

The explicit initialization gives us an opportunity to potentially rename the log file so that it's easier to match to the test that's being run, but that's probably an item for a separate PR.

@daliborfox

Copy link
Copy Markdown
Contributor Author

I've spent some time further cleaning up and optimizing the PR, and I've been able to resolve the performance problem reported by the testsuite.

By getting rid of the snprintf() call in Write_Message_Locked(), I was able to reduce the uncontended mean latency of the win32 debug test on my machine from 5.6 us down to 4.6 us, a 17% speed increase, which should be sufficient to make the logstress test pass reliably under Github CI.

I've also compared the performance to the original version of the module, and there, the latency is 4.7 us, just a tiny bit more than the modernized version.

Comparing the 32-bit release builds, the original version has an uncontended mean latency result of 3.3 us, whereas the modernized version has 3.1 us, so the performance there is also really comparable.

@daliborfox

Copy link
Copy Markdown
Contributor Author

I've done some more testing on the console output performance, and I found that the original implementation of the module had significantly faster console output than the updated one:
Screenshot 2026-11-16 151503
See one of the previous comments for the previous results with the updated code.

To alleviate this problem, I've introduced buffering and direct writing with WriteConsoleA(). With it, I've been able to get very similar performance to the original module, and better performance than the original when composing a message out of multiple fragments:
Screenshot 2026-11-16 162542

I did try to use the console buffer for the file output too, setting the File to unbuffered mode with setvbuf(File, NULL, _IONBF, 0), but surprisingly enough, it actually slowed it down significantly (from 3 us to about 5 us in the uncontended mean latency test), looks like the buffered file IO is quite well optimized in Windows.

I believe that at this point, the updated dbgprint module should be a nice improvement over the old one, with very similar performance characteristics.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants