feat(crashdump): Add crash dump functionality for fatal errors #1594

slurmlord · 2025-09-18T09:33:24Z

Merge after bugfix(system): Prevent AMD/ATI driver crash on game launch by front loading the system dbghelp.dll #1066
Merge after fix(debug): Implement reference counter and thread safety in DbgHelpLoader to allow for independent users #1804

This change introduces the option to generate crash dumps, aka. mini dumps, on fatal errors.
The main minidump functionality is done by explicitly loading the dbghelp.dll from the system directory, as the dbghelp.dll that is bundled with the game is an older version that does not include this functionality. There is an option to create small dumps or full memory dumps, currently both are created.

Small dumps

These mostly contain stacks for the process threads and some stack variables, or to create dumps with extended info. The use case for these is to quickly determine where a crash occured, the type of crash, if it was already fixed etc. In addition, if the memory allocation structures are corrupted enough, an extended info dump might not succeed while the small dump should. The size of these dumps are typically on the order of 250kB.

Full memory

These contain all the memory regions in the process, as they specify MiniDumpWithFullMemory flag to MiniDumpWriteDump. They are considerably larger than the small dumps but compress relatively well. As an example, a ~400MB dump file is compressed to ~70MB using the Windows 11 built-in 7Z compression.

Storage Location

Crash dumps are stored in a new folder called 'CrashDumps' under the userDir ("Documents\Command and Conquer Generals Zero Hour Data"), and on startup it will create this directory if it doesn't exist and delete any older dumps so only the 10 newest small and 2 newest full memory dumps are left. This is to preserve disk space, as the full memory files can be several hundred MB.

Integration points

For VS2022 builds, unhandled exceptions end up in the UnhandledExceptionFilter in WinMain, which then get a reference to the actual exception that occurred and includes that in the dump.
For VC6 builds, unhandled exceptions are caught in the catch(...) blocks of GameEngine::execute which then calls RELEASE_CRASH. As there is no exception data available in this case to populate _EXCEPTION_POINTERS from, an intentional exception is triggered to get the trace of the current thread. This makes the stack traces for VC6 a bit more cryptic than VS2022 builds as the C++ exception handling gets included in the trace.

Limitations

In the longer run we'll probably want to replace this code with a more mature solution, like CrashPad, but that currently depends on a newer compiler than VC6.
As the code is intended to be temporary, it's kept behind a new CMake feature so it can be easily removed. There are also some other decisions made with this in mind:

Minidump is created in-process. Ideally, the dump should be performed by a process external to the crashing/failing process, but to avoid having to ship an extra binary, in-process was chosen instead. It's being performed in a separate thread to hopefully have a clean stack to work with.
Depends on RTS_BUILD_OPTION_VC6_FULL_DEBUG for VC6 builds. The PDBs generated with the default VC6 compile options are lacking in information, making the mini dumps less useful. The option RTS_BUILD_OPTION_VC6_FULL_DEBUG should be enabled for VC6 builds to ensure maximum usability. VS2022 builds produce better PDBs and require no extra options.
Directory management code is contained within the MiniDumper class, not re-usable by other components.
Many Win32-specific types and functions are used directly without regards for portability.
As the MiniDump feature is not available for VC6, a lot of headers have been borrowed from minidumpapiset.h and included in the DbgHelpLoader_minidump.h file.
Globals are used for storing the current exception info.
Only enabled for the games, tools are currently not included.

OmniBlade

Looks good, tested dump generation with a 2022 build.

slurmlord · 2025-10-09T12:41:24Z

Another approach I realized after publishing could be to move the GameMemory allocations from using GlobalAlloc to instead create a separate GameMemory heap and use HeapAlloc.
The benefit would be a lot less code required to traverse the allocations for inclusions the dump, as it could be done with HeapWalk instead for only the GameMemory heap.

xezon

Looks good overall. Just a bunch of small comments.

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Core/GameEngine/Source/Common/System/Debug.cpp

Generals/Code/Main/WinMain.cpp

Core/GameEngine/Include/Common/MiniDumper.h

Generals/Code/Main/WinMain.cpp

Core/GameEngine/Include/Common/GameMemory.h

Core/GameEngine/Source/Common/System/GameMemory.cpp

Core/GameEngine/Source/Common/System/MiniDumper.cpp

slurmlord · 2025-10-14T19:54:04Z

Pushed a commit attempting to address most of the comments in the previous review:

Made TheMiniDumper a pointer, declared in MiniDumper.cpp
Moved extern TheMiniDumper to MiniDumper.h
Moved MiniDumper ctor body to MiniDumper.cpp
(Attempted to) find and correct all the stylistically misplaced "{"
Added enum for dump type
Fall back to DUMP_TYPE_FULL when DUMP_TYPE_GAMEMEMORY is requested and game memory implementation is turned off
Removed cmake feature dependency on RTS_GAMEMEMORY_ENABLE
Removed thread Id from dump file name
Replace hard-coded list of executable names to match with GetModuleFileNameW
(Attempted to) clean up log format
Added Enum values for dump thread exit codes
Removed MiniDumper::m_endRangeIter

In addition:

Added new static init/shutdown methods, using placement new on the process heap
Prefixed win32 api function calls with global namespace ::
Added asserts for dumping thread active state

Remaining:
Merge with #1066 when merged.

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Skyaero42 · 2025-11-01T12:45:39Z

I think this should be enabled by default for now and included in our weekly pre-releases and legi.cc/patch distributions. The patch still crashes now and then without being able to find a cause (replays are send to us, but there is no crash to be found). For example, Legi (and only Legi) crashed yesterday on stream while being on the patch. No cause was found.

Players on the patch (with crashdump on) can send us the crashdumps and thus help finding where the issues/instabilities are.

slurmlord · 2025-11-04T19:48:09Z

I think this should be enabled by default for now and included in our weekly pre-releases and legi.cc/patch distributions. The patch still crashes now and then without being able to find a cause (replays are send to us, but there is no crash to be found). For example, Legi (and only Legi) crashed yesterday on stream while being on the patch. No cause was found.

Players on the patch (with crashdump on) can send us the crashdumps and thus help finding where the issues/instabilities are.

Agreed, that's the main use case I was hoping this could address. Once this is merged, the remaining hurdle would be to get the weekly builds to enable the Vc6FullDebug cmake option.

Core/Libraries/Source/WWVegas/WWLib/DbgHelpLoader.h

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Core/Libraries/Source/WWVegas/WWLib/DbgHelpLoader.h

xezon

I think the cleanup in MiniDumper class needs another look. Make it simple and robust.

Core/GameEngine/Source/Common/System/MiniDumper.cpp

cmake/config-memory.cmake

Core/GameEngine/Source/Common/System/MiniDumper.cpp

xezon

Code looks good to me. A few minor comments that could be looked into.

Core/GameEngine/Source/Common/System/Debug.cpp

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Core/GameEngine/Source/Common/System/Debug.cpp

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Caball009 · 2025-11-21T20:16:54Z

Core/Libraries/Source/WWVegas/WWLib/DbgHelpLoader_minidump.h

+	MiniDumpFilterWriteCombinedMemory = 0x01000000,
+	MiniDumpValidTypeFlags = 0x01ffffff,
+	MiniDumpNoIgnoreInaccessibleMemory = 0x02000000,
+	MiniDumpValidTypeFlagsEx = 0x03ffffff,


Is this value correct?

From https://learn.microsoft.com/en-us/windows/win32/api/minidumpapiset/ne-minidumpapiset-minidump_type I'd expect this to be 0x02000001.

There does seem to be some discrepancy between the doc and the SDK headers. The SDK headers defines it explicitly as 0x03ffffff, you can see this in the original file at C:\Program Files (x86)\Windows Kits\10\Include\10.0.26100.0\um\minidumpapiset.h
My money would be on the docs being wrong, and as we don't use this value directly it should be fine for our purpose. Does that seem reasonable?
And just out of curiosity, how did you notice this?

Include\10.0.26100.0\

I only have two older versions and MiniDumpValidTypeFlags is the last value for both versions.

Does that seem reasonable?

Yeah, seems reasonable to me.

And just out of curiosity, how did you notice this?

I was wondering why we're defining all these structs / enums, so I happened to check a couple or maybe just this one to see where they were coming from.

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Core/GameEngine/Include/Common/MiniDumper.h

Core/GameEngine/Source/Common/System/MiniDumper.cpp

EndsWithCaseInsensitive needs looking

slurmlord · 2025-11-26T18:35:12Z

Rebased on main.

Added use of endsWithNoCase from refactor(string): Implement generic startsWith, endsWith functions for C strings #1898 in MiniDumper::ShouldWriteDataSegsForModule.
Reordered wanted_modules in MiniDumper::ShouldWriteDataSegsForModule to static constexpr const.

Core/GameEngine/Source/Common/System/MiniDumper.cpp

slurmlord · 2025-11-27T17:50:57Z

Refactored a bit, introducing an enum for the memory range phases and modified the output variable of the callback routine so that our callback can always return TRUE.

xezon

To me it looks like we can get rid of the concept of m_dumpObjectsSubState by caching the actual pointers to memory structures and keep iterating until they reach NULL and then go to the next until all lists reached the end.

Performance wise it is probably not the biggest of deals but it should make the logic cleaner.

Core/GameEngine/Source/Common/System/MiniDumper.cpp

slurmlord · 2025-11-27T22:36:01Z

Got rid of the m_dumpObjectsSubState and cached the various objects that's being iterated. This also reduced the changes required in GameMemory. Looks better!

xezon

This looks much cleaner indeed.

Core/GameEngine/Include/Common/MiniDumper.h

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Core/GameEngine/Include/Common/MiniDumper.h

Core/GameEngine/Source/Common/System/MiniDumper.cpp

xezon

A bunch more small comments.

Core/GameEngine/Include/Common/MiniDumper.h

Core/GameEngine/Source/Common/System/MiniDumper.cpp

Core/GameEngine/Include/Common/GameMemory.h

Core/GameEngine/Source/Common/System/GameMemory.cpp

Core/GameEngine/Source/Common/System/MiniDumper.cpp

…entations to cpp

Core/GameEngine/Include/Common/GameMemory.h

Core/GameEngine/Source/Common/System/GameMemory.cpp

Core/GameEngine/Source/Common/System/MiniDumper.cpp

xezon · 2025-12-03T18:30:06Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

 				m_currentAllocator = TheDynamicMemoryAllocator;
-				if (m_currentAllocator)
-					m_currentSingleBlock = m_currentAllocator->getFirstRawBlock();
+				MoveToNextAllocatorWithRawBlocks();


I think this is fine, but I was just wondering if the TheDynamicMemoryAllocator can have an iterator like you added for TheMemoryPoolFactory?

Basically have a MemoryPoolIterator and DynamicMemoryIterator and then use them the same way. Or is there a specific reason why Memory Pool needed the iterator while Dynamic Memory did not?

xezon · 2025-12-03T18:42:03Z

Core/GameEngine/Include/Common/GameMemory.h

 #endif

+#ifdef RTS_ENABLE_CRASHDUMP
+class AllocationRangeIterator


Perhaps also put "MemoryPool" into its name.

Alternatively, you could also make this class a member of "MemoryPool" and then simply name it "Iterator".

It then would become MemoryPool::Iterator

Unless there is a reason for it to be standalone or it is too much work to touch.

xezon · 2025-12-03T18:42:19Z

Core/GameEngine/Include/Common/GameMemory.h

+
+	AllocationRangeIterator();
+	AllocationRangeIterator(const MemoryPoolFactory* factory);
+	AllocationRangeIterator(MemoryPool& pool, MemoryPoolBlob& blob);


This constructor is redundant and unused. Can remove.

xezon · 2025-12-03T18:45:16Z

Core/GameEngine/Include/Common/GameMemory.h

+
+	void updateRange();
+	void moveToNextBlob();
+	const MemoryPoolFactory* m_factory;


This member looks not necessary

xezon · 2025-12-03T18:47:55Z

Core/GameEngine/Source/Common/System/GameMemory.cpp

+	m_currentPool = NULL;
+	m_currentBlobInPool = NULL;
+	m_factory = NULL;
+	m_range = MemoryPoolAllocatedRange();


This means m_range is still uninitialized, because MemoryPoolAllocatedRange has no initializer.

Are the constructors missing a call to updateRange() down here? It looks like the very first iterator is malformed.

xezon · 2025-12-03T18:57:32Z

Core/GameEngine/Source/Common/System/GameMemory.cpp

+	m_currentBlobInPool->fillAllocatedRange(m_range);
+}
+
+void AllocationRangeIterator::moveToNextBlob()


This function looks over-complicated to the eye. Maybe. I asked Chat to simplify it and it claims that the following is logically identical

void AllocationRangeIterator::moveToNextBlob() { if (m_currentBlobInPool) m_currentBlobInPool = m_currentBlobInPool->getNextInList(); while (!m_currentBlobInPool && m_currentPool) { m_currentPool = m_currentPool->getNextPoolInList(); if (m_currentPool) m_currentBlobInPool = m_currentPool->m_firstBlob; } }

xezon · 2025-12-03T19:04:07Z

Core/GameEngine/Source/Common/System/GameMemory.cpp

+	{
+		m_range.allocationAddr = nullptr;
+		m_range.allocationSize = 0;
+		return;


Maybe use else {} instead of return.

xezon · 2025-12-03T19:08:25Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+	}
+}
+
+void MiniDumper::MoveToNextSingleBlock()


MoveToNextAllocatorWithRawBlocks and MoveToNextSingleBlock can likely be one function like how you did it in the Memory Pool Iterator.

slurmlord · 2025-12-06T10:43:16Z

I'm considering to reduce the scope of this PR a bit to make it easier to review etc.
The proposal is to remove the GameMemory dump mode, and only have Small and Full in this PR. GameMemory can be added in a subsequent PR if we see that the Full dumps are required often and are too large.
The benefits of this reduction in scope are:

No changes required in GameMemory
Less code in MiniDumper (everything wrapped in #ifndef DISABLE_GAMEMEMORY)
Less complexity and chance for things to go wrong
Easier to review
Gives us insights in how often the crashdumps are actually useful and if it's worth investing in the GameMemory/extended type dumps before going to full CrashPad or similar solutions.

Does this seem like a reasonable path forward?

xezon · 2025-12-06T12:27:41Z

You can do that if you like. I think it would be possible to finalize this change as is as well.

xezon

Looking good.

xezon · 2025-12-06T15:16:16Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

 void MiniDumper::CreateMiniDump(DumpType dumpType)
 {
-	// Create a unique dump file name, using the path from m_dumpDir, in m_dumpFile
+	// Create a unique dump file name, using the path from m_dumpDir, in m_dumpFile 


Trailing space :)

OmniBlade approved these changes Sep 23, 2025

View reviewed changes

xezon added Major Severity: Minor < Major < Critical < Blocker Gen Relates to Generals ZH Relates to Zero Hour Debug Is mostly debug functionality System Is Systems related labels Oct 8, 2025

xezon reviewed Oct 9, 2025

View reviewed changes

xezon reviewed Oct 16, 2025

View reviewed changes

Core/GameEngine/Source/Common/System/MiniDumper.cpp Show resolved Hide resolved

Core/GameEngine/Source/Common/System/MiniDumper.cpp Outdated Show resolved Hide resolved

Core/GameEngine/Source/Common/System/MiniDumper.cpp Outdated Show resolved Hide resolved

xezon added the Approved Pull Request was approved label Oct 21, 2025

slurmlord force-pushed the feature-crashdump branch from 7bb8e8c to 90a0b49 Compare November 4, 2025 19:43

slurmlord mentioned this pull request Nov 5, 2025

build(weekly): Enable complete debug symbols for VC6 weekly builds #1803

Merged

xezon reviewed Nov 5, 2025

View reviewed changes

Core/Libraries/Source/WWVegas/WWLib/DbgHelpLoader.h Outdated Show resolved Hide resolved

Core/GameEngine/Source/Common/System/MiniDumper.cpp Outdated Show resolved Hide resolved

slurmlord commented Nov 12, 2025

View reviewed changes

Core/Libraries/Source/WWVegas/WWLib/DbgHelpLoader.h Show resolved Hide resolved

xezon reviewed Nov 16, 2025

View reviewed changes

xezon previously approved these changes Nov 19, 2025

View reviewed changes

xezon reviewed Nov 21, 2025

View reviewed changes

Core/GameEngine/Source/Common/System/Debug.cpp Outdated Show resolved Hide resolved

Core/GameEngine/Source/Common/System/Debug.cpp Outdated Show resolved Hide resolved

Core/GameEngine/Source/Common/System/MiniDumper.cpp Outdated Show resolved Hide resolved

Caball009 reviewed Nov 21, 2025

View reviewed changes

xezon reviewed Nov 22, 2025

View reviewed changes

Core/GameEngine/Source/Common/System/MiniDumper.cpp Outdated Show resolved Hide resolved

Core/GameEngine/Source/Common/System/MiniDumper.cpp Outdated Show resolved Hide resolved

slurmlord mentioned this pull request Nov 23, 2025

refactor(string): Implement generic startsWith, endsWith functions for C strings #1898

Merged

feat(crashdump): Add crash dump functionality

a241b80

slurmlord force-pushed the feature-crashdump branch from 9cbf93d to a241b80 Compare November 26, 2025 18:31

xezon reviewed Nov 26, 2025

View reviewed changes

Refactor CallbackInternal and DumpMemoryObjects

5e543ba

xezon reviewed Nov 27, 2025

View reviewed changes

Improve iteration in DumpMemoryObjects

7399cce

xezon reviewed Nov 27, 2025

View reviewed changes

Make state enums private, init all meory objs in DumpObjectsState_Begin

7cc9b3c

xezon reviewed Nov 28, 2025

View reviewed changes

Include enum name in enum values, move AllocationRangeIterator implem…

bcb86fe

…entations to cpp

xezon reviewed Dec 1, 2025

View reviewed changes

This was referenced Dec 2, 2025

[WIP] Review PR #1594 for code style consistency slurmlord/GeneralsGameCode#3

Closed

[WIP] Review code style and feedback consistency for PR #1594 slurmlord/GeneralsGameCode#4

Closed

Simplify allocator and raw block navigation

c0f1045

xezon reviewed Dec 3, 2025

View reviewed changes

Remove integration with GameMemory, offer Small and Full dump types

810371e

xezon approved these changes Dec 6, 2025

View reviewed changes

Already enough trailing whitespace, no need to add more

7ce2363

xezon added this to the Important features milestone Dec 7, 2025

xezon merged commit f4608df into TheSuperHackers:main Dec 7, 2025
23 checks passed

xezon mentioned this pull request Dec 7, 2025

refactor: modernize NULL to nullptr #1938

Open

feat(crashdump): Add crash dump functionality for fatal errors #1594

feat(crashdump): Add crash dump functionality for fatal errors #1594

Conversation

slurmlord commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Small dumps

Full memory

Storage Location

Integration points

Limitations

Uh oh!

OmniBlade left a comment

Choose a reason for hiding this comment

Uh oh!

slurmlord commented Oct 9, 2025

Uh oh!

xezon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

slurmlord commented Oct 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Skyaero42 commented Nov 1, 2025

Uh oh!

slurmlord commented Nov 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xezon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xezon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

slurmlord commented Nov 26, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

slurmlord commented Sep 18, 2025 •

edited

Loading