[rtext] Integer Overflow in `LoadFontData()` leads to Segfault in `GenImageFontAtlas()`

[oneafter]

Description

We discovered a Denial of Service (DoS) vulnerability in Raylib. The vulnerability is triggered by an Integer Overflow in LoadFontData (src/rtext.c), where a negative value is passed to calloc.

With ASAN: The tool intercepts the invalid calloc parameter (-14624 * 2) and aborts.

In Release Build (No ASAN): The invalid allocation propagates downstream, eventually causing a Segmentation Fault in GenImageFontAtlas.

Environment

• OS: Linux x86_64
• Complier: Clang
• Build Configure: Release
• Affected Version: master branch

Vulnerability Details

• Target: Raylib
• Vulnerability Type: Integer Overflow / Argument Injection to calloc
• Root Cause Location: src/raylib/src/rtext.c:747 (in LoadFontData)
• Crash Location: GenImageFontAtlas
• Root Cause Analysis: The ASAN report identifies a calloc-overflow at rtext.c:747.

calloc parameters overflow: count * size (-14624 * 2)

This suggests that the font parser reads a corrupted or malicious value from the font file, interpreting it as -14624. This negative value is passed to calloc.

In a Release build, the standard libc implementation likely handles this calloc request by returning NULL (or succeeding if interpreted as a large unsigned value but failing later). Raylib appears to proceed with this invalid state, causing GenImageFontAtlas to dereference an invalid pointer or access out-of-bounds memory, resulting in the SIGSEGV observed in GDB.

Reproduce

1. Compile the raylib font test harness with AddressSanitizer enabled (-fsanitize=address -g)
2. Run the fuzzer harness with the attached repro input:

./harness < repro

ASAN report

==10067==ERROR: AddressSanitizer: calloc parameters overflow: count * size (-14624 * 2) cannot be represented in type size_t (thread T0)
    #0 0x55d32390573d in calloc (/src/raylib/build_afl/harness_raylib_font+0x12873d) (BuildId: e4cb6e6c9d2ceaf7402b83966bc71e3ad01f720c)
    #1 0x55d323955cd1 in LoadFontData /src/raylib/src/rtext.c:747:37
    #2 0x55d323946f52 in main /src/raylib/harness_raylib_font.c:39:29
    #3 0x7efef36351c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)
    #4 0x7efef363528a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e)
    #5 0x55d323865724 in _start (/src/raylib/build_afl/harness_raylib_font+0x88724) (BuildId: e4cb6e6c9d2ceaf7402b83966bc71e3ad01f720c)

==10067==HINT: if you don't care about these errors you may set allocator_may_return_null=1
SUMMARY: AddressSanitizer: calloc-overflow (/src/raylib/build_afl/harness_raylib_font+0x12873d) (BuildId: e4cb6e6c9d2ceaf7402b83966bc71e3ad01f720c) in calloc
==10067==ABORTING

3. Recompile the raylib and font test harness without AddressSanitizer
4. Run the fuzzer harness with the attached repro input:

gdb --args ./harness_raylib_font
run < repro
bt

GDB report:

GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./harness_raylib_font...
(gdb) run < id:000007,sig:06,src:000002,time:22654,execs:18929,op:quick,pos:62972
Starting program: /src/repro/raylib/harness_raylib_font < id:000007,sig:06,src:000002,time:22654,execs:18929,op:quick,pos:62972
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x0000562a516fab36 in GenImageFontAtlas ()
(gdb) bt
#0  0x0000562a516fab36 in GenImageFontAtlas ()
#1  0x0000562a516f3c61 in main () at /src/repro/raylib/harness_raylib_font.c:49
(gdb) bt
#0  0x0000562a516fab36 in GenImageFontAtlas ()
#1  0x0000562a516f3c61 in main () at /src/repro/raylib/harness_raylib_font.c:49
(gdb) bt
#0  0x0000562a516fab36 in GenImageFontAtlas ()
#1  0x0000562a516f3c61 in main () at /src/repro/raylib/harness_raylib_font.c:49

[raysan5]

@oneafter I appreciate all those reports that you are sending but are you considering sending patches to fix those issues?

[oneafter]

@raysan5 Yes, I would be happy to contribute patches. I'm preparing a fix for this crash by adding boundary checks.

Since I am new to the codebase, I will submit the PR for review and would appreciate any feedback to ensure it matches the project's coding style.

[raysan5]

@oneafter thanks for your answer, raylib project prioritize code simplicity as much as possible. Also note that raylib implements CodeQL scanning to detect potential security issues but only critical ones are addressed. At this moment there is only one critical to be addressed detected by CodeQL:

CodeQL reports results: https://github.com/raysan5/raylib/actions/runs/20458480923