LP reader: reduce memory use and speed up reading small files by DoumarCharles · Pull Request #3446 · ERGO-Code/HiGHS

DoumarCharles · 2026-10-19T18:42:09Z

Reading a small LP file allocates about 7 MB (peak about 5 MB) during Highs::readModel, whatever its size. This matters where the heap does not shrink, e.g. WebAssembly, or when many small models are read. Causes, all in highs/io/LpReader.cpp:

Lexer::fill grows its buffer by kChunkSize (1 MB) and zero-fills it on each refill, and again on each look-ahead past the end of the file.
zstr::ifstream is opened with its default buffers, two of 1 MB, also for plain files.
renderDiagnostics reopens the file even when there are no warnings.

Changes: fill reads 16 KB first and doubles up to kChunkSize, and stops refilling once it has reached the end of the file. zstr is opened with 64 KB buffers. renderDiagnostics returns early when there is nothing to render.

file	peak heap before → after	read time before → after
258 B	5131 → 155 KB	0.81–0.89 → 0.014–0.019 ms
32 KB	5214 → 262 KB	0.86–0.88 → 0.75–0.77 ms
350 KB	6185 → 1577 KB	8.8–9.0 → 7.8–8.2 ms
3.8 MB	13053 → 11542 KB	107–114 → 105–110 ms

(Linux, clang, Release. Heap counted by replacing operator new. Read time is the median per read over 3 rounds. Gzipped files behave the same.)

Testing: ctest 169/169, unit_tests 441/441. Old and new readers give identical messages, status and model on 364 files: 300 random files, plus edge cases such as empty and truncated files, CRLF, no final newline, parse errors, warnings, a 2 MB line, sizes of 1 byte either side of each buffer boundary, and gzipped files. The result is the same with -DLP_READER_CHUNK_SIZE=7 -D_GLIBCXX_ASSERTIONS.

@odow, since you wrote the new reader: does this fit what you had in mind?

Checklist

I have read the contributing guidelines
This PR targets the latest branch
Tests are passing
Documentation was updated where relevant
This PR is not primarily AI-generated (per the AI contributions policy in CONTRIBUTING.md)

Lexer::fill grew its buffer by kChunkSize (1 MB) on each refill and again on each look-ahead past the end of the file; it now reads 16 KB first, doubles up to kChunkSize, and stops refilling at the end of the file. zstr::ifstream is opened with 64 KB buffers instead of two of 1 MB, and renderDiagnostics no longer reopens the file when there is nothing to render. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

odow · 2026-10-19T19:45:51Z

Thanks! I'll take a look. I'll admit that I didn't benchmark small files. I was mainly focused on the performance of reading very big files.

codecov · 2026-10-19T19:51:50Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.19%. Comparing base (c884431) to head (bd2532f).
⚠️ Report is 85 commits behind head on latest.

Additional details and impacted files

@@            Coverage Diff             @@
##           latest    #3446      +/-   ##
==========================================
+ Coverage   75.11%   75.19%   +0.07%     
==========================================
  Files         460      460              
  Lines      112060   112317     +257     
  Branches    17769    17804      +35     
==========================================
+ Hits        84178    84456     +278     
+ Misses      27600    27573      -27     
- Partials      282      288       +6

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

DoumarCharles · 2026-10-19T23:20:51Z

Thanks @odow, your changes fix the issues I identified.

If you're after speed, two small changes in the lexer help:

Replace the std::deque in peek()/next() with a small ring buffer. Used as a queue, the deque allocates and frees a 512-byte block about every 10 tokens. The parser never looks more than four tokens ahead, so 8 fixed slots are enough, e.g. tokens_[(first_token_ + num_tokens_++) % kMaxLookahead] = lex();.
Special-case printable ASCII before calling std::isspace, e.g. if (c != ' ' && ((c > 0x20 && c < 0x7F) || !std::isspace(c))).

Together they gave about 10% on large files in my tests, and more on mid-sized ones, with peak memory unchanged. Let me know if you'd like me to code that up.

odow · 2026-10-20T02:31:29Z

This is a nice but small gain on latest:

Files at the sizes you asked for (500 repetitions for the first two, 25 for the rest):

┌─────────┬─────────┬─────────┬─────────┬───────────────────┬────────────────┐
│  File   │ v1.15.1 │ latest  │  HEAD   │ latest vs v1.15.1 │ HEAD vs latest │
├─────────┼─────────┼─────────┼─────────┼───────────────────┼────────────────┤
│ 10 KiB  │ 154 µs  │ 137 µs  │ 121 µs  │ 1.1x              │ 12% faster     │
├─────────┼─────────┼─────────┼─────────┼───────────────────┼────────────────┤
│ 100 KiB │ 1.57 ms │ 0.83 ms │ 0.78 ms │ 1.9x              │ 6% faster      │
├─────────┼─────────┼─────────┼─────────┼───────────────────┼────────────────┤
│ 1 MiB   │ 19.4 ms │ 8.1 ms  │ 7.7 ms  │ 2.4x              │ 5% faster      │
├─────────┼─────────┼─────────┼─────────┼───────────────────┼────────────────┤
│ 10 MiB  │ 240 ms  │ 85 ms   │ 81 ms   │ 2.8x              │ 5% faster      │
└─────────┴─────────┴─────────┴─────────┴───────────────────┴────────────────┘

Claude tells me the difference might be more in linux because macOS uses a different buffer strategy for std::deque.

DoumarCharles · 2026-10-20T04:41:19Z

Thanks @odow for coding it up. I measured on Linux with GCC’s libstdc++, where std::deque frees and reallocates a 512-byte block about every 10 tokens: on a 3.8 MB file the ring buffer removes about 90,000 allocations and 24% of instructions (~10% wall time; ~25% at 350 KB). Built against libc++ the deque reuses its blocks, so the gain drops to about 10% on small-to-mid files and is within noise on large ones, which matches your 5%. So the change is mostly a Linux/libstdc++ win.

DoumarCharles · 2026-10-20T16:04:35Z

Thanks @odow. I was making sure the ring buffer holds up on bad input, so I fed the reader some corrupted and binary input (zip archives, damaged gzip/zlib streams, random bytes). The ring buffer was fine, but I found three other cases that fail on the current head (36586b9). Here are test cases for them:

lp-file-format-corrupt-gzip: zstr recognises gzip from the first bytes, not the extension, and throws on the bad block. The exception escapes Highs::readModel and aborts the caller. This one doesn't seem to come from the rewrite: the old reader didn't catch it either.
lp-file-format-long-token-of-utf8-continuation-bytes: a token of 40 × 0x80 makes Parser::quote read before the start of the buffer (ASan reports a heap-buffer-overflow). Without ASan it may pass or crash, depending on what's in memory.
lp-file-format-corrupt-gzip-after-parse-error: the parser stops early on a parse error, but renderDiagnostics then reads the rest of the line, reaches the bad checksum, and the exception escapes the same way as in 1.

diff --git a/check/TestLpFileFormat.cpp b/check/TestLpFileFormat.cpp
index 6486ac3..953ba7a 100644
--- a/check/TestLpFileFormat.cpp
+++ b/check/TestLpFileFormat.cpp
@@ -1857,3 +1857,66 @@ TEST_CASE("lp-file-format-just-over-1mib", "[LpFileFormat]") {
   REQUIRE(sum_lower == double(num_row) * (num_row - 1) / 2);
   REQUIRE(lp.row_names_.back() == "c" + std::to_string(num_row - 1));
 }
+
+TEST_CASE("lp-file-format-corrupt-gzip", "[LpFileFormat]") {
+  // A gzip header followed by a deflate block of the reserved type. zstr
+  // detects the gzip header from the content, not the file extension, and
+  // throws when it decompresses the block; readModel must return an error
+  // rather than let the exception escape.
+  const std::string test_name = Catch::getResultCapture().getCurrentTestName();
+  std::string filename = test_name + ".lp";
+  const std::string content("\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x07", 11);
+  Highs highs;
+  std::string log;
+  REQUIRE(readLp(highs, filename, content, log) == HighsStatus::kError);
+#ifdef ZLIB_FOUND
+  REQUIRE(log.find("invalid block type") != std::string::npos);
+#endif
+}
+
+TEST_CASE("lp-file-format-long-token-of-utf8-continuation-bytes",
+          "[LpFileFormat]") {
+  // Quoting a token longer than 30 bytes backs up to the start of a UTF-8
+  // character. A token made only of continuation bytes has no such start, so
+  // this must stop at the beginning of the token rather than read before it.
+  const std::string test_name = Catch::getResultCapture().getCurrentTestName();
+  std::string filename = test_name + ".lp";
+  const std::string content = std::string(40, '\x80') + "\n";
+  Highs highs;
+  std::string log;
+  REQUIRE(readLp(highs, filename, content, log) == HighsStatus::kError);
+  REQUIRE(log.find("expected a section keyword") != std::string::npos);
+}
+
+TEST_CASE("lp-file-format-corrupt-gzip-after-parse-error", "[LpFileFormat]") {
+  // A parse error at the start of a long line, in a gzip file whose checksum
+  // is wrong. The parser stops before the end of the stream, but rendering
+  // the error reads the rest of the line, which reaches the bad checksum.
+  // The tokens are short, so the parser stops reading early. The data is in
+  // stored (uncompressed) deflate blocks of 60000 bytes.
+  const std::string test_name = Catch::getResultCapture().getCurrentTestName();
+  std::string filename = test_name + ".lp";
+  std::string content("\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03", 10);
+  const size_t kBlock = 60000;
+  const size_t kNumBlock = 4;
+  for (size_t b = 0;

odow · 2026-10-20T22:33:01Z

I've fixed the first and second ones. it looks like you copied the third test only partially.

DoumarCharles · 2026-10-20T23:46:37Z

Thanks for all your wonderful fixes @odow.

... it looks like you copied the third test only partially.

lp-file-format-corrupt-gzip-after-parse-error: the parser stops early on a parse error, but renderDiagnostics then reads the rest of the line, reaches the bad checksum, and the exception escapes the same way as in 1.

diff --git a/check/TestLpFileFormat.cpp b/check/TestLpFileFormat.cpp
index 6486ac3..953ba7a 100644
--- a/check/TestLpFileFormat.cpp
+++ b/check/TestLpFileFormat.cpp
@@ -1857,3 +1857,66 @@ TEST_CASE("lp-file-format-just-over-1mib", "[LpFileFormat]") {
   REQUIRE(sum_lower == double(num_row) * (num_row - 1) / 2);
   REQUIRE(lp.row_names_.back() == "c" + std::to_string(num_row - 1));
 }
+
+TEST_CASE("lp-file-format-corrupt-gzip", "[LpFileFormat]") {
+  // A gzip header followed by a deflate block of the reserved type. zstr
+  // detects the gzip header from the content, not the file extension, and
+  // throws when it decompresses the block; readModel must return an error
+  // rather than let the exception escape.
+  const std::string test_name = Catch::getResultCapture().getCurrentTestName();
+  std::string filename = test_name + ".lp";
+  const std::string content("\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03\x07", 11);
+  Highs highs;
+  std::string log;
+  REQUIRE(readLp(highs, filename, content, log) == HighsStatus::kError);
+#ifdef ZLIB_FOUND
+  REQUIRE(log.find("invalid block type") != std::string::npos);
+#endif
+}
+
+TEST_CASE("lp-file-format-long-token-of-utf8-continuation-bytes",
+          "[LpFileFormat]") {
+  // Quoting a token longer than 30 bytes backs up to the start of a UTF-8
+  // character. A token made only of continuation bytes has no such start, so
+  // this must stop at the beginning of the token rather than read before it.
+  const std::string test_name = Catch::getResultCapture().getCurrentTestName();
+  std::string filename = test_name + ".lp";
+  const std::string content = std::string(40, '\x80') + "\n";
+  Highs highs;
+  std::string log;
+  REQUIRE(readLp(highs, filename, content, log) == HighsStatus::kError);
+  REQUIRE(log.find("expected a section keyword") != std::string::npos);
+}
+
+TEST_CASE("lp-file-format-corrupt-gzip-after-parse-error", "[LpFileFormat]") {
+  // A parse error at the start of a long line, in a gzip file whose checksum
+  // is wrong. The parser stops before the end of the stream, but rendering
+  // the error reads the rest of the line, which reaches the bad checksum.
+  // The tokens are short, so the parser stops reading early. The data is in
+  // stored (uncompressed) deflate blocks of 60000 bytes.
+  const std::string test_name = Catch::getResultCapture().getCurrentTestName();
+  std::string filename = test_name + ".lp";
+  std::string content("\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03", 10);
+  const size_t kBlock = 60000;
+  const size_t kNumBlock = 4;
+  for (size_t b = 0; b < kNumBlock; b++) {
+    std::string data;
+    while (data.size() < kBlock) data += "x ";
+    if (b == 0) data.replace(0, 4, "bad ");
+    if (b + 1 == kNumBlock) data.back() = '\n';
+    content += char(b + 1 == kNumBlock ? 1 : 0);  // BFINAL, BTYPE = stored
+    content += char(kBlock & 0xFF);
+    content += char(kBlock >> 8);
+    content += char(~kBlock & 0xFF);
+    content += char((~kBlock >> 8) & 0xFF);
+    content += data;
+  }
+  content += std::string(8, '\0');  // wrong CRC32 and length
+  Highs highs;
+  std::string log;
+  REQUIRE(readLp(highs, filename, content, log) == HighsStatus::kError);
+#ifdef ZLIB_FOUND
+  REQUIRE(log.find("expected a section keyword, found `bad`") !=
+          std::string::npos);
+#endif
+}

Remove LP_READER_CHUNK_SIZE and simplify the Lexer::fill implementation

644ad0e

odow added 2 commits October 20, 2026 14:59

Special case isspace in Lexer::lex to recognize ASCII characters

58c15e4

Replace std::deque<Token> tokens_ with a fixed-size buffer

36586b9

odow approved these changes Oct 20, 2026

View reviewed changes

Fix reading corrupted files

85d3efc

jajhall self-assigned this Oct 21, 2026

jajhall added Code-quality LP file reader labels Oct 21, 2026

Fix printing an error diagnostic when the line is corrupt

bd2532f

jajhall merged commit 438c326 into ERGO-Code:latest Oct 28, 2026
392 checks passed

Navigation Menu

Repository navigation

LP reader: reduce memory use and speed up reading small files - #3446

LP reader: reduce memory use and speed up reading small files#3446
jajhall merged 6 commits into
ERGO-Code:latestfrom
DoumarCharles:lp-reader-buffers

DoumarCharles commented Oct 19, 2026

Uh oh!

odow commented Oct 19, 2026

Uh oh!

codecov Bot commented Oct 19, 2026 •

edited

Loading

Uh oh!

DoumarCharles commented Oct 19, 2026

Uh oh!

odow commented Oct 20, 2026

Uh oh!

DoumarCharles commented Oct 20, 2026

Uh oh!

DoumarCharles commented Oct 20, 2026

Uh oh!

odow commented Oct 20, 2026

Uh oh!

DoumarCharles commented Oct 20, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

DoumarCharles commented Oct 19, 2026

Checklist

Uh oh!

odow commented Oct 19, 2026

Uh oh!

codecov Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

DoumarCharles commented Oct 19, 2026

Uh oh!

odow commented Oct 20, 2026

Uh oh!

DoumarCharles commented Oct 20, 2026

Uh oh!

DoumarCharles commented Oct 20, 2026

Uh oh!

odow commented Oct 20, 2026

Uh oh!

DoumarCharles commented Oct 20, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Oct 19, 2026 •

edited

Loading