Repository navigation
A non-UTF-8 argument reaches the program as bytes and opens its file (#1798) - #1799
Merged
Merged
Conversation
…endlang#1798) IO.args decodes argv as UTF-8, so a Latin-1 name such as "caf" E9 arrived as U+FFFD, could not be opened, and could not be told from "caf" E8. Two byte twins, as TCP, UDP and File.read have: IO.args_bytes gives each argument's bytes, and File.open_bytes opens a path given as bytes (a value past 255 fails with EINVAL, a 0 with EILSEQ). Each lives in its sibling's effect file under its own #ifdef CID. The JS lane reads the raw arguments from /proc/self/cmdline on Linux or _NSGetArgv under Bun on macOS when an argument holds a U+FFFD; node on macOS still sees EF BF BD.
Drop io_args_raw (/proc/self/cmdline, _NSGetArgv through bun:ffi) and the matching of decoded arguments against it: platform-specific, never reached by a gate, and hiding its failures behind a catch. Node and Bun decode argv before the program runs, so in JS a byte that isn't UTF-8 arrives as EF BF BD; the C lane keeps every byte. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tin-1 The doc said only node on macOS loses a byte that isn't UTF-8; with no raw argv in JS, every JS lane does. The test's Latin-1 case asserted what the file system under /tmp does with a name that isn't UTF-8 (APFS refuses it, ext4 takes it, HFS+ and utf8only ZFS differ), not what File.open_bytes does. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Lorenzobattistela
approved these changes
Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1798.
What changes
This takes the issue's first shape: byte twins, like the ones TCP, UDP and
File.readalready have.Stringstays text (#1248), and bytes cross asList<&2, U32>.IO.args_bytes() -> IO(List<List<&2, U32>>)gives the entriesIO.argsgives, as bytes, with no UTF-8 decoding. In C it isIO.args's loop withio_listwhereIO.argsusesio_str.File.open_bytes(path: List<&2, U32>, mode: String)opens the path's bytes as they are. A value past 255 fails with EINVAL, asTCP.send_bytesdoes, and a 0 fails with EILSEQ, as a NUL inFile.open's path does. It sharesFile.open's code in both lanes:io_cbufwithCID(Con)in C, andio_unlistin JS.Each twin lives in its sibling's effect file (
args.c/.js,file_open.c/.js) under its own#ifdef CID(...), asfile_read.cdoes.The JS lane. Node and Bun decode
process.argvbefore the program runs, so thereIO.args_bytesis each argument's UTF-8: a byte that isn't UTF-8 arrives as U+FFFD's UTF-8, EF BF BD. The doc comment inbase.bendsays so. The C lane keeps every byte.IO.get_envandProcess.run's program and arguments also take Strings, as do the other path effects. This PR leaves them alone: each could get a twin the same way if one is wanted.The issue's program
Rewritten for the twins:
The C binary on Linux, with
caf\xe9.txtpresent:In the JS lanes (
bun,node,bend args.bend) both print99 97 102 239 191 189 …. A UTF-8 name (caf\xc3\xa9.txt) gives99 97 102 195 169 …andopen: okin every lane.Tests
tests/io/args_bytes.bendisargs.bendfor the twin: every lane sees its program, as bytes, and no argument. The harness passes no arguments, so the gate can't reach the Latin-1 argv case; the C run above checks it by hand.tests/io/file_open_bytes.bend: UTF-8 bytes open the file thatFile.open's String names; a 256 fails with EINVAL, and a 0 with EILSEQ.Checks
Merged onto main e9055f7:
gates/test.tsPASS 1696 / 1696,gates/repo.ts54 / 54,gates/ping.ts73 / 73,gates/safe.ts0 false rejects, andgates/perf.tsthe same cells as main.comp.ts,bend.tsandmain.tsdon't change.guide/GUIDE.mdis left for a human. If you want the twin mentioned there, one sentence after theIO.argsone could read: "IO.args_bytesanswers the same entries as bytes, for a name that isn't UTF-8, andFile.open_bytesopens one."