Skip to content

A non-UTF-8 argument reaches the program as bytes and opens its file (#1798) - #1799

Merged
Lorenzobattistela merged 4 commits into
bendlang:mainfrom
viridale:fix-1798-args-bytes
Oct 21, 2026
Merged

Lorenzobattistela merged 4 commits into
bendlang:mainfrom
viridale:fix-1798-args-bytes

Conversation

@viridale

@viridale viridale commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1798.

What changes

This takes the issue's first shape: byte twins, like the ones TCP, UDP and File.read already have. String stays text (#1248), and bytes cross as List<&2, U32>.

  • IO.args_bytes() -> IO(List<List<&2, U32>>) gives the entries IO.args gives, as bytes, with no UTF-8 decoding. In C it is IO.args's loop with io_list where IO.args uses io_str.
  • File.open_bytes(path: List<&2, U32>, mode: String) opens the path's bytes as they are. A value past 255 fails with EINVAL, as TCP.send_bytes does, and a 0 fails with EILSEQ, as a NUL in File.open's path does. It shares File.open's code in both lanes: io_cbuf with CID(Con) in C, and io_unlist in JS.

Each twin lives in its sibling's effect file (args.c/.js, file_open.c/.js) under its own #ifdef CID(...), as file_read.c does.

The JS lane. Node and Bun decode process.argv before the program runs, so there IO.args_bytes is each argument's UTF-8: a byte that isn't UTF-8 arrives as U+FFFD's UTF-8, EF BF BD. The doc comment in base.bend says so. The C lane keeps every byte.

IO.get_env and Process.run's program and arguments also take Strings, as do the other path effects. This PR leaves them alone: each could get a twin the same way if one is wanted.

The issue's program

Rewritten for the twins:

import Base

def hex(xs: List<&2, U32>) -> String:
  match xs:
    case Nil{}: ""
    case x <> rest: " " ++ U32.show(x) ++ hex(rest)

def show_open(r: Result<&1, &1, U32 & String, File>) -> IO(Unit):
  match r:
    case Done{file}:
      do IO<Unit>:
        IO.print(("open: ok" : String))
        File.close(file)
    case Fail{(code, msg)}:
      IO.print(("open: " ++ msg : String))

def report(args: List<List<&2, U32>>) -> IO(Unit):
  match args:
    case Nil{}: IO.print("no argument")
    case prog <> rest:
      match rest:
        case Nil{}: IO.print("no argument")
        case +name <> more:
          do IO<Unit>:
            IO.print(("bytes:" ++ hex(name) : String))
            r : Result<&1, &1, U32 & String, File> <- File.open_bytes(name, "r")
            show_open(r)

def main() -> IO(Unit):
  do IO<Unit>:
    args : List<List<&2, U32>> <- IO.args_bytes()
    report(args)

The C binary on Linux, with caf\xe9.txt present:

$ ./args "$(printf 'caf\xe9.txt')"
bytes: 99 97 102 233 46 116 120 116
open: ok
$ ./args "$(printf 'caf\xe8.txt')"
bytes: 99 97 102 232 46 116 120 116
open: No such file or directory

In the JS lanes (bun, node, bend args.bend) both print 99 97 102 239 191 189 …. A UTF-8 name (caf\xc3\xa9.txt) gives 99 97 102 195 169 … and open: ok in every lane.

Tests

  • tests/io/args_bytes.bend is args.bend for the twin: every lane sees its program, as bytes, and no argument. The harness passes no arguments, so the gate can't reach the Latin-1 argv case; the C run above checks it by hand.
  • tests/io/file_open_bytes.bend: UTF-8 bytes open the file that File.open's String names; a 256 fails with EINVAL, and a 0 with EILSEQ.

Checks

Merged onto main e9055f7: gates/test.ts PASS 1696 / 1696, gates/repo.ts 54 / 54, gates/ping.ts 73 / 73, gates/safe.ts 0 false rejects, and gates/perf.ts the same cells as main. comp.ts, bend.ts and main.ts don't change.

guide/GUIDE.md is left for a human. If you want the twin mentioned there, one sentence after the IO.args one could read: "IO.args_bytes answers the same entries as bytes, for a name that isn't UTF-8, and File.open_bytes opens one."

…endlang#1798)

IO.args decodes argv as UTF-8, so a Latin-1 name such as "caf" E9 arrived
as U+FFFD, could not be opened, and could not be told from "caf" E8.

Two byte twins, as TCP, UDP and File.read have: IO.args_bytes gives each
argument's bytes, and File.open_bytes opens a path given as bytes (a value
past 255 fails with EINVAL, a 0 with EILSEQ). Each lives in its sibling's
effect file under its own #ifdef CID. The JS lane reads the raw arguments
from /proc/self/cmdline on Linux or _NSGetArgv under Bun on macOS when an
argument holds a U+FFFD; node on macOS still sees EF BF BD.
Lorenzobattistela and others added 3 commits October 21, 2026 06:59
Drop io_args_raw (/proc/self/cmdline, _NSGetArgv through bun:ffi) and the
matching of decoded arguments against it: platform-specific, never reached
by a gate, and hiding its failures behind a catch. Node and Bun decode argv
before the program runs, so in JS a byte that isn't UTF-8 arrives as
EF BF BD; the C lane keeps every byte.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tin-1

The doc said only node on macOS loses a byte that isn't UTF-8; with no raw
argv in JS, every JS lane does. The test's Latin-1 case asserted what the
file system under /tmp does with a name that isn't UTF-8 (APFS refuses it,
ext4 takes it, HFS+ and utf8only ZFS differ), not what File.open_bytes does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Lorenzobattistela
Lorenzobattistela merged commit e7c3de9 into bendlang:main Oct 21, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

IO.args turns non-UTF-8 arguments into U+FFFD, so a program can't open a file it was given

2 participants