Introduction of frontmatter was a theoretical breaking change (via `TokenStream::from_str`)

[fmease]

Reproducer

• rustc a.rs --crate-type proc-macro --extern proc_macro where a.rs:

  #[proc_macro]
  pub fn ck(x: proc_macro::TokenStream) -> proc_macro::TokenStream {
      _ = "---".parse::<proc_macro::TokenStream>();
      x
  }

• rustc b.rs --crate-type lib --extern a -L. where b.rs:

  a::ck!();

In 1.87 and earlier, this used to compile just fine. However, since 1.88 (more precisely since PR #140035) this fails with:

error: invalid infostring for frontmatter
 --> b.rs:1:1
  |
1 | a::ck!();
  | ^^^^^^^^
  |
  = note: frontmatter infostrings must be a single identifier immediately following the opening
  = note: this error originates in the macro `a::ck` (in Nightly builds, run with -Z macro-backtrace for more info)

error: unclosed frontmatter
 --> b.rs:1:1
  |
1 | a::ck!();
  | ^^^^^^^^
  |
note: frontmatter opening here was not closed
 --> b.rs:1:1
  |
1 | a::ck!();
  | ^^^^^^^^
  = note: this error originates in the macro `a::ck` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0658]: frontmatters are experimental
 --> b.rs:1:1
  |
1 | a::ck!();
  | ^^^^^^^^
  |
  = note: see issue #136889 <https://github.com/rust-lang/rust/issues/136889> for more information
  = note: this error originates in the macro `a::ck` (in Nightly builds, run with -Z macro-backtrace for more info)

[fmease]

I don't know why I didn't think of this when writing #137193 (comment), jeez! Indeed, "frontmatter should be unobservable to proc macros" but you don't achieve that by stripping/lexing it.

[fmease]

We do strip shebangs in TokenStream::from_str but for frontmatter I guess we should only do that starting in the next edition if ever?

Unless this is classified as WONTFIX / acceptable breakage (tho as a matter of fact, frontmatter syntax is still unstable, so it may very well still change which is concerning), the immediate fix would be to stop stripping/lexing frontmatter in TokenStream::from_str (should be as easy as passing FrontmatterAllowed::No to the relevant tokenize if possible).

[bjorn3]

Do we allow shebangs in TokenStream::from_str? IMO frontmatter should be handled the same as shebangs in TokenStream::from_str. Either both should be allowed or both should be denied.

[fmease]

We do as I said:

We do strip shebangs in TokenStream::from_str but for frontmatter I guess we should only do that starting in the next edition if ever?

[jieyouxu]

cc @fee1-dead
(as I won't have bandwidth to send a fix during weekday)