How are these hints going to be generated?

[eqrion]

I seem to recall that we expect these hints to be generated using some sort of profiling step (but can't find that easily, I might be wrong here).

Are we thinking that toolchains should modify the wasm code to collect extra information? Or is the expectation that wasm engines should offer a mode that collects this information?

One crazy idea would be to modify the JS-API so that there's a boolean collectCompilationHints on CompileOptions when compiling a module that opts into profiling, then a Instance.getCompilationHints() which grabs the profiling information seen so far.

If you enable this, you'd expect for that module to run slower for that session as it'd stay in a baseline tier while collecting hints.

This could then be used for web developers to be able to collect real profiling information by opting some small random percentage of users into collecting this information then aggregating it over time.

This could also be possible if toolchains are instrumenting their wasm code to collect this data, I'm just not sure if that's the plan.

[dschuff]

I would think it could be done either way. LLVM has infrastructure to instrument code and feed that data back into the IR for subsequent compilations. In that mode a toolchain like emscripten would need a way to exfiltrate that data from the browser like what we have with wasm-split, and it would also of course need to propagate the IR-level data into the backend and make it wasm-level data, which we don't have hooked up in the wasm backend today. But I think the V8 folks have also discussed/prototyped a way to use the existing V8 profiler to collect e.g. tiering information, and you could then attach it directly to the binary. You wouldn't necessarily need to expose a JS API for a developer to do that locally, as it could be part of devtools and be controlled like the existing profiling infrastructure.
But the idea of doing it in the field with real user runs is interesting. Is there precedent on the web platform for collecting that kind of profiling information from the field?

[ecmziegler]

I don't think there's any reason to discourage toolchains from collecting data through instrumentation, but I also think that engines providing this capability is desirable, because we collect most of the data in the baseline for optimizations anyway. It also allows developers to optimize libraries for their use case without having access to the source code as recompilation is not necessary.

My original thinking was that a command line flag should be suffice as such profiles will often be collected during automated runs. But I recall that someone else also expressed interest in collecting data in the wild. So having both options might be a good idea.

I'm not sure how much the data is skewed by profiling the baseline tier only, but V8 would also prefer that because that's where the pre-existing instrumentation lives and because that's much closer to the Wasm binary. With the data we are planning to collect at the moment, the skew should be negligible to non-existent though.

In the future, we can think about getting some of that data through sampling-based profiling too which should be less noticable to users. Especially when you combine data from many users/runs, you could likely live with a low resolution. But that's only if in-the-wild collection becomes commonplace.

[eqrion]

But the idea of doing it in the field with real user runs is interesting. Is there precedent on the web platform for collecting that kind of profiling information from the field?

I don't know of any.

My original thinking was that a command line flag should be suffice as such profiles will often be collected during automated runs. But I recall that someone else also expressed interest in collecting data in the wild. So having both options might be a good idea.

Just to be clear, I think a DevTools/CLI flag would also be fine for collecting this data (or through toolchain instrumentation). I just had the idea that this could be exposed through a Web API as well.

I like the idea of developers aggregating this information from end-user executions as I feel like a big risk is that this data turns out to not be accurate due to the badly designed profiling runs.

But also, after thinking about this more, I think the bar to exposing this instrumentation through a Web API is a lot higher.

I think first we'd need to thoroughly define all the hints so that engines agree what we report. We'd also need to cover all the edge cases like trapping, and if we allow tier-up at all. And the last thing that would worry me would be if there's any privacy concerns. I don't think there would be, because this data could just be collecting through instrumentation anyway. But I've been surprised before.

So definitely not saying this proposal needs to do this, but it seems interesting to me.