Get `examples/cross-browser.py` to run on top of Firefox’s BiDi implementation

[mathiasbynens]

With Firefox Nighty:

$ ./firefox --remote-debugging-port=9222
WebDriver BiDi listening on ws://localhost:9222
DevTools listening on ws://localhost:9222/devtools/browser/6cf65b2b-54e9-444e-a36e-0c04f12b47c9
…

$ # in another terminal / tab

$ PORT=9222 python3 examples/cross-browser.py
Traceback (most recent call last):
  File "~/projects/chromium-bidi/examples/cross-browser.py", line 150, in <module>
    result = loop.run_until_complete(main())
  File "~/homebrew/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "~/projects/chromium-bidi/examples/cross-browser.py", line 53, in main
    websocket = await get_websocket()
  File "~/projects/chromium-bidi/examples/cross-browser.py", line 28, in get_websocket
    return await websockets.connect(url)
  File "~/Library/Python/3.9/lib/python/site-packages/websockets/client.py", line 542, in __await_impl__
    await protocol.handshake(
  File "~/Library/Python/3.9/lib/python/site-packages/websockets/client.py", line 296, in handshake
    raise InvalidStatusCode(status_code)
websockets.exceptions.InvalidStatusCode: server rejected WebSocket connection: HTTP 200

We should figure out what’s going on here.

[whimboo]

@mathiasbynens this might be related to the host and origin header restrictions which have been discussed on the WebDriver BiDi issue tracker. There is w3c/webdriver-bidi#155 for getting it merged.

Basically WebSocket clients should send an empty Origin header. If that's not done by the client then the connection will be denied. Maybe enable DUMPIO=1 if it's available here as well similar to Puppeteer to see the Firefox stdout logging.

In case of a non-empty Origin header there is the --remote-allow-origins argument for Firefox which can be used to add specific origins to the allow list. Maybe your client sends a null origin? In such a case make use of --remote-allow-origins=null when starting Firefox.

Please let me know if this is the issue. If not the stdout logging from Firefox will be very helpful. Maybe you could even set the remote.log.level preference to Trace when pre-creating the Firefox profile.

[mathiasbynens]

I added some logging to my local cross-browser.py copy:

import logging

logging.basicConfig(
    format="%(message)s",
    level=logging.DEBUG,
)

Successful connection (with Chromium BiDi):

$ PORT=8080 python3 examples/cross-browser.py # with Chromium-BiDi
Using selector: KqueueSelector
client - state = CONNECTING
client - event = connection_made(<_SelectorSocketTransport fd=8 read=idle write=<idle, bufsize=0>>)
client > GET / HTTP/1.1
client > Headers([('Host', 'localhost:8080'), ('Upgrade', 'websocket'), ('Connection', 'Upgrade'), ('Sec-WebSocket-Key', 'jRrnZ+8O9mGjDrIelMc6Ng=='), ('Sec-WebSocket-Version', '13'), ('Sec-WebSocket-Extensions', 'permessage-deflate; client_max_window_bits'), ('User-Agent', 'Python/3.9 websockets/8.1')])
client - event = data_received(<129 bytes>)
client < HTTP/1.1 101 Switching Protocols
client < Headers([('Upgrade', 'websocket'), ('Connection', 'Upgrade'), ('Sec-WebSocket-Accept', '1pDVP6gqcpumF8buMDUfjACRZQY=')])
client - state = OPEN
[…]

Failed connection with Firefox:

$ PORT=9222 python3 examples/cross-browser.py # Firefox
Using selector: KqueueSelector
client - state = CONNECTING
client - event = connection_made(<_SelectorSocketTransport fd=8 read=idle write=<idle, bufsize=0>>)
client > GET / HTTP/1.1
client > Headers([('Host', 'localhost:9222'), ('Upgrade', 'websocket'), ('Connection', 'Upgrade'), ('Sec-WebSocket-Key', 'FPPJ9BiwgX6daTSuOpcviw=='), ('Sec-WebSocket-Version', '13'), ('Sec-WebSocket-Extensions', 'permessage-deflate; client_max_window_bits'), ('User-Agent', 'Python/3.9 websockets/8.1')])
client - event = data_received(<153 bytes>)
client < HTTP/1.1 200 OK
client < Headers([('content-type', 'text/html;charset=utf-8'), ('connection', 'close'), ('server', 'httpd.js'), ('date', 'Wed, 27 Apr 2022 07:16:09 GMT'), ('content-length', '361')])
client ! failing CONNECTING WebSocket connection with code 1006
client x half-closing TCP connection
client - event = data_received(<361 bytes>)
client - event = eof_received()
client - event = connection_lost(None)
client - state = CLOSED
client x code = 1006, reason = [no reason]
Traceback (most recent call last):
  File "~/projects/chromium-bidi/examples/cross-browser.py", line 157, in <module>
    result = loop.run_until_complete(main())
  File "~/homebrew/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "~/projects/chromium-bidi/examples/cross-browser.py", line 60, in main
    websocket = await get_websocket()
  File "~/projects/chromium-bidi/examples/cross-browser.py", line 35, in get_websocket
    return await websockets.connect(url)
  File "~/Library/Python/3.9/lib/python/site-packages/websockets/client.py", line 542, in __await_impl__
    await protocol.handshake(
  File "~/Library/Python/3.9/lib/python/site-packages/websockets/client.py", line 296, in handshake
    raise InvalidStatusCode(status_code)
websockets.exceptions.InvalidStatusCode: server rejected WebSocket connection: HTTP 200

The relevant difference is:

# Chromium:
client < HTTP/1.1 101 Switching Protocols
client < Headers([('Upgrade', 'websocket'), ('Connection', 'Upgrade'), ('Sec-WebSocket-Accept', '1pDVP6gqcpumF8buMDUfjACRZQY=')])
client - state = OPEN

# Firefox:
client < HTTP/1.1 200 OK
client < Headers([('content-type', 'text/html;charset=utf-8'), ('connection', 'close'), ('server', 'httpd.js'), ('date', 'Wed, 27 Apr 2022 07:16:09 GMT'), ('content-length', '361')])
client ! failing CONNECTING WebSocket connection with code 1006
client x half-closing TCP connection
client - event = data_received(<361 bytes>)
client - event = eof_received()
client - event = connection_lost(None)
client - state = CLOSED
client x code = 1006, reason = [no reason]

[whimboo]

Interesting. We seem to return the wrong HTTP header here. But so far none of the WebSocket clients that we used actually run into this problem. I'll check that.

Also I can see that you are using version 8.1 of websockets. But the latest release is 10.3. Maybe you could try again with that one in parallel?

[whimboo]

There is https://bugzilla.mozilla.org/show_bug.cgi?id=1766581 on file to get the HTTP response status fixed.

[whimboo]

@mathiasbynens actually we do send the correct response when the connection occurs for the correct client request which has to include the /session as URL. And that seems to be the problem:

https://github.com/GoogleChromeLabs/chromium-bidi/blob/main/examples/cross-browser.py#L27

[whimboo]

You can find the details in the WebDriver spec at https://w3c.github.io/webdriver-bidi/#transport under step 2. As such it looks like that Chrome requires a fix to not accept WebSocket connections for any URL (resource name).

[whimboo]

@mathiasbynens so I assume that this issue is fixed when using the right end-point? Or is something else not working yet?