What the Autopilot sees

Good news! I fiddled around and got the data from the processing chain to the browser. For testing purposes I pass the captured screen and the region of interest to the websocket. Both images are encoded as base64 string. The browser receives these strings and updates the src-fields of the <img /> tags.

captured screen (top) and region of interest (bottom)

Because the data is sent to the whole websocket connection pool you can open a second browser window and see the same two images simultaneously. Pretty cool.

I have to do some further performance testing. The data transfered to the browser is kinda big. But it works for the beginning.