CSC/ECE 517 Fall 2019 - M1951. Implement missing OffscreenCanvas APIs

Servo is an open source, Mozilla led project with the goal of creating a modern web engine with a highly parallel back-end. In this project, we are working on implementing several missing features related to drawing to offscreen HTML canvases.

Introduction

Servo

Servo is an open source layout engine for web content being developed by Mozilla and about 1000 contributors. It differs from previous engines by offering low level parallelism in rendering web components. Servo is designed to be cross platform with builds supporting Windows, Linux, Mac OS, and Android. Servo is written in Rust and tested using Javascript and HTML.

Rust

The bulk of Servo is written in Rust, an open source static compiled language that is largely similar to C++. Compared to similar languages, Rust adds fine grained control of coherency and memory layout, while offering safer threading and concurrency.

OffscreenCanvas API

The OffscreenCanvas API is a web API designed to render web content in a container not currently displayed on screen. These canvases can support a wide range of content. Our project deals with implementing and fixing some of the method implementations in Servo. As of writing, only Chrome 69+ and Opera 56+ fully support the OffscreenCanvas API. Some of the most notable uses of this API are prerendering media, such as an image, in an environment that is invisible to a user. This canvas could then be converted into a bitmap or a blob and cast onto another, visible canvas. Mozilla offers examples with code on the OffscreenCanvas API page.

Project Description

Getting servo set up and ready to run requires a few dependencies, namely in installation of the Rust Toolchain and Python 2.7. More specific setup instructions can be found in the Servo README.md. After dependencies are installed and the Servo repo cloned, the following script will build the dev version of Servo:

   cd servo
   ./mach build --dev

At this point tests can be run using the following command:

   ./mach run <path_to_test_dir>

To achieve our project goals we were assigned several issues to look at. These issues cover the high level steps to improve the support of the OffscreenCanvas API. These issues are listed here:

• Getting pixel data returns blank pixels
• Support drawing offscreen canvases onto other canvases
• Support converting offscreen canvas data into a blob
• Implement linking offscreen canvases to placeholder canvases

In addition to these issues we have also been directed to a couple more issues that also deal with the OffscreenCanvas API:

• Make offscreen canvas rendering context use offscreen canvas' size
• Add set_bitmap_dimensions method to CanvasState

In our instruction we were told to first pursue fix the image data issue, then add support for drawing offscreen canvases onto other canvases. As we began working towards getting oriented with the servo repo, we were diverted to work on the additional issues first, as they were largely foundation to improve the implementation of OffscreenCanvasAPI.

The issues resolved by our work on this project are bolded above.

OSS Progress

In each of these Github issues that we have worked on, we have a thread of our communications with the project owners. The initial pixel issue seems to have been resolved by another user. The bulk of our time in resolving the paramount issue has been spent working on the two issues that serve as prerequisites for drawing one offscreen canvas onto another canvas. Our first pull request addressed these issues and allowed us to get feedback on our work towards our primary goals. Our second pull request rebases our first and takes into account some of our feedback, in addition to finishing the refactoring issue.

For our second submission on this project, we have made changes that led to our pull request successfully being merged in Servo. During the next round of feedback, we will be focusing on continuing to implement OffscreenCanvas APIs. This pull request also modified the test metadata to indicate the 17 newly passing tests resulting from our fixes.

OSS Code Changes

The first major step in better supporting offscreen canvases was to make sure that the parameterization types were ubiquitous across all canvas contexts. In most cases this required a simple change such as changing size integers to cast to 32 or 64 bit integers. This shows up in about a dozen files with the only difference being calls to cast functions or strongly typed parameterizations. An example is given below.

This step also required writing many of the used casting functions for a given context. Again these functions span several files with an example of the functions below.

Another step towards supporting Offscreen Canvases was to allow them to properly maintain their context when setting their parameters. We do this by testing the context of the canvas in the height and width setters.

The overall theme of adding more versatility to offscreen canvases often requires implementing canvas_state with more universal functions, both on and offscreen. One example of this is refactoring repeated code to set bitmap dimensions from the specific rendering contexts to the canvas_state class.

These changes are the biggest among many other that can be seen detailed in either of our pull requests. The end result is 2 new passing tests indicating essential functionality for offscreen canvases.

Final Progress

After spending our time during our OSS project working on familiarizing ourselves with the servo repo and fixing the other OffscreenCanvasAPI issues we were directed to, our final project focuses on the primary issue currently assigned to us: Support drawing offscreen canvases onto other canvases. When properly implemented resolving this issue should result in newly passing test in the below mentioned test directory.

   ./mach test-wpt tests/wpt/web-platform-tests/offscreen-canvas/drawing-images-to-the-canvas

These tests are generated to identify how closely the servo implementation of the OffscreenCanvasAPI matches with the W3C spec.

In our implementation there were several discrete steps: 1. Enable an OffscreenCanvas as a possible CanvasImageSource type. 2. Write helper functions used to extract different parameters of the canvas_state of the OffscreenCanvas, most of which parallel their HTMLCanvasElement counterparts. 3. Implement the draw_offscreen_canvas function to copy canvas data between an OffscreenCanvas source and the destination. 4. Run test directory listed above to check for newly passing tests or any errors our changes may have caused.

These issues led to the creation of our final pull request. When the above chanegs were implemented we noticed some iregularities in the testing results, which led to the identification of an issue, where some functions in the OffscreenCanvas code were using inconsistent ordering for their height and width parameters. Once resolved our changes led to 866 newly passing tests. These changes were approved and merged from our pull request.

Our changes also, after identifying the redundancies between helper functions in HTMLCanvasState and OffscreenCanvas, led to the creation of a new refactoring issue.

Final Code Changes

These changes will largely parallel the listed changes above.

As mentioned before our first step was enabling OffscreenCanvas as a CanvasImageSource. This can be seen alongside the HTMLCanvasElement option.

The added helper functions origin_is_clean, fetch_all_data, send_canvas_2d_msg, origin_is_clean, get_canvas_id, and get_ipc_renderer were added to OffscreenCanvasContext and OffscreenCanvasRenderingContext2D to allow the calling functions to identify the state of the OffscreenCanvas being copied.

Finally we implemented the draw_offscreen_canvas function to analyze the canvas context and copy its data.

This is when we encountered the issues where width and height variables were being swapped in calls to OffscreenCanvas. The changes to fix both callers and the function implementations themselves are shown below.

These changes can be seen in effect in the video of our test directory running.

References

1. https://en.wikipedia.org/wiki/Servo_(layout_engine)
2. https://doc.rust-lang.org/book/title-page.html
3. https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas
4. https://rustup.rs/
5. https://github.com/servo/servo/blob/master/README.md
6. https://github.com/servo/servo/issues/24271
7. https://github.com/servo/servo/issues/24269
8. https://github.com/servo/servo/issues/24272
9. https://github.com/servo/servo/issues/24273
10. https://github.com/servo/servo/issues/24465
11. https://github.com/servo/servo/issues/24536
12. https://github.com/servo/servo/pull/24518
13. https://github.com/servo/servo/pull/24524
14. https://github.com/servo/servo/pull/25087
15. https://github.com/servo/servo/issues/25118
16. https://youtu.be/yzFocVSXEuw