CSC/ECE 517 Fall 2017/M1753 OffscreenCanvas API: Difference between revisions

From Expertiza_Wiki
Jump to navigation Jump to search
No edit summary
(Undo revision 112812 by Wzhang43 (talk))
 
(51 intermediate revisions by 3 users not shown)
Line 1: Line 1:
The HTML specification defines a <canvas> element that use a 2D or 3D rendering context and draw graphics. The biggest limitation of <canvas> is that all of the javascript that manipulates the <canvas> has to run on the main thread, as the applications we build increase in complexity, developers inadvertently hit the performance wall. A new specification was recently developed that defines a canvas that can be used without being associated with an in-page canvas element, allowing for more efficient rendering. The OffscreenCanvas API achieves pre-rendering that using a separate off-screen canvas to render temporary image, and then rendering the off-screen canvas back to the main thread.<ref>https://www.html5rocks.com/en/tutorials/canvas/performance/][https://github.com/servo/servo/wiki/Offscreen-canvas-project</ref>
The HTML specification defines a <canvas> element that use a 2D or 3D rendering context and to draw graphics. The biggest limitation of <canvas> is that all of the javascript that manipulates the <canvas> has to run on the main thread, as the UI complexity of the application increases, developers inadvertently hit the performance wall. A new specification was recently developed that defines a canvas that can be used without being associated with an in-page canvas element, allowing for more efficient rendering. The OffscreenCanvas API achieves pre-rendering that using a separate off-screen canvas to render temporary image, and then rendering the off-screen canvas back to the main thread.<ref>https://www.html5rocks.com/en/tutorials/canvas/performance/][https://github.com/servo/servo/wiki/Offscreen-canvas-project</ref>


== '''Background Introduction''' ==
== '''Background Introduction''' ==
'''Servo[https://github.com/servo/servo]'''  is a modern web browser engine designed to maximize the benefits of multi-core-based parallel processing. Parallelism is undoubtedly the highlight feature of the servo, besides other characteristics like GPU rendering, embedding, modularity and memory safety are also stressed. Written in the Rust language, servo was aimed for the application on Android and ARM processors, but now its applicable scope has expanded to Linux, Mac OS X, Windows, Android and Firefox OS (also known as Gonk) and other operating systems[https://servo.org/].
'''[https://github.com/servo/servo Servo]'''  is a modern web browser engine designed to maximize the benefits of multi-core-based parallel processing. Parallelism is undoubtedly the highlight feature of the servo, besides other characteristics like GPU rendering, embedding, modularity and memory safety are also stressed. Written in the Rust language, servo was aimed for the application on Android and ARM processors, but now its applicable scope has expanded to Linux, Mac OS X, Windows, Android and Firefox OS (also known as Gonk) and other operating systems<ref>https://servo.org/</ref>.




'''Rust[https://en.wikipedia.org/wiki/Rust_(programming_language)]''' is a programming language that emphasizes security, concurrency and speed. The three keywords, safety, control of memory layout, and concurrency are the core features of rust, and also the primary factors that distinguish rust from other programming languages. Rust performs most of its memory management and security checks at compile time, which makes it possible to predict the time and space requirements of the program and be embedded into low-level code , such as device driver and operating system.Rust is developed on GitHub/Git [https://github.com/rust-lang/rust], and every commit to the source code is documented by the version control system.
'''[https://en.wikipedia.org/wiki/Rust_(programming_language) Rust]''' is a programming language that emphasizes security, concurrency and speed. The three keywords, safety, control of memory layout, and concurrency are the core features of rust, and also the primary factors that distinguish rust from other programming languages. Rust performs most of its memory management and security checks at compile time, which makes it possible to predict the time and space requirements of the program and be embedded into low-level code , such as device driver and operating system.Rust is developed on [https://github.com/rust-lang/rust GitHub/Git], and every commit to the source code is documented by the version control system.




'''The OffscreenCanvas API''' supports a canvas to be rendered off screen. The performance of a webpage can be limited by using <canvas> since all of the javascript that manipulates the <canvas> has to run on the main thread, situation is not optimistic when the canvas operations are complex. The OffscreenCanvas API provides a way to interact with the same canvas APIs but in a different thread. This allows rendering to progress no matter what is going on in the main thread [https://hacks.mozilla.org/2016/01/webgl-off-the-main-thread/].
'''The OffscreenCanvas API''' supports a canvas to be rendered off screen. The performance of a webpage can be limited by using <canvas> since all of the javascript that manipulates the <canvas> has to run on the main thread, situation is not optimistic when the canvas operations are complex. The OffscreenCanvas API provides a way to interact with the same canvas APIs but in a different thread. This allows rendering to progress no matter what is going on in the main thread<ref>https://hacks.mozilla.org/2016/01/webgl-off-the-main-thread/</ref><ref>https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas</ref>.


== '''Project Description''' ==
== '''Project Description''' ==
'''Implement the OffscreenCanvas API''': Servo has an implementation of the HTML Canvas API. Our project is to implement the OffscreenCanvas API which provides a canvas that can be rendered off screen and is available in both the window and worker contexts.
'''Implement the OffscreenCanvas API''': Servo has an implementation of the HTML Canvas API, we need to port this API to OffscreenCanvas. This API will provide a canvas that can be rendered off screen and will be available in both the window and worker contexts. The rendering will be done in a separate thread which makes the overall rendering of the webpage quicker.


'''Implement the OffscreenCanvasRenderingContext2D''': Servo has an implementation of the CanvasRendering2D API, we need to port this to OffscreenCanvasRenderingContext2D. This interface will be used for drawing rectangles, text, images and other objects onto the canvas element. It provides the 2D rendering context for the drawing surface of a <canvas> element.


'''Steps'''
'''Steps:'''


* Create the OffscreenCanvas and OffscreenCanvasRenderingContext2d interfaces.
# Create the OffscreenCanvas and OffscreenCanvasRenderingContext2d interfaces.
* Hide the new interfaces by default
# Hide the new interfaces by default
* Enable the existing automated tests for this feature
# Enable the existing automated tests for this feature
* Implement the OffscreenCanvas constructor that creates a new canvas
# Implement the OffscreenCanvas constructor that creates a new canvas
* Implement the OffscreenCanvas.getContext ignoring the WebGL requirements
# Implement the OffscreenCanvas.getContext ignoring the WebGL requirements


== '''Implementation''' ==
== '''Implementation''' ==
=== Create Interfaces ===
=== Step 1: Create Interfaces ===
Create the OffscreenCanvas and OffscreenCanvasRenderingContext2d interfaces with stub method implementations.
Create the OffscreenCanvas and OffscreenCanvasRenderingContext2d interfaces with stub method implementations.
  To add the interfaces, the following steps are to be followed:
    * adding the new IDL file at components/script/dom/webidls/Foo.webidl;
    * creating components/script/dom/foo.rs;
    * listing foo.rs in components/script/dom/mod.rs; 
    * defining the DOM struct Foo with a #[dom_struct] attribute, a superclass or Reflector member, and other members as appropriate;
    * implementing the dom::bindings::codegen::Bindings::FooBindings::FooMethods trait for Foo;
    * adding/updating the match arm in create_element in components/script/dom/create.rs (only applicable to new types inheriting from HTMLElement)


  Hide newly created interfaces by default:
To [http://doc.servo.org/script/dom/index.html#adding-a-new-dom-interface create] the interfaces, the following steps are to be followed :
    * Add corresponding attribute to the interface file as well as resources/prefs.json


  Enable Tests:
:1. adding the new IDL file at ''components/script/dom/webidls/OffscreenCanvas.webidl''<ref>https://html.spec.whatwg.org/multipage/canvas.html#the-offscreencanvas-interface</ref><ref>https://html.spec.whatwg.org/multipage/canvas.html#the-offscreen-2d-rendering-context</ref>;
    * Add 'offscreen-canvas' directory to tests folder.
cd servo/components/script/dom/webidls
    * Enable test preference for offscreen-canvas by adding '__dir__.ini' file to the newly created directory.
    * Run tests and change expected results.


  Implement OffscreenCanvas Constructor:
touch OffscreenCanvas.webidl
    Syntax:  
 
      var offscreen = new OffscreenCanvas(256, 256);
touch OffscreenCanvasRenderingContext2D.webidl
   
 
  Implement the getContext() method for OffscreenCanvas element.  
:2. creating ''components/script/dom/OffscreenCanvas.rs'';
    Syntax:  
cd servo/components/script/dom
        canvas.getContext(contextType, contextAttributes);  
 
touch offscreencanvas.rs
 
touch offscreencanvasrenderingcontext2d.rs
 
:3. listing ''OffscreenCanvas.rs'' in ''components/script/dom/mod.rs'';
 
[[File:Createoffscreencanvas.png]]
 
:4. defining the DOM struct OffscreenCanvas with a ''#[dom_struct]'' attribute, a superclass or Reflector member, and other members as appropriate;
 
[[File:Definedomstruct.png]]
 
:5. implementing the ''dom::bindings::codegen::Bindings::OffsceenCanvasBindings::OffscreenCanvasMethods'' trait for OffscreenCanvas;
 
[[File:Dombindings.png]]
 
:6. adding/updating the match arm in create_element in ''components/script/dom/create.rs'' (only applicable to new types inheriting from HTMLElement)
 
===Step 2: Hide newly created interfaces by default ===
 
:1. Hide the new interfaces by default by adding a ''[Pref="dom.offscreen_canvas.enabled"]'' attribute to each one and add a corresponding preference to ''resources/prefs.json''
::* Add the attribute in file ''OffscreenCanvas'' and ''OffscreenCanvasRendering'':
[[File:Attribute.png]]
::* Add the corresponding preference in ''prefs.json'':
[[File:Attribute2.png]]
 
===Step 3: Enable Tests ===
:1. Add ''offscreen-canvas'' directory to tests folder.
cd servo/tests/wpt/metadata
 
mkdir offscreen-canvas
 
:2. Enable test preference for offscreen-canvas by adding ''__dir__.ini'' file to the newly created directory.
cd servo/tests/wpt/metadata/offscreen-canvas
 
touch __dir__.ini
 
:3. Run tests and change expected results<ref>https://github.com/servo/servo/blob/master/tests/wpt/README.md#updating-test-expectations</ref>.
[[File:preds.png]]
 
===Step 4: Implement OffscreenCanvas Constructor===
Create a new OffscreenCanvas by implementing the following Syntax:  
var offscreen = new OffscreenCanvas(256, 256);
 
===Step 5: Implement the getContext() method for OffscreenCanvas element ===
Implement the ''OffscreenCanvas.getContext'' ignoring the WebGL requirements:
  canvas.getContext(contextType, contextAttributes);


== '''Testing Details''' ==
== '''Testing Details''' ==
:'''1. Building'''
Servo is built with the Rust package manager, [https://crates.io/ Cargo].
The following commands are steps to built servo in development mode, please note the resulting binary is very slow.
<code>
git clone https://github.com/jitendra29/servo.git
</code>
<code>
cd servo
</code>
<code>
./mach build --dev
</code>
<code>
./mach run tests/html/about-mozilla.html
</code>
:'''2. Testing'''
Servo provides automated tests, adding the <code>--release</code> flag to create an optimized build:
<code>
./mach build --release
</code>
<code>
./mach run --release tests/html/about-mozilla.html
</code>
'''Notes''':
#We have not added any new tests to the test suite as servo follows TDD and tests were previously written for OffscreenCanvas. We are adjusting some of the test expectations for the tests to pass our implementation.
#Our project cannot be tested from UI since it is basically improving some javascript features (OffsereenCanvas) in servo. However you can check that it doesn't break the existing code and the browser runs correctly by running a test page on servo after performing the build as mentioned above.


== '''Pull Request''' ==
== '''Pull Request''' ==
Here is our [https://github.com/servo/servo/pull/19067 pull request]. In the link you can see all code snippets changed due to implementing the above steps, as well as integration test progression information.


== '''Future Work''' ==
== '''Future Work''' ==
*extract all relevant canvas operations from CanvasRenderingContext2d into an implementation shared with OffscreenCanvasRenderingContext2d
#Extract all relevant canvas operations from CanvasRenderingContext2d into an implementation shared with OffscreenCanvasRenderingContext2d
**create a trait that abstracts away any operation that currently uses self.canvas in the 2d canvas rendering context, since the offscreen canvas rendering context has no associated <canvas> element
#*create a trait that abstracts away any operation that currently uses self.canvas in the 2d canvas rendering context, since the offscreen canvas rendering context has no associated <canvas> element
*implement the convertToBlob API to allow testing the contents of the canvas
#Implement the convertToBlob API to allow testing the contents of the canvas
*support offscreen webgl contexts in a similar fashion to the 2d context, by sharing relevant code between OffscreenCanvasRenderingContext2d and WebGLRenderingContext
#Support offscreen webgl contexts in a similar fashion to the 2d context, by sharing relevant code between OffscreenCanvasRenderingContext2d and WebGLRenderingContext


== '''References''' ==
== '''References''' ==
<references/>
<references/>
<br>
 
1. https://html.spec.whatwg.org/multipage/canvas.html#the-offscreen-2d-rendering-context<br>
2. https://html.spec.whatwg.org/multipage/structured-data.html#transferable<br>
3. https://html.spec.whatwg.org/multipage/canvas.html#the-offscreencanvas-interface<br>
4. https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers<br>
5. https://developer.mozilla.org/en-US/docs/Web/API/Comment<br>
6. https://developer.mozilla.org/en-US/docs/Web/API/Comment/Comment<br>
7. https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/getContext<br>
8. https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D<br>
9. http://doc.servo.org/script/dom/index.html#adding-a-new-dom-interface<br>

Latest revision as of 10:51, 10 November 2017

The HTML specification defines a <canvas> element that use a 2D or 3D rendering context and to draw graphics. The biggest limitation of <canvas> is that all of the javascript that manipulates the <canvas> has to run on the main thread, as the UI complexity of the application increases, developers inadvertently hit the performance wall. A new specification was recently developed that defines a canvas that can be used without being associated with an in-page canvas element, allowing for more efficient rendering. The OffscreenCanvas API achieves pre-rendering that using a separate off-screen canvas to render temporary image, and then rendering the off-screen canvas back to the main thread.<ref>https://www.html5rocks.com/en/tutorials/canvas/performance/][https://github.com/servo/servo/wiki/Offscreen-canvas-project</ref>

Background Introduction

Servo is a modern web browser engine designed to maximize the benefits of multi-core-based parallel processing. Parallelism is undoubtedly the highlight feature of the servo, besides other characteristics like GPU rendering, embedding, modularity and memory safety are also stressed. Written in the Rust language, servo was aimed for the application on Android and ARM processors, but now its applicable scope has expanded to Linux, Mac OS X, Windows, Android and Firefox OS (also known as Gonk) and other operating systems<ref>https://servo.org/</ref>.


Rust is a programming language that emphasizes security, concurrency and speed. The three keywords, safety, control of memory layout, and concurrency are the core features of rust, and also the primary factors that distinguish rust from other programming languages. Rust performs most of its memory management and security checks at compile time, which makes it possible to predict the time and space requirements of the program and be embedded into low-level code , such as device driver and operating system.Rust is developed on GitHub/Git, and every commit to the source code is documented by the version control system.


The OffscreenCanvas API supports a canvas to be rendered off screen. The performance of a webpage can be limited by using <canvas> since all of the javascript that manipulates the <canvas> has to run on the main thread, situation is not optimistic when the canvas operations are complex. The OffscreenCanvas API provides a way to interact with the same canvas APIs but in a different thread. This allows rendering to progress no matter what is going on in the main thread<ref>https://hacks.mozilla.org/2016/01/webgl-off-the-main-thread/</ref><ref>https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas</ref>.

Project Description

Implement the OffscreenCanvas API: Servo has an implementation of the HTML Canvas API, we need to port this API to OffscreenCanvas. This API will provide a canvas that can be rendered off screen and will be available in both the window and worker contexts. The rendering will be done in a separate thread which makes the overall rendering of the webpage quicker.

Implement the OffscreenCanvasRenderingContext2D: Servo has an implementation of the CanvasRendering2D API, we need to port this to OffscreenCanvasRenderingContext2D. This interface will be used for drawing rectangles, text, images and other objects onto the canvas element. It provides the 2D rendering context for the drawing surface of a <canvas> element.

Steps:

  1. Create the OffscreenCanvas and OffscreenCanvasRenderingContext2d interfaces.
  2. Hide the new interfaces by default
  3. Enable the existing automated tests for this feature
  4. Implement the OffscreenCanvas constructor that creates a new canvas
  5. Implement the OffscreenCanvas.getContext ignoring the WebGL requirements

Implementation

Step 1: Create Interfaces

Create the OffscreenCanvas and OffscreenCanvasRenderingContext2d interfaces with stub method implementations.

To create the interfaces, the following steps are to be followed :

1. adding the new IDL file at components/script/dom/webidls/OffscreenCanvas.webidl<ref>https://html.spec.whatwg.org/multipage/canvas.html#the-offscreencanvas-interface</ref><ref>https://html.spec.whatwg.org/multipage/canvas.html#the-offscreen-2d-rendering-context</ref>;
cd servo/components/script/dom/webidls
touch OffscreenCanvas.webidl
touch OffscreenCanvasRenderingContext2D.webidl
2. creating components/script/dom/OffscreenCanvas.rs;
cd servo/components/script/dom 
touch offscreencanvas.rs
touch offscreencanvasrenderingcontext2d.rs
3. listing OffscreenCanvas.rs in components/script/dom/mod.rs;

4. defining the DOM struct OffscreenCanvas with a #[dom_struct] attribute, a superclass or Reflector member, and other members as appropriate;

5. implementing the dom::bindings::codegen::Bindings::OffsceenCanvasBindings::OffscreenCanvasMethods trait for OffscreenCanvas;

6. adding/updating the match arm in create_element in components/script/dom/create.rs (only applicable to new types inheriting from HTMLElement)

Step 2: Hide newly created interfaces by default

1. Hide the new interfaces by default by adding a [Pref="dom.offscreen_canvas.enabled"] attribute to each one and add a corresponding preference to resources/prefs.json
  • Add the attribute in file OffscreenCanvas and OffscreenCanvasRendering:

  • Add the corresponding preference in prefs.json:

Step 3: Enable Tests

1. Add offscreen-canvas directory to tests folder.
cd servo/tests/wpt/metadata
mkdir offscreen-canvas
2. Enable test preference for offscreen-canvas by adding __dir__.ini file to the newly created directory.
cd servo/tests/wpt/metadata/offscreen-canvas
touch __dir__.ini
3. Run tests and change expected results<ref>https://github.com/servo/servo/blob/master/tests/wpt/README.md#updating-test-expectations</ref>.

Step 4: Implement OffscreenCanvas Constructor

Create a new OffscreenCanvas by implementing the following Syntax:

var offscreen = new OffscreenCanvas(256, 256);

Step 5: Implement the getContext() method for OffscreenCanvas element

Implement the OffscreenCanvas.getContext ignoring the WebGL requirements:

 canvas.getContext(contextType, contextAttributes);

Testing Details

1. Building

Servo is built with the Rust package manager, Cargo.

The following commands are steps to built servo in development mode, please note the resulting binary is very slow.

git clone https://github.com/jitendra29/servo.git

cd servo

./mach build --dev

./mach run tests/html/about-mozilla.html

2. Testing

Servo provides automated tests, adding the --release flag to create an optimized build:

./mach build --release

./mach run --release tests/html/about-mozilla.html

Notes:

  1. We have not added any new tests to the test suite as servo follows TDD and tests were previously written for OffscreenCanvas. We are adjusting some of the test expectations for the tests to pass our implementation.
  2. Our project cannot be tested from UI since it is basically improving some javascript features (OffsereenCanvas) in servo. However you can check that it doesn't break the existing code and the browser runs correctly by running a test page on servo after performing the build as mentioned above.

Pull Request

Here is our pull request. In the link you can see all code snippets changed due to implementing the above steps, as well as integration test progression information.

Future Work

  1. Extract all relevant canvas operations from CanvasRenderingContext2d into an implementation shared with OffscreenCanvasRenderingContext2d
    • create a trait that abstracts away any operation that currently uses self.canvas in the 2d canvas rendering context, since the offscreen canvas rendering context has no associated <canvas> element
  2. Implement the convertToBlob API to allow testing the contents of the canvas
  3. Support offscreen webgl contexts in a similar fashion to the 2d context, by sharing relevant code between OffscreenCanvasRenderingContext2d and WebGLRenderingContext

References

<references/>