CSC/CSC 517 Spring 2020/Implement ImageBitMap WebAPI

Background Information

This project aims to contribute to Mozilla's experimental browser engine called Servo, which is implemented in a language called RUST(useful for implementing features that need concurrency and memory safety).

Many of the components of Servo are still under development and one such feature is the ImageBitmap.

Major browsers support the ImageBitmap standard which can be used to create images that are ready to be drawn efficiently to HTML canvas elements. Servo is a new, experimental browser that supports these canvas APIs.

The goal of and motivation behind this project is to implement support for image bitmaps and improve our canvas automated test coverage as a result.

About ImageBitMap and Motivation behind the project

We usually decode images for a use with canvas to allow users to customize an avatar, crop an image, or just zoom in on a picture. The problem with decoding images is that it can be CPU intensive, and that can sometimes mean jank or checkerboarding.

But the createImageBitmap() method allows us to decode the image in the background and get access to a new ImageBitmap primitive, which you can draw into a canvas in the same way you would an <img> element, another canvas, or a video.

The aim of this project is to develop the ImageBitmap for the servo environment.

This can be done in the steps mentioned in the following section.

Steps for implementation

Initial Phase

• Step 1: Add a ImageBitmap WebIDL interface to components/script/dom/webidls and Rust implementation in components/script/dom/imagebitmap.rs
• Step 2: Add the createImageBitmap method that takes no extra x/y/w/h parameters in component/script/dom/webidls/WindowOrWorkerGlobalScope.webidl and implement the method in component/script/dom/window.rs, handling the HTMLCanvasElement and OffscreenCanvas types from the possible image sources

Subsequent phase

• Step 1: Implement several remaining image source types (HTMLImageElement, ImageData, ImageBitmap)
• Step 2: Implement the createImageBitmap overload that accepts x/y/w/h parameters
• Step 3: Implement support for ImageBitmaps as canvas image sources in components/script/canvas_state.rs

Current Implementation

The steps that have been implemented so far in this project by the previous batch are:

Step 1: Added the ImageBitmap interface to components/script/dom/webidls that represents a bitmap image which can be drawn to a <canvas> without undue latency. The interface contains height and weight as its attributes which are read only unsigned long integers.

//[Exposed=(Window,Worker), Serializable, Transferable]
[Exposed=(Window,Worker)]
interface ImageBitmap {
  readonly attribute unsigned long width;
  readonly attribute unsigned long height;
  //void close();
};

typedef (CanvasImageSource or
         Blob or
         ImageData) ImageBitmapSource;

enum ImageOrientation { "none", "flipY" };
enum PremultiplyAlpha { "none", "premultiply", "default" };
enum ColorSpaceConversion { "none", "default" };
enum ResizeQuality { "pixelated", "low", "medium", "high" };

dictionary ImageBitmapOptions {
  ImageOrientation imageOrientation = "none";
  PremultiplyAlpha premultiplyAlpha = "default";
  ColorSpaceConversion colorSpaceConversion = "default";
  [EnforceRange] unsigned long resizeWidth;
  [EnforceRange] unsigned long resizeHeight;
  ResizeQuality resizeQuality = "low";
};

The ImageBitmap webidl also defines a dictionary for various ImageBitmap Options that can be used to modify the ImageBitmap object.

Step 2: Implemented the rust code for the webidl interface at components/script/dom/imagebitmap.rs. For more details on the code visit

use crate::dom::bindings::cell::DomRefCell;

use crate::dom::bindings::codegen::Bindings::ImageBitmapBinding::ImageBitmapMethods;
use crate::dom::bindings::root::DomRoot;
use crate::dom::globalscope::GlobalScope;

use crate::dom::bindings::error::Fallible;
use crate::dom::bindings::reflector::{reflect_dom_object, Reflector};
use dom_struct::dom_struct;

use std::vec::Vec;

#[dom_struct]
pub struct ImageBitmap {
    reflector_: Reflector,
    width: u32,
    height: u32,
    bitmap_data: DomRefCell<Vec<u8>>,
}

impl ImageBitmap {
    fn new_inherited(width_arg: u32, height_arg: u32) -> ImageBitmap {
        ImageBitmap {
            reflector_: Reflector::new(),
            width: width_arg,
            height: height_arg,
            bitmap_data: DomRefCell::new(vec![]),
        }
    }

    #[allow(dead_code)]
    pub fn new(global: &GlobalScope, width: u32, height: u32) -> Fallible<DomRoot<ImageBitmap>> {
        //assigning to a variable the return object of new_inherited
        let imagebitmap = Box::new(ImageBitmap::new_inherited(width, height));

        Ok(reflect_dom_object(imagebitmap, global))
    }
}

The code also implements the getter methods for height and width attribute of an ImageBitmap object.

impl ImageBitmapMethods for ImageBitmap {
    // https://html.spec.whatwg.org/multipage/#dom-imagebitmap-height
    fn Height(&self) -> u32 {
        //to do: add a condition for checking detached internal slot
        //and return 0 if set to true
        self.height
    }

    // https://html.spec.whatwg.org/multipage/#dom-imagebitmap-width
    fn Width(&self) -> u32 {
        //to do: add a condition to check detached internal slot
        ////and return 0 if set to true
        self.width
    }
}

Proposed Implementation

Among the remaining steps in the initial and subsequent phases, the focus will be on step 2 of initial phase and once there is progress made on this step, implementation of the subsequent steps will take place.

Starting with createImageBitmap()

• The createImageBitmap() method creates a bitmap from a given source, optionally cropped to contain only a portion of that source. The method exists on the global scope in both windows and workers. It accepts a variety of different image sources, and returns a Promise which resolves to an ImageBitmap.
• This method is defined in the file components/script/dom/webidls/WindowOrWorkerGlobalScope.webidl and rust code is implemented at components/script/dom/window.rs.
• The syntax of the method looks like

const imageBitmapPromise = createImageBitmap(image[, options]);

const imageBitmapPromise = createImageBitmap(image, sx, sy, sw, sh[, options]);

where the parameters indicate:

image: an image source, which can be an img element, a SVG image element, a video element, a canvas element, a blob object, an ImageData object or another ImageBitmap object.

sx, sy, sw, sh: if given, source image is cropped to the given pixels.

options (Optional): the ImageBitmap object's bitmap data is modified according to options. Available options are:

• imageOrientation
• premultiplyAlpha
• colorSpaceConversion
• resizeWidth
• resizeHeight
• resizeQuality

You can read more about these options here

The return value is a Promise that is resolved when a new ImageBitmap is created.

• As a first step, we will be implementing the method to handle canvas elements. Subsequently, we will be enhancing the method to handle other image sources and the x/y/w/h parameters.

Implementing close() method from previous work

• close() is a method in the ImageBitmap interface. It is implemented in components/script/dom/imagebitmap.rs file.
• This method disposes of all graphical resources associated with an ImageBitmap.

Current Progress

• Implemented createImageBitmap method for canvas image source(HTMLCanvasElement and OffscreenCanvas) using RUST programming language in the components/script/dom/workerglobalscope.rs and components/script/dom/window.rs files. The comments in the following code will explain the functionality that it carries out.

// https://html.spec.whatwg.org/multipage/#dom-createimagebitmap
fn CreateImageBitmap(
    &self,
    image: ImageBitmapSource,
    options: &ImageBitmapOptions,
) -> Rc<Promise> {
    let global = self.global();
    let in_realm_proof = AlreadyInRealm::assert(&global);
    // The following line creates promise element which is to be returned
    let p = Promise::new_in_current_realm(&global, InRealm::Already(&in_realm_proof));
    if options.resizeWidth.unwrap() == 0 {
        p.reject_error(Error::InvalidState)
    }

    if options.resizeHeight.unwrap() == 0 {
        p.reject_error(Error::InvalidState)
    }

    let image_bitmap = ImageBitmap::new(&global, 0, 0).unwrap();

    // match the image type to promise element
    let promise = match image {
        // Check if the imageBitmapSource is of HTMLCanvasElement type
        ImageBitmapSource::HTMLCanvasElement(ref canvas) => {
            // https://html.spec.whatwg.org/multipage/#check-the-usability-of-the-image-argument
            if !canvas.is_valid() {
                p.reject_error(Error::InvalidState)
            }
      
            // Use fetch_all_data() from canvas_state.rs to return the image data and its size and assign it to a DomRefCell as a vector
            if let Some((data, size)) = canvas.fetch_all_data() {
                let data = DomRefCell::new(
                    data.map(|data| data.to_vec())
                        .unwrap_or_else(|| vec![0; size.area() as usize * 4]),
                );
                // Set imageBitmap's bitmap data to a copy of image's bitmap data
                image_bitmap.set_bitmap_data(data);
                // Set the origin-clean flag of the imageBitmap's bitmap to the same value as the origin-clean flag of image's bitmap
                image_bitmap.set_origin_clean(canvas.origin_is_clean());
                // Resolve p with imageBitmap
                p.resolve_native(&(image_bitmap));
            }
            p
        },
        // Check if the imageBitmapSource is of OffscreenCanvas type
        ImageBitmapSource::OffscreenCanvas(ref canvas) => {
            // https://html.spec.whatwg.org/multipage/#check-the-usability-of-the-image-argument
            if !canvas.is_valid() {
                p.reject_error(Error::InvalidState)
            }
            // Use fetch_all_data() from canvas_state.rs to return the image data and its size and assign it to a DomRefCell as a vector
            if let Some((data, size)) = canvas.fetch_all_data() {
                let data = DomRefCell::new(
                    data.map(|data| data.to_vec())
                        .unwrap_or_else(|| vec![0; size.area() as usize * 4]), // set value as 0 if image data is null
                );
                // Set imageBitmap's bitmap data to a copy of image's bitmap data
                image_bitmap.set_bitmap_data(data);  
                // Set the origin-clean flag of the imageBitmap's bitmap to the same value as the origin-clean flag of image's bitmap
                image_bitmap.set_origin_clean(canvas.origin_is_clean());
                // Resolve p with imageBitmap
                p.resolve_native(&(image_bitmap));
            }
            p
        },
        _ => p,
    };
    promise
}

• Tested the build using ‘mach’ commands

Design of the createImageBitmap() method

On invoking the createImageBitmap(image, options) or the createImageBitmap(image sx, sy, sw, sh, options) the following in the sequence of actions

Switching on the "image" argument the following is the sequence of actions

If the image is an SVGImage

If the image is a video

If the image is a canvas

If the image is an ImageBitmap

If the image is an ImageData

Test Plan

The following are the respective steps in testing:

• Servo check should be successful.

The compiling time for servo build takes a lot of time. In order to make sure that our changes didn't break any of the existing features quickly, we run

For Linux or macOS:
    ./mach check

For Windows:
    mach.bat check

• Servo building test should be successful.

The compiling time is based on the CPU clock speed and the OS of the system where the file is being run. For a 64-bit Linux OS, it takes from about 1 hour to 2 hours for the entire build. It takes 2.5 hours on Windows OS. This is done to make sure that the end-to-end implementation can be successfully run in a system.

For Linux or macOS:
    ./mach build --dev

For Windows:
    mach.bat build --dev

• Check for the file tidiness (following standards of servo) using the command:

For Linux or macOS:
    ./mach test-tidy
    ./mach fmt

For Windows:
    mach.bat test-tidy
    mach.bat fmt

• Servo automated Unit-testing should be successful.

Running the following commands will run all the unit-tests

For Linux or macOS:
    ./mach test-unit

For Windows:
    mach.bat test-unit

Note: A log file can be maintained to make keep a check on the tests that are passing.

• Update the automated servo Web Platform tests(wpt):

For a DOM feature, extensive tests already exist under tests/wpt. Any change in DOM would require updating the expected results in the automated tests.

This first requires storing the log in raw format from a test run, for example by running

./mach test-wpt --log-raw /path/for/storing/log/file/<filename>.log

Once log is saved, then run to update test expectations

./mach update-wpt /path/to/logfile/<filename>.log

• Finally, to run the servo with a webpage (Google.com) to visually confirm that servo is working :

For Linux or macOS:
    ./mach run https://www.google.com

For Windows:
    mach.bat run https://www.google.com

Code submission and Pull request

Code: [1]
Pull Request: [2]

Team Details

Jayalakshmi Vishwanathan (jviswan@ncsu.edu)

Nita Radhakrishnan (nradhak2@ncsu.edu)

Ramya Ananth (rananth2@ncsu.edu)

Sandeep Kundala (skundal@ncsu.edu)

Mentor Details

Jay Modi

References

• What is servo?
• what is RUST programming?