CSC/CSC 517 Spring 2020/Implement ImageBitMap WebAPI

Background Information

This project aims to contribute to Mozilla's experimental browser engine called Servo, which is implemented in a language called RUST(useful for implementing features that need concurrency and memory safety).

Many of the components of Servo are still under development and one such feature is the ImageBitmap.

Major browsers support the ImageBitmap standard which can be used to create images that are ready to be drawn efficiently to HTML canvas elements. Servo is a new, experimental browser that supports these canvas APIs.

The goal of and motivation behind this project is to implement support for image bitmaps and improve our canvas automated test coverage as a result.

About ImageBitMap and Motivation behind the project

We usually decode images for a use with canvas to allow users to customize an avatar, crop an image, or just zoom in on a picture. The problem with decoding images is that it can be CPU intensive, and that can sometimes mean jank or checkerboarding.

But the createImageBitmap() method allows us to decode the image in the background and get access to a new ImageBitmap primitive, which you can draw into a canvas in the same way you would an <img> element, another canvas, or a video.

The aim of this project is to develop the ImageBitmap for the servo environment.

This can be done in the steps mentioned in the following section.

Steps for implementation

Initial Phase

• Step 1: Add a ImageBitmap WebIDL interface to components/script/dom/webidls and Rust implementation in components/script/dom/imagebitmap.rs
• Step 2: Add the createImageBitmap method that takes no extra x/y/w/h parameters in component/script/dom/webidls/WindowOrWorkerGlobalScope.webidl and implement the method in component/script/dom/window.rs, handling the HTMLCanvasElement and OffscreenCanvas types from the possible image sources

Subsequent phase

• Step 1: Implement several remaining image source types (HTMLImageElement, ImageData, ImageBitmap)
• Step 2: Implement the createImageBitmap overload that accepts x/y/w/h parameters
• Step 3: Implement support for ImageBitmaps as canvas image sources in components/script/canvas_state.rs

Previous Implementation

The steps that have been implemented so far in this project by the previous batch are:

Step 1: Added the ImageBitmap interface to components/script/dom/webidls that represents a bitmap image which can be drawn to a <canvas> without undue latency. The interface contains height and weight as its attributes which are read only unsigned long integers.

//[Exposed=(Window,Worker), Serializable, Transferable]
[Exposed=(Window,Worker)]
interface ImageBitmap {
  readonly attribute unsigned long width;
  readonly attribute unsigned long height;
  //void close();
};

typedef (CanvasImageSource or
         Blob or
         ImageData) ImageBitmapSource;

enum ImageOrientation { "none", "flipY" };
enum PremultiplyAlpha { "none", "premultiply", "default" };
enum ColorSpaceConversion { "none", "default" };
enum ResizeQuality { "pixelated", "low", "medium", "high" };

dictionary ImageBitmapOptions {
  ImageOrientation imageOrientation = "none";
  PremultiplyAlpha premultiplyAlpha = "default";
  ColorSpaceConversion colorSpaceConversion = "default";
  [EnforceRange] unsigned long resizeWidth;
  [EnforceRange] unsigned long resizeHeight;
  ResizeQuality resizeQuality = "low";
};

The ImageBitmap webidl also defines a dictionary for various ImageBitmap Options that can be used to modify the ImageBitmap object.

Step 2: Implemented the rust code for the webidl interface at components/script/dom/imagebitmap.rs. For more details on the code visit

use crate::dom::bindings::cell::DomRefCell;

use crate::dom::bindings::codegen::Bindings::ImageBitmapBinding::ImageBitmapMethods;
use crate::dom::bindings::root::DomRoot;
use crate::dom::globalscope::GlobalScope;

use crate::dom::bindings::error::Fallible;
use crate::dom::bindings::reflector::{reflect_dom_object, Reflector};
use dom_struct::dom_struct;

use std::vec::Vec;

#[dom_struct]
pub struct ImageBitmap {
    reflector_: Reflector,
    width: u32,
    height: u32,
    bitmap_data: DomRefCell<Vec<u8>>,
}

impl ImageBitmap {
    fn new_inherited(width_arg: u32, height_arg: u32) -> ImageBitmap {
        ImageBitmap {
            reflector_: Reflector::new(),
            width: width_arg,
            height: height_arg,
            bitmap_data: DomRefCell::new(vec![]),
        }
    }

    #[allow(dead_code)]
    pub fn new(global: &GlobalScope, width: u32, height: u32) -> Fallible<DomRoot<ImageBitmap>> {
        //assigning to a variable the return object of new_inherited
        let imagebitmap = Box::new(ImageBitmap::new_inherited(width, height));

        Ok(reflect_dom_object(imagebitmap, global))
    }
}

The code also implements the getter methods for height and width attribute of an ImageBitmap object.

impl ImageBitmapMethods for ImageBitmap {
    // https://html.spec.whatwg.org/multipage/#dom-imagebitmap-height
    fn Height(&self) -> u32 {
        //to do: add a condition for checking detached internal slot
        //and return 0 if set to true
        self.height
    }

    // https://html.spec.whatwg.org/multipage/#dom-imagebitmap-width
    fn Width(&self) -> u32 {
        //to do: add a condition to check detached internal slot
        ////and return 0 if set to true
        self.width
    }
}

Proposed Implementation

Among the remaining steps in the initial and subsequent phases, the focus will be on step 2 of initial phase and once there is progress made on this step, implementation of the subsequent steps will take place.

Implementing createImageBitmap web API

• The createImageBitmap() method creates a bitmap from a given source, optionally cropped to contain only a portion of that source. The method exists on the global scope in both windows and workers. It accepts a variety of different image sources, and returns a Promise which resolves to an ImageBitmap.
• This web interface is defined in the file components/script/dom/webidls/WindowOrWorkerGlobalScope.webidl and rust code is implemented at components/script/dom/globalscope.rs. It is then called in components/script/dom/window.rs and components/script/dom/workerglobalscope.rs
• The syntax of the method looks like

const imageBitmapPromise = createImageBitmap(image[, options]);

const imageBitmapPromise = createImageBitmap(image, sx, sy, sw, sh[, options]);

where the parameters indicate:

image: an image source, which can be an img element, a SVG image element, a video element, a canvas element, a blob object, an ImageData object or another ImageBitmap object.

sx, sy, sw, sh: if given, source image is cropped to the given pixels.

options (Optional): the ImageBitmap object's bitmap data is modified according to options. Available options are:

• imageOrientation
• premultiplyAlpha
• colorSpaceConversion
• resizeWidth
• resizeHeight
• resizeQuality

You can read more about these options here

The return value is a Promise that is resolved when a new ImageBitmap is created.

• As a first step, we will be implementing the method to handle canvas elements. Subsequently, we will be enhancing the method to handle other image sources and the x/y/w/h parameters.

Implementing close() method from previous work

• close() is a method in the ImageBitmap interface. It is implemented in components/script/dom/imagebitmap.rs file.
• This method disposes of all graphical resources associated with an ImageBitmap.

Current Progress

• Implemented createImageBitmap method for canvas image source(HTMLCanvasElement and OffscreenCanvas) using RUST programming language in the components/script/dom/globalscope.rs file. The comments in the following code will explain the functionality that it carries out.

// https://html.spec.whatwg.org/multipage/#dom-createimagebitmap
pub fn create_image_bitmap(
    &self,
    image: ImageBitmapSource,
    options: &ImageBitmapOptions,
) -> Rc<Promise> {
    let in_realm_proof = AlreadyInRealm::assert(&self);
    // Created a new promise object p
    let p = Promise::new_in_current_realm(&self, InRealm::Already(&in_realm_proof));

    // If resizewidth is present and is equal to 0, reject p with InvalidState error
    if options.resizeWidth.map_or(false, |w| w == 0) {
        p.reject_error(Error::InvalidState);
        return p;
    }

    // If resizeheight is present and is equal to 0, reject p with InvalidState error
    if options.resizeHeight.map_or(false, |w| w == 0) {
        p.reject_error(Error::InvalidState);
        return p;
    }

    // match the image to different types of image source and return the resolved p
    let promise = match image {
        // check if image source is HTMLCanvasElement
        ImageBitmapSource::HTMLCanvasElement(ref canvas) => {
            // https://html.spec.whatwg.org/multipage/#check-the-usability-of-the-image-argument
            // check if the image source is valid
            if !canvas.is_valid() {
                p.reject_error(Error::InvalidState);
                return p;
            }

            // the fetch_all_data() method returns the bitmap data and size of the image
            if let Some((data, size)) = canvas.fetch_all_data() {
                // convert the data to a <vec> type
                let data = data
                    .map(|data| data.to_vec())
                    .unwrap_or_else(|| vec![0; size.area() as usize * 4]);
                //create imageBitmap object using the height and width of the image
                let image_bitmap = ImageBitmap::new(&self, size.width, size.height).unwrap();

                // copy the data to bitmap_data of imageBitmap object
                image_bitmap.set_bitmap_data(data);
                // copy the origin clean flag of canvas image to origin_clean flag of imageBitmap object
                image_bitmap.set_origin_clean(canvas.origin_is_clean());
                // resolve p with imageBitmap object
                p.resolve_native(&(image_bitmap));
            }
            p
        },
        // check if image source is OffscreenCanvas
        ImageBitmapSource::OffscreenCanvas(ref canvas) => {
            // https://html.spec.whatwg.org/multipage/#check-the-usability-of-the-image-argument
            // check if the image source is valid
            if !canvas.is_valid() {
                p.reject_error(Error::InvalidState);
                return p;
            }

            // the fetch_all_data() method returns the bitmap data and size of the image
            if let Some((data, size)) = canvas.fetch_all_data() {
                // convert the data to a <vec> type
                let data = data
                    .map(|data| data.to_vec())
                    .unwrap_or_else(|| vec![0; size.area() as usize * 4]);
                //create imageBitmap object using the height and width of the image
                let image_bitmap = ImageBitmap::new(&self, size.width, size.height).unwrap();

                // copy the data to bitmap_data of imageBitmap object
                image_bitmap.set_bitmap_data(data);
                // copy the origin clean flag of canvas image to origin_clean flag of imageBitmap object
                image_bitmap.set_origin_clean(canvas.origin_is_clean());
                // resolve p with imageBitmap object
                p.resolve_native(&(image_bitmap));
            }
            p
        },
        _ => {
            // if the image source is not HTMLCanvasElement or OffscreenCanvas, reject p with NotSupported Error
            p.reject_error(Error::NotSupported);
            return p;
        },
    };
    promise
}

• Then we called the above method inside components/script/dom/window.rs and components/script/dom/workerglobalscope.rs.

 fn CreateImageBitmap(
    &self,
    image: ImageBitmapSource,
    options: &ImageBitmapOptions,
) -> Rc<Promise> {
    let p = self
        .upcast::<GlobalScope>()
        .create_image_bitmap(image, options);
    p
}

• We created setters for bitmap_data and origin_clean attributes in components/script/dom/imagebitmap.rs.

pub fn set_bitmap_data(&self, data: Vec<u8>) {
    *self.bitmap_data.borrow_mut() = data;
}

pub fn set_origin_clean(&self, origin_is_clean: bool) {
    self.origin_clean.set(origin_is_clean);
}

• Updated wpt(Web Platform Tests) tests results to include createImageBitmap API and reject non-canvas image source elements.

• For this implementation, we followed the DRY principle and strategy pattern.

Design of the createImageBitmap() method

On invoking the createImageBitmap(image, options) or the createImageBitmap(image sx, sy, sw, sh, options) the following in the sequence of actions

Switching on the "image" argument the following is the sequence of actions

If the image is an SVGImage

If the image is a video

If the image is a canvas

If the image is an ImageBitmap

If the image is an ImageData

Test Plan

The following are the respective steps in testing:

• Servo mach check test

The compiling time for servo build takes a lot of time. In order to make sure that our changes didn't break any of the existing features quickly, we run

For Linux or macOS:
    ./mach check

For Windows:
    mach.bat check

• Servo mach build test

The compiling time is based on the CPU clock speed and the OS of the system where the file is being run. For a 64-bit Linux OS, it takes from about 1 hour to 2 hours for the entire build. It takes 2.5 hours on Windows OS. This is done to make sure that the end-to-end implementation can be successfully run in a system.

For Linux or macOS:
    ./mach build --dev

For Windows:
    mach.bat build --dev

• Check for the file tidiness (following standards of servo) using the command:

For Linux or macOS:
    ./mach test-tidy
    ./mach fmt

For Windows:
    mach.bat test-tidy
    mach.bat fmt

• Servo automated Unit-testing

Running the following commands will run all the unit-tests

For Linux or macOS:
    ./mach test-unit

For Windows:
    mach.bat test-unit

Note: A log file can be maintained to make keep a check on the tests that are passing.

• Update the automated servo Web Platform tests(wpt):

For a DOM feature, extensive tests already exist under tests/wpt. Any change in DOM would require updating the expected results in the automated tests.

This first requires storing the log in raw format from a test run. We ran the following command:

wget https://community.taskcluster-artifacts.net/KqQWr_VwQDe55wGO_T193Q/0/public/test-wpt.log 
https://community.taskcluster-artifacts.net/PhPboOqiSa-gdhzBMJbL_w/0/public/test-wpt.log 
https://community.taskcluster-artifacts.net/Fzvx31ovS6GPF4eGGBl8rA/0/public/test-wpt.log 
https://community.taskcluster-artifacts.net/EfQsJZaXTBSTmct0hy3scw/0/public/test-wpt.log

Once log is saved, then we ran the following command to update the test expectations

./mach update-wpt test-wpt.log test-wpt.log.1 test-wpt.log.2 test-wpt.log.3

This updated the test results in 73 test files to include the createImageBitmap API.

• Finally, to run the servo with a webpage (Google.com) to visually confirm that servo is working:

For Linux or macOS:
    ./mach run https://www.google.com

For Windows:
    mach.bat run https://www.google.com

Future Work

The createImageBitmap() method can be extended to support the subsequent steps as shown in the steps for implementation.

1. The method can be extended to support other image sources such as SVGImage, video sources, ImageData source, and so on. 
2. Overload the createImageBitmap() method to accept the s/y/w/h parameters 
3. Also implement support for ImageBitmaps as canvas image sources in the canvas_state.rs file

Code submission and Pull request

Code: [1]
Pull Request: [2]

Team Details

Jayalakshmi Vishwanathan (jviswan@ncsu.edu)

Nita Radhakrishnan (nradhak2@ncsu.edu)

Ramya Ananth (rananth2@ncsu.edu)

Sandeep Kundala (skundal@ncsu.edu)

Mentor Details

Jay Modi (jmodi3@ncsu.edu)

References

• What is servo?
• what is RUST programming?