M1900: Implement missing WebAudio node support

Introduction

Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode and StereoPannerNode).

Although the GitHub description for our project is titled "Implement JS support for several missing WebAudio node types", we did not actually program in JavaScript for this project. This project involved writing Rust code to implement the backend functionality of the WebAudio API, which the end user can then write JavaScript code to utilize. Therefore, the code snippets on this page are entirely in the Rust language. The only portion of our project written in JavaScript/HTML are the tests; however, the tests are not the main focus of this project (feature implementation is).

Tracking Issues

ConstantSourceNode

StereoPannerNode

Useful Resources

• Setup for modifying the standalone media backend implementation
• Implementation of audio node processing algorithms
• Runnable examples of audio node processing algorithms
• Example pull request implementing a missing node type in Servo (another example)
• Example pull request implementing the processing backend for a missing node type
• Setup for making changes to Servo's code
• Documentation for introducing new WebIDL-based DOM interfaces to Servo
• Documentation for integrating a version of servo-media that contains your local changes into your local Servo build

Project Steps

Initial Steps (OSS Project)

• Email the mozilla.dev.servo mailing list (be sure to subscribe to it first!) introducing your group and asking any necessary questions.
• Create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext.
• Connect the DOM interface to the underlying backend node by processing the unimplemented message type.
• Update the expected test results for the relevant tests.

Subsequent Steps (Final Project)

• Implement an audio node message that is specific to OscillatorNode (use BiquadFilterNode as a model) which updates the node's oscillator type.
• Implement the type attribute setter for the OscillatorNode interface which uses the new oscillator node message.
• Implement the backend for StereoPannerNode in the media crate by creating a new node implementation using PannerNode as a guide. The processing algorithm is described in the specification. Create a runnable example based on the example for PannerNode.
• Create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext.

Design Choices

OSS Project

Our OSS project did not involve any design patterns since we were just implementing methods.

However, we faced multiple options of how to create and implement the ConstantSourceNode. Initially, we tried to model it on the OscillatorNode implementation, but faced a build error.

let node_options =
options
.parent
.unwrap_or(2, ChannelCountMode::Max, ChannelInterpretation::Speakers);

We eventually learned that the ConstantSourceNode should not have a parent field because it does not inherit from anything.

let node_options =
options
.unwrap_or(2, ChannelCountMode::Max, ChannelInterpretation::Speakers);

We faced other design choices that we had to adjust from the OscillatorNode implementation. For example, we encountered errors with this code.

use servo_media::audio::constant_source_node::ConstantSourceNodeOptions;

We then realized that ConstantSourceNodeOptions needed to be cast as a different type in order to function correctly.

use servo_media::audio::constant_source_node::ConstantSourceNodeOptions as ServoMediaConstantSourceOptions;

Final Project

For our final project, we had a lot of freedom to design our runnable StereoPannerNode example. For the example, we could have set the pan value to anything between -1 (full left pan) and 1 (full right pan) and then changed it any way we wished. At first, we considered setting the pan value to different values at different times, like this:

context.message_node(
        osc,
        AudioNodeMessage::SetParam(ParamType::Pan, UserAutomationEvent::SetValueAtTime(-1., 2.)),
    );
context.message_node(
        osc,
        AudioNodeMessage::SetParam(ParamType::Pan, UserAutomationEvent::SetValueAtTime(1., 4.)),
    );

That code would set the pan value to -1 at time = 2 seconds, and then set it to 1 at time = 4 seconds. However, it sounded very weird to have the sound jumping around from place to place.

Eventually, we decided that linearly ramping up the pan value made for the best listening experience. It made the sound continuous but still exemplified what changing the pan value could do.

context.message_node(
        pan,
        AudioNodeMessage::SetParam(
            ParamType::Pan,
            UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 0., 4.),
        ),
    );

Our final version of the StereoPannerNode example involved linearly ramping the pan value from -1 to 0 and then from 0 to 1. This meant the sound moved from left to right over time.

The rest of our final project just involved implementing methods and therefore did not involve any design choices.

Completed Work

Our Pull Requests:

OSS Project

We created the DOM interface for ConstantSourceNode and implemented the CreateConstantSource API for BaseAudio Context. The connection between the DOM interface and the underlying backend node was already implemented by other contributors before we started the project.

Final Project Step 1 - Oscillator node message

The purpose of our first pull request was to implement an audio node message specific to OscillatorNode. This was within the servo/media project. For this pull request, we had to create a new OscillatorNodeMessage within the oscillator_node.rs file. The purpose of the OscillatorNodeMessage is to set the type of the OscillatorNode. We then had to modify node.rs to utilize the new OscillatorNodeMessage.

Final Project Step 2 - Type attribute for OscillatorNode interface

The purpose of our second pull request was to implement the type attribute for the OscillatorNode interface within the main servo project. This required enabling the setter throws and OscillatorType attribute within the OscillatorNode.webidl file. More importantly, it involved modifying the oscillatornode.rs file to include methods to set and get the type of the OscillatorNode. This was challenging because we did not realize at first that we needed to wrap the oscillator_type in a cell in order to achieve interior mutability. Initially, we were handing references to the setters and getters of the DOM struct members, which did not expose the set and get methods we were calling. We had to wrap the oscillator_type in a cell to give mutable references instead.

Final Project Step 3 - StereoPannerNode backend

For our third pull request, we had to implement the backend for the StereoPannerNode in servo/media. Then we had to create a runnable example that utilizes the StereoPannerNode. We designed our example so that we would initialize a StereoPannerNode with the default pan value of 0. Then, we would set the pan value to -1. Next, we would linearly ramp up the pan value to 0 and then to 1. A user could run this example and listen to the StereoPannerNode through speakers as its pan value changed over time, causing the sound to move from left to right.

Final Project Step 4 - StereoPannerNode DOM

For our fourth and last pull request, we created the DOM interface for StereoPannerNode and implemented the createStereoPannerNode API for BaseAudioContext within the main servo project. We had to create a few new files for this, since there was no existing stereopannernode.rs or StereoPannerNode.webidl. We also had to modify audionode.rs to add constraints when creating a StereoPannerNode, such as to ensure that there are not more than two channels and that the channel count mode is not set to max. In that case, we had to return a NotSupported error.

Testing

Testing from Command Line

After following the setup instructions in the Servo readme, cloning the repository, and changing directories into it, you can build the project with the following command from the terminal (the readme includes the different build commands to use depending on your OS). Note that it may take a long time to build the project.

./mach build --release

After building, you can run the relevant tests with the following command.

./mach test-wpt tests/wpt/web-platform-tests/webaudio/the-audio-api/the-constantsourcenode-interface/

You will notice that all tests pass as expected.

Testing from UI

Our project cannot be tested from a UI since it involves the Servo project.