M1900: Implement missing WebAudio node support: Difference between revisions

From Expertiza_Wiki
Jump to navigation Jump to search
 
(23 intermediate revisions by the same user not shown)
Line 2: Line 2:
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. [https://github.com/servo/servo Servo] is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of [https://github.com/servo/servo/wiki/WebAudio-JS-interfaces-student-project this project] is to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode and StereoPannerNode).
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. [https://github.com/servo/servo Servo] is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of [https://github.com/servo/servo/wiki/WebAudio-JS-interfaces-student-project this project] is to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode and StereoPannerNode).


Although the GitHub description for our project is titled "Implement JS support for several missing WebAudio node types", we are not actually programming in JavaScript for this project. This project involves writing Rust code to implement the backend functionality of the WebAudio API, which the end user can then write JavaScript code to utilize. Therefore, the code snippets on this page are entirely in the Rust language. The only portion of our project written in JavaScript/HTML are the tests; however, the tests are not the main focus of this project (feature implementation is).
Although the GitHub description for our project is titled "Implement JS support for several missing WebAudio node types", we did not actually program in JavaScript for this project. This project involved writing Rust code to implement the backend functionality of the WebAudio API, which the end user can then write JavaScript code to utilize. Therefore, the code snippets on this page are entirely in the Rust language. The only portion of our project written in JavaScript/HTML are the tests; however, the tests are not the main focus of this project (feature implementation is).


== Tracking Issues ==
=== Tracking Issues ===


[https://github.com/servo/servo/issues/21803 ConstantSourceNode]
[https://github.com/servo/servo/issues/21803 ConstantSourceNode]
Line 10: Line 10:
[https://github.com/servo/servo/issues/21797 StereoPannerNode]
[https://github.com/servo/servo/issues/21797 StereoPannerNode]


== Useful Resources ==
=== Useful Resources ===


* [https://github.com/servo/media#servo-media Setup] for modifying the standalone media backend implementation
* [https://github.com/servo/media#servo-media Setup] for modifying the standalone media backend implementation
Line 21: Line 21:
* [https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#overriding-dependencies Documentation] for integrating a version of servo-media that contains your local changes into your local Servo build
* [https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#overriding-dependencies Documentation] for integrating a version of servo-media that contains your local changes into your local Servo build


== Initial Steps (OSS Project) ==
== Project Steps ==
 
=== Initial Steps (OSS Project) ===
* Email the mozilla.dev.servo mailing list (be sure to subscribe to it first!) introducing your group and asking any necessary questions.
* Email the mozilla.dev.servo mailing list (be sure to subscribe to it first!) introducing your group and asking any necessary questions.
* Create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext.
* Create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext.
Line 27: Line 29:
* Update the expected test results for the relevant tests.
* Update the expected test results for the relevant tests.


== Subsequent Steps (Final Project) ==
=== Subsequent Steps (Final Project) ===


* Implement an audio node message that is specific to OscillatorNode (use BiquadFilterNode as a model) which updates the node's oscillator type.
* Implement an audio node message that is specific to OscillatorNode (use BiquadFilterNode as a model) which updates the node's oscillator type.
Line 36: Line 38:
== Design Choices ==
== Design Choices ==


Our project did not involve any design patterns since we were just implementing methods.
=== OSS Project ===
 
Our OSS project did not involve any design patterns since we were just implementing methods.


However, we faced multiple options of how to create and implement the ConstantSourceNode. Initially, we tried to model it on the OscillatorNode implementation, but faced a build error.  
However, we faced multiple options of how to create and implement the ConstantSourceNode. Initially, we tried to model it on the OscillatorNode implementation, but faced a build error.  
Line 67: Line 71:
</pre>
</pre>


== Completed Work ==
=== Final Project ===
 
We created the DOM interface for ConstantSourceNode and implemented the CreateConstantSource API for BaseAudio Context. The connection between the DOM interface and the underlying backend node was already [https://github.com/servo/media/pull/164 implemented by other contributors] before we started the project.
 
[https://github.com/servo/servo/pull/23152 Our Pull Request]


DOM implementation:
For our final project, we had a lot of freedom to design our runnable StereoPannerNode example. For the example, we could have set the pan value to anything between -1 (full left pan) and 1 (full right pan) and then changed it any way we wished. At first, we considered setting the pan value to different values at different times, like this:
<pre>
use crate::dom::audioparam::AudioParam;
use crate::dom::audioscheduledsourcenode::AudioScheduledSourceNode;
use crate::dom::baseaudiocontext::BaseAudioContext;
use crate::dom::bindings::codegen::Bindings::AudioNodeBinding::{
    ChannelCountMode, ChannelInterpretation,
};
use crate::dom::bindings::codegen::Bindings::AudioParamBinding::AutomationRate;
use crate::dom::bindings::codegen::Bindings::ConstantSourceNodeBinding::ConstantSourceNodeMethods;
use crate::dom::bindings::codegen::Bindings::ConstantSourceNodeBinding::{
    self, ConstantSourceOptions,
};
use crate::dom::bindings::error::Fallible;
use crate::dom::bindings::reflector::reflect_dom_object;
use crate::dom::bindings::root::{Dom, DomRoot};
use crate::dom::window::Window;
use dom_struct::dom_struct;
use servo_media::audio::node::AudioNodeInit;
use servo_media::audio::constant_source_node::ConstantSourceNodeOptions as ServoMediaConstantSourceOptions;
use servo_media::audio::param::ParamType;
use std::f32;
#[dom_struct]
pub struct ConstantSourceNode {
    source_node: AudioScheduledSourceNode,
    offset: Dom<AudioParam>,
}
impl ConstantSourceNode {
    #[allow(unrooted_must_root)]
    pub fn new_inherited(
        window: &Window,
        context: &BaseAudioContext,
        options: &ConstantSourceOptions,
    ) -> Fallible<ConstantSourceNode> {
        let node_options =
            options
                .parent
                .unwrap_or(2, ChannelCountMode::Max, ChannelInterpretation::Speakers);
        let source_node = AudioScheduledSourceNode::new_inherited(
            AudioNodeInit::ConstantSourceNode(options.into()),
            context,
            node_options,
            0, /* inputs */
            1, /* outputs */
        )?;
        let node_id = source_node.node().node_id();
        let offset = AudioParam::new(
            window,
            context,
            node_id,
            ParamType::Offset,
            AutomationRate::A_rate,
            1.,
            f32::MIN,
            f32::MAX,
        );
        Ok(ConstantSourceNode {
            source_node,
            offset: Dom::from_ref(&offset),
        })
    }
    #[allow(unrooted_must_root)]
    pub fn new(
        window: &Window,
        context: &BaseAudioContext,
        options: &ConstantSourceOptions,
    ) -> Fallible<DomRoot<ConstantSourceNode>> {
        let node = ConstantSourceNode::new_inherited(window, context, options)?;
        Ok(reflect_dom_object(
            Box::new(node),
            window,
            ConstantSourceNodeBinding::Wrap,
        ))
    }
    pub fn Constructor(
        window: &Window,
        context: &BaseAudioContext,
        options: &ConstantSourceOptions,
    ) -> Fallible<DomRoot<ConstantSourceNode>> {
        ConstantSourceNode::new(window, context, options)
    }
}
impl ConstantSourceNodeMethods for ConstantSourceNode {
    fn Offset(&self) -> DomRoot<AudioParam> {
        DomRoot::from_ref(&self.offset)
    }
}
impl<'a> From<&'a ConstantSourceOptions> for ServoMediaConstantSourceOptions {
    fn from(options: &'a ConstantSourceOptions) -> Self {
        Self {
            offset: *options.offset,
        }
    }
}
</pre>


Constant Source Node Webidl DOM:
<pre>
<pre>
         dictionary ConstantSourceOptions: AudioNodeOptions {
context.message_node(
            float offset = 1;
        osc,
         };
         AudioNodeMessage::SetParam(ParamType::Pan, UserAutomationEvent::SetValueAtTime(-1., 2.)),
    );
         [Exposed=Window, Constructor (BaseAudioContext context, optional ConstantSourceOptions options)]
context.message_node(
        interface ConstantSourceNode : AudioScheduledSourceNode {
         osc,
            readonly attribute AudioParam offset;
         AudioNodeMessage::SetParam(ParamType::Pan, UserAutomationEvent::SetValueAtTime(1., 4.)),
        };
    );
</pre>
</pre>


CreateConstantSource API for BaseAudio Context:
That code would set the pan value to -1 at time = 2 seconds, and then set it to 1 at time = 4 seconds. However, it sounded very weird to have the sound jumping around from place to place.
<pre>
        use crate::dom::constantsourcenode::ConstantSourceNode;
        fn CreateConstantSource(&self) -> Fallible<DomRoot<ConstantSourceNode>> {
            ConstantSourceNode::new( &self.global().as_window(), &self, &ConstantSourceOptions::empty())
        }
</pre>


Connection between the DOM interface and the underlying backend node (implemented by other contributors):
Eventually, we decided that linearly ramping up the pan value made for the best listening experience. It made the sound continuous but still exemplified what changing the pan value could do.


<pre>
<pre>
         AudioNodeInit::ConstantSourceNode(options) => {
context.message_node(
             Box::new(ConstantSourceNode::new(options, ch))
        pan,
        }
         AudioNodeMessage::SetParam(
             ParamType::Pan,
            UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 0., 4.),
        ),
    );
</pre>
</pre>


== Planned Work (Design Document) ==
Our final version of the StereoPannerNode example involved linearly ramping the pan value from -1 to 0 and then from 0 to 1. This meant the sound moved from left to right over time.


As we continue our project, it will likely not involve any design patterns since we will just be implementing methods. Here is what we must do and our plan for how do it:
The rest of our final project just involved implementing methods and therefore did not involve any design choices.


* Implement an audio node message that is specific to OscillatorNode (use BiquadFilterNode as a model) which updates the node's oscillator type.
== Completed Work ==
** The audio node message specification is in servo/media/audio/src/node.rs. We will use this specification for guidance.


Our Pull Requests:


<div class="center" style="width: auto; margin-left: auto; margin-right: auto;">[[File:OscillatorNode.png]]</div>
[https://github.com/servo/servo/pull/23152 OSS Project]


We created the DOM interface for ConstantSourceNode and implemented the CreateConstantSource API for BaseAudio Context. The connection between the DOM interface and the underlying backend node was already [https://github.com/servo/media/pull/164 implemented by other contributors] before we started the project.


* Implement the type attribute setter for the OscillatorNode interface which uses the new oscillator node message.
[https://github.com/servo/media/pull/242 Final Project Step 1 - Oscillator node message]
** We will accomplish this by modifying servo/components/script/dom/webidls/OscillatorNode.webidl.


<pre>
The purpose of our first pull request was to implement an audio node message specific to OscillatorNode. This was within the servo/media project. For this pull request, we had to create a new OscillatorNodeMessage within the oscillator_node.rs file. The purpose of the OscillatorNodeMessage is to set the type of the OscillatorNode. We then had to modify node.rs to utilize the new OscillatorNodeMessage.
[Exposed=Window,
Constructor (BaseAudioContext context, optional OscillatorOptions options)]
interface OscillatorNode : AudioScheduledSourceNode {
//  utilize the new OscillatorType audio node message  
    attribute OscillatorType type;
};
</pre>
 
* Implement the backend for StereoPannerNode in the media crate by creating a new node implementation using PannerNode as a guide. The processing algorithm is described in the specification. Create a runnable example based on the example for PannerNode.
** The goal for the StereoPannerNode implementation is that the user should be able to specify a pan value between -1 (full left pan) and 1 (full right pan) and the StereoPannerNode will adjust the gain on each of the two speakers accordingly.  
** We will follow the example in servo/media/examples/panner.rs to create our runnable example.


[https://github.com/servo/servo/pull/23282 Final Project Step 2 - Type attribute for OscillatorNode interface]


<div class="center" style="width: auto; margin-left: auto; margin-right: auto;">[[File:StereoPannerNode.png]]</div>
The purpose of our second pull request was to implement the type attribute for the OscillatorNode interface within the main servo project. This required enabling the setter throws and OscillatorType attribute within the OscillatorNode.webidl file. More importantly, it involved modifying the oscillatornode.rs file to include methods to set and get the type of the OscillatorNode. This was challenging because we did not realize at first that we needed to wrap the oscillator_type in a cell in order to achieve interior mutability. Initially, we were handing references to the setters and getters of the DOM struct members, which did not expose the set and get methods we were calling. We had to wrap the oscillator_type in a cell to give mutable references instead.


[https://github.com/servo/media/pull/243 Final Project Step 3 - StereoPannerNode backend]


* Create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext.
For our third pull request, we had to implement the backend for the StereoPannerNode in servo/media. Then we had to create a runnable example that utilizes the StereoPannerNode. We designed our example so that we would initialize a StereoPannerNode with the default pan value of 0. Then, we would set the pan value to -1. Next, we would linearly ramp up the pan value to 0 and then to 1. A user could run this example and listen to the StereoPannerNode through speakers as its pan value changed over time, causing the sound to move from left to right.
** We already created the DOM interface for ConstantSourceNode and implemented the createConstantSourceNode API for BaseAudioContext, so we will follow the same procedure when creating the DOM interface for StereoPannerNode and implementing the createStereoPannerNode API.


[https://github.com/servo/servo/pull/23281 Final Project Step 4 - StereoPannerNode DOM]


<div class="center" style="width: auto; margin-left: auto; margin-right: auto;">[[File:BaseAudioContext.png]]</div>
For our fourth and last pull request, we created the DOM interface for StereoPannerNode and implemented the createStereoPannerNode API for BaseAudioContext within the main servo project. We had to create a few new files for this, since there was no existing stereopannernode.rs or StereoPannerNode.webidl. We also had to modify audionode.rs to add constraints when creating a StereoPannerNode, such as to ensure that there are not more than two channels and that the channel count mode is not set to max. In that case, we had to return a NotSupported error.


== Testing ==
== Testing ==

Latest revision as of 11:42, 3 May 2019

Introduction

Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode and StereoPannerNode).

Although the GitHub description for our project is titled "Implement JS support for several missing WebAudio node types", we did not actually program in JavaScript for this project. This project involved writing Rust code to implement the backend functionality of the WebAudio API, which the end user can then write JavaScript code to utilize. Therefore, the code snippets on this page are entirely in the Rust language. The only portion of our project written in JavaScript/HTML are the tests; however, the tests are not the main focus of this project (feature implementation is).

Tracking Issues

ConstantSourceNode

StereoPannerNode

Useful Resources

  • Setup for modifying the standalone media backend implementation
  • Implementation of audio node processing algorithms
  • Runnable examples of audio node processing algorithms
  • Example pull request implementing a missing node type in Servo (another example)
  • Example pull request implementing the processing backend for a missing node type
  • Setup for making changes to Servo's code
  • Documentation for introducing new WebIDL-based DOM interfaces to Servo
  • Documentation for integrating a version of servo-media that contains your local changes into your local Servo build

Project Steps

Initial Steps (OSS Project)

  • Email the mozilla.dev.servo mailing list (be sure to subscribe to it first!) introducing your group and asking any necessary questions.
  • Create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext.
  • Connect the DOM interface to the underlying backend node by processing the unimplemented message type.
  • Update the expected test results for the relevant tests.

Subsequent Steps (Final Project)

  • Implement an audio node message that is specific to OscillatorNode (use BiquadFilterNode as a model) which updates the node's oscillator type.
  • Implement the type attribute setter for the OscillatorNode interface which uses the new oscillator node message.
  • Implement the backend for StereoPannerNode in the media crate by creating a new node implementation using PannerNode as a guide. The processing algorithm is described in the specification. Create a runnable example based on the example for PannerNode.
  • Create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext.

Design Choices

OSS Project

Our OSS project did not involve any design patterns since we were just implementing methods.

However, we faced multiple options of how to create and implement the ConstantSourceNode. Initially, we tried to model it on the OscillatorNode implementation, but faced a build error.

let node_options =
options
.parent
.unwrap_or(2, ChannelCountMode::Max, ChannelInterpretation::Speakers);

We eventually learned that the ConstantSourceNode should not have a parent field because it does not inherit from anything.

let node_options =
options
.unwrap_or(2, ChannelCountMode::Max, ChannelInterpretation::Speakers);

We faced other design choices that we had to adjust from the OscillatorNode implementation. For example, we encountered errors with this code.

use servo_media::audio::constant_source_node::ConstantSourceNodeOptions;

We then realized that ConstantSourceNodeOptions needed to be cast as a different type in order to function correctly.

use servo_media::audio::constant_source_node::ConstantSourceNodeOptions as ServoMediaConstantSourceOptions;

Final Project

For our final project, we had a lot of freedom to design our runnable StereoPannerNode example. For the example, we could have set the pan value to anything between -1 (full left pan) and 1 (full right pan) and then changed it any way we wished. At first, we considered setting the pan value to different values at different times, like this:

context.message_node(
        osc,
        AudioNodeMessage::SetParam(ParamType::Pan, UserAutomationEvent::SetValueAtTime(-1., 2.)),
    );
context.message_node(
        osc,
        AudioNodeMessage::SetParam(ParamType::Pan, UserAutomationEvent::SetValueAtTime(1., 4.)),
    );

That code would set the pan value to -1 at time = 2 seconds, and then set it to 1 at time = 4 seconds. However, it sounded very weird to have the sound jumping around from place to place.

Eventually, we decided that linearly ramping up the pan value made for the best listening experience. It made the sound continuous but still exemplified what changing the pan value could do.

context.message_node(
        pan,
        AudioNodeMessage::SetParam(
            ParamType::Pan,
            UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 0., 4.),
        ),
    );

Our final version of the StereoPannerNode example involved linearly ramping the pan value from -1 to 0 and then from 0 to 1. This meant the sound moved from left to right over time.

The rest of our final project just involved implementing methods and therefore did not involve any design choices.

Completed Work

Our Pull Requests:

OSS Project

We created the DOM interface for ConstantSourceNode and implemented the CreateConstantSource API for BaseAudio Context. The connection between the DOM interface and the underlying backend node was already implemented by other contributors before we started the project.

Final Project Step 1 - Oscillator node message

The purpose of our first pull request was to implement an audio node message specific to OscillatorNode. This was within the servo/media project. For this pull request, we had to create a new OscillatorNodeMessage within the oscillator_node.rs file. The purpose of the OscillatorNodeMessage is to set the type of the OscillatorNode. We then had to modify node.rs to utilize the new OscillatorNodeMessage.

Final Project Step 2 - Type attribute for OscillatorNode interface

The purpose of our second pull request was to implement the type attribute for the OscillatorNode interface within the main servo project. This required enabling the setter throws and OscillatorType attribute within the OscillatorNode.webidl file. More importantly, it involved modifying the oscillatornode.rs file to include methods to set and get the type of the OscillatorNode. This was challenging because we did not realize at first that we needed to wrap the oscillator_type in a cell in order to achieve interior mutability. Initially, we were handing references to the setters and getters of the DOM struct members, which did not expose the set and get methods we were calling. We had to wrap the oscillator_type in a cell to give mutable references instead.

Final Project Step 3 - StereoPannerNode backend

For our third pull request, we had to implement the backend for the StereoPannerNode in servo/media. Then we had to create a runnable example that utilizes the StereoPannerNode. We designed our example so that we would initialize a StereoPannerNode with the default pan value of 0. Then, we would set the pan value to -1. Next, we would linearly ramp up the pan value to 0 and then to 1. A user could run this example and listen to the StereoPannerNode through speakers as its pan value changed over time, causing the sound to move from left to right.

Final Project Step 4 - StereoPannerNode DOM

For our fourth and last pull request, we created the DOM interface for StereoPannerNode and implemented the createStereoPannerNode API for BaseAudioContext within the main servo project. We had to create a few new files for this, since there was no existing stereopannernode.rs or StereoPannerNode.webidl. We also had to modify audionode.rs to add constraints when creating a StereoPannerNode, such as to ensure that there are not more than two channels and that the channel count mode is not set to max. In that case, we had to return a NotSupported error.

Testing

Testing from Command Line

After following the setup instructions in the Servo readme, cloning the repository, and changing directories into it, you can build the project with the following command from the terminal (the readme includes the different build commands to use depending on your OS). Note that it may take a long time to build the project.

./mach build --release

After building, you can run the relevant tests with the following command.

./mach test-wpt tests/wpt/web-platform-tests/webaudio/the-audio-api/the-constantsourcenode-interface/

You will notice that all tests pass as expected.

Testing from UI

Our project cannot be tested from a UI since it involves the Servo project.