Expertiza_Wiki - User contributions [en]

M1852 Implement missing WebAudio node support

2018-12-07T00:59:31Z

Pbhalas: /* Build */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

[[File:initialsteps.png]]
[[File:Subsequentsteps.png]]

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source_node.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs
* media/examples/constant_source.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source_node.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::{AudioNodeEngine, AudioScheduledSourceNodeMessage, OnEndedCallback};
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo, ShouldPlay};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioScheduledSourceNode,AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
start_at: Option<Tick>,
stop_at: Option<Tick>,
onended_callback: Option<OnEndedCallback>,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset.into()),
start_at: None,
stop_at: None,
onended_callback: None,
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {

debug_assert!(inputs.len() == 0);

inputs.blocks.push(Default::default());

let (start_at, stop_at) = match self.should_play_at(info.frame) {
ShouldPlay::No => {
return inputs;
}
ShouldPlay::Between(start, end) => (start, end),
};

{
inputs.blocks[0].explicit_silence();

let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue;
} else if tick > stop_at {
break;
}
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}
frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}
fn input_count(&self) -> u32{
0
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
make_message_handler!(AudioScheduledSourceNode: handle_source_node_message);
}
</pre>

We created the <code>constant_source.rs</code> file in the <code>examples/</code> directory which realizes the implementation of Constant Source Node
<pre>
extern crate servo_media;

use servo_media::audio::constant_source_node::ConstantSourceNodeOptions;
use servo_media::audio::gain_node::GainNodeOptions;
use servo_media::audio::param::{ParamType, RampKind, UserAutomationEvent};
use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();

let mut cs_options = ConstantSourceNodeOptions::default();
cs_options.offset = 0.;
let cs = context.create_node(
AudioNodeInit::ConstantSourceNode(cs_options.clone()),
Default::default(),
);

let mut gain_options = GainNodeOptions::default();
gain_options.gain = 0.1;
let gain = context.create_node(

AudioNodeInit::GainNode(gain_options.clone()),

Default::default(),
);

let osc = context.create_node(
AudioNodeInit::OscillatorNode(Default::default()),
Default::default(),
);

context.connect_ports(osc.output(0), gain.input(0));
context.connect_ports(cs.output(0), gain.param(ParamType::Gain));
context.connect_ports(gain.output(0), dest.input(0));

let _ = context.resume();
context.message_node(
osc,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(

gain,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 1., 1.5),
),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 0.1, 3.0),
),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 1., 4.5),
),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 0.1, 6.0),
),
);

thread::sleep(time::Duration::from_millis(9000));
let _ = context.close();

}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

=='''Build'''==
12/06- Completed the changes stated in the review for Constant Source Node example. The updated pull request can be accessed via [https://github.com/servo/media/pull/164 here]

11/9- The formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

4) Run the command <big><code>./constant_source</code></big>

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-12-07T00:59:02Z

Pbhalas: /* Build */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

[[File:initialsteps.png]]
[[File:Subsequentsteps.png]]

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source_node.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs
* media/examples/constant_source.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source_node.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::{AudioNodeEngine, AudioScheduledSourceNodeMessage, OnEndedCallback};
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo, ShouldPlay};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioScheduledSourceNode,AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
start_at: Option<Tick>,
stop_at: Option<Tick>,
onended_callback: Option<OnEndedCallback>,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset.into()),
start_at: None,
stop_at: None,
onended_callback: None,
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {

debug_assert!(inputs.len() == 0);

inputs.blocks.push(Default::default());

let (start_at, stop_at) = match self.should_play_at(info.frame) {
ShouldPlay::No => {
return inputs;
}
ShouldPlay::Between(start, end) => (start, end),
};

{
inputs.blocks[0].explicit_silence();

let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue;
} else if tick > stop_at {
break;
}
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}
frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}
fn input_count(&self) -> u32{
0
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
make_message_handler!(AudioScheduledSourceNode: handle_source_node_message);
}
</pre>

We created the <code>constant_source.rs</code> file in the <code>examples/</code> directory which realizes the implementation of Constant Source Node
<pre>
extern crate servo_media;

use servo_media::audio::constant_source_node::ConstantSourceNodeOptions;
use servo_media::audio::gain_node::GainNodeOptions;
use servo_media::audio::param::{ParamType, RampKind, UserAutomationEvent};
use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();

let mut cs_options = ConstantSourceNodeOptions::default();
cs_options.offset = 0.;
let cs = context.create_node(
AudioNodeInit::ConstantSourceNode(cs_options.clone()),
Default::default(),
);

let mut gain_options = GainNodeOptions::default();
gain_options.gain = 0.1;
let gain = context.create_node(

AudioNodeInit::GainNode(gain_options.clone()),

Default::default(),
);

let osc = context.create_node(
AudioNodeInit::OscillatorNode(Default::default()),
Default::default(),
);

context.connect_ports(osc.output(0), gain.input(0));
context.connect_ports(cs.output(0), gain.param(ParamType::Gain));
context.connect_ports(gain.output(0), dest.input(0));

let _ = context.resume();
context.message_node(
osc,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(

gain,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 1., 1.5),
),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 0.1, 3.0),
),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 1., 4.5),
),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 0.1, 6.0),
),
);

thread::sleep(time::Duration::from_millis(9000));
let _ = context.close();

}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

=='''Build'''==
12/06- Completed the changes stated in the review for Constant Source Node example. The updated pull request can be accessed via [https://github.com/servo/media/pull/164]

11/9- The formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

4) Run the command <big><code>./constant_source</code></big>

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-12-07T00:27:15Z

Pbhalas: /* Test Plan */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

[[File:initialsteps.png]]
[[File:Subsequentsteps.png]]

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source_node.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs
* media/examples/constant_source.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source_node.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::{AudioNodeEngine, AudioScheduledSourceNodeMessage, OnEndedCallback};
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo, ShouldPlay};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioScheduledSourceNode,AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
start_at: Option<Tick>,
stop_at: Option<Tick>,
onended_callback: Option<OnEndedCallback>,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset.into()),
start_at: None,
stop_at: None,
onended_callback: None,
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {

debug_assert!(inputs.len() == 0);

inputs.blocks.push(Default::default());

let (start_at, stop_at) = match self.should_play_at(info.frame) {
ShouldPlay::No => {
return inputs;
}
ShouldPlay::Between(start, end) => (start, end),
};

{
inputs.blocks[0].explicit_silence();

let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue;
} else if tick > stop_at {
break;
}
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}
frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}
fn input_count(&self) -> u32{
0
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
make_message_handler!(AudioScheduledSourceNode: handle_source_node_message);
}
</pre>

We created the <code>constant_source.rs</code> file in the <code>examples/</code> directory which realizes the implementation of Constant Source Node
<pre>
extern crate servo_media;

use servo_media::audio::constant_source_node::ConstantSourceNodeOptions;
use servo_media::audio::gain_node::GainNodeOptions;
use servo_media::audio::param::{ParamType, RampKind, UserAutomationEvent};
use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();

let mut cs_options = ConstantSourceNodeOptions::default();
cs_options.offset = 0.;
let cs = context.create_node(
AudioNodeInit::ConstantSourceNode(cs_options.clone()),
Default::default(),
);

let mut gain_options = GainNodeOptions::default();
gain_options.gain = 0.1;
let gain = context.create_node(

AudioNodeInit::GainNode(gain_options.clone()),

Default::default(),
);

let osc = context.create_node(
AudioNodeInit::OscillatorNode(Default::default()),
Default::default(),
);

context.connect_ports(osc.output(0), gain.input(0));
context.connect_ports(cs.output(0), gain.param(ParamType::Gain));
context.connect_ports(gain.output(0), dest.input(0));

let _ = context.resume();
context.message_node(
osc,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(

gain,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 1., 1.5),
),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 0.1, 3.0),
),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 1., 4.5),
),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 0.1, 6.0),
),
);

thread::sleep(time::Duration::from_millis(9000));
let _ = context.close();

}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

4) Run the command <big><code>./constant_source</code></big>

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-12-07T00:24:12Z

Pbhalas: /* Code Snippet */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

[[File:initialsteps.png]]
[[File:Subsequentsteps.png]]

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source_node.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs
* media/examples/constant_source.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source_node.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::{AudioNodeEngine, AudioScheduledSourceNodeMessage, OnEndedCallback};
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo, ShouldPlay};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioScheduledSourceNode,AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
start_at: Option<Tick>,
stop_at: Option<Tick>,
onended_callback: Option<OnEndedCallback>,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset.into()),
start_at: None,
stop_at: None,
onended_callback: None,
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {

debug_assert!(inputs.len() == 0);

inputs.blocks.push(Default::default());

let (start_at, stop_at) = match self.should_play_at(info.frame) {
ShouldPlay::No => {
return inputs;
}
ShouldPlay::Between(start, end) => (start, end),
};

{
inputs.blocks[0].explicit_silence();

let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue;
} else if tick > stop_at {
break;
}
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}
frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}
fn input_count(&self) -> u32{
0
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
make_message_handler!(AudioScheduledSourceNode: handle_source_node_message);
}
</pre>

We created the <code>constant_source.rs</code> file in the <code>examples/</code> directory which realizes the implementation of Constant Source Node
<pre>
extern crate servo_media;

use servo_media::audio::constant_source_node::ConstantSourceNodeOptions;
use servo_media::audio::gain_node::GainNodeOptions;
use servo_media::audio::param::{ParamType, RampKind, UserAutomationEvent};
use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();

let mut cs_options = ConstantSourceNodeOptions::default();
cs_options.offset = 0.;
let cs = context.create_node(
AudioNodeInit::ConstantSourceNode(cs_options.clone()),
Default::default(),
);

let mut gain_options = GainNodeOptions::default();
gain_options.gain = 0.1;
let gain = context.create_node(

AudioNodeInit::GainNode(gain_options.clone()),

Default::default(),
);

let osc = context.create_node(
AudioNodeInit::OscillatorNode(Default::default()),
Default::default(),
);

context.connect_ports(osc.output(0), gain.input(0));
context.connect_ports(cs.output(0), gain.param(ParamType::Gain));
context.connect_ports(gain.output(0), dest.input(0));

let _ = context.resume();
context.message_node(
osc,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(

gain,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 1., 1.5),
),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 0.1, 3.0),
),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 1., 4.5),
),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::RampToValueAtTime(RampKind::Linear, 0.1, 6.0),
),
);

thread::sleep(time::Duration::from_millis(9000));
let _ = context.close();

}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-12-07T00:19:39Z

Pbhalas: /* Files (to be) modified */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

[[File:initialsteps.png]]
[[File:Subsequentsteps.png]]

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source_node.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs
* media/examples/constant_source.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T01:17:07Z

Pbhalas: /* Code Snippet */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

[[File:initialsteps.png]]
[[File:Subsequentsteps.png]]

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T01:16:38Z

Pbhalas:

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

[[File:initialsteps.png]]
[[File:Subsequentsteps.png]]

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T01:15:38Z

Pbhalas: /* Subsequent Steps */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

[[File:initialsteps.png]]
[[File:Subsequentsteps.png]]

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

File:Subsequentsteps.png

2018-11-21T01:14:55Z

Pbhalas: uploaded a new version of "File:Subsequentsteps.png"

File:Subsequentsteps.png

2018-11-21T01:11:20Z

Pbhalas:

M1852 Implement missing WebAudio node support

2018-11-21T01:10:44Z

Pbhalas: /* Subsequent Steps */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

[[File:initialsteps.png]]
[[File:subsequentsteps.png]]

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

File:Initialsteps.png

2018-11-21T01:08:19Z

Pbhalas:

M1852 Implement missing WebAudio node support

2018-11-21T01:05:12Z

Pbhalas: /* Subsequent Steps */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

<center>[[File:initial_steps.png|249x599px]]</center> 

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T01:04:55Z

Pbhalas: /* Subsequent Steps */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

<center>[[File:is.png|249x599px]]</center> 

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T01:04:36Z

Pbhalas: /* Subsequent Steps */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

<center>[[File:initial_steps.png|249x599px]]</center> 

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T01:03:20Z

Pbhalas: /* Subsequent Steps */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

<center>[[File:initial_steps.png]]</center> 

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T01:02:36Z

Pbhalas: /* Subsequent Steps */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

<center>[[File:initial_steps.png|200px]]</center> 

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T01:02:16Z

Pbhalas: /* Subsequent Steps */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

<center>[[File:initial_steps.png]]</center> 

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T01:01:51Z

Pbhalas: /* Subsequent Steps */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

<center>[[File:initial_steps.png|200x200px]]</center> 

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T01:01:37Z

Pbhalas: /* Subsequent Steps */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

<center>[[File:initial_steps.png|100x200px]]</center> 

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T00:59:04Z

Pbhalas: /* Subsequent Steps */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

<center>[[File:initial_steps.png|upright=default]]</center> 

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T00:58:30Z

Pbhalas:

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

<center>[[File:initial_steps.png|upright=0.5]]</center> 

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T00:55:11Z

Pbhalas:

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

<center>[[File:initial_steps.png]]</center> 

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T00:50:24Z

Pbhalas:

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T00:49:59Z

Pbhalas: /* Subsequent Steps= */

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T00:49:40Z

Pbhalas:

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

=='''Subsequent Steps'''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T00:49:08Z

Pbhalas:

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design'''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps'''==
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

=='''Subsequent Steps'''==
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

=='''Files (to be) modified'''==
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

M1852 Implement missing WebAudio node support

2018-11-21T00:47:55Z

Pbhalas:

=='''Introduction'''==

==='''Servo'''===
Servo is a modern, high-performance browser engine designed for both application and embedded use and written in the Rust programming language.
It is currently developed on 64bit OS X, 64bit Linux, and Android.

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and high performance.

More information about Rust can be found [https://www.rust-lang.org/en-US/ here]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. Basic audio operations are performed with audio nodes, which are linked together to form an audio routing graph. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context 
2. Inside the context, create sources — such as <audio>, oscillator, stream 
3. Create effects nodes, such as reverb, biquad filter, panner, compressor 
4. Choose final destination of audio, for example your system speakers 
5. Connect the sources up to the effects, and the effects to the destination. 

=='''Problem Statement'''==
Major browsers support the WebAudio standard which can be used to create complex media playback applications from low-level building blocks. Servo is a new, experimental browser that supports some of these building blocks (called audio nodes); the goal of this project is to to improve compatibility with web content that relies on the WebAudio API by implementing missing pieces of incomplete node types (OscillatorNode) along with entire missing nodes (ConstantSourceNode, StereoPannerNode, IIRFilterNode). Additional project information is available [https://github.com/servo/servo/wiki/WebAudio-nodes-student-project#implement-support-for-several-missing-webaudio-node-types here]

=='''Proposed Design''==
Our design and plan of work is split into two set of tasks- Initial steps and Subsequent steps.

==='''Initial Steps''===
* To create a new example program- media/examples/oscillator.rs that exercises the different oscillator types
* To implement the processing algorithm for the different oscillator types like Sine, Square, Sawtooth, Triangle and a Custom wave based on user's input in media/audio/src/oscillator_node.rs so that the example programs in media/examples/oscillator.rs sound correct.
* To implement the missing ConstantSource node type in a new constant_source.rs file in the directory media/audio/src that produces a constant tone based a stored value that can be modified.
* To create the DOM interface for ConstantSourceNode and implement the createConstantSource API for BaseAudioContext
* To connect the DOM interface to the underlying backend node by processing the unimplemented message type.

==='''Subsequent Steps''===
* To implement the backend for StereoPannerNode in the media crate by creating a new node. To create a runnable example for the same.
* To create the DOM interface for StereoPannerNode and implement the createStereoPannerNode API for BaseAudioContext
* To create the DOM interface for IIRFilterNode and implement the createIIRFilterNode API for BaseAudioContext.
* To implement the backend for IIRFilterNode based on the specification's processing model and create a runnable example

==='''Files (to be) modified''===
* media/audio/src/oscillator_node.rs
* media/audio/src/constant_source.rs
* media/audio/src/node.rs
* media/audio/src/lib.rs
* media/audio/src/render_thread.rs
* media/audio/src/param.rs
* media/examples/oscillator.rs

=='''Implementation'''==
Our current implementation involves audio processing for Sine, Square, Sawtooth and Triangle waveforms and we are working on generating the Custom wave. We created a oscillator.rs file which includes examples of the given waveforms. These waveforms can be visualized using the given image: 
<center>[[File:Waveform.jpg]]</center>

The "OscillatorType" variable is a string which has the name of the waveform associated with it. For each waveform, we calculate the value which is used by the OscilaterNode class to generate a wave. Each type of wave has a particular output. The output is a function of phase, with each waveform having its own function as follows:

<center>[[File:Sine.png]]</center> 
<center>[[File:SquareTriangleSawtooth.png]]</center>

==='''Code Snippet'''===
The following changes have been done in the <code>oscillator_node.rs</code> file in the <code>audio/src/</code> directory where the function to generate the waveforms is written
<pre>
// Convert all our parameters to the target type for calculations
let vol: f32 = 1.0;
let sample_rate = info.sample_rate as f64;
let two_pi = 2.0 * PI;

// We're carrying a phase with up to 2pi around instead of working
// on the sample offset. High sample offsets cause too much inaccuracy when
// converted to floating point numbers and then iterated over in 1-steps
//
// Also, if the frequency changes the phase should not
let mut step = two_pi * self.frequency.value() as f64 / sample_rate;
while let Some(mut frame) = iter.next() {
let tick = frame.tick();
if tick < start_at {
continue
} else if tick > stop_at {
break;
}

if self.update_parameters(info, tick) {
step = two_pi * self.frequency.value() as f64 / sample_rate;
}
let mut value = vol;

//Based on the type of wave, the value is calculated using the formulae for the respective waveform
match self.oscillator_type {
OscillatorType::Sine => {
value = vol * f32::sin(NumCast::from(self.phase).unwrap());
}

OscillatorType::Square => {
if self.phase >= PI && self.phase < two_pi {
value = vol * 1.0;
}
else if self.phase > 0.0 && self.phase < PI {
value = vol * (-1.0);
}
}

OscillatorType::Sawtooth => {
value = vol * ((self.phase as f64) / (PI)) as f32;
}

OscillatorType::Triangle => {
if self.phase >= 0. && self.phase < PI/2.{
value = vol * 2.0 * ((self.phase as f64) / (PI)) as f32;
}
else if self.phase >= PI/2. && self.phase < PI {
value = vol * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= PI && self.phase < (3.* PI/2.) {
value = vol * -1. * ( 1. - ( ( (self.phase as f64) - (PI/2.)) * (2./PI) ) as f32 );
}
else if self.phase >= 3.*PI/2. && self.phase < 2.*PI {
value = vol * (-2.0) * ((self.phase as f64) / (PI)) as f32;
}
}

</pre>

The <code> oscillator.rs </code> file in the <code>media/examples/ </code> directory has been created by us which generates objects and calls methods to run the different oscillator type examples

<pre>
extern crate servo_media;

use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::oscillator_node::OscillatorNodeOptions;
use servo_media::audio::oscillator_node::OscillatorType::Sawtooth;
use servo_media::audio::oscillator_node::OscillatorType::Triangle;
//use servo_media::audio::oscillator_node::OscillatorType::Sine;
use servo_media::audio::oscillator_node::OscillatorType::Custom;
use servo_media::audio::oscillator_node::OscillatorType::Square;
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let mut options = OscillatorNodeOptions::default();
let osc1 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc1.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc1,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(3000));

options.oscillator_type = Square;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc2 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc2.output(0), dest.input(0));
let _ = context.resume();
context.message_node(
osc2,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Sawtooth;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc3 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc3.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc3,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Triangle;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc4 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc4.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc4,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

thread::sleep(time::Duration::from_millis(3000));
let _ = context.close();
thread::sleep(time::Duration::from_millis(1000));

options.oscillator_type = Custom;
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();
let osc5 = context.create_node(AudioNodeInit::OscillatorNode(options), Default::default());
context.connect_ports(osc5.output(0), dest.input(0));
thread::sleep(time::Duration::from_millis(3000));

let _ = context.resume();
context.message_node(
osc5,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);
thread::sleep(time::Duration::from_millis(3000));
}

fn main() {
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}
</pre>

After running the code, different sound patterns are created with respect to each waveform (Sine, Square, Sawtooth, Triangle)

We created the <code>constant_source.rs</code> file in the <code>audio/src/</code> directory which has the function to generate a constant tone

<pre>
use block::Chunk;
use block::Tick;
use node::AudioNodeEngine;
use node::BlockInfo;
use node::{AudioNodeType, ChannelInfo};
use param::{Param, ParamType};

#[derive(Copy, Clone, Debug)]
pub struct ConstantSourceNodeOptions {
pub offset: f32,
}

impl Default for ConstantSourceNodeOptions {
fn default() -> Self {
ConstantSourceNodeOptions { offset: 1. }
}
}

#[derive(AudioNodeCommon)]
pub(crate) struct ConstantSourceNode {
channel_info: ChannelInfo,
offset: Param,
}

impl ConstantSourceNode {
pub fn new(options: ConstantSourceNodeOptions, channel_info: ChannelInfo) -> Self {
Self {
channel_info,
offset: Param::new(options.offset),
}
}

pub fn update_parameters(&mut self, info: &BlockInfo, tick: Tick) -> bool {
self.offset.update(info, tick)
}
}

impl AudioNodeEngine for ConstantSourceNode {
fn node_type(&self) -> AudioNodeType {
AudioNodeType::ConstantSourceNode
}

fn process(&mut self, mut inputs: Chunk, info: &BlockInfo) -> Chunk {
debug_assert!(inputs.len() == 1);
inputs.blocks[0].explicit_silence();
if inputs.blocks[0].is_silence() {
return inputs;
}

{
let mut iter = inputs.blocks[0].iter();
let mut offset = self.offset.value();
while let Some(mut frame) = iter.next() {
if self.update_parameters(info, frame.tick()) {
offset = self.offset.value();
}

frame.mutate_with(|sample, _| *sample = offset);
}
}
inputs
}

fn get_param(&mut self, id: ParamType) -> &mut Param {
match id {
ParamType::Offset => &mut self.offset,
_ => panic!("Unknown param {:?} for the offset", id),
}
}
}

</pre>

=='''Build'''==

11/9- The the formatted changes have been committed as said in the review.The changes to the formatting can be seen [https://github.com/Avanthikaa/media/commit/5d8cdfc029daf69122c64f8bfcceaec6f80916a4 here]

[[File:format_commit.png]]

<big>11/8- A Servo team member (@Manishearth) approved the pull request and advised us to make some minor formatting changes.</big>

[[File:pullReq.png]]

11/8- Submitted a pull request which contained the new oscillator.rs file having the examples of implementation of the different OscillatorType or waveforms

We have opened our pull request and are working on getting it merged based on the reviews received. Please use our forked repository till the pull request is merged.

Forked repository: <big><code>https://github.com/servo/media</code></big>

Clone command using git:
<big><code>git clone https://github.com/servo/media.git</code></big>

Once you have the forked repo, please follow steps [https://github.com/servo/media#servo-media here] to do a build.

Note that build may take up to 45 minutes, based on your system configuration.

==='''Installing dependencies'''===

Installing Rust

The initial step is to install Rust. You can do so by installing Rustup, version 1.8.0 or more recent.

1. To install on Windows, download and run [https://win.rustup.rs/ rustup-init.exe] then follow the onscreen instructions.

2. To install on other systems, run:

<code>curl https://sh.rustup.rs -sSf | sh</code>

For other installation methds click [https://github.com/rust-lang-nursery/rustup.rs/#other-installation-methods here]

----

Local build instructions for Debian-based Linuxes are given below:

1. Run <code>./mach bootstrap.</code> If this fails, run the commands below:
<code>
sudo apt install git curl autoconf libx11-dev \
libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \
gperf g++ build-essential cmake virtualenv python-pip \
libssl1.0-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev \
libglu1-mesa-dev libgles2-mesa-dev libegl1-mesa-dev libdbus-1-dev \
libharfbuzz-dev ccache clang \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev autoconf2.13</code>

If you using a version prior to Ubuntu 17.04 or Debian Sid, replace libssl1.0-dev with libssl-dev. Additionally, you'll need a local copy of GStreamer with a version later than 12.0. You can place it in support/linux/gstreamer/gstreamer, or run ./mach bootstrap-gstreamer to set it up.

If you are using Ubuntu 16.04 run export HARFBUZZ_SYS_NO_PKG_CONFIG=1 before building to avoid an error with harfbuzz.

If you are on Ubuntu 14.04 and encountered errors on installing these dependencies involving libcheese, see #6158 for a workaround. You may also need to install gcc 4.9, clang 4.0, and cmake 3.2:

If virtualenv does not exist, try python-virtualenv.

----

Local build instructions for Windows environments are given below:

1. Install Python for Windows (https://www.python.org/downloads/release/python-2714/).

The Windows x86-64 MSI installer is fine. You should change the installation to install the "Add python.exe to Path" feature.

2. Install virtualenv.

In a normal Windows Shell (cmd.exe or "Command Prompt" from the start menu), do:

<big><code>pip install virtualenv</code></big>

If this does not work, you may need to reboot for the changed PATH settings (by the python installer) to take effect.

3. Install Git for Windows (https://git-scm.com/download/win). DO allow it to add git.exe to the PATH (default settings for the installer are fine).

4. Install Visual Studio Community 2017 (https://www.visualstudio.com/vs/community/).

You MUST add "Visual C++" to the list of installed components. It is not on by default. Visual Studio 2017 MUST installed to the default location or mach.bat will not find it.

If you encountered errors with the environment above, do the following for a workaround:

Download and install Build Tools for Visual Studio 2017

Install python2.7 x86-x64 and virtualenv

5. Run <big><code>mach.bat build -d</code></big> to build

If you have troubles with x64 type prompt as mach.bat set by default:

you may need to choose and launch the type manually, such as x86_x64 Cross Tools Command Prompt for VS 2017 in the Windows menu.)

cd to/the/path/servo

python mach build -d

Build instructions for all other environments are available [https://github.com/servo/servo here]

==='''Building locally'''===

Once you complete the above steps to install dependencies, following are steps to build servo:

To build Servo in development mode. This is useful for development, but the resulting binary is very slow.

<code>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html </code>

Or on Windows MSVC, in a normal Command Prompt (cmd.exe):
<code>
git clone https://github.com/servo/servo
cd servo
mach.bat build --dev </code>

For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:

<code>
./mach build --release
./mach run --release tests/html/about-mozilla.html
Note: mach build will build both servo and libsimpleservo. To make compilation a bit faster, it's possible to only compile the servo binary: ./mach build --dev -p servo.

</code>

==='''Running a build'''===

We can quickly verify if the servo build is working by running the command

<big><code>'./mach run http://www.google.com'</code></big>

This will open a browser instance rendering the Google homepage.

This should be straightforward on any environment that has rendering support - Linus, Windows, MacOS, Android

=='''Test Plan'''==

'''Testing Approach'''

As we have implemented the processing algorithm for oscillator different oscillator types, after running the rust file the user will be able to hear the different generated waveforms based on its type. (Sine, Square, Sawtooth, Triangle)

'''Testing Steps'''

Please read and perform the actions on the Build and Verification sections properly before testing.

1) Make sure you are in this directory: <big><code>target/debug</code></big>

2) Run the command <big><code>./oscillator</code></big>

3) Different sounds based on the type of oscillator (waveform) must be generated after definite interval of time.

Examples for the different sound waves can be found below:

<li>[https://upload.wikimedia.org/wikipedia/commons/d/d1/220_Hz_sine_wave.ogg Sine]
<li>[https://upload.wikimedia.org/wikipedia/commons/9/92/Square_wave_1000.ogg Square]
<li>[https://en.audiofanzine.com/medias/audio/#id:473859 Sawtooth]
<li>[https://upload.wikimedia.org/wikipedia/commons/d/df/220_Hz_anti-aliased_triangle_wave.ogg Triangle]

==='''Pull Request'''===

The pull request used to incorporate our changes upstream is available [https://github.com/servo/media/pull/163 here]

=='''References'''==

https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API

https://webaudio.github.io/web-audio-api/

https://doc.servo.org/servo/index.html

http://www.servo.org

https://doc.rust-lang.org/rust-by-example/

File:Initial steps.png

2018-11-21T00:45:52Z

Pbhalas:

E1835 Refactor delayed mailer and scheduled task

2018-11-09T19:47:28Z

Pbhalas: /* Testing using RSPEC */

== About Expertiza ==

Expertiza is an open source project based on Ruby on Rails framework. Expertiza allows the instructor to create new assignments, modify existing assignments and set deadlines. The instructor can also create a list of project topics that the students can sign up for. Expertiza allows students to peer review submissions of other students as well as do teammate reviews. Students can form team on Expertiza to work on assignments and projects.

== Problem Statement ==

The following tasks were accomplished in this project:
* Used Sidekiq gem for asynchronous processing of email jobs
* Created a new mailer class- MailWorker that uses Sidekiq's queue to hold and process jobs
* Defined the perform method to extract the Email IDs of all participants of the corresponding assignment and send an email reminder
* Replaced the code that uses the existing DelayedMailer queue to incorporate Sidekiq's queue
* Added RSPEC test cases to test the creation of assignment deadlines and background Sidekiq email job

== About Sidekiq ==

Sidekiq is a background processing framework for Ruby. It allows to scale our application by performing work in the background. In particular it consists of three parts: Client, Redis and Server. The client runs in Ruby process and allows us to create jobs for processing later. Once a job is created, a Hash representing that job is created and serialized to a Json String. This String is pushed into a queue in Redis. Redis provides data storage for Sidekiq and holds all the job data. Each Sidekiq server process pulls jobs from the queue in Redis and processes them.

== Current Implementation ==

* Uses the 'delayed_job_active_record' gem for asynchronous processing of delayed jobs
::The current implementation uses Delayed::Job for implementing a Database based asynchronous priority queue system for storing the job queue of email reminders corresponding to various deadline types.
* Email reminders were sent to users corresponding to each deadline type
::The DelayedMailer class includes a perform method that finds Email IDs of the target users corresponding to the deadline type. The subject and body of the email message is then constructed and the delayed_message method of Mailer class is used for sending the emails out to the users.
* Uses DelayedJob to enqueue email jobs
::The DelayedMailer object is enqueued into the DelayedJob queue using enqueue method. These jobs are meant to be executed as the deadline of the corresponding Email job approaches.

== Drawbacks and Solutions ==
'''Problem 1''': The perform method in the DelayedMailer class contains a big case statement to check each kind of deadline

The perform method in DelayedMailer class checks the deadline type individually using multiple if statements and populates the Email IDs of the target users by querying the database. Auxiliary methods like, for example, mail_signed_up_users is invoked which in turn calls the email_reminder method.
This makes the implementation very cumbersome and does not follow the principles of DRY code as the same email_reminder method is called multiple times under different conditions.
<pre>
def perform
assignment = Assignment.find(self.assignment_id)
if !assignment.nil? && !assignment.id.nil?
if self.deadline_type == "metareview"
mail_metareviewers
team_mails = find_team_members_email
email_reminder(team_mails, "teammate review") unless team_mails.empty?
end
if self.deadline_type == "review"
mail_reviewers # to all reviewers
end
if self.deadline_type == "submission"
mail_signed_up_users # to all signed up users
end
if self.deadline_type == "drop_topic"
if sign_up_topics?
mail_signed_up_users # reminder to signed_up users of the assignment
end
end
if self.deadline_type == "signup"
if sign_up_topics?
mail_assignment_participants # reminder to all participants
end
end
if self.deadline_type == "team_formation"
emails = get_one_member_team
email_reminder(emails, self.deadline_type)
end
if self.deadline_type == "drop_one_member_topics"
drop_one_member_topics if assignment.team_assignment?
end
drop_outstanding_reviews if self.deadline_type == "drop_outstanding_reviews"
perform_simicheck_comparisons(self.assignment_id) if self.deadline_type == "compare_files_with_simicheck"
end
end
</pre>

'''Solution''': The implementation has been changed in such a way that all Participants belonging to the assignment under consideration receive emails irrespective of the deadline type. This is found out by querying the Participant table using assignment_id and retrieving the Email ID by linking it to the User table.
* The Participant table is queried using assignment_id
* For each participant, the Email ID is retrieved by linking the Participant and User table.
* The email reminders are sent collectively to all the email addresses from the previous step.

'''Problem 2''': Scalability Issue: Delayed Job uses SQL database for storing the jobs and processes them in a single-threaded process. It's simple to set up but the performance and scalability are not great. It is not suitable for processing 100,000s of jobs a day.

'''Solution''': Sidekiq uses redis for storage which implements a distributed in-memory key-value database and processes jobs in a multithreaded process. It is efficient in terms of processing speed. The worker code needs to be thread safe.

== New Implementation ==
* In the new implementation a new class MailWorker in app/mailers/mail_worker.rb is created that uses the Sidekiq worker for the mailers queue. The three instance variables- assignment id, deadline type and due date are used in this class as well.
::* The perform method takes the above three parameters and calls the auxiliary method find_participant_emails. This method queries the Participant table to extract the Email IDs of the users participating in an assignment and returns the email list.
::* The email reminder method is unchanged from the previous implementation.

<pre>
require 'sidekiq'
class MailWorker
include Sidekiq::Worker
# ActionMailer in Rail 4 submits jobs in mailers queue instead of default queue. Rails 5 and onwards
# ActionMailer will submit mailer jobs to default queue. We need to remove the line below in that case!
sidekiq_options queue: 'mailers'
attr_accessor :assignment_id
attr_accessor :deadline_type
attr_accessor :due_at
def perform(assignment_id, deadline_type, due_at)
self.assignment_id = assignment_id
self.deadline_type = deadline_type
self.due_at = due_at
assignment = Assignment.find(self.assignment_id)
participant_mails = find_participant_emails
if ["drop_one_member_topics", "drop_outstanding_reviews", "compare_files_with_simicheck"].include?(self.deadline_type)
drop_one_member_topics if self.deadline_type == "drop_outstanding_reviews" && assignment.team_assignment
drop_outstanding_reviews if self.deadline_type == "drop_outstanding_reviews"
perform_simicheck_comparisons(self.assignment_id) if self.deadline_type == "compare_files_with_simicheck"
else
# Can we rename deadline_type(metareview) to "teammate review". If, yes then we donot need this if clause below!
deadlineText = if self.deadline_type == "metareview"
"teammate review"
else
self.deadline_type
end
email_reminder(participant_mails, deadlineText) unless participant_mails.empty?
end
end
def email_reminder(emails, deadline_type)
assignment = Assignment.find(self.assignment_id)
subject = "Message regarding #{deadline_type} for assignment #{assignment.name}"
body = "This is a reminder to complete #{deadline_type} for assignment #{assignment.name}. \
Deadline is #{self.due_at}.If you have already done the #{deadline_type}, Please ignore this mail."
emails.each do |mail|
Rails.logger.info mail
end
@mail = Mailer.delayed_message(bcc: emails, subject: subject, body: body)
@mail.deliver_now
end
def find_participant_emails
emails = []
participants = Participant.where(parent_id: self.assignment_id)
participants.each do |participant|
emails<<participant.user.email unless participant.user.nil?
end
emails
end
</pre>
* The app/models/assignment_form.rb is modified to make use of the Sidekiq's perform method to queue the job and process it when the deadline approaches.
::*The add_delayed_job method now uses Sidekiq's perform_in method to store the email job in the job queue and executes the email job after the time (in seconds) as specified in the first argument has elapsed.
::*The corresponding job_id is returned back for further processing.
::*The get_time_diff_btw_due_date_and_now method calculates the time after which the email has to be sent and is unchanged.
<pre>
def add_to_delayed_queue
duedates = AssignmentDueDate.where(parent_id: @assignment.id)
duedates.each do |due_date|
deadline_type = DeadlineType.find(due_date.deadline_type_id).name
diff_btw_time_left_and_threshold, min_left = get_time_diff_btw_due_date_and_now(due_date)
next unless diff_btw_time_left_and_threshold > 0
delayed_job_id = add_delayed_job(@assignment, deadline_type, due_date, diff_btw_time_left_and_threshold)
due_date.update_attribute(:delayed_job_id, delayed_job_id)
# If the deadline type is review, add a delayed job to drop outstanding review
add_delayed_job(@assignment, "drop_outstanding_reviews", due_date, min_left) if deadline_type == "review"
# If the deadline type is team_formation, add a delayed job to drop one member team
next unless deadline_type == "team_formation" and @assignment.team_assignment?
add_delayed_job(@assignment, "drop_one_member_topics", due_date, min_left)
end
end

def get_time_diff_btw_due_date_and_now(due_date)
due_at = due_date.due_at.to_s(:db)
Time.parse(due_at)
due_at = Time.parse(due_at)
time_left_in_min = find_min_from_now(due_at)
diff_btw_time_left_and_threshold = time_left_in_min - due_date.threshold * 60
[diff_btw_time_left_and_threshold, time_left_in_min]
end

# add DelayedJob into queue and return it
def add_delayed_job(assignment, deadline_type, due_date, min_left)
delayed_job_id = MailWorker.perform_in(min_left*60, due_date.parent_id, deadline_type, due_date.due_at )
change_item_type(delayed_job_id)
delayed_job_id
end
</pre>

== Test Plan ==
== Testing using RSPEC ==
We modified the existing test cases to replace the delayed job queue with Sidekiq, to test all modifications we have done to the code of assignment_form class. We also added RSpec test for Sidekiq mailer.
<pre>
describe MailWorker do
let(:assignment) { build(:assignment, id: 1, name: "no assignment", participants: [participant], teams: [team]) }
let(:participant) { build(:participant, id: 1, parent_id: 1, user: user) }
let(:team) { build(:assignment_team, id: 1, name: 'no team', users: [user], parent_id: 1) }
let(:user) { build(:student, id: 1, email: 'psingh22@ncsu.edu') }

before(:each) do
allow(Assignment).to receive(:find).with('1').and_return(assignment)
allow(Participant).to receive(:where).with(parent_id: '1').and_return([participant])
end

describe 'Tests mailer with sidekiq' do

it "should send email to required email address with proper content" do
Sidekiq::Testing.inline!
MailWorker.perform_async("1", "metareview", "2018-12-31 00:00:01")
email = ActionMailer::Base.deliveries.first
expect(email.from[0]).to eq("expertiza.development@gmail.com")
expect(email.bcc[0]).to eq(user.email)
expect(email.subject).to eq('Message regarding teammate review for assignment '+ assignment.name)
end

it "should expect the queue size of one" do
Sidekiq::Testing.fake!
MailWorker.perform_in(3.hours, "1", "metareview", "2018-12-31 00:00:01")
queue = Sidekiq::Queues["mailers"]
expect(queue.size).to eq(1)
end
end
end
</pre>
The tests can be executed "rspec spec" command as shown below:
<pre>
user-expertiza $rspec spec/sidekiq_mail_worker_spec.rb.

Finished in 3.19 seconds
2 examples, 0 failures
</pre>

== Testing from UI ==
1. To go to versions index page, type in the following url after logging in as instructor:
http://152.46.19.3:8080

2. Create a new assignment and add the rubric and other necessary fields and add the late policy.
http://152.46.19.3:8080/assignments/new?private=0

3. You can see the new jobs populated here
http://152.46.19.3:8080/sidekiq/scheduled

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[https://github.com/WintersLt/expertiza GitHub Project Repository Fork]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[http://bit.ly/myexpertiza Demo link]
#[http://wikis.lib.ncsu.edu/index.php/Expertiza Expertiza project documentation wiki]
#[https://relishapp.com/rspec Rspec Documentation]
#[https://github.com/mperham/sidekiq/wiki Sidekiq Documentation]
#Clean Code: A handbook of agile software craftsmanship. Author: Robert C Martin

E1835 Refactor delayed mailer and scheduled task

2018-11-09T19:25:36Z

Pbhalas: /* Testing using RSPEC */

== About Expertiza ==

Expertiza is an open source project based on Ruby on Rails framework. Expertiza allows the instructor to create new assignments, modify existing assignments and set deadlines. The instructor can also create a list of project topics that the students can sign up for. Expertiza allows students to peer review submissions of other students as well as do teammate reviews. Students can form team on Expertiza to work on assignments and projects.

== Problem Statement ==

The following tasks were accomplished in this project:
* Used Sidekiq gem for asynchronous processing of email jobs
* Created a new mailer class- MailWorker that uses Sidekiq's queue to hold and process jobs
* Defined the perform method to extract the Email IDs of all participants of the corresponding assignment and send an email reminder
* Replaced the code that uses the existing DelayedMailer queue to incorporate Sidekiq's queue
* Added RSPEC test cases to test the creation of assignment deadlines and background Sidekiq email job

== About Sidekiq ==

Sidekiq is a background processing framework for Ruby. It allows to scale our application by performing work in the background. In particular it consists of three parts: Client, Redis and Server. The client runs in Ruby process and allows us to create jobs for processing later. Once a job is created, a Hash representing that job is created and serialized to a Json String. This String is pushed into a queue in Redis. Redis provides data storage for Sidekiq and holds all the job data. Each Sidekiq server process pulls jobs from the queue in Redis and processes them.

== Current Implementation ==

* Uses the 'delayed_job_active_record' gem for asynchronous processing of delayed jobs
::The current implementation uses Delayed::Job for implementing a Database based asynchronous priority queue system for storing the job queue of email reminders corresponding to various deadline types.
* Email reminders were sent to users corresponding to each deadline type
::The DelayedMailer class includes a perform method that finds Email IDs of the target users corresponding to the deadline type. The subject and body of the email message is then constructed and the delayed_message method of Mailer class is used for sending the emails out to the users.
* Uses DelayedJob to enqueue email jobs
::The DelayedMailer object is enqueued into the DelayedJob queue using enqueue method. These jobs are meant to be executed as the deadline of the corresponding Email job approaches.

== Drawbacks and Solutions ==
'''Problem 1''': The perform method in the DelayedMailer class contains a big case statement to check each kind of deadline

The perform method in DelayedMailer class checks the deadline type individually using multiple if statements and populates the Email IDs of the target users by querying the database. Auxiliary methods like, for example, mail_signed_up_users is invoked which in turn calls the email_reminder method.
This makes the implementation very cumbersome and does not follow the principles of DRY code as the same email_reminder method is called multiple times under different conditions.
<pre>
def perform
assignment = Assignment.find(self.assignment_id)
if !assignment.nil? && !assignment.id.nil?
if self.deadline_type == "metareview"
mail_metareviewers
team_mails = find_team_members_email
email_reminder(team_mails, "teammate review") unless team_mails.empty?
end
if self.deadline_type == "review"
mail_reviewers # to all reviewers
end
if self.deadline_type == "submission"
mail_signed_up_users # to all signed up users
end
if self.deadline_type == "drop_topic"
if sign_up_topics?
mail_signed_up_users # reminder to signed_up users of the assignment
end
end
if self.deadline_type == "signup"
if sign_up_topics?
mail_assignment_participants # reminder to all participants
end
end
if self.deadline_type == "team_formation"
emails = get_one_member_team
email_reminder(emails, self.deadline_type)
end
if self.deadline_type == "drop_one_member_topics"
drop_one_member_topics if assignment.team_assignment?
end
drop_outstanding_reviews if self.deadline_type == "drop_outstanding_reviews"
perform_simicheck_comparisons(self.assignment_id) if self.deadline_type == "compare_files_with_simicheck"
end
end
</pre>

'''Solution''': The implementation has been changed in such a way that all Participants belonging to the assignment under consideration receive emails irrespective of the deadline type. This is found out by querying the Participant table using assignment_id and retrieving the Email ID by linking it to the User table.
* The Participant table is queried using assignment_id
* For each participant, the Email ID is retrieved by linking the Participant and User table.
* The email reminders are sent collectively to all the email addresses from the previous step.

'''Problem 2''': Scalability Issue: Delayed Job uses SQL database for storing the jobs and processes them in a single-threaded process. It's simple to set up but the performance and scalability are not great. It is not suitable for processing 100,000s of jobs a day.

'''Solution''': Sidekiq uses redis for storage which implements a distributed in-memory key-value database and processes jobs in a multithreaded process. It is efficient in terms of processing speed. The worker code needs to be thread safe.

== New Implementation ==
* In the new implementation a new class MailWorker in app/mailers/mail_worker.rb is created that uses the Sidekiq worker for the mailers queue. The three instance variables- assignment id, deadline type and due date are used in this class as well.
::* The perform method takes the above three parameters and calls the auxiliary method find_participant_emails. This method queries the Participant table to extract the Email IDs of the users participating in an assignment and returns the email list.
::* The email reminder method is unchanged from the previous implementation.

<pre>
require 'sidekiq'
class MailWorker
include Sidekiq::Worker
# ActionMailer in Rail 4 submits jobs in mailers queue instead of default queue. Rails 5 and onwards
# ActionMailer will submit mailer jobs to default queue. We need to remove the line below in that case!
sidekiq_options queue: 'mailers'
attr_accessor :assignment_id
attr_accessor :deadline_type
attr_accessor :due_at
def perform(assignment_id, deadline_type, due_at)
self.assignment_id = assignment_id
self.deadline_type = deadline_type
self.due_at = due_at
assignment = Assignment.find(self.assignment_id)
participant_mails = find_participant_emails
if ["drop_one_member_topics", "drop_outstanding_reviews", "compare_files_with_simicheck"].include?(self.deadline_type)
drop_one_member_topics if self.deadline_type == "drop_outstanding_reviews" && assignment.team_assignment
drop_outstanding_reviews if self.deadline_type == "drop_outstanding_reviews"
perform_simicheck_comparisons(self.assignment_id) if self.deadline_type == "compare_files_with_simicheck"
else
# Can we rename deadline_type(metareview) to "teammate review". If, yes then we donot need this if clause below!
deadlineText = if self.deadline_type == "metareview"
"teammate review"
else
self.deadline_type
end
email_reminder(participant_mails, deadlineText) unless participant_mails.empty?
end
end
def email_reminder(emails, deadline_type)
assignment = Assignment.find(self.assignment_id)
subject = "Message regarding #{deadline_type} for assignment #{assignment.name}"
body = "This is a reminder to complete #{deadline_type} for assignment #{assignment.name}. \
Deadline is #{self.due_at}.If you have already done the #{deadline_type}, Please ignore this mail."
emails.each do |mail|
Rails.logger.info mail
end
@mail = Mailer.delayed_message(bcc: emails, subject: subject, body: body)
@mail.deliver_now
end
def find_participant_emails
emails = []
participants = Participant.where(parent_id: self.assignment_id)
participants.each do |participant|
emails<<participant.user.email unless participant.user.nil?
end
emails
end
</pre>
* The app/models/assignment_form.rb is modified to make use of the Sidekiq's perform method to queue the job and process it when the deadline approaches.
::*The add_delayed_job method now uses Sidekiq's perform_in method to store the email job in the job queue and executes the email job after the time (in seconds) as specified in the first argument has elapsed.
::*The corresponding job_id is returned back for further processing.
::*The get_time_diff_btw_due_date_and_now method calculates the time after which the email has to be sent and is unchanged.
<pre>
def add_to_delayed_queue
duedates = AssignmentDueDate.where(parent_id: @assignment.id)
duedates.each do |due_date|
deadline_type = DeadlineType.find(due_date.deadline_type_id).name
diff_btw_time_left_and_threshold, min_left = get_time_diff_btw_due_date_and_now(due_date)
next unless diff_btw_time_left_and_threshold > 0
delayed_job_id = add_delayed_job(@assignment, deadline_type, due_date, diff_btw_time_left_and_threshold)
due_date.update_attribute(:delayed_job_id, delayed_job_id)
# If the deadline type is review, add a delayed job to drop outstanding review
add_delayed_job(@assignment, "drop_outstanding_reviews", due_date, min_left) if deadline_type == "review"
# If the deadline type is team_formation, add a delayed job to drop one member team
next unless deadline_type == "team_formation" and @assignment.team_assignment?
add_delayed_job(@assignment, "drop_one_member_topics", due_date, min_left)
end
end

def get_time_diff_btw_due_date_and_now(due_date)
due_at = due_date.due_at.to_s(:db)
Time.parse(due_at)
due_at = Time.parse(due_at)
time_left_in_min = find_min_from_now(due_at)
diff_btw_time_left_and_threshold = time_left_in_min - due_date.threshold * 60
[diff_btw_time_left_and_threshold, time_left_in_min]
end

# add DelayedJob into queue and return it
def add_delayed_job(assignment, deadline_type, due_date, min_left)
delayed_job_id = MailWorker.perform_in(min_left*60, due_date.parent_id, deadline_type, due_date.due_at )
change_item_type(delayed_job_id)
delayed_job_id
end
</pre>

== Test Plan ==
== Testing using RSPEC ==
We modified the existing test cases to replace the delayed job queue with Sidekiq, to test all modifications we have done to the code of assignment_form class.We also added RSpec test for Sidekiq mailer. The tests can be executed "rspec spec" command as shown below:
<pre>
user-expertiza $rspec spec/sidekiq_mail_worker_spec.rb.

Finished in 3.19 seconds
2 examples, 0 failures
</pre>

== Testing from UI ==
1. To go to versions index page, type in the following url after logging in as instructor:
http://152.46.19.3:8080

2. Create a new assignment and add the rubric and other necessary fields and add the late policy.
http://152.46.19.3:8080/assignments/new?private=0

3. You can see the new jobs populated here
http://152.46.19.3:8080/sidekiq/scheduled

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[https://github.com/WintersLt/expertiza GitHub Project Repository Fork]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[http://bit.ly/myexpertiza Demo link]
#[http://wikis.lib.ncsu.edu/index.php/Expertiza Expertiza project documentation wiki]
#[https://relishapp.com/rspec Rspec Documentation]
#[https://github.com/mperham/sidekiq/wiki Sidekiq Documentation]
#Clean Code: A handbook of agile software craftsmanship. Author: Robert C Martin

E1835 Refactor delayed mailer and scheduled task

2018-11-09T19:24:26Z

Pbhalas: /* Testing using RSPEC */

== About Expertiza ==

Expertiza is an open source project based on Ruby on Rails framework. Expertiza allows the instructor to create new assignments, modify existing assignments and set deadlines. The instructor can also create a list of project topics that the students can sign up for. Expertiza allows students to peer review submissions of other students as well as do teammate reviews. Students can form team on Expertiza to work on assignments and projects.

== Problem Statement ==

The following tasks were accomplished in this project:
* Used Sidekiq gem for asynchronous processing of email jobs
* Created a new mailer class- MailWorker that uses Sidekiq's queue to hold and process jobs
* Defined the perform method to extract the Email IDs of all participants of the corresponding assignment and send an email reminder
* Replaced the code that uses the existing DelayedMailer queue to incorporate Sidekiq's queue
* Added RSPEC test cases to test the creation of assignment deadlines and background Sidekiq email job

== About Sidekiq ==

Sidekiq is a background processing framework for Ruby. It allows to scale our application by performing work in the background. In particular it consists of three parts: Client, Redis and Server. The client runs in Ruby process and allows us to create jobs for processing later. Once a job is created, a Hash representing that job is created and serialized to a Json String. This String is pushed into a queue in Redis. Redis provides data storage for Sidekiq and holds all the job data. Each Sidekiq server process pulls jobs from the queue in Redis and processes them.

== Current Implementation ==

* Uses the 'delayed_job_active_record' gem for asynchronous processing of delayed jobs
::The current implementation uses Delayed::Job for implementing a Database based asynchronous priority queue system for storing the job queue of email reminders corresponding to various deadline types.
* Email reminders were sent to users corresponding to each deadline type
::The DelayedMailer class includes a perform method that finds Email IDs of the target users corresponding to the deadline type. The subject and body of the email message is then constructed and the delayed_message method of Mailer class is used for sending the emails out to the users.
* Uses DelayedJob to enqueue email jobs
::The DelayedMailer object is enqueued into the DelayedJob queue using enqueue method. These jobs are meant to be executed as the deadline of the corresponding Email job approaches.

== Drawbacks and Solutions ==
'''Problem 1''': The perform method in the DelayedMailer class contains a big case statement to check each kind of deadline

The perform method in DelayedMailer class checks the deadline type individually using multiple if statements and populates the Email IDs of the target users by querying the database. Auxiliary methods like, for example, mail_signed_up_users is invoked which in turn calls the email_reminder method.
This makes the implementation very cumbersome and does not follow the principles of DRY code as the same email_reminder method is called multiple times under different conditions.
<pre>
def perform
assignment = Assignment.find(self.assignment_id)
if !assignment.nil? && !assignment.id.nil?
if self.deadline_type == "metareview"
mail_metareviewers
team_mails = find_team_members_email
email_reminder(team_mails, "teammate review") unless team_mails.empty?
end
if self.deadline_type == "review"
mail_reviewers # to all reviewers
end
if self.deadline_type == "submission"
mail_signed_up_users # to all signed up users
end
if self.deadline_type == "drop_topic"
if sign_up_topics?
mail_signed_up_users # reminder to signed_up users of the assignment
end
end
if self.deadline_type == "signup"
if sign_up_topics?
mail_assignment_participants # reminder to all participants
end
end
if self.deadline_type == "team_formation"
emails = get_one_member_team
email_reminder(emails, self.deadline_type)
end
if self.deadline_type == "drop_one_member_topics"
drop_one_member_topics if assignment.team_assignment?
end
drop_outstanding_reviews if self.deadline_type == "drop_outstanding_reviews"
perform_simicheck_comparisons(self.assignment_id) if self.deadline_type == "compare_files_with_simicheck"
end
end
</pre>

'''Solution''': The implementation has been changed in such a way that all Participants belonging to the assignment under consideration receive emails irrespective of the deadline type. This is found out by querying the Participant table using assignment_id and retrieving the Email ID by linking it to the User table.
* The Participant table is queried using assignment_id
* For each participant, the Email ID is retrieved by linking the Participant and User table.
* The email reminders are sent collectively to all the email addresses from the previous step.

'''Problem 2''': Scalability Issue: Delayed Job uses SQL database for storing the jobs and processes them in a single-threaded process. It's simple to set up but the performance and scalability are not great. It is not suitable for processing 100,000s of jobs a day.

'''Solution''': Sidekiq uses redis for storage which implements a distributed in-memory key-value database and processes jobs in a multithreaded process. It is efficient in terms of processing speed. The worker code needs to be thread safe.

== New Implementation ==
* In the new implementation a new class MailWorker in app/mailers/mail_worker.rb is created that uses the Sidekiq worker for the mailers queue. The three instance variables- assignment id, deadline type and due date are used in this class as well.
::* The perform method takes the above three parameters and calls the auxiliary method find_participant_emails. This method queries the Participant table to extract the Email IDs of the users participating in an assignment and returns the email list.
::* The email reminder method is unchanged from the previous implementation.

<pre>
require 'sidekiq'
class MailWorker
include Sidekiq::Worker
# ActionMailer in Rail 4 submits jobs in mailers queue instead of default queue. Rails 5 and onwards
# ActionMailer will submit mailer jobs to default queue. We need to remove the line below in that case!
sidekiq_options queue: 'mailers'
attr_accessor :assignment_id
attr_accessor :deadline_type
attr_accessor :due_at
def perform(assignment_id, deadline_type, due_at)
self.assignment_id = assignment_id
self.deadline_type = deadline_type
self.due_at = due_at
assignment = Assignment.find(self.assignment_id)
participant_mails = find_participant_emails
if ["drop_one_member_topics", "drop_outstanding_reviews", "compare_files_with_simicheck"].include?(self.deadline_type)
drop_one_member_topics if self.deadline_type == "drop_outstanding_reviews" && assignment.team_assignment
drop_outstanding_reviews if self.deadline_type == "drop_outstanding_reviews"
perform_simicheck_comparisons(self.assignment_id) if self.deadline_type == "compare_files_with_simicheck"
else
# Can we rename deadline_type(metareview) to "teammate review". If, yes then we donot need this if clause below!
deadlineText = if self.deadline_type == "metareview"
"teammate review"
else
self.deadline_type
end
email_reminder(participant_mails, deadlineText) unless participant_mails.empty?
end
end
def email_reminder(emails, deadline_type)
assignment = Assignment.find(self.assignment_id)
subject = "Message regarding #{deadline_type} for assignment #{assignment.name}"
body = "This is a reminder to complete #{deadline_type} for assignment #{assignment.name}. \
Deadline is #{self.due_at}.If you have already done the #{deadline_type}, Please ignore this mail."
emails.each do |mail|
Rails.logger.info mail
end
@mail = Mailer.delayed_message(bcc: emails, subject: subject, body: body)
@mail.deliver_now
end
def find_participant_emails
emails = []
participants = Participant.where(parent_id: self.assignment_id)
participants.each do |participant|
emails<<participant.user.email unless participant.user.nil?
end
emails
end
</pre>
* The app/models/assignment_form.rb is modified to make use of the Sidekiq's perform method to queue the job and process it when the deadline approaches.
::*The add_delayed_job method now uses Sidekiq's perform_in method to store the email job in the job queue and executes the email job after the time (in seconds) as specified in the first argument has elapsed.
::*The corresponding job_id is returned back for further processing.
::*The get_time_diff_btw_due_date_and_now method calculates the time after which the email has to be sent and is unchanged.
<pre>
def add_to_delayed_queue
duedates = AssignmentDueDate.where(parent_id: @assignment.id)
duedates.each do |due_date|
deadline_type = DeadlineType.find(due_date.deadline_type_id).name
diff_btw_time_left_and_threshold, min_left = get_time_diff_btw_due_date_and_now(due_date)
next unless diff_btw_time_left_and_threshold > 0
delayed_job_id = add_delayed_job(@assignment, deadline_type, due_date, diff_btw_time_left_and_threshold)
due_date.update_attribute(:delayed_job_id, delayed_job_id)
# If the deadline type is review, add a delayed job to drop outstanding review
add_delayed_job(@assignment, "drop_outstanding_reviews", due_date, min_left) if deadline_type == "review"
# If the deadline type is team_formation, add a delayed job to drop one member team
next unless deadline_type == "team_formation" and @assignment.team_assignment?
add_delayed_job(@assignment, "drop_one_member_topics", due_date, min_left)
end
end

def get_time_diff_btw_due_date_and_now(due_date)
due_at = due_date.due_at.to_s(:db)
Time.parse(due_at)
due_at = Time.parse(due_at)
time_left_in_min = find_min_from_now(due_at)
diff_btw_time_left_and_threshold = time_left_in_min - due_date.threshold * 60
[diff_btw_time_left_and_threshold, time_left_in_min]
end

# add DelayedJob into queue and return it
def add_delayed_job(assignment, deadline_type, due_date, min_left)
delayed_job_id = MailWorker.perform_in(min_left*60, due_date.parent_id, deadline_type, due_date.due_at )
change_item_type(delayed_job_id)
delayed_job_id
end
</pre>

== Test Plan ==
== Testing using RSPEC ==
We modified the existing test cases to replace the delayed job queue with Sidekiq, to test all modifications we have done to the code of assignment_form class.We also added RSpec test for Sidekiq mailer. The tests can be executed "rpec spec" command as shown below:
<pre>
user-expertiza $rspec spec/sidekiq_mail_worker_spec.rb.

Finished in 3.19 seconds
2 examples, 0 failures
</pre>

== Testing from UI ==
1. To go to versions index page, type in the following url after logging in as instructor:
http://152.46.19.3:8080

2. Create a new assignment and add the rubric and other necessary fields and add the late policy.
http://152.46.19.3:8080/assignments/new?private=0

3. You can see the new jobs populated here
http://152.46.19.3:8080/sidekiq/scheduled

== References ==
#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[https://github.com/WintersLt/expertiza GitHub Project Repository Fork]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[http://bit.ly/myexpertiza Demo link]
#[http://wikis.lib.ncsu.edu/index.php/Expertiza Expertiza project documentation wiki]
#[https://relishapp.com/rspec Rspec Documentation]
#[https://github.com/mperham/sidekiq/wiki Sidekiq Documentation]
#Clean Code: A handbook of agile software craftsmanship. Author: Robert C Martin

E1835 Refactor delayed mailer and scheduled task

2018-11-02T18:37:22Z

Pbhalas: /* About Expertiza */

== '''About Expertiza''' ==