Expertiza_Wiki - User contributions [en]

CSC/ECE 517 Spring 2019 - Project M1901 Implement missing WebAudio automation support

2019-04-13T03:12:26Z

Amadhus2: /* List of issues */

=='''Introduction'''==

==='''Servo'''===
Servo is an developmental browser engine designed to make use of the memory safety properties and concurrency features of the Rust programming language. The project was initiated by Mozilla Research with effort from Samsung to port it to Android and ARM processors

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language which focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and provides high performance. Rust is brain child of Mozilla and simple as this
<pre>
fn main() {
println!("Hello World");
}
</pre>

More information about Rust can be found [https://www.rust-lang.org/en-US/ here] and here is why we love [https://thenewstack.io/safer-future-rust/ Rust], [https://insights.stackoverflow.com/survey/2018/#most-loved-dreaded-and-wanted SF Loved]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. An audio routing graph has been created by linking together the Basic Audio operations performed with audio nodes. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context<br>
2. Inside the context, create sources — such as <audio>, oscillator, stream<br>
3. Create effects nodes, such as reverb, biquad filter, panner, compressor<br>
4. Choose final destination of audio, for example your system speakers<br>
5. Connect the sources up to the effects, and the effects to the destination.<br>

==='''Problem Statement: Web Audio Automation'''===

AudioParam is used to control the AudioNode functioning, say, Volume (a Gain parameter). These values can either be scheduled to be changed at precise times or set to particular value immediately following an event or at an event. The schedule to change when used with AudioContext.currentTime can helps in volume fades, filter sweeps etc. This would work aims to implement [https://github.com/servo/media/issues/204 SetValueCurveAtTime()].

[https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime setValueCurveAtTime()] is one such method in AudioParam that takes an array as input and schedules a change. The array is usually a curve in wedAudio and achieved my linear interpolation between the values from the floating-point values array and for the duration, d from the startTime, s

<code>v(t) = values[N * (t - s) / d]</code>, where N is the length of the values in the array.
And after the end of the curve time interval <code>(t >= s + d)</code>, the value will remain constant at the final curve value. This would persist to happen until the next automation event.

One of the application of setValueCurveAtTime() it to create tremolo effect. Suppose linear nor an exponential curve satisfy the needs then user can create curve based on values to setValueCurveAtTime() with an array of timing values. Its a much loved approached against multiple calls to setValueAtTime().

==='''Build'''===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.

=='''Implementation of the Project'''==

==='''Code'''===
The code is implemented as in [https://github.com/JHBalaji/media/blob/master/audio/src/param.rs#L418-L436 here]. The code below is the main code snippet for the SetValueCurveAtTime function. The values and function behavior is define in the param.rs file.

<pre>

AutomationEvent::SetValueCurveAtTime(ref values, start, duration) => {
let time_diff = ((duration.0 as f32) - (start.0 as f32)) as f32;
let mut progress = ((((current_tick.0 as f32) - (start.0 as f32)) as f32) / time_diff) as f32;
if progress < 0.0 {
progress = 0.0 as f32;
}
let n = values.len() as f32;
let k = (((n - 1.) * progress) as f32).floor();
let next = k + 1. as f32;
let step = time_diff / (n - 1.);
if next < n {
let time_k = (k * step) as f32;
let time_k_next = (next * step) as f32;
let v_k = values[k as usize];
let v_k_next = values[next as usize];
*value = v_k + (v_k_next - v_k) * (((current_tick.0 as f32) - time_k) / (time_k_next - time_k));
}
true
}
AutomationEvent::CancelAndHoldAtTime(..) => false,
AutomationEvent::CancelScheduledValues(..) | AutomationEvent::SetValue(..) => {
unreachable!("CancelScheduledValues/SetValue should never appear in the timeline")
}

</pre>

Let's walk through the code.

<code>AutomationEvent::SetValueCurveAtTime()</code> is a fn and takes three arguments: values, start, duration.

progress, computes the difference between the first value in start_time and [https://github.com/JHBalaji/media/blob/5a981dff94f4dfaf2ebf0f0e09f599fab9d1a13b/audio/src/param.rs#L384 time] and makes explicit conversion to f32.

<pre>
let time_diff = ((duration.0 as f32) - (start.0 as f32)) as f32;
let mut progress = ((((current_tick.0 as f32) - (start.0 as f32)) as f32) / time_diff) as f32;
</pre>

time_diff computes the total difference between duration and start_time, this is required when the values that needs to be computed goes beyond time_diff from start_time to set a constant value later.

<pre>
if progress < 0.0 {
progress = 0.0 as f32;
}
</pre>
Sets 0 if the progress from values array and present time being negative.

<pre>
let n = values.len() as f32;
let k = (((n - 1.) * progress) as f32).floor();
let next = k + 1. as f32;
let step = time_diff / (n - 1.);
</pre>
n, total length of values array. k, compute as k as in [[https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime here] and <code>next</code> is k+1, the delta or step is computed that would be added to generate the sound wave from the values array. Say, the frequency start at 44KHz and the <code>step</code> would be used to change the frequency of the curve.

<pre>
if next < n {
let time_k = (k * step) as f32;
let time_k_next = (next * step) as f32;
let v_k = values[k as usize];
let v_k_next = values[next as usize];
*value = v_k + (v_k_next - v_k) * (((current_tick.0 as f32) - time_k) / (time_k_next - time_k));
}
</pre>

The code block generates sequences of values from <code>values</code array and using <code>step</code> value we computed. And does that till it reaches the total length of the array.

<pre>
true
</pre>

[https://github.com/JHBalaji/media/blob/f32a71c51f05f44d1e8ca217fbaa4dfb0b55d11e/audio/src/param.rs#L435 Used] to loop the function call made from [https://github.com/JHBalaji/media/blob/f32a71c51f05f44d1e8ca217fbaa4dfb0b55d11e/audio/src/param.rs#L117 here].

<pre>
AutomationEvent::CancelAndHoldAtTime(..) => false,
AutomationEvent::CancelScheduledValues(..) | AutomationEvent::SetValue(..) => {
unreachable!("CancelScheduledValues/SetValue should never appear in the timeline")
}
</pre>

[https://github.com/JHBalaji/media/blob/5a981dff94f4dfaf2ebf0f0e09f599fab9d1a13b/audio/src/param.rs#L466 Code] instructs to cancel this event or end of this function and proceed to next instructions. In Audio API unless you explicitly say the block to end, the block would end and hardware device would be stopped from consumption.

==='''Examples'''===

Examples are unique aspect to Rust. This instructs that code to make use of the function we just wrote and generates a sound output. Refer our example [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs here] for the SetValueCurveAtTime()

And example is usually called by using cargo build in rust or cargo run.

[https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml Cargo.toml] dictates the naming and the file to be called as.

<pre>
cargo run --example set_value_curve
</pre>

The above code instructs to run the example [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs set_value_curve] to generate a sound output through our terminal.

==='''Compile'''===
[https://github.com/JHBalaji/media-1/blob/master/Cargo.lock Cargo.lock] instructs the dependencies that would be required to build this project.

[https://github.com/JHBalaji/media-1/blob/master/Cargo.toml Cargo.toml] calls all the folders or members to be complied. Usually the compilation would be at /media

Cargo.lock installs number of packages called crates that would be used by the project to access the modules. Each of the member in [https://github.com/JHBalaji/media-1/blob/master/Cargo.toml Cargo.toml] would have [https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml Cargo.toml] in them that would instruct [https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml#L6 dependencies] during the build. build-dependencies would be used if you have dependencies before the build.

[https://github.com/JHBalaji/media-1/blob/master/.cargo/config .cargo config] is similar to a dotenv file for Rust

==='''Test'''===

If you would like to test our build please look into [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Issues_and_Fixes Fixes] before proceeding. We would suggest MacOS or Linux 16.04 LTS for any development and testing. Virtual Machine or VirtualBox could be another option. We spent countless days in frustration with libraries and never ending apt installs so avoid Linux 18.04 for this build test.

Once <code>cargo build</code> is completed. Open /target/debug to view the examples that are generated.

[[File:Servo-initial-build.PNG]]

To run all examples after the cargo build is complete, use <code>cargo run --example <bin_name></code>

[[File:Examples.PNG]]

==='''Status of the Project'''===

The [https://github.com/JHBalaji/media-1 build] is ready and the Pull request is [https://github.com/servo/media/pull/230 merged] with Servo/media.

[[File:Pr.PNG]]

=='''Normal build from Master Repo '''==
To build Servo in development mode. This is useful for development, but the resulting binary is very slow.
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html
</pre>
For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:
<pre>
./mach build --release
./mach run --release tests/html/about-mozilla.html
</pre>

==='''Compiling and Running the code'''===

====Mac OS====

Follow the instructions posted in this [https://github.com/JHBalaji/media-1 URL] and install GStreamer and Rust. After that follow the following lines for compiling on MacOS.

<pre>
1. brew install gstreamer gst-plugins-base gst-plugins-good gst-plugins-bad gst-plugins-ugly gst-libav gst-rtsp-server gst-editing-services

2. export PKG_CONFIG_PATH=/usr/local/Cellar/libffi/3.2.1/lib/pkgconfig/

3. rustup override set nightly
</pre>

After this run the following command to compile servo/media
<pre>
cargo build
</pre>

After compiling you will be able to test our example that is places in the examples folder.

====Windows and Linux====

Head over here to install [https://rustup.rs/ Rust ToolChain]. If you need any additional details we would suggest reading the [https://github.com/rust-lang/rustup.rs/blob/master/README.md ReadMe on Rust]. Some Servo media specific build issues are [https://github.com/servo/media/blob/master/README.md advised].

We would suggest Ubuntu 18.04 LTS with updates [https://github.com/servo/media/blob/master/README.md gstreamer package].

==='''Servo Media'''===
<b>Requirements</b>
In order to build this crate you need to install gstreamer which can be found here [https://github.com/sdroege/gstreamer-rs]
==='''Test Case'''===
After you run cargo build, you should be able to see the executable of the example inside the target/debug folder. Following is the test we have written file to check the functioning of the [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs SetValueCurveAtTIme].
<pre>
extern crate servo_media;
extern crate servo_media_auto;

use servo_media::audio::constant_source_node::ConstantSourceNodeOptions;
use servo_media::audio::gain_node::GainNodeOptions;
use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::param::{ParamType, UserAutomationEvent};
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();

//Initializing the values vector for SetValueCurve function
let mut values: Vec<f32> = Vec::new();
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(1.0);
values.push(1.0);
values.push(1.0);
values.push(1.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.5);
values.push(0.5);
values.push(0.5);
values.push(0.5);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
let start_time = 0.;
let end_time = 5.;
let n = values.len() as f32;
let value_next = values[(n - 1.) as usize];

let cs = context.create_node(
AudioNodeInit::ConstantSourceNode(ConstantSourceNodeOptions::default()),
Default::default(),
);

let mut gain_options = GainNodeOptions::default();
gain_options.gain = 0.0;
let gain = context.create_node(
AudioNodeInit::GainNode(gain_options.clone()),
Default::default(),
);

let osc = context.create_node(
AudioNodeInit::OscillatorNode(Default::default()),
Default::default(),
);

context.connect_ports(osc.output(0), gain.input(0));
context.connect_ports(cs.output(0), gain.param(ParamType::Gain));
context.connect_ports(gain.output(0), dest.input(0));

let _ = context.resume();
context.message_node(
osc,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
gain,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::SetValueCurveAtTime(values, start_time, end_time),
),

);

context.message_node(
osc,
AudioNodeMessage::SetParam(
ParamType::Frequency,
UserAutomationEvent::SetValueAtTime(value_next, end_time),
),
);

thread::sleep(time::Duration::from_millis(7000));
let _ = context.close();
}

fn main() {
ServoMedia::init::<servo_media_auto::Backend>();
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}

</pre>

In the above program, the values vector is passed into the SetValueCurveAtTime function to modulate the output. The output of the same can also be checked in the test-case video at the end of this page.
We can modify the values function as we want and observe the output. One more modification that we used was by using a sine curve. It is implemented as follows. Change the values vector in the set-value-curve function with the following:

<pre>
use std::f64::consts::PI;
let mut values: Vec<f32> = Vec::new();
let curvelength = 44100;
let mut i = 0;
while i < curvelength {
values.push(f32::sin((PI *i as f64/curvelength as f64) as f32));
i += 1;
}
</pre>

The output of this function can be seen in the test video attached to at the end of this page.
<references></references>

=='''Video of Testing Audio Output'''==
[https://drive.google.com/file/d/1jzEZwAZE-7mi6N5N_JcEE6ISM-wc9910/view?usp=sharing Test-1]
[https://drive.google.com/file/d/1UsPgyv29Zf29d6pEDgLpYX86DI6hzOqp/view?usp=sharing Test-2]

=='''Issues and Fixes'''==

Windows Specific during build

https://github.com/holochain/holochain-cmd/issues/29

https://stackoverflow.com/questions/53136717/errore0554-feature-may-not-be-used-on-the-stable-release-channel-couldnt

https://github.com/servo/servo/issues/21429

=='''Final Project'''==
If you are a peer reviewer please look into this [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#For_Peer_Reviewers_Note section]

The final [https://github.com/servo/servo/wiki/Implement-missing-WebAudio-automation-student-project project] rely on our previous [https://github.com/servo/media/pull/230 work]. The next part would focus on implementing the code [https://github.com/servo/media/issues/204 written] to be accessible by Servo browser [https://github.com/servo/servo/issues/22897 engine].
This would require interfacing the DOM [https://github.com/servo/servo/blob/master/components/script/dom/webidls/AudioParam.webidl file] reference the Param.rs [https://github.com/JHBalaji/media/blob/master/audio/src/param.rs#L259 setValueCurveAtTime] to be interfaced with [https://github.com/servo/servo/blob/master/components/script/dom/audioparam.rs AudioParam.rs]. The audioparam.rs would call the UserAutomationEvent in the param.rs setValueCurveAtTime.

Once the initial function call with arguments are updated, then the [https://github.com/servo/servo/blob/master/tests/wpt/README.md#updating-test-expectations testing] would be performed on ''./mach test-wpt tests/wpt/web-platform-tests/webaudio/the-audio-api/[https://github.com/servo/servo/tree/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audioparam-interface the-audioparam-interface]''. The [https://github.com/servo/servo/tree/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audioparam-interface the-audioparam-interface] is a folder of series of test cases that emulate the behavior of a function call, here the setValueCurveAtTime would need to be necessitated to mock the values as tested in previous [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Test_Case example] but would require to be performed over a .html webpage with DOM elements to test the complete functionality.

Servo webAudio tests are performed using [https://github.com/web-platform-tests/wpt/blob/master/resources/readme.md testharness.js].

<pre>
./mach create-wpt tests/wpt/path/to/new/test.html
</pre>

would create a test.html using the WPT template for javascript tests. Once the skeleton code is completed with necessary function call, them the WPT tests would be executed using

<pre>
./mach test-wpt tests/wpt/path/to/new/test.html
./mach test-wpt tests/wpt/path/to/new/reftest.html
</pre>

[https://web-platform-tests.org/writing-tests/reftests.html reftest] are used in Servo to test behavior related to rendering; that are related to interaction made up of several webpages [think passing or referencing values to another webpage] with assertions to test cases that are expected from the interaction.

<pre>
./mach create-wpt --reftest tests/wpt/path/to/new/reftest.html --reference tests/wpt/path/to/reference.html
</pre>

reference.html and reftest.html will be created using WPT reftest template. Once the reftest is run the upstream test would need to be updated using

<pre>
./mach update-wpt --sync
./mach test-wpt --log-raw=update.log
./mach update-wpt update.log
</pre>

When reftest fails to diagnose the behavior reftest raw log is required which is generate using the below command. The .log is fed into [https://github.com/Gankro/live-reftest-analyzer reftest analyzer] to find those failing test. The reftest is also capable of pixel level comparison of the test and reference screenshots (incase of rendering)
<pre>
./mach test-wpt --log-raw wpt.log
</pre>

==='''List of issues'''===
This project has listed 4 known issues under the WebAudio title, each solving a different issue.

1. [https://github.com/servo/media/issues/204 Implement ValueCurveAtTime automation]

This is the first issue, which has been addressed and successfully closed during the initial parts of this project. The steps followed are shown in the following diagram.

[[File:WebAudio-Issue1.png]]

2. [https://github.com/servo/servo/issues/22897 Implement AudioParam.setValueCurveAtTime]

This second issue builds up on the first one, and focuses on implementing yet another method called [https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime setValueCurveAtTime]. The next focus of this project will likely be to work on this issue. This deals with the javascript API values for the curves.

[[File:WebAudio-Issue2.png]]

3. [https://github.com/servo/servo/issues/22898 Implement deprecated setPosition and setOrientation methods for AudioListener]

The third issue deals with a few missing APIs for AudioListener. We need to accordingly add the new methods, i.e. [https://webaudio.github.io/web-audio-api/#dom-pannernode-setposition setPosition] and [https://webaudio.github.io/web-audio-api/#dom-pannernode-setorientation setOrientation] to the audiolistener.rs file so that they properly update the values of the relevant AudioParam members of AudioListener along with the passed arguments.This should help stopping an endless list of javascript errors which would occur when these deprecated methods weren't implemented.

[[File:WebAudio-Issue3.png]]

4. [https://github.com/servo/media/issues/205 Implement WaveShaper node]

This final issue listed tries to implement a new node called the [https://webaudio.github.io/web-audio-api/#waveshapernode WaveShaper node]. This is an independant issue and may be done before or after the other issues have been solved.

[[File:WebAudio-Issue4.png]]

=='''For Peer Reviewers Note'''==

''Common peer review question clarifications are below,''

"Does the design incorporate all of the functionality required?" or "Check the changes that the authors proposed. Rate the overall quality of the explanations. Give comments if you find the explanation hard to follow or inadequate."

:The [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Final_Project section] clarifies your question.

"Check the designs proposed by the authors."

:There is no explicit design is involved as going by [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Final_Project proposal], there are three files to be added with new functionality in addition to test cases and files.

"Check the use of visual aids"
:Visual aids not applicable

"Give comments on the completeness of the design doc."
: Let us know what you think about them!

CSC/ECE 517 Spring 2019 - Project M1901 Implement missing WebAudio automation support

2019-04-13T03:10:03Z

Amadhus2: /* List of issues */

=='''Introduction'''==

==='''Servo'''===
Servo is an developmental browser engine designed to make use of the memory safety properties and concurrency features of the Rust programming language. The project was initiated by Mozilla Research with effort from Samsung to port it to Android and ARM processors

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language which focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and provides high performance. Rust is brain child of Mozilla and simple as this
<pre>
fn main() {
println!("Hello World");
}
</pre>

More information about Rust can be found [https://www.rust-lang.org/en-US/ here] and here is why we love [https://thenewstack.io/safer-future-rust/ Rust], [https://insights.stackoverflow.com/survey/2018/#most-loved-dreaded-and-wanted SF Loved]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. An audio routing graph has been created by linking together the Basic Audio operations performed with audio nodes. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context<br>
2. Inside the context, create sources — such as <audio>, oscillator, stream<br>
3. Create effects nodes, such as reverb, biquad filter, panner, compressor<br>
4. Choose final destination of audio, for example your system speakers<br>
5. Connect the sources up to the effects, and the effects to the destination.<br>

==='''Problem Statement: Web Audio Automation'''===

AudioParam is used to control the AudioNode functioning, say, Volume (a Gain parameter). These values can either be scheduled to be changed at precise times or set to particular value immediately following an event or at an event. The schedule to change when used with AudioContext.currentTime can helps in volume fades, filter sweeps etc. This would work aims to implement [https://github.com/servo/media/issues/204 SetValueCurveAtTime()].

[https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime setValueCurveAtTime()] is one such method in AudioParam that takes an array as input and schedules a change. The array is usually a curve in wedAudio and achieved my linear interpolation between the values from the floating-point values array and for the duration, d from the startTime, s

<code>v(t) = values[N * (t - s) / d]</code>, where N is the length of the values in the array.
And after the end of the curve time interval <code>(t >= s + d)</code>, the value will remain constant at the final curve value. This would persist to happen until the next automation event.

One of the application of setValueCurveAtTime() it to create tremolo effect. Suppose linear nor an exponential curve satisfy the needs then user can create curve based on values to setValueCurveAtTime() with an array of timing values. Its a much loved approached against multiple calls to setValueAtTime().

==='''Build'''===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.

=='''Implementation of the Project'''==

==='''Code'''===
The code is implemented as in [https://github.com/JHBalaji/media/blob/master/audio/src/param.rs#L418-L436 here]. The code below is the main code snippet for the SetValueCurveAtTime function. The values and function behavior is define in the param.rs file.

<pre>

AutomationEvent::SetValueCurveAtTime(ref values, start, duration) => {
let time_diff = ((duration.0 as f32) - (start.0 as f32)) as f32;
let mut progress = ((((current_tick.0 as f32) - (start.0 as f32)) as f32) / time_diff) as f32;
if progress < 0.0 {
progress = 0.0 as f32;
}
let n = values.len() as f32;
let k = (((n - 1.) * progress) as f32).floor();
let next = k + 1. as f32;
let step = time_diff / (n - 1.);
if next < n {
let time_k = (k * step) as f32;
let time_k_next = (next * step) as f32;
let v_k = values[k as usize];
let v_k_next = values[next as usize];
*value = v_k + (v_k_next - v_k) * (((current_tick.0 as f32) - time_k) / (time_k_next - time_k));
}
true
}
AutomationEvent::CancelAndHoldAtTime(..) => false,
AutomationEvent::CancelScheduledValues(..) | AutomationEvent::SetValue(..) => {
unreachable!("CancelScheduledValues/SetValue should never appear in the timeline")
}

</pre>

Let's walk through the code.

<code>AutomationEvent::SetValueCurveAtTime()</code> is a fn and takes three arguments: values, start, duration.

progress, computes the difference between the first value in start_time and [https://github.com/JHBalaji/media/blob/5a981dff94f4dfaf2ebf0f0e09f599fab9d1a13b/audio/src/param.rs#L384 time] and makes explicit conversion to f32.

<pre>
let time_diff = ((duration.0 as f32) - (start.0 as f32)) as f32;
let mut progress = ((((current_tick.0 as f32) - (start.0 as f32)) as f32) / time_diff) as f32;
</pre>

time_diff computes the total difference between duration and start_time, this is required when the values that needs to be computed goes beyond time_diff from start_time to set a constant value later.

<pre>
if progress < 0.0 {
progress = 0.0 as f32;
}
</pre>
Sets 0 if the progress from values array and present time being negative.

<pre>
let n = values.len() as f32;
let k = (((n - 1.) * progress) as f32).floor();
let next = k + 1. as f32;
let step = time_diff / (n - 1.);
</pre>
n, total length of values array. k, compute as k as in [[https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime here] and <code>next</code> is k+1, the delta or step is computed that would be added to generate the sound wave from the values array. Say, the frequency start at 44KHz and the <code>step</code> would be used to change the frequency of the curve.

<pre>
if next < n {
let time_k = (k * step) as f32;
let time_k_next = (next * step) as f32;
let v_k = values[k as usize];
let v_k_next = values[next as usize];
*value = v_k + (v_k_next - v_k) * (((current_tick.0 as f32) - time_k) / (time_k_next - time_k));
}
</pre>

The code block generates sequences of values from <code>values</code array and using <code>step</code> value we computed. And does that till it reaches the total length of the array.

<pre>
true
</pre>

[https://github.com/JHBalaji/media/blob/f32a71c51f05f44d1e8ca217fbaa4dfb0b55d11e/audio/src/param.rs#L435 Used] to loop the function call made from [https://github.com/JHBalaji/media/blob/f32a71c51f05f44d1e8ca217fbaa4dfb0b55d11e/audio/src/param.rs#L117 here].

<pre>
AutomationEvent::CancelAndHoldAtTime(..) => false,
AutomationEvent::CancelScheduledValues(..) | AutomationEvent::SetValue(..) => {
unreachable!("CancelScheduledValues/SetValue should never appear in the timeline")
}
</pre>

[https://github.com/JHBalaji/media/blob/5a981dff94f4dfaf2ebf0f0e09f599fab9d1a13b/audio/src/param.rs#L466 Code] instructs to cancel this event or end of this function and proceed to next instructions. In Audio API unless you explicitly say the block to end, the block would end and hardware device would be stopped from consumption.

==='''Examples'''===

Examples are unique aspect to Rust. This instructs that code to make use of the function we just wrote and generates a sound output. Refer our example [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs here] for the SetValueCurveAtTime()

And example is usually called by using cargo build in rust or cargo run.

[https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml Cargo.toml] dictates the naming and the file to be called as.

<pre>
cargo run --example set_value_curve
</pre>

The above code instructs to run the example [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs set_value_curve] to generate a sound output through our terminal.

==='''Compile'''===
[https://github.com/JHBalaji/media-1/blob/master/Cargo.lock Cargo.lock] instructs the dependencies that would be required to build this project.

[https://github.com/JHBalaji/media-1/blob/master/Cargo.toml Cargo.toml] calls all the folders or members to be complied. Usually the compilation would be at /media

Cargo.lock installs number of packages called crates that would be used by the project to access the modules. Each of the member in [https://github.com/JHBalaji/media-1/blob/master/Cargo.toml Cargo.toml] would have [https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml Cargo.toml] in them that would instruct [https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml#L6 dependencies] during the build. build-dependencies would be used if you have dependencies before the build.

[https://github.com/JHBalaji/media-1/blob/master/.cargo/config .cargo config] is similar to a dotenv file for Rust

==='''Test'''===

If you would like to test our build please look into [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Issues_and_Fixes Fixes] before proceeding. We would suggest MacOS or Linux 16.04 LTS for any development and testing. Virtual Machine or VirtualBox could be another option. We spent countless days in frustration with libraries and never ending apt installs so avoid Linux 18.04 for this build test.

Once <code>cargo build</code> is completed. Open /target/debug to view the examples that are generated.

[[File:Servo-initial-build.PNG]]

To run all examples after the cargo build is complete, use <code>cargo run --example <bin_name></code>

[[File:Examples.PNG]]

==='''Status of the Project'''===

The [https://github.com/JHBalaji/media-1 build] is ready and the Pull request is [https://github.com/servo/media/pull/230 merged] with Servo/media.

[[File:Pr.PNG]]

=='''Normal build from Master Repo '''==
To build Servo in development mode. This is useful for development, but the resulting binary is very slow.
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html
</pre>
For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:
<pre>
./mach build --release
./mach run --release tests/html/about-mozilla.html
</pre>

==='''Compiling and Running the code'''===

====Mac OS====

Follow the instructions posted in this [https://github.com/JHBalaji/media-1 URL] and install GStreamer and Rust. After that follow the following lines for compiling on MacOS.

<pre>
1. brew install gstreamer gst-plugins-base gst-plugins-good gst-plugins-bad gst-plugins-ugly gst-libav gst-rtsp-server gst-editing-services

2. export PKG_CONFIG_PATH=/usr/local/Cellar/libffi/3.2.1/lib/pkgconfig/

3. rustup override set nightly
</pre>

After this run the following command to compile servo/media
<pre>
cargo build
</pre>

After compiling you will be able to test our example that is places in the examples folder.

====Windows and Linux====

Head over here to install [https://rustup.rs/ Rust ToolChain]. If you need any additional details we would suggest reading the [https://github.com/rust-lang/rustup.rs/blob/master/README.md ReadMe on Rust]. Some Servo media specific build issues are [https://github.com/servo/media/blob/master/README.md advised].

We would suggest Ubuntu 18.04 LTS with updates [https://github.com/servo/media/blob/master/README.md gstreamer package].

==='''Servo Media'''===
<b>Requirements</b>
In order to build this crate you need to install gstreamer which can be found here [https://github.com/sdroege/gstreamer-rs]
==='''Test Case'''===
After you run cargo build, you should be able to see the executable of the example inside the target/debug folder. Following is the test we have written file to check the functioning of the [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs SetValueCurveAtTIme].
<pre>
extern crate servo_media;
extern crate servo_media_auto;

use servo_media::audio::constant_source_node::ConstantSourceNodeOptions;
use servo_media::audio::gain_node::GainNodeOptions;
use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::param::{ParamType, UserAutomationEvent};
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();

//Initializing the values vector for SetValueCurve function
let mut values: Vec<f32> = Vec::new();
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(1.0);
values.push(1.0);
values.push(1.0);
values.push(1.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.5);
values.push(0.5);
values.push(0.5);
values.push(0.5);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
let start_time = 0.;
let end_time = 5.;
let n = values.len() as f32;
let value_next = values[(n - 1.) as usize];

let cs = context.create_node(
AudioNodeInit::ConstantSourceNode(ConstantSourceNodeOptions::default()),
Default::default(),
);

let mut gain_options = GainNodeOptions::default();
gain_options.gain = 0.0;
let gain = context.create_node(
AudioNodeInit::GainNode(gain_options.clone()),
Default::default(),
);

let osc = context.create_node(
AudioNodeInit::OscillatorNode(Default::default()),
Default::default(),
);

context.connect_ports(osc.output(0), gain.input(0));
context.connect_ports(cs.output(0), gain.param(ParamType::Gain));
context.connect_ports(gain.output(0), dest.input(0));

let _ = context.resume();
context.message_node(
osc,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
gain,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::SetValueCurveAtTime(values, start_time, end_time),
),

);

context.message_node(
osc,
AudioNodeMessage::SetParam(
ParamType::Frequency,
UserAutomationEvent::SetValueAtTime(value_next, end_time),
),
);

thread::sleep(time::Duration::from_millis(7000));
let _ = context.close();
}

fn main() {
ServoMedia::init::<servo_media_auto::Backend>();
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}

</pre>

In the above program, the values vector is passed into the SetValueCurveAtTime function to modulate the output. The output of the same can also be checked in the test-case video at the end of this page.
We can modify the values function as we want and observe the output. One more modification that we used was by using a sine curve. It is implemented as follows. Change the values vector in the set-value-curve function with the following:

<pre>
use std::f64::consts::PI;
let mut values: Vec<f32> = Vec::new();
let curvelength = 44100;
let mut i = 0;
while i < curvelength {
values.push(f32::sin((PI *i as f64/curvelength as f64) as f32));
i += 1;
}
</pre>

The output of this function can be seen in the test video attached to at the end of this page.
<references></references>

=='''Video of Testing Audio Output'''==
[https://drive.google.com/file/d/1jzEZwAZE-7mi6N5N_JcEE6ISM-wc9910/view?usp=sharing Test-1]
[https://drive.google.com/file/d/1UsPgyv29Zf29d6pEDgLpYX86DI6hzOqp/view?usp=sharing Test-2]

=='''Issues and Fixes'''==

Windows Specific during build

https://github.com/holochain/holochain-cmd/issues/29

https://stackoverflow.com/questions/53136717/errore0554-feature-may-not-be-used-on-the-stable-release-channel-couldnt

https://github.com/servo/servo/issues/21429

=='''Final Project'''==
If you are a peer reviewer please look into this [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#For_Peer_Reviewers_Note section]

The final [https://github.com/servo/servo/wiki/Implement-missing-WebAudio-automation-student-project project] rely on our previous [https://github.com/servo/media/pull/230 work]. The next part would focus on implementing the code [https://github.com/servo/media/issues/204 written] to be accessible by Servo browser [https://github.com/servo/servo/issues/22897 engine].
This would require interfacing the DOM [https://github.com/servo/servo/blob/master/components/script/dom/webidls/AudioParam.webidl file] reference the Param.rs [https://github.com/JHBalaji/media/blob/master/audio/src/param.rs#L259 setValueCurveAtTime] to be interfaced with [https://github.com/servo/servo/blob/master/components/script/dom/audioparam.rs AudioParam.rs]. The audioparam.rs would call the UserAutomationEvent in the param.rs setValueCurveAtTime.

Once the initial function call with arguments are updated, then the [https://github.com/servo/servo/blob/master/tests/wpt/README.md#updating-test-expectations testing] would be performed on ''./mach test-wpt tests/wpt/web-platform-tests/webaudio/the-audio-api/[https://github.com/servo/servo/tree/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audioparam-interface the-audioparam-interface]''. The [https://github.com/servo/servo/tree/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audioparam-interface the-audioparam-interface] is a folder of series of test cases that emulate the behavior of a function call, here the setValueCurveAtTime would need to be necessitated to mock the values as tested in previous [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Test_Case example] but would require to be performed over a .html webpage with DOM elements to test the complete functionality.

Servo webAudio tests are performed using [https://github.com/web-platform-tests/wpt/blob/master/resources/readme.md testharness.js].

<pre>
./mach create-wpt tests/wpt/path/to/new/test.html
</pre>

would create a test.html using the WPT template for javascript tests. Once the skeleton code is completed with necessary function call, them the WPT tests would be executed using

<pre>
./mach test-wpt tests/wpt/path/to/new/test.html
./mach test-wpt tests/wpt/path/to/new/reftest.html
</pre>

[https://web-platform-tests.org/writing-tests/reftests.html reftest] are used in Servo to test behavior related to rendering; that are related to interaction made up of several webpages [think passing or referencing values to another webpage] with assertions to test cases that are expected from the interaction.

<pre>
./mach create-wpt --reftest tests/wpt/path/to/new/reftest.html --reference tests/wpt/path/to/reference.html
</pre>

reference.html and reftest.html will be created using WPT reftest template. Once the reftest is run the upstream test would need to be updated using

<pre>
./mach update-wpt --sync
./mach test-wpt --log-raw=update.log
./mach update-wpt update.log
</pre>

When reftest fails to diagnose the behavior reftest raw log is required which is generate using the below command. The .log is fed into [https://github.com/Gankro/live-reftest-analyzer reftest analyzer] to find those failing test. The reftest is also capable of pixel level comparison of the test and reference screenshots (incase of rendering)
<pre>
./mach test-wpt --log-raw wpt.log
</pre>

==='''List of issues'''===
This project has listed 4 known issues under the WebAudio title, each solving a different issue.

1. [https://github.com/servo/media/issues/204 Implement ValueCurveAtTime automation]

This is the first issue, which has been addressed and successfully closed during the initial parts of this project. The steps followed are shown in the following diagram.

[[File:WebAudio-Issue1.png]]

2. [https://github.com/servo/servo/issues/22897 Implement AudioParam.setValueCurveAtTime]

This second issue builds up on the first one, and focuses on implementing yet another method called [https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime setValueCurveAtTime]. The next focus of this project will likely be to work on this issue. This deals with the javascript API values for the curves.

[[File:WebAudio-Issue2.png]]

3. [https://github.com/servo/servo/issues/22898 Implement deprecated setPosition and setOrientation methods for AudioListener]

The third issue deals with a few missing APIs for AudioListener. We need to accordingly add the new methods, i.e. [https://webaudio.github.io/web-audio-api/#dom-pannernode-setposition setPosition] and [https://webaudio.github.io/web-audio-api/#dom-pannernode-setorientation setOrientation] to the audiolistener.rs file so that they properly update the values of the relevant AudioParam members of AudioListener along with the passed arguments.This should help stopping an endless list of javascript errors when these deprecated methods weren't implemented.

[[File:WebAudio-Issue3.png]]

4. [https://github.com/servo/media/issues/205 Implement WaveShaper node]

This final issue listed tries to implement a new node called the [https://webaudio.github.io/web-audio-api/#waveshapernode WaveShaper node]. This is an independant issue and may be done before or after the other issues have been solved.

[[File:WebAudio-Issue4.png]]

=='''For Peer Reviewers Note'''==

''Common peer review question clarifications are below,''

"Does the design incorporate all of the functionality required?" or "Check the changes that the authors proposed. Rate the overall quality of the explanations. Give comments if you find the explanation hard to follow or inadequate."

:The [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Final_Project section] clarifies your question.

"Check the designs proposed by the authors."

:There is no explicit design is involved as going by [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Final_Project proposal], there are three files to be added with new functionality in addition to test cases and files.

"Check the use of visual aids"
:Visual aids not applicable

"Give comments on the completeness of the design doc."
: Let us know what you think about them!

File:WebAudio-Issue4.png

2019-04-13T03:09:47Z

Amadhus2:

CSC/ECE 517 Spring 2019 - Project M1901 Implement missing WebAudio automation support

2019-04-13T03:01:24Z

Amadhus2: /* List of issues */

=='''Introduction'''==

==='''Servo'''===
Servo is an developmental browser engine designed to make use of the memory safety properties and concurrency features of the Rust programming language. The project was initiated by Mozilla Research with effort from Samsung to port it to Android and ARM processors

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language which focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and provides high performance. Rust is brain child of Mozilla and simple as this
<pre>
fn main() {
println!("Hello World");
}
</pre>

More information about Rust can be found [https://www.rust-lang.org/en-US/ here] and here is why we love [https://thenewstack.io/safer-future-rust/ Rust], [https://insights.stackoverflow.com/survey/2018/#most-loved-dreaded-and-wanted SF Loved]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. An audio routing graph has been created by linking together the Basic Audio operations performed with audio nodes. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context<br>
2. Inside the context, create sources — such as <audio>, oscillator, stream<br>
3. Create effects nodes, such as reverb, biquad filter, panner, compressor<br>
4. Choose final destination of audio, for example your system speakers<br>
5. Connect the sources up to the effects, and the effects to the destination.<br>

==='''Problem Statement: Web Audio Automation'''===

AudioParam is used to control the AudioNode functioning, say, Volume (a Gain parameter). These values can either be scheduled to be changed at precise times or set to particular value immediately following an event or at an event. The schedule to change when used with AudioContext.currentTime can helps in volume fades, filter sweeps etc. This would work aims to implement [https://github.com/servo/media/issues/204 SetValueCurveAtTime()].

[https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime setValueCurveAtTime()] is one such method in AudioParam that takes an array as input and schedules a change. The array is usually a curve in wedAudio and achieved my linear interpolation between the values from the floating-point values array and for the duration, d from the startTime, s

<code>v(t) = values[N * (t - s) / d]</code>, where N is the length of the values in the array.
And after the end of the curve time interval <code>(t >= s + d)</code>, the value will remain constant at the final curve value. This would persist to happen until the next automation event.

One of the application of setValueCurveAtTime() it to create tremolo effect. Suppose linear nor an exponential curve satisfy the needs then user can create curve based on values to setValueCurveAtTime() with an array of timing values. Its a much loved approached against multiple calls to setValueAtTime().

==='''Build'''===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.

=='''Implementation of the Project'''==

==='''Code'''===
The code is implemented as in [https://github.com/JHBalaji/media/blob/master/audio/src/param.rs#L418-L436 here]. The code below is the main code snippet for the SetValueCurveAtTime function. The values and function behavior is define in the param.rs file.

<pre>

AutomationEvent::SetValueCurveAtTime(ref values, start, duration) => {
let time_diff = ((duration.0 as f32) - (start.0 as f32)) as f32;
let mut progress = ((((current_tick.0 as f32) - (start.0 as f32)) as f32) / time_diff) as f32;
if progress < 0.0 {
progress = 0.0 as f32;
}
let n = values.len() as f32;
let k = (((n - 1.) * progress) as f32).floor();
let next = k + 1. as f32;
let step = time_diff / (n - 1.);
if next < n {
let time_k = (k * step) as f32;
let time_k_next = (next * step) as f32;
let v_k = values[k as usize];
let v_k_next = values[next as usize];
*value = v_k + (v_k_next - v_k) * (((current_tick.0 as f32) - time_k) / (time_k_next - time_k));
}
true
}
AutomationEvent::CancelAndHoldAtTime(..) => false,
AutomationEvent::CancelScheduledValues(..) | AutomationEvent::SetValue(..) => {
unreachable!("CancelScheduledValues/SetValue should never appear in the timeline")
}

</pre>

Let's walk through the code.

<code>AutomationEvent::SetValueCurveAtTime()</code> is a fn and takes three arguments: values, start, duration.

progress, computes the difference between the first value in start_time and [https://github.com/JHBalaji/media/blob/5a981dff94f4dfaf2ebf0f0e09f599fab9d1a13b/audio/src/param.rs#L384 time] and makes explicit conversion to f32.

<pre>
let time_diff = ((duration.0 as f32) - (start.0 as f32)) as f32;
let mut progress = ((((current_tick.0 as f32) - (start.0 as f32)) as f32) / time_diff) as f32;
</pre>

time_diff computes the total difference between duration and start_time, this is required when the values that needs to be computed goes beyond time_diff from start_time to set a constant value later.

<pre>
if progress < 0.0 {
progress = 0.0 as f32;
}
</pre>
Sets 0 if the progress from values array and present time being negative.

<pre>
let n = values.len() as f32;
let k = (((n - 1.) * progress) as f32).floor();
let next = k + 1. as f32;
let step = time_diff / (n - 1.);
</pre>
n, total length of values array. k, compute as k as in [[https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime here] and <code>next</code> is k+1, the delta or step is computed that would be added to generate the sound wave from the values array. Say, the frequency start at 44KHz and the <code>step</code> would be used to change the frequency of the curve.

<pre>
if next < n {
let time_k = (k * step) as f32;
let time_k_next = (next * step) as f32;
let v_k = values[k as usize];
let v_k_next = values[next as usize];
*value = v_k + (v_k_next - v_k) * (((current_tick.0 as f32) - time_k) / (time_k_next - time_k));
}
</pre>

The code block generates sequences of values from <code>values</code array and using <code>step</code> value we computed. And does that till it reaches the total length of the array.

<pre>
true
</pre>

[https://github.com/JHBalaji/media/blob/f32a71c51f05f44d1e8ca217fbaa4dfb0b55d11e/audio/src/param.rs#L435 Used] to loop the function call made from [https://github.com/JHBalaji/media/blob/f32a71c51f05f44d1e8ca217fbaa4dfb0b55d11e/audio/src/param.rs#L117 here].

<pre>
AutomationEvent::CancelAndHoldAtTime(..) => false,
AutomationEvent::CancelScheduledValues(..) | AutomationEvent::SetValue(..) => {
unreachable!("CancelScheduledValues/SetValue should never appear in the timeline")
}
</pre>

[https://github.com/JHBalaji/media/blob/5a981dff94f4dfaf2ebf0f0e09f599fab9d1a13b/audio/src/param.rs#L466 Code] instructs to cancel this event or end of this function and proceed to next instructions. In Audio API unless you explicitly say the block to end, the block would end and hardware device would be stopped from consumption.

==='''Examples'''===

Examples are unique aspect to Rust. This instructs that code to make use of the function we just wrote and generates a sound output. Refer our example [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs here] for the SetValueCurveAtTime()

And example is usually called by using cargo build in rust or cargo run.

[https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml Cargo.toml] dictates the naming and the file to be called as.

<pre>
cargo run --example set_value_curve
</pre>

The above code instructs to run the example [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs set_value_curve] to generate a sound output through our terminal.

==='''Compile'''===
[https://github.com/JHBalaji/media-1/blob/master/Cargo.lock Cargo.lock] instructs the dependencies that would be required to build this project.

[https://github.com/JHBalaji/media-1/blob/master/Cargo.toml Cargo.toml] calls all the folders or members to be complied. Usually the compilation would be at /media

Cargo.lock installs number of packages called crates that would be used by the project to access the modules. Each of the member in [https://github.com/JHBalaji/media-1/blob/master/Cargo.toml Cargo.toml] would have [https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml Cargo.toml] in them that would instruct [https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml#L6 dependencies] during the build. build-dependencies would be used if you have dependencies before the build.

[https://github.com/JHBalaji/media-1/blob/master/.cargo/config .cargo config] is similar to a dotenv file for Rust

==='''Test'''===

If you would like to test our build please look into [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Issues_and_Fixes Fixes] before proceeding. We would suggest MacOS or Linux 16.04 LTS for any development and testing. Virtual Machine or VirtualBox could be another option. We spent countless days in frustration with libraries and never ending apt installs so avoid Linux 18.04 for this build test.

Once <code>cargo build</code> is completed. Open /target/debug to view the examples that are generated.

[[File:Servo-initial-build.PNG]]

To run all examples after the cargo build is complete, use <code>cargo run --example <bin_name></code>

[[File:Examples.PNG]]

==='''Status of the Project'''===

The [https://github.com/JHBalaji/media-1 build] is ready and the Pull request is [https://github.com/servo/media/pull/230 merged] with Servo/media.

[[File:Pr.PNG]]

=='''Normal build from Master Repo '''==
To build Servo in development mode. This is useful for development, but the resulting binary is very slow.
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html
</pre>
For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:
<pre>
./mach build --release
./mach run --release tests/html/about-mozilla.html
</pre>

==='''Compiling and Running the code'''===

====Mac OS====

Follow the instructions posted in this [https://github.com/JHBalaji/media-1 URL] and install GStreamer and Rust. After that follow the following lines for compiling on MacOS.

<pre>
1. brew install gstreamer gst-plugins-base gst-plugins-good gst-plugins-bad gst-plugins-ugly gst-libav gst-rtsp-server gst-editing-services

2. export PKG_CONFIG_PATH=/usr/local/Cellar/libffi/3.2.1/lib/pkgconfig/

3. rustup override set nightly
</pre>

After this run the following command to compile servo/media
<pre>
cargo build
</pre>

After compiling you will be able to test our example that is places in the examples folder.

====Windows and Linux====

Head over here to install [https://rustup.rs/ Rust ToolChain]. If you need any additional details we would suggest reading the [https://github.com/rust-lang/rustup.rs/blob/master/README.md ReadMe on Rust]. Some Servo media specific build issues are [https://github.com/servo/media/blob/master/README.md advised].

We would suggest Ubuntu 18.04 LTS with updates [https://github.com/servo/media/blob/master/README.md gstreamer package].

==='''Servo Media'''===
<b>Requirements</b>
In order to build this crate you need to install gstreamer which can be found here [https://github.com/sdroege/gstreamer-rs]
==='''Test Case'''===
After you run cargo build, you should be able to see the executable of the example inside the target/debug folder. Following is the test we have written file to check the functioning of the [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs SetValueCurveAtTIme].
<pre>
extern crate servo_media;
extern crate servo_media_auto;

use servo_media::audio::constant_source_node::ConstantSourceNodeOptions;
use servo_media::audio::gain_node::GainNodeOptions;
use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::param::{ParamType, UserAutomationEvent};
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();

//Initializing the values vector for SetValueCurve function
let mut values: Vec<f32> = Vec::new();
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(1.0);
values.push(1.0);
values.push(1.0);
values.push(1.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.5);
values.push(0.5);
values.push(0.5);
values.push(0.5);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
let start_time = 0.;
let end_time = 5.;
let n = values.len() as f32;
let value_next = values[(n - 1.) as usize];

let cs = context.create_node(
AudioNodeInit::ConstantSourceNode(ConstantSourceNodeOptions::default()),
Default::default(),
);

let mut gain_options = GainNodeOptions::default();
gain_options.gain = 0.0;
let gain = context.create_node(
AudioNodeInit::GainNode(gain_options.clone()),
Default::default(),
);

let osc = context.create_node(
AudioNodeInit::OscillatorNode(Default::default()),
Default::default(),
);

context.connect_ports(osc.output(0), gain.input(0));
context.connect_ports(cs.output(0), gain.param(ParamType::Gain));
context.connect_ports(gain.output(0), dest.input(0));

let _ = context.resume();
context.message_node(
osc,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
gain,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::SetValueCurveAtTime(values, start_time, end_time),
),

);

context.message_node(
osc,
AudioNodeMessage::SetParam(
ParamType::Frequency,
UserAutomationEvent::SetValueAtTime(value_next, end_time),
),
);

thread::sleep(time::Duration::from_millis(7000));
let _ = context.close();
}

fn main() {
ServoMedia::init::<servo_media_auto::Backend>();
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}

</pre>

In the above program, the values vector is passed into the SetValueCurveAtTime function to modulate the output. The output of the same can also be checked in the test-case video at the end of this page.
We can modify the values function as we want and observe the output. One more modification that we used was by using a sine curve. It is implemented as follows. Change the values vector in the set-value-curve function with the following:

<pre>
use std::f64::consts::PI;
let mut values: Vec<f32> = Vec::new();
let curvelength = 44100;
let mut i = 0;
while i < curvelength {
values.push(f32::sin((PI *i as f64/curvelength as f64) as f32));
i += 1;
}
</pre>

The output of this function can be seen in the test video attached to at the end of this page.
<references></references>

=='''Video of Testing Audio Output'''==
[https://drive.google.com/file/d/1jzEZwAZE-7mi6N5N_JcEE6ISM-wc9910/view?usp=sharing Test-1]
[https://drive.google.com/file/d/1UsPgyv29Zf29d6pEDgLpYX86DI6hzOqp/view?usp=sharing Test-2]

=='''Issues and Fixes'''==

Windows Specific during build

https://github.com/holochain/holochain-cmd/issues/29

https://stackoverflow.com/questions/53136717/errore0554-feature-may-not-be-used-on-the-stable-release-channel-couldnt

https://github.com/servo/servo/issues/21429

=='''Final Project'''==
If you are a peer reviewer please look into this [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#For_Peer_Reviewers_Note section]

The final [https://github.com/servo/servo/wiki/Implement-missing-WebAudio-automation-student-project project] rely on our previous [https://github.com/servo/media/pull/230 work]. The next part would focus on implementing the code [https://github.com/servo/media/issues/204 written] to be accessible by Servo browser [https://github.com/servo/servo/issues/22897 engine].
This would require interfacing the DOM [https://github.com/servo/servo/blob/master/components/script/dom/webidls/AudioParam.webidl file] reference the Param.rs [https://github.com/JHBalaji/media/blob/master/audio/src/param.rs#L259 setValueCurveAtTime] to be interfaced with [https://github.com/servo/servo/blob/master/components/script/dom/audioparam.rs AudioParam.rs]. The audioparam.rs would call the UserAutomationEvent in the param.rs setValueCurveAtTime.

Once the initial function call with arguments are updated, then the [https://github.com/servo/servo/blob/master/tests/wpt/README.md#updating-test-expectations testing] would be performed on ''./mach test-wpt tests/wpt/web-platform-tests/webaudio/the-audio-api/[https://github.com/servo/servo/tree/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audioparam-interface the-audioparam-interface]''. The [https://github.com/servo/servo/tree/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audioparam-interface the-audioparam-interface] is a folder of series of test cases that emulate the behavior of a function call, here the setValueCurveAtTime would need to be necessitated to mock the values as tested in previous [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Test_Case example] but would require to be performed over a .html webpage with DOM elements to test the complete functionality.

Servo webAudio tests are performed using [https://github.com/web-platform-tests/wpt/blob/master/resources/readme.md testharness.js].

<pre>
./mach create-wpt tests/wpt/path/to/new/test.html
</pre>

would create a test.html using the WPT template for javascript tests. Once the skeleton code is completed with necessary function call, them the WPT tests would be executed using

<pre>
./mach test-wpt tests/wpt/path/to/new/test.html
./mach test-wpt tests/wpt/path/to/new/reftest.html
</pre>

[https://web-platform-tests.org/writing-tests/reftests.html reftest] are used in Servo to test behavior related to rendering; that are related to interaction made up of several webpages [think passing or referencing values to another webpage] with assertions to test cases that are expected from the interaction.

<pre>
./mach create-wpt --reftest tests/wpt/path/to/new/reftest.html --reference tests/wpt/path/to/reference.html
</pre>

reference.html and reftest.html will be created using WPT reftest template. Once the reftest is run the upstream test would need to be updated using

<pre>
./mach update-wpt --sync
./mach test-wpt --log-raw=update.log
./mach update-wpt update.log
</pre>

When reftest fails to diagnose the behavior reftest raw log is required which is generate using the below command. The .log is fed into [https://github.com/Gankro/live-reftest-analyzer reftest analyzer] to find those failing test. The reftest is also capable of pixel level comparison of the test and reference screenshots (incase of rendering)
<pre>
./mach test-wpt --log-raw wpt.log
</pre>

==='''List of issues'''===
This project has listed 4 known issues under the WebAudio title, each solving a different issue.

1. [https://github.com/servo/media/issues/204 Implement ValueCurveAtTime automation]

This is the first issue, which has been addressed and successfully closed during the initial parts of this project. The steps followed are shown in the following diagram.

[[File:WebAudio-Issue1.png]]

2. [https://github.com/servo/servo/issues/22897 Implement AudioParam.setValueCurveAtTime]

This second issue builds up on the first one, and focuses on implementing yet another method called [https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime setValueCurveAtTime]. The next focus of this project will likely be to work on this issue. This deals with the javascript API values for the curves.

[[File:WebAudio-Issue2.png]]

3. [https://github.com/servo/servo/issues/22898 Implement deprecated setPosition and setOrientation methods for AudioListener]

The third issue deals with a few missing APIs for AudioListener. We need to accordingly add the new methods, i.e. setPosition and setOrientation to the audiolistener.rs file so that they properly update the values of the relevant AudioParam members of AudioListener along with the passed arguments.This should help stopping an endless list of javascript errors when these deprecated methods weren't implemented.

[[File:WebAudio-Issue3.png]]

=='''For Peer Reviewers Note'''==

''Common peer review question clarifications are below,''

"Does the design incorporate all of the functionality required?" or "Check the changes that the authors proposed. Rate the overall quality of the explanations. Give comments if you find the explanation hard to follow or inadequate."

:The [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Final_Project section] clarifies your question.

"Check the designs proposed by the authors."

:There is no explicit design is involved as going by [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Final_Project proposal], there are three files to be added with new functionality in addition to test cases and files.

"Check the use of visual aids"
:Visual aids not applicable

"Give comments on the completeness of the design doc."
: Let us know what you think about them!

File:WebAudio-Issue3.png

2019-04-13T03:01:04Z

Amadhus2:

CSC/ECE 517 Spring 2019 - Project M1901 Implement missing WebAudio automation support

2019-04-13T02:52:52Z

Amadhus2: /* Final Project */

=='''Introduction'''==

==='''Servo'''===
Servo is an developmental browser engine designed to make use of the memory safety properties and concurrency features of the Rust programming language. The project was initiated by Mozilla Research with effort from Samsung to port it to Android and ARM processors

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language which focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and provides high performance. Rust is brain child of Mozilla and simple as this
<pre>
fn main() {
println!("Hello World");
}
</pre>

More information about Rust can be found [https://www.rust-lang.org/en-US/ here] and here is why we love [https://thenewstack.io/safer-future-rust/ Rust], [https://insights.stackoverflow.com/survey/2018/#most-loved-dreaded-and-wanted SF Loved]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. An audio routing graph has been created by linking together the Basic Audio operations performed with audio nodes. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context<br>
2. Inside the context, create sources — such as <audio>, oscillator, stream<br>
3. Create effects nodes, such as reverb, biquad filter, panner, compressor<br>
4. Choose final destination of audio, for example your system speakers<br>
5. Connect the sources up to the effects, and the effects to the destination.<br>

==='''Problem Statement: Web Audio Automation'''===

AudioParam is used to control the AudioNode functioning, say, Volume (a Gain parameter). These values can either be scheduled to be changed at precise times or set to particular value immediately following an event or at an event. The schedule to change when used with AudioContext.currentTime can helps in volume fades, filter sweeps etc. This would work aims to implement [https://github.com/servo/media/issues/204 SetValueCurveAtTime()].

[https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime setValueCurveAtTime()] is one such method in AudioParam that takes an array as input and schedules a change. The array is usually a curve in wedAudio and achieved my linear interpolation between the values from the floating-point values array and for the duration, d from the startTime, s

<code>v(t) = values[N * (t - s) / d]</code>, where N is the length of the values in the array.
And after the end of the curve time interval <code>(t >= s + d)</code>, the value will remain constant at the final curve value. This would persist to happen until the next automation event.

One of the application of setValueCurveAtTime() it to create tremolo effect. Suppose linear nor an exponential curve satisfy the needs then user can create curve based on values to setValueCurveAtTime() with an array of timing values. Its a much loved approached against multiple calls to setValueAtTime().

==='''Build'''===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.

=='''Implementation of the Project'''==

==='''Code'''===
The code is implemented as in [https://github.com/JHBalaji/media/blob/master/audio/src/param.rs#L418-L436 here]. The code below is the main code snippet for the SetValueCurveAtTime function. The values and function behavior is define in the param.rs file.

<pre>

AutomationEvent::SetValueCurveAtTime(ref values, start, duration) => {
let time_diff = ((duration.0 as f32) - (start.0 as f32)) as f32;
let mut progress = ((((current_tick.0 as f32) - (start.0 as f32)) as f32) / time_diff) as f32;
if progress < 0.0 {
progress = 0.0 as f32;
}
let n = values.len() as f32;
let k = (((n - 1.) * progress) as f32).floor();
let next = k + 1. as f32;
let step = time_diff / (n - 1.);
if next < n {
let time_k = (k * step) as f32;
let time_k_next = (next * step) as f32;
let v_k = values[k as usize];
let v_k_next = values[next as usize];
*value = v_k + (v_k_next - v_k) * (((current_tick.0 as f32) - time_k) / (time_k_next - time_k));
}
true
}
AutomationEvent::CancelAndHoldAtTime(..) => false,
AutomationEvent::CancelScheduledValues(..) | AutomationEvent::SetValue(..) => {
unreachable!("CancelScheduledValues/SetValue should never appear in the timeline")
}

</pre>

Let's walk through the code.

<code>AutomationEvent::SetValueCurveAtTime()</code> is a fn and takes three arguments: values, start, duration.

progress, computes the difference between the first value in start_time and [https://github.com/JHBalaji/media/blob/5a981dff94f4dfaf2ebf0f0e09f599fab9d1a13b/audio/src/param.rs#L384 time] and makes explicit conversion to f32.

<pre>
let time_diff = ((duration.0 as f32) - (start.0 as f32)) as f32;
let mut progress = ((((current_tick.0 as f32) - (start.0 as f32)) as f32) / time_diff) as f32;
</pre>

time_diff computes the total difference between duration and start_time, this is required when the values that needs to be computed goes beyond time_diff from start_time to set a constant value later.

<pre>
if progress < 0.0 {
progress = 0.0 as f32;
}
</pre>
Sets 0 if the progress from values array and present time being negative.

<pre>
let n = values.len() as f32;
let k = (((n - 1.) * progress) as f32).floor();
let next = k + 1. as f32;
let step = time_diff / (n - 1.);
</pre>
n, total length of values array. k, compute as k as in [[https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime here] and <code>next</code> is k+1, the delta or step is computed that would be added to generate the sound wave from the values array. Say, the frequency start at 44KHz and the <code>step</code> would be used to change the frequency of the curve.

<pre>
if next < n {
let time_k = (k * step) as f32;
let time_k_next = (next * step) as f32;
let v_k = values[k as usize];
let v_k_next = values[next as usize];
*value = v_k + (v_k_next - v_k) * (((current_tick.0 as f32) - time_k) / (time_k_next - time_k));
}
</pre>

The code block generates sequences of values from <code>values</code array and using <code>step</code> value we computed. And does that till it reaches the total length of the array.

<pre>
true
</pre>

[https://github.com/JHBalaji/media/blob/f32a71c51f05f44d1e8ca217fbaa4dfb0b55d11e/audio/src/param.rs#L435 Used] to loop the function call made from [https://github.com/JHBalaji/media/blob/f32a71c51f05f44d1e8ca217fbaa4dfb0b55d11e/audio/src/param.rs#L117 here].

<pre>
AutomationEvent::CancelAndHoldAtTime(..) => false,
AutomationEvent::CancelScheduledValues(..) | AutomationEvent::SetValue(..) => {
unreachable!("CancelScheduledValues/SetValue should never appear in the timeline")
}
</pre>

[https://github.com/JHBalaji/media/blob/5a981dff94f4dfaf2ebf0f0e09f599fab9d1a13b/audio/src/param.rs#L466 Code] instructs to cancel this event or end of this function and proceed to next instructions. In Audio API unless you explicitly say the block to end, the block would end and hardware device would be stopped from consumption.

==='''Examples'''===

Examples are unique aspect to Rust. This instructs that code to make use of the function we just wrote and generates a sound output. Refer our example [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs here] for the SetValueCurveAtTime()

And example is usually called by using cargo build in rust or cargo run.

[https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml Cargo.toml] dictates the naming and the file to be called as.

<pre>
cargo run --example set_value_curve
</pre>

The above code instructs to run the example [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs set_value_curve] to generate a sound output through our terminal.

==='''Compile'''===
[https://github.com/JHBalaji/media-1/blob/master/Cargo.lock Cargo.lock] instructs the dependencies that would be required to build this project.

[https://github.com/JHBalaji/media-1/blob/master/Cargo.toml Cargo.toml] calls all the folders or members to be complied. Usually the compilation would be at /media

Cargo.lock installs number of packages called crates that would be used by the project to access the modules. Each of the member in [https://github.com/JHBalaji/media-1/blob/master/Cargo.toml Cargo.toml] would have [https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml Cargo.toml] in them that would instruct [https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml#L6 dependencies] during the build. build-dependencies would be used if you have dependencies before the build.

[https://github.com/JHBalaji/media-1/blob/master/.cargo/config .cargo config] is similar to a dotenv file for Rust

==='''Test'''===

If you would like to test our build please look into [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Issues_and_Fixes Fixes] before proceeding. We would suggest MacOS or Linux 16.04 LTS for any development and testing. Virtual Machine or VirtualBox could be another option. We spent countless days in frustration with libraries and never ending apt installs so avoid Linux 18.04 for this build test.

Once <code>cargo build</code> is completed. Open /target/debug to view the examples that are generated.

[[File:Servo-initial-build.PNG]]

To run all examples after the cargo build is complete, use <code>cargo run --example <bin_name></code>

[[File:Examples.PNG]]

==='''Status of the Project'''===

The [https://github.com/JHBalaji/media-1 build] is ready and the Pull request is [https://github.com/servo/media/pull/230 merged] with Servo/media.

[[File:Pr.PNG]]

=='''Normal build from Master Repo '''==
To build Servo in development mode. This is useful for development, but the resulting binary is very slow.
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html
</pre>
For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:
<pre>
./mach build --release
./mach run --release tests/html/about-mozilla.html
</pre>

==='''Compiling and Running the code'''===

====Mac OS====

Follow the instructions posted in this [https://github.com/JHBalaji/media-1 URL] and install GStreamer and Rust. After that follow the following lines for compiling on MacOS.

<pre>
1. brew install gstreamer gst-plugins-base gst-plugins-good gst-plugins-bad gst-plugins-ugly gst-libav gst-rtsp-server gst-editing-services

2. export PKG_CONFIG_PATH=/usr/local/Cellar/libffi/3.2.1/lib/pkgconfig/

3. rustup override set nightly
</pre>

After this run the following command to compile servo/media
<pre>
cargo build
</pre>

After compiling you will be able to test our example that is places in the examples folder.

====Windows and Linux====

Head over here to install [https://rustup.rs/ Rust ToolChain]. If you need any additional details we would suggest reading the [https://github.com/rust-lang/rustup.rs/blob/master/README.md ReadMe on Rust]. Some Servo media specific build issues are [https://github.com/servo/media/blob/master/README.md advised].

We would suggest Ubuntu 18.04 LTS with updates [https://github.com/servo/media/blob/master/README.md gstreamer package].

==='''Servo Media'''===
<b>Requirements</b>
In order to build this crate you need to install gstreamer which can be found here [https://github.com/sdroege/gstreamer-rs]
==='''Test Case'''===
After you run cargo build, you should be able to see the executable of the example inside the target/debug folder. Following is the test we have written file to check the functioning of the [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs SetValueCurveAtTIme].
<pre>
extern crate servo_media;
extern crate servo_media_auto;

use servo_media::audio::constant_source_node::ConstantSourceNodeOptions;
use servo_media::audio::gain_node::GainNodeOptions;
use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::param::{ParamType, UserAutomationEvent};
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();

//Initializing the values vector for SetValueCurve function
let mut values: Vec<f32> = Vec::new();
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(1.0);
values.push(1.0);
values.push(1.0);
values.push(1.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.5);
values.push(0.5);
values.push(0.5);
values.push(0.5);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
let start_time = 0.;
let end_time = 5.;
let n = values.len() as f32;
let value_next = values[(n - 1.) as usize];

let cs = context.create_node(
AudioNodeInit::ConstantSourceNode(ConstantSourceNodeOptions::default()),
Default::default(),
);

let mut gain_options = GainNodeOptions::default();
gain_options.gain = 0.0;
let gain = context.create_node(
AudioNodeInit::GainNode(gain_options.clone()),
Default::default(),
);

let osc = context.create_node(
AudioNodeInit::OscillatorNode(Default::default()),
Default::default(),
);

context.connect_ports(osc.output(0), gain.input(0));
context.connect_ports(cs.output(0), gain.param(ParamType::Gain));
context.connect_ports(gain.output(0), dest.input(0));

let _ = context.resume();
context.message_node(
osc,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
gain,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::SetValueCurveAtTime(values, start_time, end_time),
),

);

context.message_node(
osc,
AudioNodeMessage::SetParam(
ParamType::Frequency,
UserAutomationEvent::SetValueAtTime(value_next, end_time),
),
);

thread::sleep(time::Duration::from_millis(7000));
let _ = context.close();
}

fn main() {
ServoMedia::init::<servo_media_auto::Backend>();
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}

</pre>

In the above program, the values vector is passed into the SetValueCurveAtTime function to modulate the output. The output of the same can also be checked in the test-case video at the end of this page.
We can modify the values function as we want and observe the output. One more modification that we used was by using a sine curve. It is implemented as follows. Change the values vector in the set-value-curve function with the following:

<pre>
use std::f64::consts::PI;
let mut values: Vec<f32> = Vec::new();
let curvelength = 44100;
let mut i = 0;
while i < curvelength {
values.push(f32::sin((PI *i as f64/curvelength as f64) as f32));
i += 1;
}
</pre>

The output of this function can be seen in the test video attached to at the end of this page.
<references></references>

=='''Video of Testing Audio Output'''==
[https://drive.google.com/file/d/1jzEZwAZE-7mi6N5N_JcEE6ISM-wc9910/view?usp=sharing Test-1]
[https://drive.google.com/file/d/1UsPgyv29Zf29d6pEDgLpYX86DI6hzOqp/view?usp=sharing Test-2]

=='''Issues and Fixes'''==

Windows Specific during build

https://github.com/holochain/holochain-cmd/issues/29

https://stackoverflow.com/questions/53136717/errore0554-feature-may-not-be-used-on-the-stable-release-channel-couldnt

https://github.com/servo/servo/issues/21429

=='''Final Project'''==
If you are a peer reviewer please look into this [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#For_Peer_Reviewers_Note section]

The final [https://github.com/servo/servo/wiki/Implement-missing-WebAudio-automation-student-project project] rely on our previous [https://github.com/servo/media/pull/230 work]. The next part would focus on implementing the code [https://github.com/servo/media/issues/204 written] to be accessible by Servo browser [https://github.com/servo/servo/issues/22897 engine].
This would require interfacing the DOM [https://github.com/servo/servo/blob/master/components/script/dom/webidls/AudioParam.webidl file] reference the Param.rs [https://github.com/JHBalaji/media/blob/master/audio/src/param.rs#L259 setValueCurveAtTime] to be interfaced with [https://github.com/servo/servo/blob/master/components/script/dom/audioparam.rs AudioParam.rs]. The audioparam.rs would call the UserAutomationEvent in the param.rs setValueCurveAtTime.

Once the initial function call with arguments are updated, then the [https://github.com/servo/servo/blob/master/tests/wpt/README.md#updating-test-expectations testing] would be performed on ''./mach test-wpt tests/wpt/web-platform-tests/webaudio/the-audio-api/[https://github.com/servo/servo/tree/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audioparam-interface the-audioparam-interface]''. The [https://github.com/servo/servo/tree/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audioparam-interface the-audioparam-interface] is a folder of series of test cases that emulate the behavior of a function call, here the setValueCurveAtTime would need to be necessitated to mock the values as tested in previous [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Test_Case example] but would require to be performed over a .html webpage with DOM elements to test the complete functionality.

Servo webAudio tests are performed using [https://github.com/web-platform-tests/wpt/blob/master/resources/readme.md testharness.js].

<pre>
./mach create-wpt tests/wpt/path/to/new/test.html
</pre>

would create a test.html using the WPT template for javascript tests. Once the skeleton code is completed with necessary function call, them the WPT tests would be executed using

<pre>
./mach test-wpt tests/wpt/path/to/new/test.html
./mach test-wpt tests/wpt/path/to/new/reftest.html
</pre>

[https://web-platform-tests.org/writing-tests/reftests.html reftest] are used in Servo to test behavior related to rendering; that are related to interaction made up of several webpages [think passing or referencing values to another webpage] with assertions to test cases that are expected from the interaction.

<pre>
./mach create-wpt --reftest tests/wpt/path/to/new/reftest.html --reference tests/wpt/path/to/reference.html
</pre>

reference.html and reftest.html will be created using WPT reftest template. Once the reftest is run the upstream test would need to be updated using

<pre>
./mach update-wpt --sync
./mach test-wpt --log-raw=update.log
./mach update-wpt update.log
</pre>

When reftest fails to diagnose the behavior reftest raw log is required which is generate using the below command. The .log is fed into [https://github.com/Gankro/live-reftest-analyzer reftest analyzer] to find those failing test. The reftest is also capable of pixel level comparison of the test and reference screenshots (incase of rendering)
<pre>
./mach test-wpt --log-raw wpt.log
</pre>

==='''List of issues'''===
This project has listed 4 known issues under the WebAudio title, each solving a different issue.

1. [https://github.com/servo/media/issues/204 Implement ValueCurveAtTime automation]

This is the first issue, which has been addressed and successfully closed during the initial parts of this project. The steps followed are shown in the following diagram.

[[File:WebAudio-Issue1.png]]

2. [https://github.com/servo/servo/issues/22897 Implement AudioParam.setValueCurveAtTime]

This second issue builds up on the first one, and focuses on implementing yet another method called [https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime setValueCurveAtTime]. The next focus of this project will likely be to work on this issue.

[[File:WebAudio-Issue2.png]]

=='''For Peer Reviewers Note'''==

''Common peer review question clarifications are below,''

"Does the design incorporate all of the functionality required?" or "Check the changes that the authors proposed. Rate the overall quality of the explanations. Give comments if you find the explanation hard to follow or inadequate."

:The [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Final_Project section] clarifies your question.

"Check the designs proposed by the authors."

:There is no explicit design is involved as going by [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Final_Project proposal], there are three files to be added with new functionality in addition to test cases and files.

"Check the use of visual aids"
:Visual aids not applicable

"Give comments on the completeness of the design doc."
: Let us know what you think about them!

CSC/ECE 517 Spring 2019 - Project M1901 Implement missing WebAudio automation support

2019-04-13T02:52:02Z

Amadhus2: /* List of issues */

=='''Introduction'''==

==='''Servo'''===
Servo is an developmental browser engine designed to make use of the memory safety properties and concurrency features of the Rust programming language. The project was initiated by Mozilla Research with effort from Samsung to port it to Android and ARM processors

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language which focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and provides high performance. Rust is brain child of Mozilla and simple as this
<pre>
fn main() {
println!("Hello World");
}
</pre>

More information about Rust can be found [https://www.rust-lang.org/en-US/ here] and here is why we love [https://thenewstack.io/safer-future-rust/ Rust], [https://insights.stackoverflow.com/survey/2018/#most-loved-dreaded-and-wanted SF Loved]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. An audio routing graph has been created by linking together the Basic Audio operations performed with audio nodes. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context<br>
2. Inside the context, create sources — such as <audio>, oscillator, stream<br>
3. Create effects nodes, such as reverb, biquad filter, panner, compressor<br>
4. Choose final destination of audio, for example your system speakers<br>
5. Connect the sources up to the effects, and the effects to the destination.<br>

==='''Problem Statement: Web Audio Automation'''===

AudioParam is used to control the AudioNode functioning, say, Volume (a Gain parameter). These values can either be scheduled to be changed at precise times or set to particular value immediately following an event or at an event. The schedule to change when used with AudioContext.currentTime can helps in volume fades, filter sweeps etc. This would work aims to implement [https://github.com/servo/media/issues/204 SetValueCurveAtTime()].

[https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime setValueCurveAtTime()] is one such method in AudioParam that takes an array as input and schedules a change. The array is usually a curve in wedAudio and achieved my linear interpolation between the values from the floating-point values array and for the duration, d from the startTime, s

<code>v(t) = values[N * (t - s) / d]</code>, where N is the length of the values in the array.
And after the end of the curve time interval <code>(t >= s + d)</code>, the value will remain constant at the final curve value. This would persist to happen until the next automation event.

One of the application of setValueCurveAtTime() it to create tremolo effect. Suppose linear nor an exponential curve satisfy the needs then user can create curve based on values to setValueCurveAtTime() with an array of timing values. Its a much loved approached against multiple calls to setValueAtTime().

==='''Build'''===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.

=='''Implementation of the Project'''==

==='''Code'''===
The code is implemented as in [https://github.com/JHBalaji/media/blob/master/audio/src/param.rs#L418-L436 here]. The code below is the main code snippet for the SetValueCurveAtTime function. The values and function behavior is define in the param.rs file.

<pre>

AutomationEvent::SetValueCurveAtTime(ref values, start, duration) => {
let time_diff = ((duration.0 as f32) - (start.0 as f32)) as f32;
let mut progress = ((((current_tick.0 as f32) - (start.0 as f32)) as f32) / time_diff) as f32;
if progress < 0.0 {
progress = 0.0 as f32;
}
let n = values.len() as f32;
let k = (((n - 1.) * progress) as f32).floor();
let next = k + 1. as f32;
let step = time_diff / (n - 1.);
if next < n {
let time_k = (k * step) as f32;
let time_k_next = (next * step) as f32;
let v_k = values[k as usize];
let v_k_next = values[next as usize];
*value = v_k + (v_k_next - v_k) * (((current_tick.0 as f32) - time_k) / (time_k_next - time_k));
}
true
}
AutomationEvent::CancelAndHoldAtTime(..) => false,
AutomationEvent::CancelScheduledValues(..) | AutomationEvent::SetValue(..) => {
unreachable!("CancelScheduledValues/SetValue should never appear in the timeline")
}

</pre>

Let's walk through the code.

<code>AutomationEvent::SetValueCurveAtTime()</code> is a fn and takes three arguments: values, start, duration.

progress, computes the difference between the first value in start_time and [https://github.com/JHBalaji/media/blob/5a981dff94f4dfaf2ebf0f0e09f599fab9d1a13b/audio/src/param.rs#L384 time] and makes explicit conversion to f32.

<pre>
let time_diff = ((duration.0 as f32) - (start.0 as f32)) as f32;
let mut progress = ((((current_tick.0 as f32) - (start.0 as f32)) as f32) / time_diff) as f32;
</pre>

time_diff computes the total difference between duration and start_time, this is required when the values that needs to be computed goes beyond time_diff from start_time to set a constant value later.

<pre>
if progress < 0.0 {
progress = 0.0 as f32;
}
</pre>
Sets 0 if the progress from values array and present time being negative.

<pre>
let n = values.len() as f32;
let k = (((n - 1.) * progress) as f32).floor();
let next = k + 1. as f32;
let step = time_diff / (n - 1.);
</pre>
n, total length of values array. k, compute as k as in [[https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime here] and <code>next</code> is k+1, the delta or step is computed that would be added to generate the sound wave from the values array. Say, the frequency start at 44KHz and the <code>step</code> would be used to change the frequency of the curve.

<pre>
if next < n {
let time_k = (k * step) as f32;
let time_k_next = (next * step) as f32;
let v_k = values[k as usize];
let v_k_next = values[next as usize];
*value = v_k + (v_k_next - v_k) * (((current_tick.0 as f32) - time_k) / (time_k_next - time_k));
}
</pre>

The code block generates sequences of values from <code>values</code array and using <code>step</code> value we computed. And does that till it reaches the total length of the array.

<pre>
true
</pre>

[https://github.com/JHBalaji/media/blob/f32a71c51f05f44d1e8ca217fbaa4dfb0b55d11e/audio/src/param.rs#L435 Used] to loop the function call made from [https://github.com/JHBalaji/media/blob/f32a71c51f05f44d1e8ca217fbaa4dfb0b55d11e/audio/src/param.rs#L117 here].

<pre>
AutomationEvent::CancelAndHoldAtTime(..) => false,
AutomationEvent::CancelScheduledValues(..) | AutomationEvent::SetValue(..) => {
unreachable!("CancelScheduledValues/SetValue should never appear in the timeline")
}
</pre>

[https://github.com/JHBalaji/media/blob/5a981dff94f4dfaf2ebf0f0e09f599fab9d1a13b/audio/src/param.rs#L466 Code] instructs to cancel this event or end of this function and proceed to next instructions. In Audio API unless you explicitly say the block to end, the block would end and hardware device would be stopped from consumption.

==='''Examples'''===

Examples are unique aspect to Rust. This instructs that code to make use of the function we just wrote and generates a sound output. Refer our example [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs here] for the SetValueCurveAtTime()

And example is usually called by using cargo build in rust or cargo run.

[https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml Cargo.toml] dictates the naming and the file to be called as.

<pre>
cargo run --example set_value_curve
</pre>

The above code instructs to run the example [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs set_value_curve] to generate a sound output through our terminal.

==='''Compile'''===
[https://github.com/JHBalaji/media-1/blob/master/Cargo.lock Cargo.lock] instructs the dependencies that would be required to build this project.

[https://github.com/JHBalaji/media-1/blob/master/Cargo.toml Cargo.toml] calls all the folders or members to be complied. Usually the compilation would be at /media

Cargo.lock installs number of packages called crates that would be used by the project to access the modules. Each of the member in [https://github.com/JHBalaji/media-1/blob/master/Cargo.toml Cargo.toml] would have [https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml Cargo.toml] in them that would instruct [https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml#L6 dependencies] during the build. build-dependencies would be used if you have dependencies before the build.

[https://github.com/JHBalaji/media-1/blob/master/.cargo/config .cargo config] is similar to a dotenv file for Rust

==='''Test'''===

If you would like to test our build please look into [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Issues_and_Fixes Fixes] before proceeding. We would suggest MacOS or Linux 16.04 LTS for any development and testing. Virtual Machine or VirtualBox could be another option. We spent countless days in frustration with libraries and never ending apt installs so avoid Linux 18.04 for this build test.

Once <code>cargo build</code> is completed. Open /target/debug to view the examples that are generated.

[[File:Servo-initial-build.PNG]]

To run all examples after the cargo build is complete, use <code>cargo run --example <bin_name></code>

[[File:Examples.PNG]]

''''''Bold text''''''==='''Status of the Project'''===

The [https://github.com/JHBalaji/media-1 build] is ready and the Pull request is [https://github.com/servo/media/pull/230 merged] with Servo/media.

[[File:Pr.PNG]]
[[File:Example.jpg]]

=='''Normal build from Master Repo '''==
To build Servo in development mode. This is useful for development, but the resulting binary is very slow.
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html
</pre>
For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:
<pre>
./mach build --release
./mach run --release tests/html/about-mozilla.html
</pre>

==='''Compiling and Running the code'''===

====Mac OS====

Follow the instructions posted in this [https://github.com/JHBalaji/media-1 URL] and install GStreamer and Rust. After that follow the following lines for compiling on MacOS.

<pre>
1. brew install gstreamer gst-plugins-base gst-plugins-good gst-plugins-bad gst-plugins-ugly gst-libav gst-rtsp-server gst-editing-services

2. export PKG_CONFIG_PATH=/usr/local/Cellar/libffi/3.2.1/lib/pkgconfig/

3. rustup override set nightly
</pre>

After this run the following command to compile servo/media
<pre>
cargo build
</pre>

After compiling you will be able to test our example that is places in the examples folder.

====Windows and Linux====

Head over here to install [https://rustup.rs/ Rust ToolChain]. If you need any additional details we would suggest reading the [https://github.com/rust-lang/rustup.rs/blob/master/README.md ReadMe on Rust]. Some Servo media specific build issues are [https://github.com/servo/media/blob/master/README.md advised].

We would suggest Ubuntu 18.04 LTS with updates [https://github.com/servo/media/blob/master/README.md gstreamer package].

==='''Servo Media'''===
<b>Requirements</b>
In order to build this crate you need to install gstreamer which can be found here [https://github.com/sdroege/gstreamer-rs]
==='''Test Case'''===
After you run cargo build, you should be able to see the executable of the example inside the target/debug folder. Following is the test we have written file to check the functioning of the [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs SetValueCurveAtTIme].
<pre>
extern crate servo_media;
extern crate servo_media_auto;

use servo_media::audio::constant_source_node::ConstantSourceNodeOptions;
use servo_media::audio::gain_node::GainNodeOptions;
use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::param::{ParamType, UserAutomationEvent};
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();

//Initializing the values vector for SetValueCurve function
let mut values: Vec<f32> = Vec::new();
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(1.0);
values.push(1.0);
values.push(1.0);
values.push(1.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.5);
values.push(0.5);
values.push(0.5);
values.push(0.5);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
let start_time = 0.;
let end_time = 5.;
let n = values.len() as f32;
let value_next = values[(n - 1.) as usize];

let cs = context.create_node(
AudioNodeInit::ConstantSourceNode(ConstantSourceNodeOptions::default()),
Default::default(),
);

let mut gain_options = GainNodeOptions::default();
gain_options.gain = 0.0;
let gain = context.create_node(
AudioNodeInit::GainNode(gain_options.clone()),
Default::default(),
);

let osc = context.create_node(
AudioNodeInit::OscillatorNode(Default::default()),
Default::default(),
);

context.connect_ports(osc.output(0), gain.input(0));
context.connect_ports(cs.output(0), gain.param(ParamType::Gain));
context.connect_ports(gain.output(0), dest.input(0));

let _ = context.resume();
context.message_node(
osc,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
gain,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::SetValueCurveAtTime(values, start_time, end_time),
),

);

context.message_node(
osc,
AudioNodeMessage::SetParam(
ParamType::Frequency,
UserAutomationEvent::SetValueAtTime(value_next, end_time),
),
);

thread::sleep(time::Duration::from_millis(7000));
let _ = context.close();
}

fn main() {
ServoMedia::init::<servo_media_auto::Backend>();
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}

</pre>

In the above program, the values vector is passed into the SetValueCurveAtTime function to modulate the output. The output of the same can also be checked in the test-case video at the end of this page.
We can modify the values function as we want and observe the output. One more modification that we used was by using a sine curve. It is implemented as follows. Change the values vector in the set-value-curve function with the following:

<pre>
use std::f64::consts::PI;
let mut values: Vec<f32> = Vec::new();
let curvelength = 44100;
let mut i = 0;
while i < curvelength {
values.push(f32::sin((PI *i as f64/curvelength as f64) as f32));
i += 1;
}
</pre>

The output of this function can be seen in the test video attached to at the end of this page.
<references></references>

=='''Video of Testing Audio Output'''==
[https://drive.google.com/file/d/1jzEZwAZE-7mi6N5N_JcEE6ISM-wc9910/view?usp=sharing Test-1]
[https://drive.google.com/file/d/1UsPgyv29Zf29d6pEDgLpYX86DI6hzOqp/view?usp=sharing Test-2]

=='''Issues and Fixes'''==

Windows Specific during build

https://github.com/holochain/holochain-cmd/issues/29

https://stackoverflow.com/questions/53136717/errore0554-feature-may-not-be-used-on-the-stable-release-channel-couldnt

https://github.com/servo/servo/issues/21429

=='''Final Project'''==
If you are a peer reviewer please look into this [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#For_Peer_Reviewers_Note section]

The final [https://github.com/servo/servo/wiki/Implement-missing-WebAudio-automation-student-project project] rely on our previous [https://github.com/servo/media/pull/230 work]. The next part would focus on implementing the code [https://github.com/servo/media/issues/204 written] to be accessible by Servo browser [https://github.com/servo/servo/issues/22897 engine].
This would require interfacing the DOM [https://github.com/servo/servo/blob/master/components/script/dom/webidls/AudioParam.webidl file] reference the Param.rs [https://github.com/JHBalaji/media/blob/master/audio/src/param.rs#L259 setValueCurveAtTime] to be interfaced with [https://github.com/servo/servo/blob/master/components/script/dom/audioparam.rs AudioParam.rs]. The audioparam.rs would call the UserAutomationEvent in the param.rs setValueCurveAtTime.

Once the initial function call with arguments are updated, then the [https://github.com/servo/servo/blob/master/tests/wpt/README.md#updating-test-expectations testing] would be performed on ''./mach test-wpt tests/wpt/web-platform-tests/webaudio/the-audio-api/[https://github.com/servo/servo/tree/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audioparam-interface the-audioparam-interface]''. The [https://github.com/servo/servo/tree/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audioparam-interface the-audioparam-interface] is a folder of series of test cases that emulate the behavior of a function call, here the setValueCurveAtTime would need to be necessitated to mock the values as tested in previous [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Test_Case example] but would require to be performed over a .html webpage with DOM elements to test the complete functionality.

Servo webAudio tests are performed using [https://github.com/web-platform-tests/wpt/blob/master/resources/readme.md testharness.js].

<pre>
./mach create-wpt tests/wpt/path/to/new/test.html
</pre>

would create a test.html using the WPT template for javascript tests. Once the skeleton code is completed with necessary function call, them the WPT tests would be executed using

<pre>
./mach test-wpt tests/wpt/path/to/new/test.html
./mach test-wpt tests/wpt/path/to/new/reftest.html
</pre>

[https://web-platform-tests.org/writing-tests/reftests.html reftest] are used in Servo to test behavior related to rendering; that are related to interaction made up of several webpages [think passing or referencing values to another webpage] with assertions to test cases that are expected from the interaction.

<pre>
./mach create-wpt --reftest tests/wpt/path/to/new/reftest.html --reference tests/wpt/path/to/reference.html
</pre>

reference.html and reftest.html will be created using WPT reftest template. Once the reftest is run the upstream test would need to be updated using

<pre>
./mach update-wpt --sync
./mach test-wpt --log-raw=update.log
./mach update-wpt update.log
</pre>

When reftest fails to diagnose the behavior reftest raw log is required which is generate using the below command. The .log is fed into [https://github.com/Gankro/live-reftest-analyzer reftest analyzer] to find those failing test. The reftest is also capable of pixel level comparison of the test and reference screenshots (incase of rendering)
<pre>
./mach test-wpt --log-raw wpt.log
</pre>

==='''List of issues'''===
This project has listed 4 known issues under the WebAudio title, each solving a different issue.

1. [https://github.com/servo/media/issues/204 Implement ValueCurveAtTime automation]
This is the first issue, which has been addressed and successfully closed during the initial parts of this project. The steps followed are shown in the following diagram.

[[File:WebAudio-Issue1.png]]

2. [https://github.com/servo/servo/issues/22897 Implement AudioParam.setValueCurveAtTime]
This second issue builds up on the first one, and focuses on implementing yet another method called [https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime setValueCurveAtTime]. The next focus of this project will likely be to work on this issue.

[[File:WebAudio-Issue2.png]]

=='''For Peer Reviewers Note'''==

''Common peer review question clarifications are below,''

"Does the design incorporate all of the functionality required?" or "Check the changes that the authors proposed. Rate the overall quality of the explanations. Give comments if you find the explanation hard to follow or inadequate."

:The [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Final_Project section] clarifies your question.

"Check the designs proposed by the authors."

:There is no explicit design is involved as going by [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Final_Project proposal], there are three files to be added with new functionality in addition to test cases and files.

"Check the use of visual aids"
:Visual aids not applicable

"Give comments on the completeness of the design doc."
: Let us know what you think about them!

CSC/ECE 517 Spring 2019 - Project M1901 Implement missing WebAudio automation support

2019-04-13T02:51:34Z

Amadhus2: /* List of issues */

=='''Introduction'''==

==='''Servo'''===
Servo is an developmental browser engine designed to make use of the memory safety properties and concurrency features of the Rust programming language. The project was initiated by Mozilla Research with effort from Samsung to port it to Android and ARM processors

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language which focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and provides high performance. Rust is brain child of Mozilla and simple as this
<pre>
fn main() {
println!("Hello World");
}
</pre>

More information about Rust can be found [https://www.rust-lang.org/en-US/ here] and here is why we love [https://thenewstack.io/safer-future-rust/ Rust], [https://insights.stackoverflow.com/survey/2018/#most-loved-dreaded-and-wanted SF Loved]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. An audio routing graph has been created by linking together the Basic Audio operations performed with audio nodes. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context<br>
2. Inside the context, create sources — such as <audio>, oscillator, stream<br>
3. Create effects nodes, such as reverb, biquad filter, panner, compressor<br>
4. Choose final destination of audio, for example your system speakers<br>
5. Connect the sources up to the effects, and the effects to the destination.<br>

==='''Problem Statement: Web Audio Automation'''===

AudioParam is used to control the AudioNode functioning, say, Volume (a Gain parameter). These values can either be scheduled to be changed at precise times or set to particular value immediately following an event or at an event. The schedule to change when used with AudioContext.currentTime can helps in volume fades, filter sweeps etc. This would work aims to implement [https://github.com/servo/media/issues/204 SetValueCurveAtTime()].

[https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime setValueCurveAtTime()] is one such method in AudioParam that takes an array as input and schedules a change. The array is usually a curve in wedAudio and achieved my linear interpolation between the values from the floating-point values array and for the duration, d from the startTime, s

<code>v(t) = values[N * (t - s) / d]</code>, where N is the length of the values in the array.
And after the end of the curve time interval <code>(t >= s + d)</code>, the value will remain constant at the final curve value. This would persist to happen until the next automation event.

One of the application of setValueCurveAtTime() it to create tremolo effect. Suppose linear nor an exponential curve satisfy the needs then user can create curve based on values to setValueCurveAtTime() with an array of timing values. Its a much loved approached against multiple calls to setValueAtTime().

==='''Build'''===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.

=='''Implementation of the Project'''==

==='''Code'''===
The code is implemented as in [https://github.com/JHBalaji/media/blob/master/audio/src/param.rs#L418-L436 here]. The code below is the main code snippet for the SetValueCurveAtTime function. The values and function behavior is define in the param.rs file.

<pre>

AutomationEvent::SetValueCurveAtTime(ref values, start, duration) => {
let time_diff = ((duration.0 as f32) - (start.0 as f32)) as f32;
let mut progress = ((((current_tick.0 as f32) - (start.0 as f32)) as f32) / time_diff) as f32;
if progress < 0.0 {
progress = 0.0 as f32;
}
let n = values.len() as f32;
let k = (((n - 1.) * progress) as f32).floor();
let next = k + 1. as f32;
let step = time_diff / (n - 1.);
if next < n {
let time_k = (k * step) as f32;
let time_k_next = (next * step) as f32;
let v_k = values[k as usize];
let v_k_next = values[next as usize];
*value = v_k + (v_k_next - v_k) * (((current_tick.0 as f32) - time_k) / (time_k_next - time_k));
}
true
}
AutomationEvent::CancelAndHoldAtTime(..) => false,
AutomationEvent::CancelScheduledValues(..) | AutomationEvent::SetValue(..) => {
unreachable!("CancelScheduledValues/SetValue should never appear in the timeline")
}

</pre>

Let's walk through the code.

<code>AutomationEvent::SetValueCurveAtTime()</code> is a fn and takes three arguments: values, start, duration.

progress, computes the difference between the first value in start_time and [https://github.com/JHBalaji/media/blob/5a981dff94f4dfaf2ebf0f0e09f599fab9d1a13b/audio/src/param.rs#L384 time] and makes explicit conversion to f32.

<pre>
let time_diff = ((duration.0 as f32) - (start.0 as f32)) as f32;
let mut progress = ((((current_tick.0 as f32) - (start.0 as f32)) as f32) / time_diff) as f32;
</pre>

time_diff computes the total difference between duration and start_time, this is required when the values that needs to be computed goes beyond time_diff from start_time to set a constant value later.

<pre>
if progress < 0.0 {
progress = 0.0 as f32;
}
</pre>
Sets 0 if the progress from values array and present time being negative.

<pre>
let n = values.len() as f32;
let k = (((n - 1.) * progress) as f32).floor();
let next = k + 1. as f32;
let step = time_diff / (n - 1.);
</pre>
n, total length of values array. k, compute as k as in [[https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime here] and <code>next</code> is k+1, the delta or step is computed that would be added to generate the sound wave from the values array. Say, the frequency start at 44KHz and the <code>step</code> would be used to change the frequency of the curve.

<pre>
if next < n {
let time_k = (k * step) as f32;
let time_k_next = (next * step) as f32;
let v_k = values[k as usize];
let v_k_next = values[next as usize];
*value = v_k + (v_k_next - v_k) * (((current_tick.0 as f32) - time_k) / (time_k_next - time_k));
}
</pre>

The code block generates sequences of values from <code>values</code array and using <code>step</code> value we computed. And does that till it reaches the total length of the array.

<pre>
true
</pre>

[https://github.com/JHBalaji/media/blob/f32a71c51f05f44d1e8ca217fbaa4dfb0b55d11e/audio/src/param.rs#L435 Used] to loop the function call made from [https://github.com/JHBalaji/media/blob/f32a71c51f05f44d1e8ca217fbaa4dfb0b55d11e/audio/src/param.rs#L117 here].

<pre>
AutomationEvent::CancelAndHoldAtTime(..) => false,
AutomationEvent::CancelScheduledValues(..) | AutomationEvent::SetValue(..) => {
unreachable!("CancelScheduledValues/SetValue should never appear in the timeline")
}
</pre>

[https://github.com/JHBalaji/media/blob/5a981dff94f4dfaf2ebf0f0e09f599fab9d1a13b/audio/src/param.rs#L466 Code] instructs to cancel this event or end of this function and proceed to next instructions. In Audio API unless you explicitly say the block to end, the block would end and hardware device would be stopped from consumption.

==='''Examples'''===

Examples are unique aspect to Rust. This instructs that code to make use of the function we just wrote and generates a sound output. Refer our example [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs here] for the SetValueCurveAtTime()

And example is usually called by using cargo build in rust or cargo run.

[https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml Cargo.toml] dictates the naming and the file to be called as.

<pre>
cargo run --example set_value_curve
</pre>

The above code instructs to run the example [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs set_value_curve] to generate a sound output through our terminal.

==='''Compile'''===
[https://github.com/JHBalaji/media-1/blob/master/Cargo.lock Cargo.lock] instructs the dependencies that would be required to build this project.

[https://github.com/JHBalaji/media-1/blob/master/Cargo.toml Cargo.toml] calls all the folders or members to be complied. Usually the compilation would be at /media

Cargo.lock installs number of packages called crates that would be used by the project to access the modules. Each of the member in [https://github.com/JHBalaji/media-1/blob/master/Cargo.toml Cargo.toml] would have [https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml Cargo.toml] in them that would instruct [https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml#L6 dependencies] during the build. build-dependencies would be used if you have dependencies before the build.

[https://github.com/JHBalaji/media-1/blob/master/.cargo/config .cargo config] is similar to a dotenv file for Rust

==='''Test'''===

If you would like to test our build please look into [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Issues_and_Fixes Fixes] before proceeding. We would suggest MacOS or Linux 16.04 LTS for any development and testing. Virtual Machine or VirtualBox could be another option. We spent countless days in frustration with libraries and never ending apt installs so avoid Linux 18.04 for this build test.

Once <code>cargo build</code> is completed. Open /target/debug to view the examples that are generated.

[[File:Servo-initial-build.PNG]]

To run all examples after the cargo build is complete, use <code>cargo run --example <bin_name></code>

[[File:Examples.PNG]]

''''''Bold text''''''==='''Status of the Project'''===

The [https://github.com/JHBalaji/media-1 build] is ready and the Pull request is [https://github.com/servo/media/pull/230 merged] with Servo/media.

[[File:Pr.PNG]]
[[File:Example.jpg]]

=='''Normal build from Master Repo '''==
To build Servo in development mode. This is useful for development, but the resulting binary is very slow.
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html
</pre>
For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:
<pre>
./mach build --release
./mach run --release tests/html/about-mozilla.html
</pre>

==='''Compiling and Running the code'''===

====Mac OS====

Follow the instructions posted in this [https://github.com/JHBalaji/media-1 URL] and install GStreamer and Rust. After that follow the following lines for compiling on MacOS.

<pre>
1. brew install gstreamer gst-plugins-base gst-plugins-good gst-plugins-bad gst-plugins-ugly gst-libav gst-rtsp-server gst-editing-services

2. export PKG_CONFIG_PATH=/usr/local/Cellar/libffi/3.2.1/lib/pkgconfig/

3. rustup override set nightly
</pre>

After this run the following command to compile servo/media
<pre>
cargo build
</pre>

After compiling you will be able to test our example that is places in the examples folder.

====Windows and Linux====

Head over here to install [https://rustup.rs/ Rust ToolChain]. If you need any additional details we would suggest reading the [https://github.com/rust-lang/rustup.rs/blob/master/README.md ReadMe on Rust]. Some Servo media specific build issues are [https://github.com/servo/media/blob/master/README.md advised].

We would suggest Ubuntu 18.04 LTS with updates [https://github.com/servo/media/blob/master/README.md gstreamer package].

==='''Servo Media'''===
<b>Requirements</b>
In order to build this crate you need to install gstreamer which can be found here [https://github.com/sdroege/gstreamer-rs]
==='''Test Case'''===
After you run cargo build, you should be able to see the executable of the example inside the target/debug folder. Following is the test we have written file to check the functioning of the [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs SetValueCurveAtTIme].
<pre>
extern crate servo_media;
extern crate servo_media_auto;

use servo_media::audio::constant_source_node::ConstantSourceNodeOptions;
use servo_media::audio::gain_node::GainNodeOptions;
use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::param::{ParamType, UserAutomationEvent};
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();

//Initializing the values vector for SetValueCurve function
let mut values: Vec<f32> = Vec::new();
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(1.0);
values.push(1.0);
values.push(1.0);
values.push(1.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.5);
values.push(0.5);
values.push(0.5);
values.push(0.5);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
let start_time = 0.;
let end_time = 5.;
let n = values.len() as f32;
let value_next = values[(n - 1.) as usize];

let cs = context.create_node(
AudioNodeInit::ConstantSourceNode(ConstantSourceNodeOptions::default()),
Default::default(),
);

let mut gain_options = GainNodeOptions::default();
gain_options.gain = 0.0;
let gain = context.create_node(
AudioNodeInit::GainNode(gain_options.clone()),
Default::default(),
);

let osc = context.create_node(
AudioNodeInit::OscillatorNode(Default::default()),
Default::default(),
);

context.connect_ports(osc.output(0), gain.input(0));
context.connect_ports(cs.output(0), gain.param(ParamType::Gain));
context.connect_ports(gain.output(0), dest.input(0));

let _ = context.resume();
context.message_node(
osc,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
gain,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::SetValueCurveAtTime(values, start_time, end_time),
),

);

context.message_node(
osc,
AudioNodeMessage::SetParam(
ParamType::Frequency,
UserAutomationEvent::SetValueAtTime(value_next, end_time),
),
);

thread::sleep(time::Duration::from_millis(7000));
let _ = context.close();
}

fn main() {
ServoMedia::init::<servo_media_auto::Backend>();
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}

</pre>

In the above program, the values vector is passed into the SetValueCurveAtTime function to modulate the output. The output of the same can also be checked in the test-case video at the end of this page.
We can modify the values function as we want and observe the output. One more modification that we used was by using a sine curve. It is implemented as follows. Change the values vector in the set-value-curve function with the following:

<pre>
use std::f64::consts::PI;
let mut values: Vec<f32> = Vec::new();
let curvelength = 44100;
let mut i = 0;
while i < curvelength {
values.push(f32::sin((PI *i as f64/curvelength as f64) as f32));
i += 1;
}
</pre>

The output of this function can be seen in the test video attached to at the end of this page.
<references></references>

=='''Video of Testing Audio Output'''==
[https://drive.google.com/file/d/1jzEZwAZE-7mi6N5N_JcEE6ISM-wc9910/view?usp=sharing Test-1]
[https://drive.google.com/file/d/1UsPgyv29Zf29d6pEDgLpYX86DI6hzOqp/view?usp=sharing Test-2]

=='''Issues and Fixes'''==

Windows Specific during build

https://github.com/holochain/holochain-cmd/issues/29

https://stackoverflow.com/questions/53136717/errore0554-feature-may-not-be-used-on-the-stable-release-channel-couldnt

https://github.com/servo/servo/issues/21429

=='''Final Project'''==
If you are a peer reviewer please look into this [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#For_Peer_Reviewers_Note section]

The final [https://github.com/servo/servo/wiki/Implement-missing-WebAudio-automation-student-project project] rely on our previous [https://github.com/servo/media/pull/230 work]. The next part would focus on implementing the code [https://github.com/servo/media/issues/204 written] to be accessible by Servo browser [https://github.com/servo/servo/issues/22897 engine].
This would require interfacing the DOM [https://github.com/servo/servo/blob/master/components/script/dom/webidls/AudioParam.webidl file] reference the Param.rs [https://github.com/JHBalaji/media/blob/master/audio/src/param.rs#L259 setValueCurveAtTime] to be interfaced with [https://github.com/servo/servo/blob/master/components/script/dom/audioparam.rs AudioParam.rs]. The audioparam.rs would call the UserAutomationEvent in the param.rs setValueCurveAtTime.

Once the initial function call with arguments are updated, then the [https://github.com/servo/servo/blob/master/tests/wpt/README.md#updating-test-expectations testing] would be performed on ''./mach test-wpt tests/wpt/web-platform-tests/webaudio/the-audio-api/[https://github.com/servo/servo/tree/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audioparam-interface the-audioparam-interface]''. The [https://github.com/servo/servo/tree/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audioparam-interface the-audioparam-interface] is a folder of series of test cases that emulate the behavior of a function call, here the setValueCurveAtTime would need to be necessitated to mock the values as tested in previous [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Test_Case example] but would require to be performed over a .html webpage with DOM elements to test the complete functionality.

Servo webAudio tests are performed using [https://github.com/web-platform-tests/wpt/blob/master/resources/readme.md testharness.js].

<pre>
./mach create-wpt tests/wpt/path/to/new/test.html
</pre>

would create a test.html using the WPT template for javascript tests. Once the skeleton code is completed with necessary function call, them the WPT tests would be executed using

<pre>
./mach test-wpt tests/wpt/path/to/new/test.html
./mach test-wpt tests/wpt/path/to/new/reftest.html
</pre>

[https://web-platform-tests.org/writing-tests/reftests.html reftest] are used in Servo to test behavior related to rendering; that are related to interaction made up of several webpages [think passing or referencing values to another webpage] with assertions to test cases that are expected from the interaction.

<pre>
./mach create-wpt --reftest tests/wpt/path/to/new/reftest.html --reference tests/wpt/path/to/reference.html
</pre>

reference.html and reftest.html will be created using WPT reftest template. Once the reftest is run the upstream test would need to be updated using

<pre>
./mach update-wpt --sync
./mach test-wpt --log-raw=update.log
./mach update-wpt update.log
</pre>

When reftest fails to diagnose the behavior reftest raw log is required which is generate using the below command. The .log is fed into [https://github.com/Gankro/live-reftest-analyzer reftest analyzer] to find those failing test. The reftest is also capable of pixel level comparison of the test and reference screenshots (incase of rendering)
<pre>
./mach test-wpt --log-raw wpt.log
</pre>

==='''List of issues'''===
This project has listed 4 known issues under the WebAudio title, each solving a different issue.

1. [https://github.com/servo/media/issues/204 Implement ValueCurveAtTime automation]
This is the first issue, which has been addressed and successfully closed during the initial parts of this project. The steps followed are shown in the following diagram.

[[File:WebAudio-Issue1.png]]

2. [https://github.com/servo/servo/issues/22897 Implement AudioParam.setValueCurveAtTime]
This second issue builds up on the first one, and focuses on implementing yet another method called [https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime setValueCurveAtTime]. The next focus of this project will likely be to work on this issue.

[[File:WebAudio-Issue2.png]]

=='''For Peer Reviewers Note'''==

''Common peer review question clarifications are below,''

"Does the design incorporate all of the functionality required?" or "Check the changes that the authors proposed. Rate the overall quality of the explanations. Give comments if you find the explanation hard to follow or inadequate."

:The [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Final_Project section] clarifies your question.

"Check the designs proposed by the authors."

:There is no explicit design is involved as going by [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Final_Project proposal], there are three files to be added with new functionality in addition to test cases and files.

"Check the use of visual aids"
:Visual aids not applicable

"Give comments on the completeness of the design doc."
: Let us know what you think about them!

File:WebAudio-Issue2.png

2019-04-13T02:48:18Z

Amadhus2:

CSC/ECE 517 Spring 2019 - Project M1901 Implement missing WebAudio automation support

2019-04-13T02:46:28Z

Amadhus2: /* List of issues */

=='''Introduction'''==

==='''Servo'''===
Servo is an developmental browser engine designed to make use of the memory safety properties and concurrency features of the Rust programming language. The project was initiated by Mozilla Research with effort from Samsung to port it to Android and ARM processors

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language which focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and provides high performance. Rust is brain child of Mozilla and simple as this
<pre>
fn main() {
println!("Hello World");
}
</pre>

More information about Rust can be found [https://www.rust-lang.org/en-US/ here] and here is why we love [https://thenewstack.io/safer-future-rust/ Rust], [https://insights.stackoverflow.com/survey/2018/#most-loved-dreaded-and-wanted SF Loved]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. An audio routing graph has been created by linking together the Basic Audio operations performed with audio nodes. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context<br>
2. Inside the context, create sources — such as <audio>, oscillator, stream<br>
3. Create effects nodes, such as reverb, biquad filter, panner, compressor<br>
4. Choose final destination of audio, for example your system speakers<br>
5. Connect the sources up to the effects, and the effects to the destination.<br>

==='''Problem Statement: Web Audio Automation'''===

AudioParam is used to control the AudioNode functioning, say, Volume (a Gain parameter). These values can either be scheduled to be changed at precise times or set to particular value immediately following an event or at an event. The schedule to change when used with AudioContext.currentTime can helps in volume fades, filter sweeps etc. This would work aims to implement [https://github.com/servo/media/issues/204 SetValueCurveAtTime()].

[https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime setValueCurveAtTime()] is one such method in AudioParam that takes an array as input and schedules a change. The array is usually a curve in wedAudio and achieved my linear interpolation between the values from the floating-point values array and for the duration, d from the startTime, s

<code>v(t) = values[N * (t - s) / d]</code>, where N is the length of the values in the array.
And after the end of the curve time interval <code>(t >= s + d)</code>, the value will remain constant at the final curve value. This would persist to happen until the next automation event.

One of the application of setValueCurveAtTime() it to create tremolo effect. Suppose linear nor an exponential curve satisfy the needs then user can create curve based on values to setValueCurveAtTime() with an array of timing values. Its a much loved approached against multiple calls to setValueAtTime().

==='''Build'''===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.

=='''Implementation of the Project'''==

==='''Code'''===
The code is implemented as in [https://github.com/JHBalaji/media/blob/master/audio/src/param.rs#L418-L436 here]. The code below is the main code snippet for the SetValueCurveAtTime function. The values and function behavior is define in the param.rs file.

<pre>

AutomationEvent::SetValueCurveAtTime(ref values, start, duration) => {
let time_diff = ((duration.0 as f32) - (start.0 as f32)) as f32;
let mut progress = ((((current_tick.0 as f32) - (start.0 as f32)) as f32) / time_diff) as f32;
if progress < 0.0 {
progress = 0.0 as f32;
}
let n = values.len() as f32;
let k = (((n - 1.) * progress) as f32).floor();
let next = k + 1. as f32;
let step = time_diff / (n - 1.);
if next < n {
let time_k = (k * step) as f32;
let time_k_next = (next * step) as f32;
let v_k = values[k as usize];
let v_k_next = values[next as usize];
*value = v_k + (v_k_next - v_k) * (((current_tick.0 as f32) - time_k) / (time_k_next - time_k));
}
true
}
AutomationEvent::CancelAndHoldAtTime(..) => false,
AutomationEvent::CancelScheduledValues(..) | AutomationEvent::SetValue(..) => {
unreachable!("CancelScheduledValues/SetValue should never appear in the timeline")
}

</pre>

Let's walk through the code.

<code>AutomationEvent::SetValueCurveAtTime()</code> is a fn and takes three arguments: values, start, duration.

progress, computes the difference between the first value in start_time and [https://github.com/JHBalaji/media/blob/5a981dff94f4dfaf2ebf0f0e09f599fab9d1a13b/audio/src/param.rs#L384 time] and makes explicit conversion to f32.

<pre>
let time_diff = ((duration.0 as f32) - (start.0 as f32)) as f32;
let mut progress = ((((current_tick.0 as f32) - (start.0 as f32)) as f32) / time_diff) as f32;
</pre>

time_diff computes the total difference between duration and start_time, this is required when the values that needs to be computed goes beyond time_diff from start_time to set a constant value later.

<pre>
if progress < 0.0 {
progress = 0.0 as f32;
}
</pre>
Sets 0 if the progress from values array and present time being negative.

<pre>
let n = values.len() as f32;
let k = (((n - 1.) * progress) as f32).floor();
let next = k + 1. as f32;
let step = time_diff / (n - 1.);
</pre>
n, total length of values array. k, compute as k as in [[https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime here] and <code>next</code> is k+1, the delta or step is computed that would be added to generate the sound wave from the values array. Say, the frequency start at 44KHz and the <code>step</code> would be used to change the frequency of the curve.

<pre>
if next < n {
let time_k = (k * step) as f32;
let time_k_next = (next * step) as f32;
let v_k = values[k as usize];
let v_k_next = values[next as usize];
*value = v_k + (v_k_next - v_k) * (((current_tick.0 as f32) - time_k) / (time_k_next - time_k));
}
</pre>

The code block generates sequences of values from <code>values</code array and using <code>step</code> value we computed. And does that till it reaches the total length of the array.

<pre>
true
</pre>

[https://github.com/JHBalaji/media/blob/f32a71c51f05f44d1e8ca217fbaa4dfb0b55d11e/audio/src/param.rs#L435 Used] to loop the function call made from [https://github.com/JHBalaji/media/blob/f32a71c51f05f44d1e8ca217fbaa4dfb0b55d11e/audio/src/param.rs#L117 here].

<pre>
AutomationEvent::CancelAndHoldAtTime(..) => false,
AutomationEvent::CancelScheduledValues(..) | AutomationEvent::SetValue(..) => {
unreachable!("CancelScheduledValues/SetValue should never appear in the timeline")
}
</pre>

[https://github.com/JHBalaji/media/blob/5a981dff94f4dfaf2ebf0f0e09f599fab9d1a13b/audio/src/param.rs#L466 Code] instructs to cancel this event or end of this function and proceed to next instructions. In Audio API unless you explicitly say the block to end, the block would end and hardware device would be stopped from consumption.

==='''Examples'''===

Examples are unique aspect to Rust. This instructs that code to make use of the function we just wrote and generates a sound output. Refer our example [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs here] for the SetValueCurveAtTime()

And example is usually called by using cargo build in rust or cargo run.

[https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml Cargo.toml] dictates the naming and the file to be called as.

<pre>
cargo run --example set_value_curve
</pre>

The above code instructs to run the example [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs set_value_curve] to generate a sound output through our terminal.

==='''Compile'''===
[https://github.com/JHBalaji/media-1/blob/master/Cargo.lock Cargo.lock] instructs the dependencies that would be required to build this project.

[https://github.com/JHBalaji/media-1/blob/master/Cargo.toml Cargo.toml] calls all the folders or members to be complied. Usually the compilation would be at /media

Cargo.lock installs number of packages called crates that would be used by the project to access the modules. Each of the member in [https://github.com/JHBalaji/media-1/blob/master/Cargo.toml Cargo.toml] would have [https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml Cargo.toml] in them that would instruct [https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml#L6 dependencies] during the build. build-dependencies would be used if you have dependencies before the build.

[https://github.com/JHBalaji/media-1/blob/master/.cargo/config .cargo config] is similar to a dotenv file for Rust

==='''Test'''===

If you would like to test our build please look into [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Issues_and_Fixes Fixes] before proceeding. We would suggest MacOS or Linux 16.04 LTS for any development and testing. Virtual Machine or VirtualBox could be another option. We spent countless days in frustration with libraries and never ending apt installs so avoid Linux 18.04 for this build test.

Once <code>cargo build</code> is completed. Open /target/debug to view the examples that are generated.

[[File:Servo-initial-build.PNG]]

To run all examples after the cargo build is complete, use <code>cargo run --example <bin_name></code>

[[File:Examples.PNG]]

==='''Status of the Project'''===

The [https://github.com/JHBalaji/media-1 build] is ready and the Pull request is [https://github.com/servo/media/pull/230 merged] with Servo/media.

[[File:Pr.PNG]]

=='''Normal build from Master Repo '''==
To build Servo in development mode. This is useful for development, but the resulting binary is very slow.
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html
</pre>
For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:
<pre>
./mach build --release
./mach run --release tests/html/about-mozilla.html
</pre>

==='''Compiling and Running the code'''===

====Mac OS====

Follow the instructions posted in this [https://github.com/JHBalaji/media-1 URL] and install GStreamer and Rust. After that follow the following lines for compiling on MacOS.

<pre>
1. brew install gstreamer gst-plugins-base gst-plugins-good gst-plugins-bad gst-plugins-ugly gst-libav gst-rtsp-server gst-editing-services

2. export PKG_CONFIG_PATH=/usr/local/Cellar/libffi/3.2.1/lib/pkgconfig/

3. rustup override set nightly
</pre>

After this run the following command to compile servo/media
<pre>
cargo build
</pre>

After compiling you will be able to test our example that is places in the examples folder.

====Windows and Linux====

Head over here to install [https://rustup.rs/ Rust ToolChain]. If you need any additional details we would suggest reading the [https://github.com/rust-lang/rustup.rs/blob/master/README.md ReadMe on Rust]. Some Servo media specific build issues are [https://github.com/servo/media/blob/master/README.md advised].

We would suggest Ubuntu 18.04 LTS with updates [https://github.com/servo/media/blob/master/README.md gstreamer package].

==='''Servo Media'''===
<b>Requirements</b>
In order to build this crate you need to install gstreamer which can be found here [https://github.com/sdroege/gstreamer-rs]
==='''Test Case'''===
After you run cargo build, you should be able to see the executable of the example inside the target/debug folder. Following is the test we have written file to check the functioning of the [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs SetValueCurveAtTIme].
<pre>
extern crate servo_media;
extern crate servo_media_auto;

use servo_media::audio::constant_source_node::ConstantSourceNodeOptions;
use servo_media::audio::gain_node::GainNodeOptions;
use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::param::{ParamType, UserAutomationEvent};
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();

//Initializing the values vector for SetValueCurve function
let mut values: Vec<f32> = Vec::new();
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(1.0);
values.push(1.0);
values.push(1.0);
values.push(1.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.5);
values.push(0.5);
values.push(0.5);
values.push(0.5);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
let start_time = 0.;
let end_time = 5.;
let n = values.len() as f32;
let value_next = values[(n - 1.) as usize];

let cs = context.create_node(
AudioNodeInit::ConstantSourceNode(ConstantSourceNodeOptions::default()),
Default::default(),
);

let mut gain_options = GainNodeOptions::default();
gain_options.gain = 0.0;
let gain = context.create_node(
AudioNodeInit::GainNode(gain_options.clone()),
Default::default(),
);

let osc = context.create_node(
AudioNodeInit::OscillatorNode(Default::default()),
Default::default(),
);

context.connect_ports(osc.output(0), gain.input(0));
context.connect_ports(cs.output(0), gain.param(ParamType::Gain));
context.connect_ports(gain.output(0), dest.input(0));

let _ = context.resume();
context.message_node(
osc,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
gain,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::SetValueCurveAtTime(values, start_time, end_time),
),

);

context.message_node(
osc,
AudioNodeMessage::SetParam(
ParamType::Frequency,
UserAutomationEvent::SetValueAtTime(value_next, end_time),
),
);

thread::sleep(time::Duration::from_millis(7000));
let _ = context.close();
}

fn main() {
ServoMedia::init::<servo_media_auto::Backend>();
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}

</pre>

In the above program, the values vector is passed into the SetValueCurveAtTime function to modulate the output. The output of the same can also be checked in the test-case video at the end of this page.
We can modify the values function as we want and observe the output. One more modification that we used was by using a sine curve. It is implemented as follows. Change the values vector in the set-value-curve function with the following:

<pre>
use std::f64::consts::PI;
let mut values: Vec<f32> = Vec::new();
let curvelength = 44100;
let mut i = 0;
while i < curvelength {
values.push(f32::sin((PI *i as f64/curvelength as f64) as f32));
i += 1;
}
</pre>

The output of this function can be seen in the test video attached to at the end of this page.
<references></references>

=='''Video of Testing Audio Output'''==
[https://drive.google.com/file/d/1jzEZwAZE-7mi6N5N_JcEE6ISM-wc9910/view?usp=sharing Test-1]
[https://drive.google.com/file/d/1UsPgyv29Zf29d6pEDgLpYX86DI6hzOqp/view?usp=sharing Test-2]

=='''Issues and Fixes'''==

Windows Specific during build

https://github.com/holochain/holochain-cmd/issues/29

https://stackoverflow.com/questions/53136717/errore0554-feature-may-not-be-used-on-the-stable-release-channel-couldnt

https://github.com/servo/servo/issues/21429

=='''Final Project'''==
If you are a peer reviewer please look into this [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#For_Peer_Reviewers_Note section]

The final [https://github.com/servo/servo/wiki/Implement-missing-WebAudio-automation-student-project project] rely on our previous [https://github.com/servo/media/pull/230 work]. The next part would focus on implementing the code [https://github.com/servo/media/issues/204 written] to be accessible by Servo browser [https://github.com/servo/servo/issues/22897 engine].
This would require interfacing the DOM [https://github.com/servo/servo/blob/master/components/script/dom/webidls/AudioParam.webidl file] reference the Param.rs [https://github.com/JHBalaji/media/blob/master/audio/src/param.rs#L259 setValueCurveAtTime] to be interfaced with [https://github.com/servo/servo/blob/master/components/script/dom/audioparam.rs AudioParam.rs]. The audioparam.rs would call the UserAutomationEvent in the param.rs setValueCurveAtTime.

Once the initial function call with arguments are updated, then the [https://github.com/servo/servo/blob/master/tests/wpt/README.md#updating-test-expectations testing] would be performed on ''./mach test-wpt tests/wpt/web-platform-tests/webaudio/the-audio-api/[https://github.com/servo/servo/tree/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audioparam-interface the-audioparam-interface]''. The [https://github.com/servo/servo/tree/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audioparam-interface the-audioparam-interface] is a folder of series of test cases that emulate the behavior of a function call, here the setValueCurveAtTime would need to be necessitated to mock the values as tested in previous [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Test_Case example] but would require to be performed over a .html webpage with DOM elements to test the complete functionality.

Servo webAudio tests are performed using [https://github.com/web-platform-tests/wpt/blob/master/resources/readme.md testharness.js].

<pre>
./mach create-wpt tests/wpt/path/to/new/test.html
</pre>

would create a test.html using the WPT template for javascript tests. Once the skeleton code is completed with necessary function call, them the WPT tests would be executed using

<pre>
./mach test-wpt tests/wpt/path/to/new/test.html
./mach test-wpt tests/wpt/path/to/new/reftest.html
</pre>

[https://web-platform-tests.org/writing-tests/reftests.html reftest] are used in Servo to test behavior related to rendering; that are related to interaction made up of several webpages [think passing or referencing values to another webpage] with assertions to test cases that are expected from the interaction.

<pre>
./mach create-wpt --reftest tests/wpt/path/to/new/reftest.html --reference tests/wpt/path/to/reference.html
</pre>

reference.html and reftest.html will be created using WPT reftest template. Once the reftest is run the upstream test would need to be updated using

<pre>
./mach update-wpt --sync
./mach test-wpt --log-raw=update.log
./mach update-wpt update.log
</pre>

When reftest fails to diagnose the behavior reftest raw log is required which is generate using the below command. The .log is fed into [https://github.com/Gankro/live-reftest-analyzer reftest analyzer] to find those failing test. The reftest is also capable of pixel level comparison of the test and reference screenshots (incase of rendering)
<pre>
./mach test-wpt --log-raw wpt.log
</pre>

==='''List of issues'''===
This project has listed 4 known issues under the WebAudio title, each solving a different issue.

1. [https://github.com/servo/media/issues/204 Implement ValueCurveAtTime automation]
This is the first issue, which has been addressed and successfully closed during the initial parts of this project. The steps followed are shown in the following diagram.
[[File:WebAudio-Issue1.png]]

=='''For Peer Reviewers Note'''==

''Common peer review question clarifications are below,''

"Does the design incorporate all of the functionality required?" or "Check the changes that the authors proposed. Rate the overall quality of the explanations. Give comments if you find the explanation hard to follow or inadequate."

:The [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Final_Project section] clarifies your question.

"Check the designs proposed by the authors."

:There is no explicit design is involved as going by [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Final_Project proposal], there are three files to be added with new functionality in addition to test cases and files.

"Check the use of visual aids"
:Visual aids not applicable

"Give comments on the completeness of the design doc."
: Let us know what you think about them!

File:WebAudio-Issue1.png

2019-04-13T02:45:14Z

Amadhus2:

CSC/ECE 517 Spring 2019 - Project M1901 Implement missing WebAudio automation support

2019-04-13T02:38:14Z

Amadhus2: /* Final Project */

=='''Introduction'''==

==='''Servo'''===
Servo is an developmental browser engine designed to make use of the memory safety properties and concurrency features of the Rust programming language. The project was initiated by Mozilla Research with effort from Samsung to port it to Android and ARM processors

More information about Servo is available [https://servo.org here]

==='''Rust'''===
Rust is a systems programming language which focuses on memory safety and concurrency. It is similar to C++ but ensures memory safety and provides high performance. Rust is brain child of Mozilla and simple as this
<pre>
fn main() {
println!("Hello World");
}
</pre>

More information about Rust can be found [https://www.rust-lang.org/en-US/ here] and here is why we love [https://thenewstack.io/safer-future-rust/ Rust], [https://insights.stackoverflow.com/survey/2018/#most-loved-dreaded-and-wanted SF Loved]

==='''Web Audio API'''===
The Web Audio API involves handling audio operations inside an audio context, and has been designed to allow modular routing. An audio routing graph has been created by linking together the Basic Audio operations performed with audio nodes. Several sources — with different types of channel layout — are supported even within a single context. This modular design provides the flexibility to create complex audio functions with dynamic effects.

Audio nodes are linked into chains and simple webs by their inputs and outputs. They typically start with one or more sources. Sources provide arrays of sound intensities (samples) at very small timeslices, often tens of thousands of them per second. These could be either computed mathematically (such as OscillatorNode), or they can be recordings from sound/video files (like AudioBufferSourceNode and MediaElementAudioSourceNode) and audio streams (MediaStreamAudioSourceNode). In fact, sound files are just recordings of sound intensities themselves, which come in from microphones or electric instruments, and get mixed down into a single, complicated wave.

Outputs of these nodes could be linked to inputs of others, which mix or modify these streams of sound samples into different streams. A common modification is multiplying the samples by a value to make them louder or quieter (as is the case with GainNode). Once the sound has been sufficiently processed for the intended effect, it can be linked to the input of a destination (AudioContext.destination), which sends the sound to the speakers or headphones. This last connection is only necessary if the user is supposed to hear the audio.

A simple, typical workflow for web audio would look something like this:

1. Create audio context<br>
2. Inside the context, create sources — such as <audio>, oscillator, stream<br>
3. Create effects nodes, such as reverb, biquad filter, panner, compressor<br>
4. Choose final destination of audio, for example your system speakers<br>
5. Connect the sources up to the effects, and the effects to the destination.<br>

==='''Problem Statement: Web Audio Automation'''===

AudioParam is used to control the AudioNode functioning, say, Volume (a Gain parameter). These values can either be scheduled to be changed at precise times or set to particular value immediately following an event or at an event. The schedule to change when used with AudioContext.currentTime can helps in volume fades, filter sweeps etc. This would work aims to implement [https://github.com/servo/media/issues/204 SetValueCurveAtTime()].

[https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime setValueCurveAtTime()] is one such method in AudioParam that takes an array as input and schedules a change. The array is usually a curve in wedAudio and achieved my linear interpolation between the values from the floating-point values array and for the duration, d from the startTime, s

<code>v(t) = values[N * (t - s) / d]</code>, where N is the length of the values in the array.
And after the end of the curve time interval <code>(t >= s + d)</code>, the value will remain constant at the final curve value. This would persist to happen until the next automation event.

One of the application of setValueCurveAtTime() it to create tremolo effect. Suppose linear nor an exponential curve satisfy the needs then user can create curve based on values to setValueCurveAtTime() with an array of timing values. Its a much loved approached against multiple calls to setValueAtTime().

==='''Build'''===
Servo is built with Cargo, the Rust package manager. We also use Mozilla's Mach tools to orchestrate the build and other tasks.

=='''Implementation of the Project'''==

==='''Code'''===
The code is implemented as in [https://github.com/JHBalaji/media/blob/master/audio/src/param.rs#L418-L436 here]. The code below is the main code snippet for the SetValueCurveAtTime function. The values and function behavior is define in the param.rs file.

<pre>

AutomationEvent::SetValueCurveAtTime(ref values, start, duration) => {
let time_diff = ((duration.0 as f32) - (start.0 as f32)) as f32;
let mut progress = ((((current_tick.0 as f32) - (start.0 as f32)) as f32) / time_diff) as f32;
if progress < 0.0 {
progress = 0.0 as f32;
}
let n = values.len() as f32;
let k = (((n - 1.) * progress) as f32).floor();
let next = k + 1. as f32;
let step = time_diff / (n - 1.);
if next < n {
let time_k = (k * step) as f32;
let time_k_next = (next * step) as f32;
let v_k = values[k as usize];
let v_k_next = values[next as usize];
*value = v_k + (v_k_next - v_k) * (((current_tick.0 as f32) - time_k) / (time_k_next - time_k));
}
true
}
AutomationEvent::CancelAndHoldAtTime(..) => false,
AutomationEvent::CancelScheduledValues(..) | AutomationEvent::SetValue(..) => {
unreachable!("CancelScheduledValues/SetValue should never appear in the timeline")
}

</pre>

Let's walk through the code.

<code>AutomationEvent::SetValueCurveAtTime()</code> is a fn and takes three arguments: values, start, duration.

progress, computes the difference between the first value in start_time and [https://github.com/JHBalaji/media/blob/5a981dff94f4dfaf2ebf0f0e09f599fab9d1a13b/audio/src/param.rs#L384 time] and makes explicit conversion to f32.

<pre>
let time_diff = ((duration.0 as f32) - (start.0 as f32)) as f32;
let mut progress = ((((current_tick.0 as f32) - (start.0 as f32)) as f32) / time_diff) as f32;
</pre>

time_diff computes the total difference between duration and start_time, this is required when the values that needs to be computed goes beyond time_diff from start_time to set a constant value later.

<pre>
if progress < 0.0 {
progress = 0.0 as f32;
}
</pre>
Sets 0 if the progress from values array and present time being negative.

<pre>
let n = values.len() as f32;
let k = (((n - 1.) * progress) as f32).floor();
let next = k + 1. as f32;
let step = time_diff / (n - 1.);
</pre>
n, total length of values array. k, compute as k as in [[https://webaudio.github.io/web-audio-api/#dom-audioparam-setvaluecurveattime here] and <code>next</code> is k+1, the delta or step is computed that would be added to generate the sound wave from the values array. Say, the frequency start at 44KHz and the <code>step</code> would be used to change the frequency of the curve.

<pre>
if next < n {
let time_k = (k * step) as f32;
let time_k_next = (next * step) as f32;
let v_k = values[k as usize];
let v_k_next = values[next as usize];
*value = v_k + (v_k_next - v_k) * (((current_tick.0 as f32) - time_k) / (time_k_next - time_k));
}
</pre>

The code block generates sequences of values from <code>values</code array and using <code>step</code> value we computed. And does that till it reaches the total length of the array.

<pre>
true
</pre>

[https://github.com/JHBalaji/media/blob/f32a71c51f05f44d1e8ca217fbaa4dfb0b55d11e/audio/src/param.rs#L435 Used] to loop the function call made from [https://github.com/JHBalaji/media/blob/f32a71c51f05f44d1e8ca217fbaa4dfb0b55d11e/audio/src/param.rs#L117 here].

<pre>
AutomationEvent::CancelAndHoldAtTime(..) => false,
AutomationEvent::CancelScheduledValues(..) | AutomationEvent::SetValue(..) => {
unreachable!("CancelScheduledValues/SetValue should never appear in the timeline")
}
</pre>

[https://github.com/JHBalaji/media/blob/5a981dff94f4dfaf2ebf0f0e09f599fab9d1a13b/audio/src/param.rs#L466 Code] instructs to cancel this event or end of this function and proceed to next instructions. In Audio API unless you explicitly say the block to end, the block would end and hardware device would be stopped from consumption.

==='''Examples'''===

Examples are unique aspect to Rust. This instructs that code to make use of the function we just wrote and generates a sound output. Refer our example [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs here] for the SetValueCurveAtTime()

And example is usually called by using cargo build in rust or cargo run.

[https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml Cargo.toml] dictates the naming and the file to be called as.

<pre>
cargo run --example set_value_curve
</pre>

The above code instructs to run the example [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs set_value_curve] to generate a sound output through our terminal.

==='''Compile'''===
[https://github.com/JHBalaji/media-1/blob/master/Cargo.lock Cargo.lock] instructs the dependencies that would be required to build this project.

[https://github.com/JHBalaji/media-1/blob/master/Cargo.toml Cargo.toml] calls all the folders or members to be complied. Usually the compilation would be at /media

Cargo.lock installs number of packages called crates that would be used by the project to access the modules. Each of the member in [https://github.com/JHBalaji/media-1/blob/master/Cargo.toml Cargo.toml] would have [https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml Cargo.toml] in them that would instruct [https://github.com/JHBalaji/media-1/blob/master/examples/Cargo.toml#L6 dependencies] during the build. build-dependencies would be used if you have dependencies before the build.

[https://github.com/JHBalaji/media-1/blob/master/.cargo/config .cargo config] is similar to a dotenv file for Rust

==='''Test'''===

If you would like to test our build please look into [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Issues_and_Fixes Fixes] before proceeding. We would suggest MacOS or Linux 16.04 LTS for any development and testing. Virtual Machine or VirtualBox could be another option. We spent countless days in frustration with libraries and never ending apt installs so avoid Linux 18.04 for this build test.

Once <code>cargo build</code> is completed. Open /target/debug to view the examples that are generated.

[[File:Servo-initial-build.PNG]]

To run all examples after the cargo build is complete, use <code>cargo run --example <bin_name></code>

[[File:Examples.PNG]]

==='''Status of the Project'''===

The [https://github.com/JHBalaji/media-1 build] is ready and the Pull request is [https://github.com/servo/media/pull/230 merged] with Servo/media.

[[File:Pr.PNG]]

=='''Normal build from Master Repo '''==
To build Servo in development mode. This is useful for development, but the resulting binary is very slow.
<pre>
git clone https://github.com/servo/servo
cd servo
./mach build --dev
./mach run tests/html/about-mozilla.html
</pre>
For benchmarking, performance testing, or real-world use, add the --release flag to create an optimized build:
<pre>
./mach build --release
./mach run --release tests/html/about-mozilla.html
</pre>

==='''Compiling and Running the code'''===

====Mac OS====

Follow the instructions posted in this [https://github.com/JHBalaji/media-1 URL] and install GStreamer and Rust. After that follow the following lines for compiling on MacOS.

<pre>
1. brew install gstreamer gst-plugins-base gst-plugins-good gst-plugins-bad gst-plugins-ugly gst-libav gst-rtsp-server gst-editing-services

2. export PKG_CONFIG_PATH=/usr/local/Cellar/libffi/3.2.1/lib/pkgconfig/

3. rustup override set nightly
</pre>

After this run the following command to compile servo/media
<pre>
cargo build
</pre>

After compiling you will be able to test our example that is places in the examples folder.

====Windows and Linux====

Head over here to install [https://rustup.rs/ Rust ToolChain]. If you need any additional details we would suggest reading the [https://github.com/rust-lang/rustup.rs/blob/master/README.md ReadMe on Rust]. Some Servo media specific build issues are [https://github.com/servo/media/blob/master/README.md advised].

We would suggest Ubuntu 18.04 LTS with updates [https://github.com/servo/media/blob/master/README.md gstreamer package].

==='''Servo Media'''===
<b>Requirements</b>
In order to build this crate you need to install gstreamer which can be found here [https://github.com/sdroege/gstreamer-rs]
==='''Test Case'''===
After you run cargo build, you should be able to see the executable of the example inside the target/debug folder. Following is the test we have written file to check the functioning of the [https://github.com/JHBalaji/media-1/blob/master/examples/set_value_curve.rs SetValueCurveAtTIme].
<pre>
extern crate servo_media;
extern crate servo_media_auto;

use servo_media::audio::constant_source_node::ConstantSourceNodeOptions;
use servo_media::audio::gain_node::GainNodeOptions;
use servo_media::audio::node::{AudioNodeInit, AudioNodeMessage, AudioScheduledSourceNodeMessage};
use servo_media::audio::param::{ParamType, UserAutomationEvent};
use servo_media::ServoMedia;
use std::sync::Arc;
use std::{thread, time};

fn run_example(servo_media: Arc<ServoMedia>) {
let context = servo_media.create_audio_context(Default::default());
let dest = context.dest_node();

//Initializing the values vector for SetValueCurve function
let mut values: Vec<f32> = Vec::new();
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(1.0);
values.push(1.0);
values.push(1.0);
values.push(1.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.5);
values.push(0.5);
values.push(0.5);
values.push(0.5);
values.push(0.0);
values.push(0.0);
values.push(0.0);
values.push(0.0);
let start_time = 0.;
let end_time = 5.;
let n = values.len() as f32;
let value_next = values[(n - 1.) as usize];

let cs = context.create_node(
AudioNodeInit::ConstantSourceNode(ConstantSourceNodeOptions::default()),
Default::default(),
);

let mut gain_options = GainNodeOptions::default();
gain_options.gain = 0.0;
let gain = context.create_node(
AudioNodeInit::GainNode(gain_options.clone()),
Default::default(),
);

let osc = context.create_node(
AudioNodeInit::OscillatorNode(Default::default()),
Default::default(),
);

context.connect_ports(osc.output(0), gain.input(0));
context.connect_ports(cs.output(0), gain.param(ParamType::Gain));
context.connect_ports(gain.output(0), dest.input(0));

let _ = context.resume();
context.message_node(
osc,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
gain,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::AudioScheduledSourceNode(AudioScheduledSourceNodeMessage::Start(0.)),
);

context.message_node(
cs,
AudioNodeMessage::SetParam(
ParamType::Offset,
UserAutomationEvent::SetValueCurveAtTime(values, start_time, end_time),
),

);

context.message_node(
osc,
AudioNodeMessage::SetParam(
ParamType::Frequency,
UserAutomationEvent::SetValueAtTime(value_next, end_time),
),
);

thread::sleep(time::Duration::from_millis(7000));
let _ = context.close();
}

fn main() {
ServoMedia::init::<servo_media_auto::Backend>();
if let Ok(servo_media) = ServoMedia::get() {
run_example(servo_media);
} else {
unreachable!();
}
}

</pre>

In the above program, the values vector is passed into the SetValueCurveAtTime function to modulate the output. The output of the same can also be checked in the test-case video at the end of this page.
We can modify the values function as we want and observe the output. One more modification that we used was by using a sine curve. It is implemented as follows. Change the values vector in the set-value-curve function with the following:

<pre>
use std::f64::consts::PI;
let mut values: Vec<f32> = Vec::new();
let curvelength = 44100;
let mut i = 0;
while i < curvelength {
values.push(f32::sin((PI *i as f64/curvelength as f64) as f32));
i += 1;
}
</pre>

The output of this function can be seen in the test video attached to at the end of this page.
<references></references>

=='''Video of Testing Audio Output'''==
[https://drive.google.com/file/d/1jzEZwAZE-7mi6N5N_JcEE6ISM-wc9910/view?usp=sharing Test-1]
[https://drive.google.com/file/d/1UsPgyv29Zf29d6pEDgLpYX86DI6hzOqp/view?usp=sharing Test-2]

=='''Issues and Fixes'''==

Windows Specific during build

https://github.com/holochain/holochain-cmd/issues/29

https://stackoverflow.com/questions/53136717/errore0554-feature-may-not-be-used-on-the-stable-release-channel-couldnt

https://github.com/servo/servo/issues/21429

=='''Final Project'''==
If you are a peer reviewer please look into this [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#For_Peer_Reviewers_Note section]

The final [https://github.com/servo/servo/wiki/Implement-missing-WebAudio-automation-student-project project] rely on our previous [https://github.com/servo/media/pull/230 work]. The next part would focus on implementing the code [https://github.com/servo/media/issues/204 written] to be accessible by Servo browser [https://github.com/servo/servo/issues/22897 engine].
This would require interfacing the DOM [https://github.com/servo/servo/blob/master/components/script/dom/webidls/AudioParam.webidl file] reference the Param.rs [https://github.com/JHBalaji/media/blob/master/audio/src/param.rs#L259 setValueCurveAtTime] to be interfaced with [https://github.com/servo/servo/blob/master/components/script/dom/audioparam.rs AudioParam.rs]. The audioparam.rs would call the UserAutomationEvent in the param.rs setValueCurveAtTime.

Once the initial function call with arguments are updated, then the [https://github.com/servo/servo/blob/master/tests/wpt/README.md#updating-test-expectations testing] would be performed on ''./mach test-wpt tests/wpt/web-platform-tests/webaudio/the-audio-api/[https://github.com/servo/servo/tree/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audioparam-interface the-audioparam-interface]''. The [https://github.com/servo/servo/tree/master/tests/wpt/web-platform-tests/webaudio/the-audio-api/the-audioparam-interface the-audioparam-interface] is a folder of series of test cases that emulate the behavior of a function call, here the setValueCurveAtTime would need to be necessitated to mock the values as tested in previous [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Test_Case example] but would require to be performed over a .html webpage with DOM elements to test the complete functionality.

Servo webAudio tests are performed using [https://github.com/web-platform-tests/wpt/blob/master/resources/readme.md testharness.js].

<pre>
./mach create-wpt tests/wpt/path/to/new/test.html
</pre>

would create a test.html using the WPT template for javascript tests. Once the skeleton code is completed with necessary function call, them the WPT tests would be executed using

<pre>
./mach test-wpt tests/wpt/path/to/new/test.html
./mach test-wpt tests/wpt/path/to/new/reftest.html
</pre>

[https://web-platform-tests.org/writing-tests/reftests.html reftest] are used in Servo to test behavior related to rendering; that are related to interaction made up of several webpages [think passing or referencing values to another webpage] with assertions to test cases that are expected from the interaction.

<pre>
./mach create-wpt --reftest tests/wpt/path/to/new/reftest.html --reference tests/wpt/path/to/reference.html
</pre>

reference.html and reftest.html will be created using WPT reftest template. Once the reftest is run the upstream test would need to be updated using

<pre>
./mach update-wpt --sync
./mach test-wpt --log-raw=update.log
./mach update-wpt update.log
</pre>

When reftest fails to diagnose the behavior reftest raw log is required which is generate using the below command. The .log is fed into [https://github.com/Gankro/live-reftest-analyzer reftest analyzer] to find those failing test. The reftest is also capable of pixel level comparison of the test and reference screenshots (incase of rendering)
<pre>
./mach test-wpt --log-raw wpt.log
</pre>

==='''List of issues'''===
This project has listed 4 known issues under the WebAudio title, each solving a different issue.

1. [https://github.com/servo/media/issues/204 Implement ValueCurveAtTime automation]

=='''For Peer Reviewers Note'''==

''Common peer review question clarifications are below,''

"Does the design incorporate all of the functionality required?" or "Check the changes that the authors proposed. Rate the overall quality of the explanations. Give comments if you find the explanation hard to follow or inadequate."

:The [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Final_Project section] clarifies your question.

"Check the designs proposed by the authors."

:There is no explicit design is involved as going by [http://wiki.expertiza.ncsu.edu/index.php/CSC/ECE_517_Spring_2019_-_Project_M1901_Implement_missing_WebAudio_automation_support#Final_Project proposal], there are three files to be added with new functionality in addition to test cases and files.

"Check the use of visual aids"
:Visual aids not applicable

"Give comments on the completeness of the design doc."
: Let us know what you think about them!

E1911 Refactor Criterion

2019-04-02T00:49:31Z

Amadhus2: /* Testing */

=E1911 Refactoring criterion.rb=

This page provides a description of the Expertiza based OSS project.

__TOC__

==About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

==Problem Statement==
The bulk of the code in criterion is HTML. It is the violation of MVC architecture - Model should not concern itself with how the data is displayed. This code needs to be moved to a partial file, and the partial file needs to be called in all appropriate places which call the criterion’s model methods. Once the logic for the view is moved out of the model, the model should only be left with business logic. This business logic code can also be refactored. Plus there are virtually no comments, properly comment the code.

==About Criterion.rb==
Criterion is one of the types of questions that can be added to a questionnaire. There are many other types of question objects like dropdown that have the implementation of similar methods. As of now, the criterion model holds 4 methods that are called from different places in the application. Each of these methods is storing a string value which contains the necessary HTML lines to display the required functionalities to the user. This string value is then returned to the calling methods from the respective views. This HTML string is then entered wherever necessary. This is ideally the exact function of a partial, and this page needs to be heavily reformatted to use partials instead of returning a string containing html code.

===The pull request===
https://github.com/expertiza/expertiza/pull/1407

===The edit method===
This method is one of the several other methods in other files like criterion.rb which gets called when the type of the question being created in the respective model. In the case of this problem, which deals with criterion type questions, the edit method defined in criterion.rb is called when the question.type is equal to "Criterion". This method allows for a user to create the criterion question and enter the necessary details. This method is called only from two view files - one for creating (_questionnaire.html.erb partial file) and another for editing (edit.html.erb). Both these files are in questionnaires view. Verified this using both RubyMine and the grep command. Another interesting thing to note is the use of "self." to get the attributes associated with question object. When we move this code to a partial, we need to change this in the erb file. The outcome of this method's refactoring will completely remove this method from the model file criterion.rb as there is no business logic involved here.

<pre>
def edit(_count)
html = '<td align="center"><a rel="nofollow" data-method="delete" href="/questions/' + self.id.to_s + '">Remove</a></td>'

html += '<td><input size="6" value="' + self.seq.to_s + '" name="question[' + self.id.to_s + '][seq]"'
html += ' id="question_' + self.id.to_s + '_seq" type="text"></td>'

html += '<td><textarea cols="50" rows="1" name="question[' + self.id.to_s + '][txt]"'
html += ' id="question_' + self.id.to_s + '_txt" placeholder="Edit question content here">' + self.txt + '</textarea></td>'

html += '<td><input size="10" disabled="disabled" value="' + self.type + '" name="question[' + self.id.to_s + '][type]"'
html += ' id="question_' + self.id.to_s + '_type" type="text"></td>'

html += '<td><input size="2" value="' + self.weight.to_s
html += '" name="question[' + self.id.to_s + '][weight]" id="question_' + self.id.to_s + '_weight" type="text"></td>'

html += '<td>text area size <input size="3" value="' + self.size.to_s
html += '" name="question[' + self.id.to_s + '][size]" id="question_' + self.id.to_s + '_size" type="text"></td>'

html += '<td> max_label <input size="10" value="' + self.max_label.to_s + '" name="question[' + self.id.to_s
html += '][max_label]" id="question_' + self.id.to_s + '_max_label" type="text"> min_label <input size="12" value="' + self.min_label.to_s
html += '" name="question[' + self.id.to_s + '][min_label]" id="question_' + self.id.to_s + '_min_label" type="text"></td>'

safe_join(["<tr>".html_safe, "</tr>".html_safe], html.html_safe)
end
</pre>

The safe_join is later used to return the HTML string to the calling view.

===The view_question_text method===
This method has the same explanation as the edit method except the fact that this method is called only during viewing of questionnaires by an instructor. We observed that this method is also called by in student_quiz view. On further analysis, we found that the student_quiz does not use criterion object and has only three options: TrueFalse, MultipleChoiceRadio and MultipleChoiceCheckbox. So no refactoring is required here. Verified this method isn't called anywhere else using RubyMine and grep command. The outcome of this method's refactoring will completely remove this method from the model file criterion.rb as there is no business logic in this method and is completely html code.

<pre>
def view_question_text
html = '<TD align="left"> ' + self.txt + ' </TD>'
html += '<TD align="left">' + self.type + '</TD>'
html += '<td align="center">' + self.weight.to_s + '</TD>'
questionnaire = self.questionnaire
if !self.max_label.nil? && !self.min_label.nil?
html += '<TD align="center"> (' + self.min_label + ') ' + questionnaire.min_question_score.to_s
html += ' to ' + questionnaire.max_question_score.to_s + ' (' + self.max_label + ')</TD>'
else
html += '<TD align="center">' + questionnaire.min_question_score.to_s + ' to ' + questionnaire.max_question_score.to_s + '</TD>'
end

safe_join(["<TR>".html_safe, "</TR>".html_safe], html.html_safe)
end
</pre>

===The complete method===

This method is one of the several other methods in other files like criterion.rb which gets called when the user responds to the question being created in the respective model. In the case of this problem, which deals with criterion type question responses, the complete method defined in criterion.rb is called when the question.type is equal to "Criterion". This method allows for a user to respond to the criterion question and enter the necessary details. This method is called only from one view file i.e. response.html.erb. This file is in response view. Verified this using both RubyMine and the grep command. Unlike edit or view_question_text the self object does not refer to a global object instead it refers to a copy of the local object. So, we had passed the required object as locals to the partial, along with the parameters required by the method namely question, count, answer, questionnaire_min, questionnaire_max, dropdown_or_scale. When we move this code to a partial, we need to change this in the erb file. The outcome of this method's refactoring will completely remove this method from the model file criterion.rb as there is no business logic involved here.

<pre>
def complete(count, answer = nil, questionnaire_min, questionnaire_max, dropdown_or_scale)
if self.size.nil?
cols = '70'
rows = '1'
else
cols = self.size.split(',')[0]
rows = self.size.split(',')[1]
end

html = '<div><label for="responses_' + count.to_s + '">' + self.txt + '</label></div>'
# show advice for each criterion question
question_advices = QuestionAdvice.where(question_id: self.id).sort_by(&:id)
advice_total_length = 0

question_advices.each do |question_advice|
advice_total_length += question_advice.advice.length if question_advice.advice && question_advice.advice != ""
end

if !question_advices.empty? and advice_total_length > 0
html += '<a id="showAdivce_' + self.id.to_s + '" onclick="showAdvice(' + self.id.to_s + ')">Show advice</a>'
html += '<script>'
html += 'function showAdvice(i){'
html += 'var element = document.getElementById("showAdivce_" + i.toString());'
html += 'var show = element.innerHTML == "Hide advice";'
html += 'if (show){'
html += 'element.innerHTML="Show advice";'
html += '}else{'
html += 'element.innerHTML="Hide advice";}'
html += 'toggleAdvice(i);}'

html += 'function toggleAdvice(i) {'
html += 'var elem = document.getElementById(i.toString() + "_myDiv");'
html += 'if (elem.style.display == "none") {'
html += 'elem.style.display = "";'
html += '} else {'
html += 'elem.style.display = "none";}}'
html += '</script>'

html += '<div id="' + self.id.to_s + '_myDiv" style="display: none;">'
# [2015-10-26] Zhewei:
# best to order advices high to low, e.g., 5 to 1
# each level used to be a link;
# clicking on the link caused the dropbox to be filled in with the corresponding number
question_advices.reverse.each_with_index do |question_advice, index|
html += '<a id="changeScore_>' + self.id.to_s + '" onclick="changeScore(' + count.to_s + ',' + index.to_s + ')">'
html += (self.questionnaire.max_question_score - index).to_s + ' - ' + question_advice.advice + '</a><br/>'
html += '<script>'
html += 'function changeScore(i, j) {'
html += 'var elem = jQuery("#responses_" + i.toString() + "_score");'
html += 'var opts = elem.children("option").length;'
html += 'elem.val((' + self.questionnaire.max_question_score.to_s + ' - j).toString());}'
html += '</script>'
end
html += '</div>'
end

if dropdown_or_scale == 'dropdown'
current_value = ""
current_value += 'data-current-rating =' + answer.answer.to_s if !answer.nil?
html += '<div><select id="responses_' + count.to_s + '_score" name="responses[' + count.to_s + '][score]" class="review-rating" ' + current_value + '>'
html += "<option value = ''>--</option>"
questionnaire_min.upto(questionnaire_max).each do |j|
html += if !answer.nil? and j == answer.answer
'<option value=' + j.to_s + ' selected="selected">'
else
'<option value=' + j.to_s + '>'
end

html += j.to_s
if j == questionnaire_min
html += "-" + self.min_label if self.min_label.present?
elsif j == questionnaire_max
html += "-" + self.max_label if self.max_label.present?
end
html += "</option>"
end
html += "</select></div><br><br>"
html += '<textarea' + ' id="responses_' + count.to_s + '_comments"' \
' name="responses[' + count.to_s + '][comment]" class="tinymce">'
html += answer.comments unless answer.nil?
html += '</textarea></td>'
elsif dropdown_or_scale == 'scale'
html += '<input id="responses_' + count.to_s + '_score" name="responses[' + count.to_s + '][score]" type="hidden"'
html += 'value="' + answer.answer.to_s + '"' unless answer.nil?
html += '>'

html += '<table>'
html += '<tr><td width="10%"></td>'
(questionnaire_min..questionnaire_max).each do |j|
html += '<td width="10%"><label>' + j.to_s + '</label></td>'
end
html += '<td width="10%"></td></tr><tr>'

html += if !self.min_label.nil?
'<td width="10%">' + self.min_label + '</td>'
else
'<td width="10%"></td>'
end
(questionnaire_min..questionnaire_max).each do |j|
html += '<td width="10%"><input type="radio" id="' + j.to_s + '" value="' + j.to_s + '" name="Radio_' + self.id.to_s + '"'
html += 'checked="checked"' if (!answer.nil? and answer.answer == j) or (answer.nil? and questionnaire_min == j)
html += '></td>'
end
html += '<script>jQuery("input[name=Radio_' + self.id.to_s + ']:radio").change(function() {'
html += 'var response_score = jQuery("#responses_' + count.to_s + '_score");'
html += 'var checked_value = jQuery("input[name=Radio_' + self.id.to_s + ']:checked").val();'
html += 'response_score.val(checked_value);});</script>'

html += if !self.max_label.nil?
'<td width="10%">' + self.max_label + '</td>'
else
'<td width="10%"></td>'
end

html += '<td width="10%"></td></tr></table>'
html += '<textarea cols=' + cols + ' rows=' + rows + ' id="responses_' + count.to_s + '_comments"' \
' name="responses[' + count.to_s + '][comment]" class="tinymce">'
html += answer.comments unless answer.nil?
html += '</textarea>'

end
safe_join(["".html_safe, "".html_safe], html.html_safe)
end
</pre>

===The view_completed_question method===

==Proposed Solution==
All these methods contain HTML text. What we propose to do is to pick these lines of code and move them into an appropriate partial, in the required format. The next task is to find out where in the entire application do these methods get called. The multiple overriding of the method calls in this poorly structured application makes it a challenging task. These method-calls then have to be replaced with the appropriate rendering of a partial in the views. This would make the structure more MVC oriented, and help keep it clean and understandable for the next developer who accesses this file.

==Implementation==
The changes proposed above were implemented as described below.

===The Edit Method===
All the code in the model was just html and thus was completely removed from there. It was moved to the partial file (views/questionnaires/_criterion_edit.html.rb) and the necessary changes were made. Since it was all HTML code, no comments were actually required. However, we still added comments describing the partial file itself. Other comments were added whenever changes were made.
====Affected View Files====
'''1. views/questionnaires/edit.html.erb -''' This file calls the edit method for all types of question objects and the object type takes care of invoking the right overridden method. We modified this file to call the partial instead of edit method only for Criterion type of question objects. This view is invoked during edit on the questionnaire. Here are the changes:

<pre>
<% for @question in @questionnaire.questions %>
- <%-# The following line call certain method of the object, which returns html string-%>
+ <%# E1911: Call the criterion_edit partial if question is of type Criterion -%>
+ <% if @question.is_a? Criterion %>
+ <%= render :partial => 'criterion_edit' %>
+ <% else %>
<%=@question.edit%>
+ <% end %>
...
</pre>

'''2. views/questionnaires/_questionnaire.html.erb -''' This partial file is called by new.html.erb when anew question is created. Same logic and changes as above.

'''3. views/questionnaires/_criterion_edit.html.erb -''' This file has all the html code that was earlier in the edit method of the criterion model. We changed the variables from "self.xxx" to "@question.xxx" because self was referring to question object in this context. Apart from this we changed the code to erb format. No more changes were deemed necessary. Please refer to the pull request to refer to the changes in this file as they are pure html and really long to be put in this wiki.

===The view_question_text Method===
This method did not contain any business logic and thus all code was formatted and moved to a new partial file (views/questionnaires/_criterion_view.html.erb). Again, no comments were required due to complete html code and a comment describing the partial was added in addition to other necessary comments in other files.

====Affected View Files====
'''1. views/questionnaires/view.html.erb -''' Modified to invoke the partial if question is of type Criterion only. Also changed the question variable to a instance variable (question => @question) to extend it's scope to the partial. This view is invoked during the view of questions in a questionnaire. Here are the changes:

<pre>
- <%questions.each do |question| %>
- <%if question.is_a? Question%>
- <%=question.view_question_text.html_safe%>
+ <% for @question in questions %>
+ <%# E1911: Call the criterion_view partial if question is of type Criterion %>
+ <%if @question.is_a? Criterion %>
+ <%= render :partial => 'criterion_view' %>
+ <%elsif @question.is_a? Question%>
+ <%= @question.view_question_text.html_safe %>
<%end%>
</pre>

'''2. views/questionnaires/_criterion_view.html.erb -''' This partial file has the moved html code from view_question_text. Note that the "self.xxx" have been updated to "@question.xxx" as self referred to question object discussed in 1. Comment added to describe the partial file. Refer the pull request to look at the code.

===The complete Method===
All the code in the model included html thus was completely removed from there. It was moved to the partial file (app/views/response/_criterion_complete.html.erb) and the necessary changes were made. Since it was all HTML code, no comments were actually required. However, we still added comments describing the partial file itself. Other comments were added whenever changes were made.
====Affected View Files====
'''1. views/repsonse/response.html.erb -''' This file calls the complete method for all types of question objects and the object type takes care of invoking the right overridden method. We modified this file to call the partial instead of the complete method only for Criterion type of question objects. This view is invoked during user responds to the questionnaire. Here are the changes:

<pre>
<% elsif question.instance_of? Scale %>

<%= render partial: 'criterion_complete', :locals => {:question => question, :count => i, :answer => answer, :questionnaire_min => @questionnaire.min_question_score, :questionnaire_max => @questionnaire.max_question_score,
:dropdown_or_scale => @dropdown_or_scale} %>

<% elsif question.instance_of? Scale %>
...
</pre>

'''2. views/response/_criterion_complete.html.erb -''' This file has all the html code that was earlier in the complete method of the criterion model. We changed the variables from "self.xxx" to "question.xxx" because self was referring to copy of question object in this context. Apart from this we changed the code to erb format. No more changes were deemed necessary. Please refer to the pull request to refer to the changes in this file as they are pure html and really long to be put in this wiki.

===The view_completed_question Method===
This method returns a constructed html string that is used to display a particular criterion question. Although this is similar to the other 3 methods, the html variable returned from this function is not directly called into a view. If that was the case, this could be simply moved into a partial as well.

Instead, this html string is appended to another larger string variable named code in the response model, which builds the entire webpage by iterative appending. The html code for a criterion question being completed is a mere "part" of this entire string, which also includes other things such as the navbar of the page, links to the buttons, other questions, the header and the footer and such.

<pre>
def add_table_rows questionnaire_max, questions, answers, code, tag_prompt_deployments = nil, current_user = nil
count = 0
# loop through questions so the the questions are displayed in order based on seq (sequence number)
questions.each do |question|
count += 1 if !question.is_a? QuestionnaireHeader and question.break_before == true
answer = answers.find {|a| a.question_id == question.id }
row_class = count.even? ? "info" : "warning"
row_class = "" if question.is_a? QuestionnaireHeader
code += '<tr class="' + row_class + '"><td>'
if !answer.nil? or question.is_a? QuestionnaireHeader
code += if question.instance_of? Criterion
#Answer Tags are enabled only for Criterion questions at the moment.
question.view_completed_question(count, answer, questionnaire_max, tag_prompt_deployments, current_user) || ''
elsif question.instance_of? Scale
question.view_completed_question(count, answer, questionnaire_max) || ''
else
question.view_completed_question(count, answer) || ''
end
end
code += '</td></tr>'
end
code
end
</pre>

This "code" variable is then used to display the entire page through the view method under response.rb.

A partial cannot be rendered halfway through the computation of "code", neither can it be called from a model. Since code is containing the html string, and not erb, we cannot append the render partial line to the code variable as well.

To refactor this method, we need to first refactor the entire response.rb to not append the html string for an entire webpage in a single variable, but rather use appropriate views to render the webpage.

Refactoring an entire model would be beyond the scope of this project as the scale of the task is that of an entire project by itself.

Thus, this method has been left untouched and still remains in criterion.rb itself.

==Testing==

Since our project only dealt with refactoring existing code, there was no particular need to test any specific components of the application. That being said, there were previous tests that were testing if the html variable returned by those methods was in the expected format or not.
<pre>
describe "#edit" do
it "returns the html " do
html = criterion.edit(0).to_s
expect(html).to eq("<tr><td align=\"center\"><a rel=\"nofollow\" data-method=\"delete\" href=\"/questions/1\">Remove</a></td><td>
<input size=\"6\" value=\"1.0\" name=\"question[1][seq]\" id=\"question_1_seq\" type=\"text\"></td><td><textarea cols=\"50\" rows=\"1\"
name=\"question[1][txt]\" id=\"question_1_txt\" placeholder=\"Edit question content here\">test txt</textarea></td><td><input size=\"10\"
disabled=\"disabled\" value=\"Criterion\" name=\"question[1][type]\" id=\"question_1_type\" type=\"text\"></td><td><input size=\"2\"
value=\"1\" name=\"question[1][weight]\" id=\"question_1_weight\" type=\"text\"></td><td>text area size <input size=\"3\" value=\"\"
name=\"question[1][size]\" id=\"question_1_size\" type=\"text\"></td><td> max_label <input size=\"10\" value=\"\" name=\"question[1]
[max_label]\" id=\"question_1_max_label\" type=\"text\"> min_label <input size=\"12\" value=\"\" name=\"question[1][min_label]\"
id=\"question_1_min_label\" type=\"text\"></td></tr>")
end
end
</pre>

<pre>
describe "#view_question_text" do
it "returns the html " do
html = criterion.view_question_text.to_s
expect(html).to eq("<TR><TD align=\"left\"> test txt </TD><TD align=\"left\">Criterion</TD><td align=\"center\">1</TD><TD align=\"center\"> () 0 to 5 ()</TD></TR>")
end
end
</pre>

<pre>
describe "#complete" do
it "returns the html " do
html = criterion.complete(0, nil, 0, 5).to_s
expect(html).to eq("<div><label for=\"responses_0\">test txt</label></div>")
end
end
</pre>

Now that these methods do not exist, there is no need to keep these test files as there is no html variable being returned or parsed. Therefore, we have removed these redundant test cases.

E1911 Refactor Criterion

2019-04-02T00:46:54Z

Amadhus2: /* Testing */

E1911 Refactor Criterion

2019-04-02T00:46:35Z

Amadhus2: /* Testing */

=E1911 Refactoring criterion.rb=

This page provides a description of the Expertiza based OSS project.

__TOC__

==About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

==Problem Statement==
The bulk of the code in criterion is HTML. It is the violation of MVC architecture - Model should not concern itself with how the data is displayed. This code needs to be moved to a partial file, and the partial file needs to be called in all appropriate places which call the criterion’s model methods. Once the logic for the view is moved out of the model, the model should only be left with business logic. This business logic code can also be refactored. Plus there are virtually no comments, properly comment the code.

==About Criterion.rb==
Criterion is one of the types of questions that can be added to a questionnaire. There are many other types of question objects like dropdown that have the implementation of similar methods. As of now, the criterion model holds 4 methods that are called from different places in the application. Each of these methods is storing a string value which contains the necessary HTML lines to display the required functionalities to the user. This string value is then returned to the calling methods from the respective views. This HTML string is then entered wherever necessary. This is ideally the exact function of a partial, and this page needs to be heavily reformatted to use partials instead of returning a string containing html code.

===The pull request===
https://github.com/expertiza/expertiza/pull/1407

===The edit method===
This method is one of the several other methods in other files like criterion.rb which gets called when the type of the question being created in the respective model. In the case of this problem, which deals with criterion type questions, the edit method defined in criterion.rb is called when the question.type is equal to "Criterion". This method allows for a user to create the criterion question and enter the necessary details. This method is called only from two view files - one for creating (_questionnaire.html.erb partial file) and another for editing (edit.html.erb). Both these files are in questionnaires view. Verified this using both RubyMine and the grep command. Another interesting thing to note is the use of "self." to get the attributes associated with question object. When we move this code to a partial, we need to change this in the erb file. The outcome of this method's refactoring will completely remove this method from the model file criterion.rb as there is no business logic involved here.

<pre>
def edit(_count)
html = '<td align="center"><a rel="nofollow" data-method="delete" href="/questions/' + self.id.to_s + '">Remove</a></td>'

html += '<td><input size="6" value="' + self.seq.to_s + '" name="question[' + self.id.to_s + '][seq]"'
html += ' id="question_' + self.id.to_s + '_seq" type="text"></td>'

html += '<td><textarea cols="50" rows="1" name="question[' + self.id.to_s + '][txt]"'
html += ' id="question_' + self.id.to_s + '_txt" placeholder="Edit question content here">' + self.txt + '</textarea></td>'

html += '<td><input size="10" disabled="disabled" value="' + self.type + '" name="question[' + self.id.to_s + '][type]"'
html += ' id="question_' + self.id.to_s + '_type" type="text"></td>'

html += '<td><input size="2" value="' + self.weight.to_s
html += '" name="question[' + self.id.to_s + '][weight]" id="question_' + self.id.to_s + '_weight" type="text"></td>'

html += '<td>text area size <input size="3" value="' + self.size.to_s
html += '" name="question[' + self.id.to_s + '][size]" id="question_' + self.id.to_s + '_size" type="text"></td>'

html += '<td> max_label <input size="10" value="' + self.max_label.to_s + '" name="question[' + self.id.to_s
html += '][max_label]" id="question_' + self.id.to_s + '_max_label" type="text"> min_label <input size="12" value="' + self.min_label.to_s
html += '" name="question[' + self.id.to_s + '][min_label]" id="question_' + self.id.to_s + '_min_label" type="text"></td>'

safe_join(["<tr>".html_safe, "</tr>".html_safe], html.html_safe)
end
</pre>

The safe_join is later used to return the HTML string to the calling view.

===The view_question_text method===
This method has the same explanation as the edit method except the fact that this method is called only during viewing of questionnaires by an instructor. We observed that this method is also called by in student_quiz view. On further analysis, we found that the student_quiz does not use criterion object and has only three options: TrueFalse, MultipleChoiceRadio and MultipleChoiceCheckbox. So no refactoring is required here. Verified this method isn't called anywhere else using RubyMine and grep command. The outcome of this method's refactoring will completely remove this method from the model file criterion.rb as there is no business logic in this method and is completely html code.

<pre>
def view_question_text
html = '<TD align="left"> ' + self.txt + ' </TD>'
html += '<TD align="left">' + self.type + '</TD>'
html += '<td align="center">' + self.weight.to_s + '</TD>'
questionnaire = self.questionnaire
if !self.max_label.nil? && !self.min_label.nil?
html += '<TD align="center"> (' + self.min_label + ') ' + questionnaire.min_question_score.to_s
html += ' to ' + questionnaire.max_question_score.to_s + ' (' + self.max_label + ')</TD>'
else
html += '<TD align="center">' + questionnaire.min_question_score.to_s + ' to ' + questionnaire.max_question_score.to_s + '</TD>'
end

safe_join(["<TR>".html_safe, "</TR>".html_safe], html.html_safe)
end
</pre>

===The complete method===

This method is one of the several other methods in other files like criterion.rb which gets called when the user responds to the question being created in the respective model. In the case of this problem, which deals with criterion type question responses, the complete method defined in criterion.rb is called when the question.type is equal to "Criterion". This method allows for a user to respond to the criterion question and enter the necessary details. This method is called only from one view file i.e. response.html.erb. This file is in response view. Verified this using both RubyMine and the grep command. Unlike edit or view_question_text the self object does not refer to a global object instead it refers to a copy of the local object. So, we had passed the required object as locals to the partial, along with the parameters required by the method namely question, count, answer, questionnaire_min, questionnaire_max, dropdown_or_scale. When we move this code to a partial, we need to change this in the erb file. The outcome of this method's refactoring will completely remove this method from the model file criterion.rb as there is no business logic involved here.

<pre>
def complete(count, answer = nil, questionnaire_min, questionnaire_max, dropdown_or_scale)
if self.size.nil?
cols = '70'
rows = '1'
else
cols = self.size.split(',')[0]
rows = self.size.split(',')[1]
end

html = '<div><label for="responses_' + count.to_s + '">' + self.txt + '</label></div>'
# show advice for each criterion question
question_advices = QuestionAdvice.where(question_id: self.id).sort_by(&:id)
advice_total_length = 0

question_advices.each do |question_advice|
advice_total_length += question_advice.advice.length if question_advice.advice && question_advice.advice != ""
end

if !question_advices.empty? and advice_total_length > 0
html += '<a id="showAdivce_' + self.id.to_s + '" onclick="showAdvice(' + self.id.to_s + ')">Show advice</a>'
html += '<script>'
html += 'function showAdvice(i){'
html += 'var element = document.getElementById("showAdivce_" + i.toString());'
html += 'var show = element.innerHTML == "Hide advice";'
html += 'if (show){'
html += 'element.innerHTML="Show advice";'
html += '}else{'
html += 'element.innerHTML="Hide advice";}'
html += 'toggleAdvice(i);}'

html += 'function toggleAdvice(i) {'
html += 'var elem = document.getElementById(i.toString() + "_myDiv");'
html += 'if (elem.style.display == "none") {'
html += 'elem.style.display = "";'
html += '} else {'
html += 'elem.style.display = "none";}}'
html += '</script>'

html += '<div id="' + self.id.to_s + '_myDiv" style="display: none;">'
# [2015-10-26] Zhewei:
# best to order advices high to low, e.g., 5 to 1
# each level used to be a link;
# clicking on the link caused the dropbox to be filled in with the corresponding number
question_advices.reverse.each_with_index do |question_advice, index|
html += '<a id="changeScore_>' + self.id.to_s + '" onclick="changeScore(' + count.to_s + ',' + index.to_s + ')">'
html += (self.questionnaire.max_question_score - index).to_s + ' - ' + question_advice.advice + '</a><br/>'
html += '<script>'
html += 'function changeScore(i, j) {'
html += 'var elem = jQuery("#responses_" + i.toString() + "_score");'
html += 'var opts = elem.children("option").length;'
html += 'elem.val((' + self.questionnaire.max_question_score.to_s + ' - j).toString());}'
html += '</script>'
end
html += '</div>'
end

if dropdown_or_scale == 'dropdown'
current_value = ""
current_value += 'data-current-rating =' + answer.answer.to_s if !answer.nil?
html += '<div><select id="responses_' + count.to_s + '_score" name="responses[' + count.to_s + '][score]" class="review-rating" ' + current_value + '>'
html += "<option value = ''>--</option>"
questionnaire_min.upto(questionnaire_max).each do |j|
html += if !answer.nil? and j == answer.answer
'<option value=' + j.to_s + ' selected="selected">'
else
'<option value=' + j.to_s + '>'
end

html += j.to_s
if j == questionnaire_min
html += "-" + self.min_label if self.min_label.present?
elsif j == questionnaire_max
html += "-" + self.max_label if self.max_label.present?
end
html += "</option>"
end
html += "</select></div><br><br>"
html += '<textarea' + ' id="responses_' + count.to_s + '_comments"' \
' name="responses[' + count.to_s + '][comment]" class="tinymce">'
html += answer.comments unless answer.nil?
html += '</textarea></td>'
elsif dropdown_or_scale == 'scale'
html += '<input id="responses_' + count.to_s + '_score" name="responses[' + count.to_s + '][score]" type="hidden"'
html += 'value="' + answer.answer.to_s + '"' unless answer.nil?
html += '>'

html += '<table>'
html += '<tr><td width="10%"></td>'
(questionnaire_min..questionnaire_max).each do |j|
html += '<td width="10%"><label>' + j.to_s + '</label></td>'
end
html += '<td width="10%"></td></tr><tr>'

html += if !self.min_label.nil?
'<td width="10%">' + self.min_label + '</td>'
else
'<td width="10%"></td>'
end
(questionnaire_min..questionnaire_max).each do |j|
html += '<td width="10%"><input type="radio" id="' + j.to_s + '" value="' + j.to_s + '" name="Radio_' + self.id.to_s + '"'
html += 'checked="checked"' if (!answer.nil? and answer.answer == j) or (answer.nil? and questionnaire_min == j)
html += '></td>'
end
html += '<script>jQuery("input[name=Radio_' + self.id.to_s + ']:radio").change(function() {'
html += 'var response_score = jQuery("#responses_' + count.to_s + '_score");'
html += 'var checked_value = jQuery("input[name=Radio_' + self.id.to_s + ']:checked").val();'
html += 'response_score.val(checked_value);});</script>'

html += if !self.max_label.nil?
'<td width="10%">' + self.max_label + '</td>'
else
'<td width="10%"></td>'
end

html += '<td width="10%"></td></tr></table>'
html += '<textarea cols=' + cols + ' rows=' + rows + ' id="responses_' + count.to_s + '_comments"' \
' name="responses[' + count.to_s + '][comment]" class="tinymce">'
html += answer.comments unless answer.nil?
html += '</textarea>'

end
safe_join(["".html_safe, "".html_safe], html.html_safe)
end
</pre>

===The view_completed_question method===

==Proposed Solution==
All these methods contain HTML text. What we propose to do is to pick these lines of code and move them into an appropriate partial, in the required format. The next task is to find out where in the entire application do these methods get called. The multiple overriding of the method calls in this poorly structured application makes it a challenging task. These method-calls then have to be replaced with the appropriate rendering of a partial in the views. This would make the structure more MVC oriented, and help keep it clean and understandable for the next developer who accesses this file.

==Implementation==
The changes proposed above were implemented as described below.

===The Edit Method===
All the code in the model was just html and thus was completely removed from there. It was moved to the partial file (views/questionnaires/_criterion_edit.html.rb) and the necessary changes were made. Since it was all HTML code, no comments were actually required. However, we still added comments describing the partial file itself. Other comments were added whenever changes were made.
====Affected View Files====
'''1. views/questionnaires/edit.html.erb -''' This file calls the edit method for all types of question objects and the object type takes care of invoking the right overridden method. We modified this file to call the partial instead of edit method only for Criterion type of question objects. This view is invoked during edit on the questionnaire. Here are the changes:

<pre>
<% for @question in @questionnaire.questions %>
- <%-# The following line call certain method of the object, which returns html string-%>
+ <%# E1911: Call the criterion_edit partial if question is of type Criterion -%>
+ <% if @question.is_a? Criterion %>
+ <%= render :partial => 'criterion_edit' %>
+ <% else %>
<%=@question.edit%>
+ <% end %>
...
</pre>

'''2. views/questionnaires/_questionnaire.html.erb -''' This partial file is called by new.html.erb when anew question is created. Same logic and changes as above.

'''3. views/questionnaires/_criterion_edit.html.erb -''' This file has all the html code that was earlier in the edit method of the criterion model. We changed the variables from "self.xxx" to "@question.xxx" because self was referring to question object in this context. Apart from this we changed the code to erb format. No more changes were deemed necessary. Please refer to the pull request to refer to the changes in this file as they are pure html and really long to be put in this wiki.

===The view_question_text Method===
This method did not contain any business logic and thus all code was formatted and moved to a new partial file (views/questionnaires/_criterion_view.html.erb). Again, no comments were required due to complete html code and a comment describing the partial was added in addition to other necessary comments in other files.

====Affected View Files====
'''1. views/questionnaires/view.html.erb -''' Modified to invoke the partial if question is of type Criterion only. Also changed the question variable to a instance variable (question => @question) to extend it's scope to the partial. This view is invoked during the view of questions in a questionnaire. Here are the changes:

<pre>
- <%questions.each do |question| %>
- <%if question.is_a? Question%>
- <%=question.view_question_text.html_safe%>
+ <% for @question in questions %>
+ <%# E1911: Call the criterion_view partial if question is of type Criterion %>
+ <%if @question.is_a? Criterion %>
+ <%= render :partial => 'criterion_view' %>
+ <%elsif @question.is_a? Question%>
+ <%= @question.view_question_text.html_safe %>
<%end%>
</pre>

'''2. views/questionnaires/_criterion_view.html.erb -''' This partial file has the moved html code from view_question_text. Note that the "self.xxx" have been updated to "@question.xxx" as self referred to question object discussed in 1. Comment added to describe the partial file. Refer the pull request to look at the code.

===The complete Method===
All the code in the model included html thus was completely removed from there. It was moved to the partial file (app/views/response/_criterion_complete.html.erb) and the necessary changes were made. Since it was all HTML code, no comments were actually required. However, we still added comments describing the partial file itself. Other comments were added whenever changes were made.
====Affected View Files====
'''1. views/repsonse/response.html.erb -''' This file calls the complete method for all types of question objects and the object type takes care of invoking the right overridden method. We modified this file to call the partial instead of the complete method only for Criterion type of question objects. This view is invoked during user responds to the questionnaire. Here are the changes:

<pre>
<% elsif question.instance_of? Scale %>

<%= render partial: 'criterion_complete', :locals => {:question => question, :count => i, :answer => answer, :questionnaire_min => @questionnaire.min_question_score, :questionnaire_max => @questionnaire.max_question_score,
:dropdown_or_scale => @dropdown_or_scale} %>

<% elsif question.instance_of? Scale %>
...
</pre>

'''2. views/response/_criterion_complete.html.erb -''' This file has all the html code that was earlier in the complete method of the criterion model. We changed the variables from "self.xxx" to "question.xxx" because self was referring to copy of question object in this context. Apart from this we changed the code to erb format. No more changes were deemed necessary. Please refer to the pull request to refer to the changes in this file as they are pure html and really long to be put in this wiki.

===The view_completed_question Method===
This method returns a constructed html string that is used to display a particular criterion question. Although this is similar to the other 3 methods, the html variable returned from this function is not directly called into a view. If that was the case, this could be simply moved into a partial as well.

Instead, this html string is appended to another larger string variable named code in the response model, which builds the entire webpage by iterative appending. The html code for a criterion question being completed is a mere "part" of this entire string, which also includes other things such as the navbar of the page, links to the buttons, other questions, the header and the footer and such.

<pre>
def add_table_rows questionnaire_max, questions, answers, code, tag_prompt_deployments = nil, current_user = nil
count = 0
# loop through questions so the the questions are displayed in order based on seq (sequence number)
questions.each do |question|
count += 1 if !question.is_a? QuestionnaireHeader and question.break_before == true
answer = answers.find {|a| a.question_id == question.id }
row_class = count.even? ? "info" : "warning"
row_class = "" if question.is_a? QuestionnaireHeader
code += '<tr class="' + row_class + '"><td>'
if !answer.nil? or question.is_a? QuestionnaireHeader
code += if question.instance_of? Criterion
#Answer Tags are enabled only for Criterion questions at the moment.
question.view_completed_question(count, answer, questionnaire_max, tag_prompt_deployments, current_user) || ''
elsif question.instance_of? Scale
question.view_completed_question(count, answer, questionnaire_max) || ''
else
question.view_completed_question(count, answer) || ''
end
end
code += '</td></tr>'
end
code
end
</pre>

This "code" variable is then used to display the entire page through the view method under response.rb.

A partial cannot be rendered halfway through the computation of "code", neither can it be called from a model. Since code is containing the html string, and not erb, we cannot append the render partial line to the code variable as well.

To refactor this method, we need to first refactor the entire response.rb to not append the html string for an entire webpage in a single variable, but rather use appropriate views to render the webpage.

Refactoring an entire model would be beyond the scope of this project as the scale of the task is that of an entire project by itself.

Thus, this method has been left untouched and still remains in criterion.rb itself.

==Testing==

Since our project only dealt with refactoring existing code, there was no particular need to test any specific components of the application. That being said, there were previous tests that were testing if the html variable returned by those methods was in the expected format or not.
<pre>
describe "#edit" do
it "returns the html " do
html = criterion.edit(0).to_s
expect(html).to eq("<tr><td align=\"center\"><a rel=\"nofollow\" data-method=\"delete\" href=\"/questions/1\">Remove</a></td><td><input size=\"6\" value=\"1.0\" name=\"question[1][seq]\" id=\"question_1_seq\" type=\"text\"></td><td><textarea cols=\"50\" rows=\"1\" name=\"question[1][txt]\" id=\"question_1_txt\" placeholder=\"Edit question content here\">test txt</textarea></td><td><input size=\"10\" disabled=\"disabled\" value=\"Criterion\" name=\"question[1][type]\" id=\"question_1_type\" type=\"text\"></td><td><input size=\"2\" value=\"1\" name=\"question[1][weight]\" id=\"question_1_weight\" type=\"text\"></td><td>text area size <input size=\"3\" value=\"\" name=\"question[1][size]\" id=\"question_1_size\" type=\"text\"></td><td> max_label <input size=\"10\" value=\"\" name=\"question[1][max_label]\" id=\"question_1_max_label\" type=\"text\"> min_label <input size=\"12\" value=\"\" name=\"question[1][min_label]\" id=\"question_1_min_label\" type=\"text\"></td></tr>")
end
end
</pre>

<pre>
describe "#view_question_text" do
it "returns the html " do
html = criterion.view_question_text.to_s
expect(html).to eq("<TR><TD align=\"left\"> test txt </TD><TD align=\"left\">Criterion</TD><td align=\"center\">1</TD><TD align=\"center\"> () 0 to 5 ()</TD></TR>")
end
end
</pre>

<pre>
describe "#complete" do
it "returns the html " do
html = criterion.complete(0, nil, 0, 5).to_s
expect(html).to eq("<div><label for=\"responses_0\">test txt</label></div>")
end
end
</pre>

E1911 Refactor Criterion

2019-04-02T00:28:43Z

Amadhus2: /* The view_completed_question Method */

E1911 Refactor Criterion

2019-04-02T00:28:04Z

Amadhus2: /* The view_completed_question Method */

=E1911 Refactoring criterion.rb=

This page provides a description of the Expertiza based OSS project.

__TOC__

==About Expertiza==

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

==Problem Statement==
The bulk of the code in criterion is HTML. It is the violation of MVC architecture - Model should not concern itself with how the data is displayed. This code needs to be moved to a partial file, and the partial file needs to be called in all appropriate places which call the criterion’s model methods. Once the logic for the view is moved out of the model, the model should only be left with business logic. This business logic code can also be refactored. Plus there are virtually no comments, properly comment the code.

==About Criterion.rb==
Criterion is one of the types of questions that can be added to a questionnaire. There are many other types of question objects like dropdown that have the implementation of similar methods. As of now, the criterion model holds 4 methods that are called from different places in the application. Each of these methods is storing a string value which contains the necessary HTML lines to display the required functionalities to the user. This string value is then returned to the calling methods from the respective views. This HTML string is then entered wherever necessary. This is ideally the exact function of a partial, and this page needs to be heavily reformatted to use partials instead of returning a string containing html code.

===The pull request===
https://github.com/expertiza/expertiza/pull/1407

===The edit method===
This method is one of the several other methods in other files like criterion.rb which gets called when the type of the question being created in the respective model. In the case of this problem, which deals with criterion type questions, the edit method defined in criterion.rb is called when the question.type is equal to "Criterion". This method allows for a user to create the criterion question and enter the necessary details. This method is called only from two view files - one for creating (_questionnaire.html.erb partial file) and another for editing (edit.html.erb). Both these files are in questionnaires view. Verified this using both RubyMine and the grep command. Another interesting thing to note is the use of "self." to get the attributes associated with question object. When we move this code to a partial, we need to change this in the erb file. The outcome of this method's refactoring will completely remove this method from the model file criterion.rb as there is no business logic involved here.

<pre>
def edit(_count)
html = '<td align="center"><a rel="nofollow" data-method="delete" href="/questions/' + self.id.to_s + '">Remove</a></td>'

html += '<td><input size="6" value="' + self.seq.to_s + '" name="question[' + self.id.to_s + '][seq]"'
html += ' id="question_' + self.id.to_s + '_seq" type="text"></td>'

html += '<td><textarea cols="50" rows="1" name="question[' + self.id.to_s + '][txt]"'
html += ' id="question_' + self.id.to_s + '_txt" placeholder="Edit question content here">' + self.txt + '</textarea></td>'

html += '<td><input size="10" disabled="disabled" value="' + self.type + '" name="question[' + self.id.to_s + '][type]"'
html += ' id="question_' + self.id.to_s + '_type" type="text"></td>'

html += '<td><input size="2" value="' + self.weight.to_s
html += '" name="question[' + self.id.to_s + '][weight]" id="question_' + self.id.to_s + '_weight" type="text"></td>'

html += '<td>text area size <input size="3" value="' + self.size.to_s
html += '" name="question[' + self.id.to_s + '][size]" id="question_' + self.id.to_s + '_size" type="text"></td>'

html += '<td> max_label <input size="10" value="' + self.max_label.to_s + '" name="question[' + self.id.to_s
html += '][max_label]" id="question_' + self.id.to_s + '_max_label" type="text"> min_label <input size="12" value="' + self.min_label.to_s
html += '" name="question[' + self.id.to_s + '][min_label]" id="question_' + self.id.to_s + '_min_label" type="text"></td>'

safe_join(["<tr>".html_safe, "</tr>".html_safe], html.html_safe)
end
</pre>

The safe_join is later used to return the HTML string to the calling view.

===The view_question_text method===
This method has the same explanation as the edit method except the fact that this method is called only during viewing of questionnaires by an instructor. We observed that this method is also called by in student_quiz view. On further analysis, we found that the student_quiz does not use criterion object and has only three options: TrueFalse, MultipleChoiceRadio and MultipleChoiceCheckbox. So no refactoring is required here. Verified this method isn't called anywhere else using RubyMine and grep command. The outcome of this method's refactoring will completely remove this method from the model file criterion.rb as there is no business logic in this method and is completely html code.

<pre>
def view_question_text
html = '<TD align="left"> ' + self.txt + ' </TD>'
html += '<TD align="left">' + self.type + '</TD>'
html += '<td align="center">' + self.weight.to_s + '</TD>'
questionnaire = self.questionnaire
if !self.max_label.nil? && !self.min_label.nil?
html += '<TD align="center"> (' + self.min_label + ') ' + questionnaire.min_question_score.to_s
html += ' to ' + questionnaire.max_question_score.to_s + ' (' + self.max_label + ')</TD>'
else
html += '<TD align="center">' + questionnaire.min_question_score.to_s + ' to ' + questionnaire.max_question_score.to_s + '</TD>'
end

safe_join(["<TR>".html_safe, "</TR>".html_safe], html.html_safe)
end
</pre>

===The complete method===

This method is one of the several other methods in other files like criterion.rb which gets called when the user responds to the question being created in the respective model. In the case of this problem, which deals with criterion type question responses, the complete method defined in criterion.rb is called when the question.type is equal to "Criterion". This method allows for a user to respond to the criterion question and enter the necessary details. This method is called only from one view file i.e. response.html.erb. This file is in response view. Verified this using both RubyMine and the grep command. Unlike edit or view_question_text the self object does not refer to a global object instead it refers to a copy of the local object. So, we had passed the required object as locals to the partial, along with the parameters required by the method namely question, count, answer, questionnaire_min, questionnaire_max, dropdown_or_scale. When we move this code to a partial, we need to change this in the erb file. The outcome of this method's refactoring will completely remove this method from the model file criterion.rb as there is no business logic involved here.

<pre>
def complete(count, answer = nil, questionnaire_min, questionnaire_max, dropdown_or_scale)
if self.size.nil?
cols = '70'
rows = '1'
else
cols = self.size.split(',')[0]
rows = self.size.split(',')[1]
end

html = '<div><label for="responses_' + count.to_s + '">' + self.txt + '</label></div>'
# show advice for each criterion question
question_advices = QuestionAdvice.where(question_id: self.id).sort_by(&:id)
advice_total_length = 0

question_advices.each do |question_advice|
advice_total_length += question_advice.advice.length if question_advice.advice && question_advice.advice != ""
end

if !question_advices.empty? and advice_total_length > 0
html += '<a id="showAdivce_' + self.id.to_s + '" onclick="showAdvice(' + self.id.to_s + ')">Show advice</a>'
html += '<script>'
html += 'function showAdvice(i){'
html += 'var element = document.getElementById("showAdivce_" + i.toString());'
html += 'var show = element.innerHTML == "Hide advice";'
html += 'if (show){'
html += 'element.innerHTML="Show advice";'
html += '}else{'
html += 'element.innerHTML="Hide advice";}'
html += 'toggleAdvice(i);}'

html += 'function toggleAdvice(i) {'
html += 'var elem = document.getElementById(i.toString() + "_myDiv");'
html += 'if (elem.style.display == "none") {'
html += 'elem.style.display = "";'
html += '} else {'
html += 'elem.style.display = "none";}}'
html += '</script>'

html += '<div id="' + self.id.to_s + '_myDiv" style="display: none;">'
# [2015-10-26] Zhewei:
# best to order advices high to low, e.g., 5 to 1
# each level used to be a link;
# clicking on the link caused the dropbox to be filled in with the corresponding number
question_advices.reverse.each_with_index do |question_advice, index|
html += '<a id="changeScore_>' + self.id.to_s + '" onclick="changeScore(' + count.to_s + ',' + index.to_s + ')">'
html += (self.questionnaire.max_question_score - index).to_s + ' - ' + question_advice.advice + '</a><br/>'
html += '<script>'
html += 'function changeScore(i, j) {'
html += 'var elem = jQuery("#responses_" + i.toString() + "_score");'
html += 'var opts = elem.children("option").length;'
html += 'elem.val((' + self.questionnaire.max_question_score.to_s + ' - j).toString());}'
html += '</script>'
end
html += '</div>'
end

if dropdown_or_scale == 'dropdown'
current_value = ""
current_value += 'data-current-rating =' + answer.answer.to_s if !answer.nil?
html += '<div><select id="responses_' + count.to_s + '_score" name="responses[' + count.to_s + '][score]" class="review-rating" ' + current_value + '>'
html += "<option value = ''>--</option>"
questionnaire_min.upto(questionnaire_max).each do |j|
html += if !answer.nil? and j == answer.answer
'<option value=' + j.to_s + ' selected="selected">'
else
'<option value=' + j.to_s + '>'
end

html += j.to_s
if j == questionnaire_min
html += "-" + self.min_label if self.min_label.present?
elsif j == questionnaire_max
html += "-" + self.max_label if self.max_label.present?
end
html += "</option>"
end
html += "</select></div><br><br>"
html += '<textarea' + ' id="responses_' + count.to_s + '_comments"' \
' name="responses[' + count.to_s + '][comment]" class="tinymce">'
html += answer.comments unless answer.nil?
html += '</textarea></td>'
elsif dropdown_or_scale == 'scale'
html += '<input id="responses_' + count.to_s + '_score" name="responses[' + count.to_s + '][score]" type="hidden"'
html += 'value="' + answer.answer.to_s + '"' unless answer.nil?
html += '>'

html += '<table>'
html += '<tr><td width="10%"></td>'
(questionnaire_min..questionnaire_max).each do |j|
html += '<td width="10%"><label>' + j.to_s + '</label></td>'
end
html += '<td width="10%"></td></tr><tr>'

html += if !self.min_label.nil?
'<td width="10%">' + self.min_label + '</td>'
else
'<td width="10%"></td>'
end
(questionnaire_min..questionnaire_max).each do |j|
html += '<td width="10%"><input type="radio" id="' + j.to_s + '" value="' + j.to_s + '" name="Radio_' + self.id.to_s + '"'
html += 'checked="checked"' if (!answer.nil? and answer.answer == j) or (answer.nil? and questionnaire_min == j)
html += '></td>'
end
html += '<script>jQuery("input[name=Radio_' + self.id.to_s + ']:radio").change(function() {'
html += 'var response_score = jQuery("#responses_' + count.to_s + '_score");'
html += 'var checked_value = jQuery("input[name=Radio_' + self.id.to_s + ']:checked").val();'
html += 'response_score.val(checked_value);});</script>'

html += if !self.max_label.nil?
'<td width="10%">' + self.max_label + '</td>'
else
'<td width="10%"></td>'
end

html += '<td width="10%"></td></tr></table>'
html += '<textarea cols=' + cols + ' rows=' + rows + ' id="responses_' + count.to_s + '_comments"' \
' name="responses[' + count.to_s + '][comment]" class="tinymce">'
html += answer.comments unless answer.nil?
html += '</textarea>'

end
safe_join(["".html_safe, "".html_safe], html.html_safe)
end
</pre>

===The view_completed_question method===

==Proposed Solution==
All these methods contain HTML text. What we propose to do is to pick these lines of code and move them into an appropriate partial, in the required format. The next task is to find out where in the entire application do these methods get called. The multiple overriding of the method calls in this poorly structured application makes it a challenging task. These method-calls then have to be replaced with the appropriate rendering of a partial in the views. This would make the structure more MVC oriented, and help keep it clean and understandable for the next developer who accesses this file.

==Implementation==
The changes proposed above were implemented as described below.

===The Edit Method===
All the code in the model was just html and thus was completely removed from there. It was moved to the partial file (views/questionnaires/_criterion_edit.html.rb) and the necessary changes were made. Since it was all HTML code, no comments were actually required. However, we still added comments describing the partial file itself. Other comments were added whenever changes were made.
====Affected View Files====
'''1. views/questionnaires/edit.html.erb -''' This file calls the edit method for all types of question objects and the object type takes care of invoking the right overridden method. We modified this file to call the partial instead of edit method only for Criterion type of question objects. This view is invoked during edit on the questionnaire. Here are the changes:

<pre>
<% for @question in @questionnaire.questions %>
- <%-# The following line call certain method of the object, which returns html string-%>
+ <%# E1911: Call the criterion_edit partial if question is of type Criterion -%>
+ <% if @question.is_a? Criterion %>
+ <%= render :partial => 'criterion_edit' %>
+ <% else %>
<%=@question.edit%>
+ <% end %>
...
</pre>

'''2. views/questionnaires/_questionnaire.html.erb -''' This partial file is called by new.html.erb when anew question is created. Same logic and changes as above.

'''3. views/questionnaires/_criterion_edit.html.erb -''' This file has all the html code that was earlier in the edit method of the criterion model. We changed the variables from "self.xxx" to "@question.xxx" because self was referring to question object in this context. Apart from this we changed the code to erb format. No more changes were deemed necessary. Please refer to the pull request to refer to the changes in this file as they are pure html and really long to be put in this wiki.

===The view_question_text Method===
This method did not contain any business logic and thus all code was formatted and moved to a new partial file (views/questionnaires/_criterion_view.html.erb). Again, no comments were required due to complete html code and a comment describing the partial was added in addition to other necessary comments in other files.

====Affected View Files====
'''1. views/questionnaires/view.html.erb -''' Modified to invoke the partial if question is of type Criterion only. Also changed the question variable to a instance variable (question => @question) to extend it's scope to the partial. This view is invoked during the view of questions in a questionnaire. Here are the changes:

<pre>
- <%questions.each do |question| %>
- <%if question.is_a? Question%>
- <%=question.view_question_text.html_safe%>
+ <% for @question in questions %>
+ <%# E1911: Call the criterion_view partial if question is of type Criterion %>
+ <%if @question.is_a? Criterion %>
+ <%= render :partial => 'criterion_view' %>
+ <%elsif @question.is_a? Question%>
+ <%= @question.view_question_text.html_safe %>
<%end%>
</pre>

'''2. views/questionnaires/_criterion_view.html.erb -''' This partial file has the moved html code from view_question_text. Note that the "self.xxx" have been updated to "@question.xxx" as self referred to question object discussed in 1. Comment added to describe the partial file. Refer the pull request to look at the code.

===The complete Method===
All the code in the model included html thus was completely removed from there. It was moved to the partial file (app/views/response/_criterion_complete.html.erb) and the necessary changes were made. Since it was all HTML code, no comments were actually required. However, we still added comments describing the partial file itself. Other comments were added whenever changes were made.
====Affected View Files====
'''1. views/repsonse/response.html.erb -''' This file calls the complete method for all types of question objects and the object type takes care of invoking the right overridden method. We modified this file to call the partial instead of the complete method only for Criterion type of question objects. This view is invoked during user responds to the questionnaire. Here are the changes:

<pre>
<% elsif question.instance_of? Scale %>

<%= render partial: 'criterion_complete', :locals => {:question => question, :count => i, :answer => answer, :questionnaire_min => @questionnaire.min_question_score, :questionnaire_max => @questionnaire.max_question_score,
:dropdown_or_scale => @dropdown_or_scale} %>

<% elsif question.instance_of? Scale %>
...
</pre>

'''2. views/response/_criterion_complete.html.erb -''' This file has all the html code that was earlier in the complete method of the criterion model. We changed the variables from "self.xxx" to "question.xxx" because self was referring to copy of question object in this context. Apart from this we changed the code to erb format. No more changes were deemed necessary. Please refer to the pull request to refer to the changes in this file as they are pure html and really long to be put in this wiki.

===The view_completed_question Method===
This method returns a constructed html string that is used to display a particular criterion question. Although this is similar to the other 3 methods, the html variable returned from this function is not directly called into a view. If that was the case, this could be simply moved into a partial as well.

Instead, this html string is appended to another larger string variable named code in the response model, which builds the entire webpage by iterative appending. The html code for a criterion question being completed is a mere "part" of this entire string, which also includes other things such as the navbar of the page, links to the buttons, other questions, the header and the footer and such.

<pre>
def add_table_rows questionnaire_max, questions, answers, code, tag_prompt_deployments = nil, current_user = nil
count = 0
# loop through questions so the the questions are displayed in order based on seq (sequence number)
questions.each do |question|
count += 1 if !question.is_a? QuestionnaireHeader and question.break_before == true
answer = answers.find {|a| a.question_id == question.id }
row_class = count.even? ? "info" : "warning"
row_class = "" if question.is_a? QuestionnaireHeader
code += '<tr class="' + row_class + '"><td>'
if !answer.nil? or question.is_a? QuestionnaireHeader
code += if question.instance_of? Criterion
#Answer Tags are enabled only for Criterion questions at the moment.
<b>question.view_completed_question(count, answer, questionnaire_max, tag_prompt_deployments, current_user) || ''</b>
elsif question.instance_of? Scale
question.view_completed_question(count, answer, questionnaire_max) || ''
else
question.view_completed_question(count, answer) || ''
end
end
code += '</td></tr>'
end
code
end
</pre>

This "code" variable is then used to display the entire page through the view method under response.rb.

A partial cannot be rendered halfway through the computation of "code", neither can it be called from a model. Since code is containing the html string, and not erb, we cannot append the render partial line to the code variable as well.

To refactor this method, we need to first refactor the entire response.rb to not append the html string for an entire webpage in a single variable, but rather use appropriate views to render the webpage.

Refactoring an entire model would be beyond the scope of this project as the scale of the task is that of an entire project by itself.

Thus, this method has been left untouched and still remains in criterion.rb itself.

==Testing==

E1911 Refactor Criterion

2019-04-02T00:27:26Z

Amadhus2: /* E1911 Refactoring criterion.rb */

E1911 Refactor Criterion

2019-04-01T22:29:05Z

Amadhus2: /* Affected View Files */

E1911 Refactor Criterion

2019-04-01T22:27:46Z

Amadhus2: /* Affected View Files */

E1911 Refactor Criterion

2019-04-01T22:26:20Z

Amadhus2: /* Affected View Files */

E1911 Refactor Criterion

2019-04-01T22:26:06Z

Amadhus2: /* Affected View Files */

E1911 Refactor Criterion

2019-04-01T22:23:45Z

Amadhus2: /* Affected View Files */

E1911 Refactor Criterion

2019-04-01T22:23:21Z

Amadhus2: /* Affected View Files */

E1911 Refactor Criterion

2019-04-01T22:22:25Z

Amadhus2:

E1911 Refactor Criterion

2019-03-26T00:29:31Z

Amadhus2:

==E1911 Refactoring criterion.rb==

This page provides a description of the Expertiza based OSS project.

__TOC__

===About Expertiza===

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

===Problem Statement===
The bulk of the code in criterion is HTML. It is the violation of MVC architecture - Model should not concern itself with how the data is displayed. This code needs to be moved to a partial file, and the partial file needs to be called in all appropriate places which call the criterion’s model methods. Once the logic for view is moved out of the model, model should only be left with business logic. This business logic code can also be refactored. Plus there are virtually no comments, properly comment the code.

E1911 Refactor Criterion

2019-03-26T00:08:48Z

Amadhus2:

==E1553. Refactoring the Versions Controller==

This page provides a description of the Expertiza based OSS project.

__TOC__

===About Expertiza===

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

===Problem Statement===
The bulk of the code in criterion is HTML. It is the violation of MVC architecture - Model should not concern itself with how the data is displayed. This code needs to be moved to a partial file, and the partial file needs to be called in all appropriate places which call the criterion’s model methods. Once the logic for view is moved out of the model, model should only be left with business logic. This business logic code can also be refactored. Plus there are virtually no comments, properly comment the code.

===About Versions Controller===
This class manages different versions of reviews. If a reviewer reviews a submission, and after that, the author revises the submission, the next time the reviewer does a review (s)he will create a new version. Sometimes it’s necessary to find the current version of a review; sometimes it’s necessary to find all versions. Similarly, a user may want to delete the current version of a review, or all versions of a review. Pagination of versions helps the user to view a subset of versions at a time. Considering the huge number of versions in the system, it is very useful to have a pagination mechanism and a filtering mechanism which can be applied on the whole set of versions. The idea is to display the versions in an ordered, comprehensible and logical manner. In Expertiza the gem ‘will_paginate’ is used to achieve pagination.

===Current Implementation===

=====Functionality=====
* Any user irrespective of his/ her privileges can view all the versions.
::The versions which a particular user can view should be restricted based on the privileges of the user. For instance, only a user with Administrator privileges should be able to view all the versions in the system. However, that is not the case now. Every user can view all the versions irrespective of whether the user is a student or an administrator.
* Any user can delete any version
::The versions which a particular user can delete should be restricted based on the privileges of the user. For instance, a student should not be allowed to delete any version. According to the current implementation any user can delete any version in the system.
* Filtering of versions were restricted to the current user
::The filtering options on versions were restricted to the current user. Sometimes a user might want to view versions associated with other users. For instance, an instructor might want to view the list of versions created by a particular student. This is not possible with the current implementation.

=====Drawbacks and Solutions=====
* '''Problem 1''': The method paginate_list is doing more than one thing.
::The method paginate_list was building a complex search criteria based on the input params, getting the list of versions from the Database matching this search criteria and then calling the Page API. All these tasks in a single method made it difficult to understand.
<pre>
# For filtering the versions list with proper search and pagination.
def paginate_list(id, user_id, item_type, event, datetime)
# Set up the search criteria
criteria = ''
criteria = criteria + "id = #{id} AND " if id && id.to_i > 0
if current_user_role? == 'Super-Administrator'
criteria = criteria + "whodunnit = #{user_id} AND " if user_id && user_id.to_i > 0
end
criteria = criteria + "whodunnit = #{current_user.try(:id)} AND " if current_user.try(:id) && current_user.try(:id).to_i > 0
criteria = criteria + "item_type = '#{item_type}' AND " if item_type && !(item_type.eql? 'Any')
criteria = criteria + "event = '#{event}' AND " if event && !(event.eql? 'Any')
criteria = criteria + "created_at >= '#{time_to_string(params[:start_time])}' AND "
criteria = criteria + "created_at <= '#{time_to_string(params[:end_time])}' AND "

if current_role == 'Instructor' || current_role == 'Administrator'

end

# Remove the last ' AND '
criteria = criteria[0..-5]

versions = Version.page(params[:page]).order('id').per_page(25).where(criteria)
versions
end
</pre>
* '''Solution''': The implementation has been changed in such a way that the versions which a user is allowed to see depends on the privileges of the user. The approach we have taken is as follows:
**An administrator can see all the versions
**An instructor can see all the versions created by him and other users who are in his course or are participants in the assignments he creates.
**A TA can see all the versions created by him and other users who are in the course for which he/ she assists.
**A Student can see all the versions created by him/ her.
* '''Problem 2''': The search criteria created in the method paginate_list was difficult to comprehend.
::The code which builds the search criteria in the method paginate_list uses many string literals and conditions and is hardly intuitive. The programmer will have to spend some time to understand what the code is really doing.
* '''Solution''': The implementation has been changed. A student is not allowed to delete any versions now. Other types of users, for instance administrators, instructors and TAs are allowed to delete only the versions they are authorized to view.
* '''Problem 3''': The paginate method can be moved to a helper class.
::VersionsController is not the only component which require to paginate items. There are other components too. For instance, the UsersController has to paginate the list of users. Hence the Paginate method can be moved to a helper class which can be accessed by other components as well.
* '''Solution''': The filtering options has also been enhanced. The current user can now choose as part of the version search filter any user from a list of users if the current user is authorized to see the versions created by that user.

===New Implementation===
*The method paginate_list has been split into 2 methods now.
** BuildSearchCriteria – as the name suggests the sole purpose of this method is to build a search criteria based on the input search filters when the current user initiates a search in versions.
** paginate_list – this method will call the paginate API.
:First the search criteria is built, then the criteria is applied to versions in the database to get all versions which matches the criteria and then the retrieved versions are paginated.
<pre>
# pagination.
def paginate_list(versions)
paginate(versions, VERSIONS_PER_PAGE);
end

def BuildSearchCriteria(id, user_id, item_type, event)
# Set up the search criteria
search_criteria = ''
search_criteria = search_criteria + add_id_filter_if_valid(id).to_s
if current_user_role? == 'Super-Administrator'
search_criteria = search_criteria + add_user_filter_for_super_admin(user_id).to_s
end
search_criteria = search_criteria + add_user_filter
search_criteria = search_criteria + add_version_type_filter(item_type).to_s
search_criteria = search_criteria + add_event_filter(event).to_s
search_criteria = search_criteria + add_date_time_filter
search_criteria
end
</pre>
* The string literals and conditions in the method paginate_list were replaced with methods with intuitive names so that the programmer can understand the code more easily. We also removed an empty if clause and a redundant statement.
<pre>
def add_id_filter_if_valid (id)
"id = #{id} AND " if id && id.to_i > 0
end

def add_user_filter_for_super_admin (user_id)
"whodunnit = #{user_id} AND " if user_id && user_id.to_i > 0
end

def add_user_filter
"whodunnit = #{current_user.try(:id)} AND " if current_user.try(:id) && current_user.try(:id).to_i > 0
end

def add_event_filter (event)
"event = '#{event}' AND " if event && !(event.eql? 'Any')
end

def add_date_time_filter
"created_at >= '#{time_to_string(params[:start_time])}' AND " +
"created_at <= '#{time_to_string(params[:end_time])}'"
end

def add_version_type_filter (version_type)
"item_type = '#{version_type}' AND " if version_type && !(version_type.eql? 'Any')
end
</pre>
* The paginate method has been moved to the helper class Pagination_Helper. This new method can be now reused by the different components like UsersController etc. The method receives two parameters, first the list to paginate and second the number of items to be displayed in a page.

<pre>
module PaginationHelper

def paginate (items, number_of_items_per_page)
items.page(params[:page]).per_page(number_of_items_per_page)
end

end
</pre>

===Code improvements===
* Introduced a constant VERSIONS_PER_PAGE and assigned the value 25 to it. The pagination algorithm for VersionsController displays at most 25 versions in a page. The existing implementation uses the value 25 straight in the code and there are few problems associated with such an approach.
** It is not easy to understand what 25 is unless the programmer takes a close look at the code.
** In case if the value 25 is used at more than one places and in future a new requirement comes to show at most 30 versions in a page, all the values will have to be modified. It is not very DRY.
* The VersionsController was overriding AccessHelper - action_allowed? method to return true in all the cases. This was violating the whole purpose of the method action_allowed?. The purpose of this method is to determine whether the user who is triggering a CRUD operation is allowed to do so. So when the current user invokes a CRUD operation, the action_allowed? method is invoked first and if the method returns true the CRUD operation is triggered or else the user is intimated with a message and gracefully exited. Hence, when the action_allowed? method is overridden to return true always, it results in providing unauthorized access to certain users.

<pre>
def action_allowed?
true
end
</pre>

:With the new implementation the AccessHelper - action_allowed? method has been modified in such a way that unauthorized access is prevented. As per the new algorithm, 'new', 'create', 'edit', 'update' cannot be invoked by any user. These operations can be accessed only by ‘papertrail’ gem. Only an ‘Administrator’ or ‘Super-Administrator’ can call 'destroy_all' method. All the other methods are accessible to ‘Administrator’, ‘Super-Administrator’, ‘Instructor’, ‘Teaching Assistant’ and ‘Student’.

<pre>
def action_allowed?
case params[:action]
when 'new', 'create', 'edit', 'update'
#Modifications can only be done by papertrail
return false
when 'destroy_all'
['Super-Administrator',
'Administrator'].include? current_role_name
else
#Allow all others
['Super-Administrator',
'Administrator',
'Instructor',
'Teaching Assistant',
'Student'].include? current_role_name
end
end
</pre>

===Automated Testing using RSPEC===
The current version of expertiza did not have any test for VersionsController. Using the test driven development(TDD) approach, we have added an exhaustive set of RSPEC tests for VersionsController, to test all the modifications we have done to the code of the controller class. The tests use double and stub features of rspec-rails gem, to fake the log in by different users - Administrator, Instructor, Student etc. The tests can be executed "rpec spec" command as shown below.
<pre>
user-expertiza $rspec spec
.
.
.
Finished in 5.39 seconds (files took 25.33 seconds to load)
66 examples, 0 failures

Randomized with seed 19254
.
.
</pre>

===Testing from UI===
Following are a few testcases with respectto our code changes that can be tried from UI:
1. To go to versions index page, type in the following url after logging in:
http://152.46.16.81:3000/versions

2. After logging in as student/instructor or admin : Try accessing the new, create, edit, update actions. These actions are not allowed to any of the users.
http://152.46.16.81:3000/versions/new
This calls the new action. In the current production version of expertiza, it is unhandled and application gives a default 404 page.

3. Another feature that can be tested from UI is Pagination. Try searching for a user's versions and see if the results are paginated or not. Search here:
http://152.46.16.81:3000/versions/search

4. Visit the same URL as step 3, you should see only the students under that instructor in the users dropdown.

===References===

#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[https://github.com/WintersLt/expertiza GitHub Project Repository Fork]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[http://bit.ly/myexpertiza Demo link]
#[http://wikis.lib.ncsu.edu/index.php/Expertiza Expertiza project documentation wiki]
#[https://relishapp.com/rspec Rspec Documentation]
#Clean Code: A handbook of agile software craftsmanship. Author: Robert.C.Martin

E1911 Refactor Criterion

2019-03-25T22:38:51Z

Amadhus2:

==E1553. Refactoring the Versions Controller==

This page provides a description of the Expertiza based OSS project.

__TOC__

===About Expertiza===

[http://expertiza.ncsu.edu/ Expertiza] is an open source project based on [http://rubyonrails.org/ Ruby on Rails] framework. Expertiza allows the instructor to create new assignments and customize new or existing assignments. It also allows the instructor to create a list of topics the students can sign up for. Students can form teams in Expertiza to work on various projects and assignments. Students can also peer review other students' submissions. Expertiza supports submission across various document types, including the URLs and wiki pages.

===Problem Statement===
The following tasks were accomplished in this project:

* Improved the clarity of code by improving the variable and parameter names.
* Long character strings were taken and given appropriate names.
* Handled pagination by a separate helper module, which can be used by multiple controllers.
* Implemented action_allowed for access_control to prevent unauthorized access of methods.
* Prevented displaying of all versions for all users and tables when a user views the index page.
* Added missing CRUD methods to Versions Controller
* Added RSPEC testcases for testing changes done in Versions Controller

===About Versions Controller===
This class manages different versions of reviews. If a reviewer reviews a submission, and after that, the author revises the submission, the next time the reviewer does a review (s)he will create a new version. Sometimes it’s necessary to find the current version of a review; sometimes it’s necessary to find all versions. Similarly, a user may want to delete the current version of a review, or all versions of a review. Pagination of versions helps the user to view a subset of versions at a time. Considering the huge number of versions in the system, it is very useful to have a pagination mechanism and a filtering mechanism which can be applied on the whole set of versions. The idea is to display the versions in an ordered, comprehensible and logical manner. In Expertiza the gem ‘will_paginate’ is used to achieve pagination.

===Current Implementation===

=====Functionality=====
* Any user irrespective of his/ her privileges can view all the versions.
::The versions which a particular user can view should be restricted based on the privileges of the user. For instance, only a user with Administrator privileges should be able to view all the versions in the system. However, that is not the case now. Every user can view all the versions irrespective of whether the user is a student or an administrator.
* Any user can delete any version
::The versions which a particular user can delete should be restricted based on the privileges of the user. For instance, a student should not be allowed to delete any version. According to the current implementation any user can delete any version in the system.
* Filtering of versions were restricted to the current user
::The filtering options on versions were restricted to the current user. Sometimes a user might want to view versions associated with other users. For instance, an instructor might want to view the list of versions created by a particular student. This is not possible with the current implementation.

=====Drawbacks and Solutions=====
* '''Problem 1''': The method paginate_list is doing more than one thing.
::The method paginate_list was building a complex search criteria based on the input params, getting the list of versions from the Database matching this search criteria and then calling the Page API. All these tasks in a single method made it difficult to understand.
<pre>
# For filtering the versions list with proper search and pagination.
def paginate_list(id, user_id, item_type, event, datetime)
# Set up the search criteria
criteria = ''
criteria = criteria + "id = #{id} AND " if id && id.to_i > 0
if current_user_role? == 'Super-Administrator'
criteria = criteria + "whodunnit = #{user_id} AND " if user_id && user_id.to_i > 0
end
criteria = criteria + "whodunnit = #{current_user.try(:id)} AND " if current_user.try(:id) && current_user.try(:id).to_i > 0
criteria = criteria + "item_type = '#{item_type}' AND " if item_type && !(item_type.eql? 'Any')
criteria = criteria + "event = '#{event}' AND " if event && !(event.eql? 'Any')
criteria = criteria + "created_at >= '#{time_to_string(params[:start_time])}' AND "
criteria = criteria + "created_at <= '#{time_to_string(params[:end_time])}' AND "

if current_role == 'Instructor' || current_role == 'Administrator'

end

# Remove the last ' AND '
criteria = criteria[0..-5]

versions = Version.page(params[:page]).order('id').per_page(25).where(criteria)
versions
end
</pre>
* '''Solution''': The implementation has been changed in such a way that the versions which a user is allowed to see depends on the privileges of the user. The approach we have taken is as follows:
**An administrator can see all the versions
**An instructor can see all the versions created by him and other users who are in his course or are participants in the assignments he creates.
**A TA can see all the versions created by him and other users who are in the course for which he/ she assists.
**A Student can see all the versions created by him/ her.
* '''Problem 2''': The search criteria created in the method paginate_list was difficult to comprehend.
::The code which builds the search criteria in the method paginate_list uses many string literals and conditions and is hardly intuitive. The programmer will have to spend some time to understand what the code is really doing.
* '''Solution''': The implementation has been changed. A student is not allowed to delete any versions now. Other types of users, for instance administrators, instructors and TAs are allowed to delete only the versions they are authorized to view.
* '''Problem 3''': The paginate method can be moved to a helper class.
::VersionsController is not the only component which require to paginate items. There are other components too. For instance, the UsersController has to paginate the list of users. Hence the Paginate method can be moved to a helper class which can be accessed by other components as well.
* '''Solution''': The filtering options has also been enhanced. The current user can now choose as part of the version search filter any user from a list of users if the current user is authorized to see the versions created by that user.

===New Implementation===
*The method paginate_list has been split into 2 methods now.
** BuildSearchCriteria – as the name suggests the sole purpose of this method is to build a search criteria based on the input search filters when the current user initiates a search in versions.
** paginate_list – this method will call the paginate API.
:First the search criteria is built, then the criteria is applied to versions in the database to get all versions which matches the criteria and then the retrieved versions are paginated.
<pre>
# pagination.
def paginate_list(versions)
paginate(versions, VERSIONS_PER_PAGE);
end

def BuildSearchCriteria(id, user_id, item_type, event)
# Set up the search criteria
search_criteria = ''
search_criteria = search_criteria + add_id_filter_if_valid(id).to_s
if current_user_role? == 'Super-Administrator'
search_criteria = search_criteria + add_user_filter_for_super_admin(user_id).to_s
end
search_criteria = search_criteria + add_user_filter
search_criteria = search_criteria + add_version_type_filter(item_type).to_s
search_criteria = search_criteria + add_event_filter(event).to_s
search_criteria = search_criteria + add_date_time_filter
search_criteria
end
</pre>
* The string literals and conditions in the method paginate_list were replaced with methods with intuitive names so that the programmer can understand the code more easily. We also removed an empty if clause and a redundant statement.
<pre>
def add_id_filter_if_valid (id)
"id = #{id} AND " if id && id.to_i > 0
end

def add_user_filter_for_super_admin (user_id)
"whodunnit = #{user_id} AND " if user_id && user_id.to_i > 0
end

def add_user_filter
"whodunnit = #{current_user.try(:id)} AND " if current_user.try(:id) && current_user.try(:id).to_i > 0
end

def add_event_filter (event)
"event = '#{event}' AND " if event && !(event.eql? 'Any')
end

def add_date_time_filter
"created_at >= '#{time_to_string(params[:start_time])}' AND " +
"created_at <= '#{time_to_string(params[:end_time])}'"
end

def add_version_type_filter (version_type)
"item_type = '#{version_type}' AND " if version_type && !(version_type.eql? 'Any')
end
</pre>
* The paginate method has been moved to the helper class Pagination_Helper. This new method can be now reused by the different components like UsersController etc. The method receives two parameters, first the list to paginate and second the number of items to be displayed in a page.

<pre>
module PaginationHelper

def paginate (items, number_of_items_per_page)
items.page(params[:page]).per_page(number_of_items_per_page)
end

end
</pre>

===Code improvements===
* Introduced a constant VERSIONS_PER_PAGE and assigned the value 25 to it. The pagination algorithm for VersionsController displays at most 25 versions in a page. The existing implementation uses the value 25 straight in the code and there are few problems associated with such an approach.
** It is not easy to understand what 25 is unless the programmer takes a close look at the code.
** In case if the value 25 is used at more than one places and in future a new requirement comes to show at most 30 versions in a page, all the values will have to be modified. It is not very DRY.
* The VersionsController was overriding AccessHelper - action_allowed? method to return true in all the cases. This was violating the whole purpose of the method action_allowed?. The purpose of this method is to determine whether the user who is triggering a CRUD operation is allowed to do so. So when the current user invokes a CRUD operation, the action_allowed? method is invoked first and if the method returns true the CRUD operation is triggered or else the user is intimated with a message and gracefully exited. Hence, when the action_allowed? method is overridden to return true always, it results in providing unauthorized access to certain users.

<pre>
def action_allowed?
true
end
</pre>

:With the new implementation the AccessHelper - action_allowed? method has been modified in such a way that unauthorized access is prevented. As per the new algorithm, 'new', 'create', 'edit', 'update' cannot be invoked by any user. These operations can be accessed only by ‘papertrail’ gem. Only an ‘Administrator’ or ‘Super-Administrator’ can call 'destroy_all' method. All the other methods are accessible to ‘Administrator’, ‘Super-Administrator’, ‘Instructor’, ‘Teaching Assistant’ and ‘Student’.

<pre>
def action_allowed?
case params[:action]
when 'new', 'create', 'edit', 'update'
#Modifications can only be done by papertrail
return false
when 'destroy_all'
['Super-Administrator',
'Administrator'].include? current_role_name
else
#Allow all others
['Super-Administrator',
'Administrator',
'Instructor',
'Teaching Assistant',
'Student'].include? current_role_name
end
end
</pre>

===Automated Testing using RSPEC===
The current version of expertiza did not have any test for VersionsController. Using the test driven development(TDD) approach, we have added an exhaustive set of RSPEC tests for VersionsController, to test all the modifications we have done to the code of the controller class. The tests use double and stub features of rspec-rails gem, to fake the log in by different users - Administrator, Instructor, Student etc. The tests can be executed "rpec spec" command as shown below.
<pre>
user-expertiza $rspec spec
.
.
.
Finished in 5.39 seconds (files took 25.33 seconds to load)
66 examples, 0 failures

Randomized with seed 19254
.
.
</pre>

===Testing from UI===
Following are a few testcases with respectto our code changes that can be tried from UI:
1. To go to versions index page, type in the following url after logging in:
http://152.46.16.81:3000/versions

2. After logging in as student/instructor or admin : Try accessing the new, create, edit, update actions. These actions are not allowed to any of the users.
http://152.46.16.81:3000/versions/new
This calls the new action. In the current production version of expertiza, it is unhandled and application gives a default 404 page.

3. Another feature that can be tested from UI is Pagination. Try searching for a user's versions and see if the results are paginated or not. Search here:
http://152.46.16.81:3000/versions/search

4. Visit the same URL as step 3, you should see only the students under that instructor in the users dropdown.

===References===

#[https://github.com/expertiza/expertiza Expertiza on GitHub]
#[https://github.com/WintersLt/expertiza GitHub Project Repository Fork]
#[http://expertiza.ncsu.edu/ The live Expertiza website]
#[http://bit.ly/myexpertiza Demo link]
#[http://wikis.lib.ncsu.edu/index.php/Expertiza Expertiza project documentation wiki]
#[https://relishapp.com/rspec Rspec Documentation]
#Clean Code: A handbook of agile software craftsmanship. Author: Robert.C.Martin