PeerLogic Web Services: Rainbow Graph Service

The Rainbow Graph

Critique Results Visualization

Created by David Tinapple, Dmytro Babik, Jaharsh Venkata Sadha, Ferry Pramudianto

In this overview, we show the current state of our attempt at visualizing the results of peer review critique session including an explanation of how we arrived at the current design iteration.

The data from this critique are from a single project assignment run in a real classroom using CritViz.com as the platform for running the peer review session.

In this assignment for a “creative coding” course, students had to explore the creative possibilities using a specific type of programming language to create an interactive piece of artwork. The assignment required the students to learn the technical aspects of the coding language and techniques, but also implement their own project in a creative and engaging way. They uploaded their resulting projects and documentation for their peers to review.

The critique session unfolded the the following way:

• Students completed their projects and uploaded their work (code and documentation) to CritViz.com by a specific deadline, answering all of required questions in the assignment.
• A second “critique assignment” was initiated, including only those students who completed the first assignment. Students who didn’t complete the original assignment were not included in the critique.
• CritViz.com automatically assigned each critic a set of peer works to review. The procedure for assigning the critiques is randomized and non-reciprocal. The students were not placed into critique groups, but rather presented with a set of works to review. Each student has a unique randomized set of works to review. The matching algorithm ensures that each student receives their set of works to review, and also has their own work placed into the same number of sets being reviewed by others.
• Each critic was then presented with the critique assignment questions to answer about each of the works in their set. The identities of the authors of the works is temporarily hidden from the critics in order to afford more objectivity and focus on the work itself. These questions included some text feedback in response to prompts, and also a single “ranking” question asking them to rank order their set overall.
• Care was taken to explain to students that this ranking task had no bearing on the recipient’s actual grade, but rather was a way for the class to “self curate” the work. The ranking task was explained as a general ranking from “strongest to weakest” and to take into account both technical and conceptual/aesthetic considerations.
• Most (but not all) students completed this critique assignment by the deadline. Each student then received a number of reviews, although due to the fact that some critics did not complete their tasks by the deadline, not all students received the exact same number of reviews.
• As soon as the critique deadline is met, all the reviews become visible to the recipients of the reviews as well as the rest of the class, and all names of the authors and critics become visible. The ranking scores are visible too.

The ranking data received consists of a simple rank position number. No “point values” are assigned to the rank scores. A rank of “1” means their critic positioned them “first place” in their set of items to rank. The resulting data, color coded (1=green, 5=red) is as follows:

Each students' column is populated with their individual rank scores sorted so that weaker scores “sink” to the bottom of the column. The columns themselves are sorted left to right by the average of all the individual scores.

This graph is a good starting place and is already useful as a diagnostic tool, however it fails to make visible the rank average scores, even though the columns are sorted by it.

The spreadsheet above also has the disadvantage of appearing to be a bar chart, when in fact the different heights for the different columns is only due to the differing numbers of critiques received by each students (remember, some students don’t complete their critiques). In the spreadsheet above, some students received 14 critiques, and others only 10.

To alleviate this false appearance of a bar chart, we could visually normalize the height of each column by showing the following:

The data is the same as the initial spreadsheet view, but each column is adjusted to be the same height, leading to some individual cells appearing bigger than others.

This graphic is an improvement, and is informative, but it fails to visually depict the overall rank average score for each column.

A simple bar chart of the rank averages, looks like this:

A rank average bar chart makes visible the pattern in rank averages, making it obvious that the columns are sorted by the average score, and also making visible the shape of the downward slope of the graph. For instance, it’s clear that the first three students received very high averages, with a drop off and then a gradual decline.

The advantage of the bar chart of average scores is that it makes the sorting clear, and shows the larger trends well, but it occludes the individual scores that go into determining these averages. Some students averages might be influenced by one outlier score, while others with a similar average might consist of uniform scores.

Our solution was to combine the raw spreadsheet view and the rank average chart into a “decomposable average” chart view.

This shows the same data as the initial spreadsheet, but each of the columns is sized according to the average of the scores received in that column. In this way we are looking at the bar chart of the rank average, but can also see the individual scores that are included in that average. This has the advantage of the average, but also depicting the variance of the scores that determine the average. For instance, the “Kyle” column has a strong variability in the scores received, whereas the “Alisha” column has a more consistent distribution of scores, even though their overall averages are quite similar.

How to use the Rainbow Graph web service

This section describes how to use the rainbow graph web service and how to format the JSON data file appropriately given the kind of data your peer review system uses.

This web service is used by sending a POST request with a properly formatted JSON data to the web service host at this link: http://peerlogic.csc.ncsu.edu/rainbowgraph/configure. What will be returned to you is the unique URL of a page with the resulting graph rendered as a 100% width interactive SVG. The URL is returned in a JSON format as the following:

{
    "url":"http://peerlogic.csc.ncsu.edu/rainbowgraph/xxxx-xxxx-xxxx-xxxx
}

Once you extract the URL, you can render this graph in your system by placing the returned URL in an IFRAME in your website as the following example.

<iframe id="rainbowGraph" src="http://peerlogic.csc.ncsu.edu/rainbowgraph/viz/dd4343b1-29e2-4a27-aed4-888fa3b6a2a2" width="800" height="380" frameborder="0" scrolling="no"></iframe>

The following code snippet is an example of a Javascript function that uses JQuery Ajax to call the web service:

<html lang="en">
<head>
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
</head>
<body>
<script>
    sendJson = function(json_file){
        $.getJSON(json_file, function(json){
            $.ajax({
                type: "POST",
                url: "http://peerlogic.csc.ncsu.edu/rainbowgraph/configure",
                dataType: "json",
                contentType: "application/json",
                data: JSON.stringify(json),
                success: function (resp) {
                    if (resp.url) {
                        $('#chart').append('<iframe id="rainbowGraph" src="' + resp.url + '" width="800" height="380" frameborder="0" scrolling="no"></iframe>')
                    }else
                        $('#chart').append(resp.error);
                },
            });
        });
    }
</script>

<button onclick="sendJson('data.json')">Get Rainbow Graph</button>
<div id="chart"></div>
</body>
</html>

see a demo here : http://peerlogic.csc.ncsu.edu/rainbowgraph/developer

JSON Data Format

The JSON configuration that you have to POST to the service have two sections:

(See an example of a valid JSON file at this link, if your system is ranking based. If your system is rating based, you can forego the “student_id” and critic_comparer_vector element in each student, as they are not relevant for you at the moment.)

[{
"metadata": {
    "title": "Intro to Digital Media, Assignment 1",  
    //primary value is the main score (rank average)
     "primary_value_label": "rank average",
     "higher_primary_value_better": false,
    //values array (the individual received scores)
     "values_label": "ranks",
     "best_value_possible": 1, //denotes the color transition from e.g., green (best value) to red (worst value)
     "worst_value_possible": 5, //denotes the color transition from e.g., green (best value) to red (worst value)
     "best_primary_value_possible": 1, //denotes the top range of the Y-axis
     "worst_primary_value_possible": 5, //denotes the bottom range of the Y-axis
     "y_axis_label": "Rank Average",
     "x_axis_label": "Students",
     "color_scheme": 5b,
    //secondary value is just an additional
     "secondary_value_label": "variance",
     "critic_comparer_flag": "yes",
     "self_assess_flag": "yes"
},"data": [{
    // a student's data
   }, {
    // another student's data, see below....
   }]
}]

The first section of the JSON is the “metadata”. Here is where you describe some properties of the graph and the meanings of the numbers in your data section.

The first element in the metadata section is “title”. This is simply the title label drawn at the top of the graph. You can leave this blank if you prefer no title.

"title":"Intro to Digital Media, Assignment 1"

The next entries in the JSON file refer to the “primary-value” data and “values” data that will appear for each column (each student) in the graph. It’s important to understand what these two things mean. To understand why these are important, let’s look at the graph itself.

At it’s core, this graph is simply a bar graph. Each student is represented by a column, and the height of the column is determined by that student’s “primary-value” entry in the JSON file. Think of “primary-value” as their overall aggregate score. This isn’t calculated by the web service but rather is provided by you.

Primary-value determines two things:

• The height of each column
• The order in which the columns are drawn.

In addition, each of these columns consists of a number of smaller elements stacked together. These elements refer to the individual scores that student received from other peers in a peer review session. These individual scores we call simply “values”. They are stored as a list of numbers that can be any length, and can be listed in any order. Some students may have more “values” than other students and that’s OK. The overall column height will always be drawn such that its height corresponds to the “primary-value” provided by you.

Primary-value is not derived automatically from the individual values you submit since your system might have a unique way to calculate this value such as weighted average based on the reviewer competencies. Thus, you have to provide the primary values in addition to the individual values in the data section of the JSON configuration.

In the example above, each student receives a few rank scores from peers, and the “primary-value” is simply the average of those ranks. Your system might use ratings instead of rankings, or if it uses ranks it might use a formula more complex than simple rank average to determine “primary-value”. How you decide to calculate “primary-value” is entirely up to you, however it’s important that you indicate in the JSON metadata whether a higher or a lower primary-value is “better”. You can do this by defining the “best_primary_value_possible” and “worst_primary_value_possible”, which helps us to understand whether the Y-axis should be rendered with an increasing / decreasing scale. These two values will also set the lower and upper bound of the Y-axis

"best_primary_value_possible": 1, 
"worst_primary_value_possible": 5,

Alternatively, you can set “higher_primary_value_better” instead of those two values, which still help us to understand how we should render the Y-axis. However, we will try to determine the lower and upper bound of the scale based on highest (max) and lowest (min) primary values in the data section.

In our case, we set “higher_primary_value_better” to false because a lower rank average is better (the best score is 1, the worst is 5). You can put any label you want. Other labels for primary-value might be things like “cumulative rating”, “average rating” or something else entirely.

"primary_value_label": "rank average",
"higher_primary_value_better": false,

Likewise, we give a label describing what the individual values that comprise a column are. In the example graph above, each student receives a few rank scores from peers, where lower ranks are better than higher ranks. (1 = first place). This is depicted by the color of individual cell in the graph, which changes from “best” to “worst” colors. The color scheme can be seen as explained in the Color Scheme section. To help us map the values to the right color, you have to let us know whether the higher values are better or worst than the lower values. This can be done by setting the “best_value_possible” and “worst_value_possible” elements as shown below.

"best_value_possible": 1, //this is rendered as green cell, when 5b color scheme is used
"worst_value_possible": 5, //this is rendered as red cell, when 5b color scheme is used

Alternatively, you can just use “higher_values_better” element and let the us figure out the best and worst values from the values in the data section. Please note that this method will spread the worst to best colors only to the worst to best values in your data.

In our example, we define “higher_values_better” as false because actually lower numbers are better.

"values_label": "ranks",
"higher_values_better": false,

Next you can set a label for the x and the y axis, or simply leave them blank.

"y_axis_label": "Rank Average",
"x_axis_label": "Students",

The “color_scheme” element allow you to change the colors of the rectangles that represent student’s scores by entering the color scheme code as defined below.

"color_scheme": 5b,

Color Scheme:

“2a”:[“#636363“,”#bdbdbd“],
“2b”:[“#99d594“,”#fc8d59“],
“2c”:[“#91bfdb“,”#fc8d59“],
“2d”:[“#31a354“,”#a1d99b“],
“2e”:[“#e6550d“,”#a1d99b“],
“2f”:[“#3182bd“,”#9ecae1“],
“3a”:[“#636363“,”#f0f0f0“,”#bdbdbd“],
“3b”:[“#91cf60“,”#fee090“,”#fc8d59“],
“3c”:[“#3288bd“,”#99d594“,”#fc8d59“],
“3d”:[“#238b45“,”#66c2a4“,”#b2e2e2“],
“3e”:[“#fb6a4a“,”#fcae91“,”#fee5d9“],
“3f”:[“#2171b5“,”#6baed6“,”#bdd7e7“],
“4a”:[“#252525“,”#636363“,”#969696“,”#bdbdbd“],
“4b”:[“#91cf60“,”#d9ef8b“,”#fee08b“,”#fc8d59“],
“4c”:[“#4575b4“,”#91bfdb“,”#fc8d59“,”#d73027“],
“4d”:[“#006d2c“,”#31a354“,”#74c476“,”#a1d99b“],
“4e”:[“#fb6a4a“,”#fc9272“,”#fcbba1“,”#fee5d9“],
“4f”:[“#3182bd“,”#6baed6“,”#9ecae1“,”#c6dbef“],
“5a”: [“#252525“,”#525252“,”#737373“,”#969696“,”#bdbdbd“],
“5b”: [“#1a9850“,”#a6d96a“,”#d9ef8b“,”#fdae61“,”#f46d43“],
“5c”: [“#4575b4“,”#abd9e9“,”#fee090“,”#fdae61“,”#f46d43“],
“5d”: [“#00441b“,”#238b45“,”#41ab5d“,”#74c476“,”#c7e9c0“],
“5e”: [“#67000d“,”#a50f15“,”#ef3b2c“,”#fc9272“,”#fee0d2“],
“5f”: [“#08306b“,”#08519c“,”#4292c6“,”#9ecae1“,”#deebf7“],
“5x”: [“rgb(178,220,143)“,”rgb(235,241,149)“,”rgb(255,247,154)“,”rgb(254,215,125)“,”rgb(252,99,86)“],
“5z”: [“#d7191c“,”#fdae61“,”#ffffbf“,”#fc9272“,”#fee0d2“],
“7b”: [“#1A9850“,”#1FA728“,”#51B625“,”#92C62C“,”#D5D033“,”#E4A13A“,”#F46C43“],
“11b”: [“#1A9850“,”#1DA138“,”#22AA20“,”#45B324“,”#6BBC28“,”#93C62C“,”#BDCF30“,”#D8C634“,”#E1AA39“,”#EA8C3E“, “#F46B43“]

Finally, we left room for a “secondary_value_label” in order to show some other data to the students when they roll over the columns in the display. This secondary value might have something to do with the variance in the values they received from peers, or the uncertainty level in calculating their primary-value. It may be something altogether different, such as their overall current percentile standing in the course. You can choose to ignore this “secondary-value” for the students, and ignore the “secondary_value_label”.

"secondary_value_label": "Variance",

The last two elements,”critic_comparer_flag” and”self_assess_flag” are flags to activate the new features. Now, the graph is able to show ranking data with more details. When you mouse over a rectangle that shows the ranking that a student gets, the graphs can show the peer who performed the assessment (marked with a purple circle), and the peers that this student was compared with (marked with blue circles). To enable this function, you have to set “critic_comparer_flag” to “yes”, and provide detailed information in the data section.

The second feature “self_assess_flag” is used to display the score that the student estimated for their own performance, we called it simply the self-assessment data. To enable this, you have to set “self_assess_flag” to “yes” in the metadata, and add an “self_assess_value” element for each student with their self assessment score.

"critic_comparer_flag": "yes",
"self_assess_flag": "yes"

Data

With the metadata section finished, the rest of the JSON file will consist of elements, one per students, in the following format.

[{
"metadata": {
   //metadata.. see example above
},"data": [{
"first_name": "John",
 "last_name": "Doe",
 "column_url": "http://localhost:3000/assignments/584/responses/42846/showcrit?crit_assignment_id=102",
 "primary_value": 1.0,
 "secondary_value": 0.0,
 "values": [1, 1, 1, 1, 1], 
 "self_assess_value": 3,
 "student_id": 744,
 "critic_comparer_vector": [{
    "rank": 1,
    "critic_id": 1222,
    "critic_peers": [679, 744, 708, 1360, 724]
   }, {
    "rank": 1,
    "critic_id": 797,
    "critic_peers": [1238, 81, 1245, 679, 744]
   }, {
    "rank": 1,
    "critic_id": 1223,
    "critic_peers": [1225, 679, 744, 708, 1360]
   }, {
    "rank": 1,
    "critic_id": 1249,
    "critic_peers": [744, 708, 1360, 1222, 724]
   }, {
    "rank": 1,
    "critic_id": 1252,
    "critic_peers": [1238, 1245, 679, 744, 708]
   }]
  },{ //another student's data }
 ]
}]

The “self_assess_flag” attribute is used for showing self assessment data that is depicted as a dark grey rounded rectangle on each column

"self_assess_value": 3,

To enable the graph showing more detailed information about how a student was ranked, you have to provide the needed information that is the student_id, and the critic_comparer_vector element that contains the id of the reviewer and the ID of the peers that this student was compared against. The critic_comparer_vector element include (1) “rank”, which is the rank that this student gets compared to his/her peers. (2) “critic_id”, which is the ID of the student who performed the assessment and ranked the students in that group. (3) “critic_peers”, which is an array of student_ids, belong to the other students (peers) to whom this student was compared against.

 "student_id": 744,
 "critic_comparer_vector": [
 {
 "rank": 1,
 "critic_id": 1222,
 "critic_peers": [ 679, 744, 708, 1360, 724 ]
 }, 
 { .. }]