View source for CSC/ECE 517 Fall 2018/E1879 Student Generated Questions Added To Rubric

'''Introduction'''
----
The rubrics in expertiza are created by instructors. These Rubrics only contain questions that are related to the existed topics. Now, Expertiza only supports students to pull questionnaires is specific rubrics to get some help. Even that they can ask about anything that is relevant to all the projects that will be submitted But when students encounter difficult problems which are not in the existed topics, and they want to get special advice on that new field, the demand for creating supplementary review questionnaire raises. This project (E1879) aims to solve this problem by allowing students to add questions to the standard instructor generated rubric so that they can get specific feedback on from the reviewers.


'''Proposed solution'''
----

We will add the Supplementary Review Questions to the current Review Questions, and show these student-generated questions under the rubric given by instructor. Even though these questions won't be graded, they will eventually provide diversity for feedback. It will increase the benefit that each team gets because they can get feedback that is specific to their project. 


'''Implementation'''
----

''Design Strategy''

In Expertiza, all kinds of rubrics and surveys are subclasses of Questionnaire. A Questionnaire can contain “questions” of several types (e.g., checkboxes, dropdowns, text boxes). So that we will add a new subclass of Questionnaire named SupplementaryReviewQuestionnaire to achieve our goal.
 
First, we want the professor to control if the students can create their own supplementary questions by using a checkbox to indicate. If the professor allow students to create their own questions, there will be a button in the students "Your work" page which can let the students jump to the same page that an instructor lands on when creating a new rubric. Then we will add the questionnaire id of this new SupplementaryReviewQuestionnaire to the Team table when students create their own questionnaire in order to show the corresponding supplementary questions to the reviewers. And when showing review questions to the reviewers we will look up the questionnaire id in Team table and modify the view to add the supplementary questions to review page. Finally, after the reviewers finish the review and submit their responds, we will change the view and let the team member can see the feedback of their supplementary questions.

''Changes in the User Interface''

1. Assignment Page

We add a checkbox for the professor to indicate if student can add their own questions.

[[File:Yzhao47_1.png]]
   
2. A button called "Create Supplementary Rubric" will appear in the student's "Your Work" section when the Instructor has allowed students to create supplementary review questions. So when a student clicks the button he/she can add the desired questions.

[[File:Yzhao47_2.png]]

3. The page where student will be directed to when he/she clicks the Create Supplementary Rubric button. Students can add Questions to the created Supplementary Review questionnaire here.

[[File:Yzhao47_3.png]]

4. Review Page

The students own supplementary questions are leading by a heading "Supplementary Review Questions". 

[[File:Yzhao47_4.png]]

5. Review Results Page

The supplementary review results will be showed to the team like other results. 

[[File:Yzhao47_5.png]]

 
'''Use Cases'''
----

1. Use case of creating a Supplementary Review Questionnaire

[[File:case1.jpg]]

2. Use case of reviewing questions for reviewers.

[[File:case2.jpg]]

3. Use case of reviewing responses of questions.

[[File:case3.jpg]]





'''Test Plan'''
----
Actually, we need to add tests for the following:

1. To check the link for "Supplementary Review Questionnaire" appears in the "Your Work" section of a student.

2. To check if the link for "Supplementary Review Questionnaire" redirects to page which allows to create questionnaire.

3. To check if the questionnaire ID is stored in the Team table.

4. To check if the reviewers can see the supplementary questions that were added by the team as part of the review questions.

5. To check if the responses of the Supplementary Review Questions have been added to the responses of the existing review questions.



'''File which are added/modified'''
----

Note: Because the pull request contains some code irrelevant to this project which make it confusing, we post our code here for you to read clearly.

1. Models

app/models/team.rb

Look up the questionnaire ID of a team.

<pre>
  def self.get_supplementary_review_questionnaire_id_of_team(team_id)
    team = Team.find(team_id)
    if team.blank?
      nil
    else
      team.supplementary_review_questionnaire_id
    end
  end
</pre>

app/models/supplementary_review_questionnaire.rb

Add the SupplementaryReviewQuestionnaire class which is the subclass of Questionnaire.

<pre>

 # This is one type of Questionnaire and  as intuitively expected this model class
 # derives from Questionnaire.
 class SupplementaryReviewQuestionnaire < Questionnaire
   # Make me better.
   attr_accessible
 end
</pre>

app/models/response.rb

Add supplementary review responses to "Show Review" section.

<pre>
  def construct_review_response code, self_id, show_tags = nil, current_user = nil
    code += '<table id="review_' + self_id + '" style="display: none;" class="table table-bordered">'
    answers = Answer.where(response_id: self.response_id)
    team_id = response_map.reviewee_id
    supplementary_review_questionnaire_id = Team.get_supplementary_review_questionnaire_id_of_team(team_id)
    unless answers.empty?
      questionnaire = self.questionnaire_by_answer(answers.first)

      unless supplementary_review_questionnaire_id.nil?
        supplementary_review_questionnaire = Questionnaire.find(supplementary_review_questionnaire_id)
        supplementary_review_questions = supplementary_review_questionnaire.questions.sort_by(&:seq)
      end

      questionnaire_max = questionnaire.max_question_score
      questions = questionnaire.questions.sort_by(&:seq)

      unless supplementary_review_questionnaire.nil?
        questions += supplementary_review_questions
      end

      # get the tag settings this questionnaire
      tag_prompt_deployments = show_tags ? TagPromptDeployment.where(questionnaire_id: questionnaire.id, assignment_id: self.map.assignment.id) : nil
      code = add_table_rows questionnaire_max, questions, answers, code, tag_prompt_deployments, current_user
    end
    comment = if !self.additional_comment.nil?
                self.additional_comment.gsub('^p', '').gsub(/\n/, '<BR/>')
              else
                ''
              end
    code += '<tr><td><b>Additional Comment: </b>' + comment + '</td></tr>'
    code += '</table>'
  end
</pre>

2. Views:

app/views/assignments/edit/_rubrics.html.erb

Add checkbox which professor used to indicate whether students can create their own questions. 

<pre>
 <!--Add a allow studuent-generated questionnaires checkbox. -->
 <input name="assignment_form[assignment][supplementary_review_questionnaire]" type="hidden" value="false"/>
 <%= check_box_tag('assignment_form[assignment][supplementary_review_questionnaire]', 'true', @assignment_form.assignment.supplementary_review_questionnaire) %>
 <%= label_tag('assignment_form[assignment][supplementary_review_questionnaire]', 'Allow student-generated questions added to rubric?') %>
</pre>

app/views/response/response.html.erb

Add code to show the supplementary questions to the reviewers.

<pre>
      <br>
       <% unless @supplementary_review_questions.nil? %>
       <tr>
         <td>
           <h2>Supplementary Review Questions</h2>
         </td>
       </tr>
         <% @supplementary_review_questions.each do |question| %>
         <tr>
           <td>
             <% answer = Answer.where(question_id: question.id, response_id: @response.id).first if !@response.nil? %>
             <% if question.instance_of? Criterion %>
               <%= question.complete(i, answer, @questionnaire.min_question_score, @questionnaire.max_question_score, @dropdown_or_scale) %>
               <%= tinymce %>
             <% elsif question.instance_of? Scale %>
               <%= question.complete(i, answer, @questionnaire.min_question_score, @questionnaire.max_question_score) %>
               <%= tinymce %>
             <% elsif question.instance_of? UploadFile %>
             <% else %>
               <%= question.complete(i, answer) %>
             <% end %>
           </td>
           <td width="25%">
             <% if [Criterion, TextArea].include? question.class %>
                 <div id="analysis_<%= i.to_s %>_comments"></div>
             <% end %>
           </td>
         </tr>
         <% i += 1 %>
         <% end %>
       <% end %>
</pre>

app/views/submitted_content/_main.html.erb

Add button for "your work" page to let students create their own questions.

<pre>
<% if stage != "Finished" and controller.controller_name == 'submitted_content' and @can_submit %>
  <h2>Add Supplementary Review Questions:</h2>
    <% if @assignment.supplementary_review_questionnaire? %>
      <%= button_to "Create Supplementary Rubric", {:controller => "questionnaires", :action => "create_supplementary_review_questionnaire", :id => participant.id } %>
    <% else %>
      The instructor hasn't enabled this feature for this assignment
    <% end %>
<% end %>
</pre>

3. Controllers:

app/controllers/response_controller.rb

Add supplementary review questionnaires part to the create, edit, update methods and some other methods which getting the questionnaires.  

<pre>
  # Prepare the parameters when student clicks "Edit"
  def edit
    assign_instance_vars
    get_all_responses
    if @prev.present?
      @sorted = @review_scores.sort {|m1, m2| m1.version_num.to_i && m2.version_num.to_i ? m2.version_num.to_i <=> m1.version_num.to_i : (m1.version_num ? -1 : 1) }
      @largest_version_num = @sorted[0]
    end
    @modified_object = @response.response_id
    # set more handy variables for the view
    set_content
    @review_scores = []
    @questions.each do |question|
      @review_scores << Answer.where(response_id: @response.response_id, question_id: question.id).first
    end

    @supplementary_review_questions.each do |question|
      @review_scores << Answer.where(response_id: @response.response_id, question_id: question.id).first
    end

    @questionnaire = set_questionnaire
    render action: 'response'
  end

  # Update the response and answers when student "edit" existing response
  def update
    render nothing: true unless action_allowed?
    # the response to be updated
    @response = Response.find(params[:id])
    msg = ""
    begin
      @map = @response.map
      @response.update_attribute('additional_comment', params[:review][:comments])

      set_questionnaire
      questions = sort_questions(@questionnaire.questions)
      supplementary_review_questions = sort_questions(@supplementary_review_questionnaire.questions)
      unless @supplementary_review_questionnaire.nil?
        questions += supplementary_review_questions
      end
      create_answers(params, questions) unless params[:responses].nil?

      @response.update_attribute('is_submitted', true) if params['isSubmit'] && params['isSubmit'] == 'Yes'
      @response.notify_instructor_on_difference if (@map.is_a? ReviewResponseMap) && @response.is_submitted && @response.significant_difference?
    rescue StandardError
      msg = "Your response was not saved. Cause:189 #{$ERROR_INFO}"
    end
    ExpertizaLogger.info LoggerMessage.new(controller_name, session[:user].name, "Your response was submitted: #{@response.is_submitted}", request)
    redirect_to controller: 'response', action: 'save', id: @map.map_id,
                return: params[:return], msg: msg, review: params[:review], save_options: params[:save_options]
  end

 def create
    map_id = params[:id]
    map_id = params[:map_id] unless params[:map_id].nil? # pass map_id as a hidden field in the review form
    @map = ResponseMap.find(map_id)
    @team_id = @map.reviewee_id

    get_all_responses
    if params[:review][:questionnaire_id]
      @questionnaire = Questionnaire.find(params[:review][:questionnaire_id])
      @round = params[:review][:round]
      @supplementary_review_questionnaire_id = Team.get_supplementary_review_questionnaire_id_of_team(@team_id)
      unless @supplementary_review_questionnaire_id.nil?
        @supplementary_review_questionnaire = Questionnaire.find(@supplementary_review_questionnaire_id)
      end
    else
      @round = nil
    end
    is_submitted = (params[:isSubmit] == 'Yes')
    was_submitted = false
    @response = Response.where(map_id: @map.id, round: @round.to_i).first
    if @response.nil?
      @response = Response.create(
        map_id: @map.id,
        additional_comment: params[:review][:comments],
        round: @round.to_i,
        is_submitted: is_submitted
      )
    end
    was_submitted = @response.is_submitted
    @response.update(additional_comment: params[:review][:comments], is_submitted: is_submitted) # ignore if autoupdate try to save when the response object is not yet created.

    # ,:version_num=>@version)
    # Change the order for displaying questions for editing response views.
    questions = sort_questions(@questionnaire.questions)

    unless @supplementary_review_questionnaire.nil?
      supplementary_review_questions = sort_questions(@supplementary_review_questionnaire.questions)
      questions += supplementary_review_questions
    end

    create_answers(params, questions) if params[:responses]
    msg = "Your response was successfully saved."
    error_msg = ""
    # only notify if is_submitted changes from false to true
    if (@map.is_a? ReviewResponseMap) && (was_submitted == false && @response.is_submitted) && @response.significant_difference?
      @response.notify_instructor_on_difference
      @response.email
    end
    redirect_to controller: 'response', action: 'save', id: @map.map_id,
                return: params[:return], msg: msg, error_msg: error_msg, review: params[:review], save_options: params[:save_options]
  end


  # new_response if a flag parameter indicating that if user is requesting a new rubric to fill
  # if true: we figure out which questionnaire to use based on current time and records in assignment_questionnaires table
  # e.g. student click "Begin" or "Update" to start filling out a rubric for others' work
  # if false: we figure out which questionnaire to display base on @response object
  # e.g. student click "Edit" or "View"
  def set_content(new_response = false)
    @title = @map.get_title
    if @map.survey?
      @survey_parent = @map.survey_parent
    else
      @assignment = @map.assignment
    end
    @participant = @map.reviewer
    @contributor = @map.contributor
    new_response ? set_questionnaire_for_new_response : set_questionnaire
    set_dropdown_or_scale
    @questions = sort_questions(@questionnaire.questions)

    unless @supplementary_review_questionnaire.nil?
      @supplementary_review_questions = sort_questions(@supplementary_review_questionnaire.questions)
    end

    @min = @questionnaire.min_question_score
    @max = @questionnaire.max_question_score
  end

  def set_questionnaire_for_new_response
    case @map.type
    when "ReviewResponseMap", "SelfReviewResponseMap"
      reviewees_topic = SignedUpTeam.topic_id_by_team_id(@contributor.id)
      @current_round = @assignment.number_of_current_round(reviewees_topic)
      @questionnaire = @map.questionnaire(@current_round)

      @supplementary_review_questionnaire_id = Team.get_supplementary_review_questionnaire_id_of_team(@contributor.id)
      unless @supplementary_review_questionnaire_id.nil?
        @supplementary_review_questionnaire = Questionnaire.find(@supplementary_review_questionnaire_id)
      end

    when
      "MetareviewResponseMap",
      "TeammateReviewResponseMap",
      "FeedbackResponseMap",
      "CourseSurveyResponseMap",
      "AssignmentSurveyResponseMap",
      "GlobalSurveyResponseMap"
      @questionnaire = @map.questionnaire
    end
  end

  def set_questionnaire
    # if user is not filling a new rubric, the @response object should be available.
    # we can find the questionnaire from the question_id in answers
    answer = @response.scores.first
    @questionnaire = @response.questionnaire_by_answer(answer)

    supplementary_review_answer = @response.scores.last
    @supplementary_review_questionnaire = @response.questionnaire_by_answer(supplementary_review_answer)

  end

</pre>


app/controllers/grades_controller.rb

Add supplementary review responses to "Your Scores" section.

<pre>
  def view_team
    @participant = AssignmentParticipant.find(params[:id])
    @assignment = @participant.assignment
    @team = @participant.team
    @team_id = @team.id
    @questions = {}
    questionnaires = @assignment.questionnaires
    retrieve_questions questionnaires
    @pscore = @participant.scores(@questions)
    @vmlist = []

    # loop through each questionnaire, and populate the view model for all data necessary
    # to render the html tables.
    counter_for_same_rubric = 0
    questionnaires.each do |questionnaire|
      @round = nil
      if @assignment.varying_rubrics_by_round? && questionnaire.type == "ReviewQuestionnaire"
        questionnaires = AssignmentQuestionnaire.where(assignment_id: @assignment.id, questionnaire_id: questionnaire.id)
        if questionnaires.count > 1
          @round = questionnaires[counter_for_same_rubric].used_in_round
          counter_for_same_rubric += 1
        else
          @round = questionnaires[0].used_in_round
          counter_for_same_rubric = 0
        end
      end
      vm = VmQuestionResponse.new(questionnaire, @assignment, @round)
      supplementary_review_questionnaire_id = Team.get_supplementary_review_questionnaire_id_of_team(@team_id)
      unless supplementary_review_questionnaire_id.nil?
        supplementary_review_questionnaire = Questionnaire.find(supplementary_review_questionnaire_id)
        unless supplementary_review_questionnaire.nil?
          supplementary_review_questions = supplementary_review_questionnaire.questions
          questionnaire.questions += supplementary_review_questions
        end
      end
      vmquestions = questionnaire.questions
      vm.add_questions(vmquestions)
      vm.add_team_members(@team)
      vm.add_reviews(@participant, @team, @assignment.varying_rubrics_by_round?)
      vm.get_number_of_comments_greater_than_10_words
      @vmlist << vm
    end
    @current_role_name = current_role_name
  end
</pre>

app/controllers/questionnaires_controller.rb

Method for creating Supplementary Review Questionnaire.

<pre>
   def create_supplementary_review_questionnaire
     @participant = AssignmentParticipant.find(params[:id])
     @team = Team.find(@participant.team.id)
     if @team.supplementary_review_questionnaire_id.nil?
       @questionnaire = Questionnaire.create(privete: false, name: "supplementary_review_questionnaire_" + @team.id.to_s,
         instructor_id: @team_id, min_question_score: 0, max_question_score: 5, type: "Questionnaire", display_type: "Review"
         instruction_loc: Questionnaire::DEFAULT_QUESTIONNAIRE_URL)
       if @questionnaire.save
         @team.supplementary_review_questionnaire_id = @questionnaire.id
         @team.save
         flash[:success] = 'You have successfully created a rubric!'
       else
         flash[:error] = $ERROR_INFO
       end
     else
       @questionnaire = Questionnaire.find(@team.supplementary_review_questionnaire_id)
     end
     redirect_to controller: 'questionnaires', action: 'edit', id: @questionnaire.id
   end
</pre>

4. Config:

routes.rb
<pre>

  resources :questionnaires, only: %i[new create edit update] do
    collection do
      get :copy
      get :list
      post :list_questionnaires
      get :new_quiz
      get :select_questionnaire_type
      post :select_questionnaire_type
      get :toggle_access
      get :view
      post :create_quiz_questionnaire
      post :update_quiz
      post :add_new_questions
      post :save_all_questions
      post :create_supplementary_review_questionnaire
    end
  end

  resources :author_feedback_questionnaires, controller: :questionnaires
  resources :review_questionnaires, controller: :questionnaires
  resources :metareview_questionnaires, controller: :questionnaires
  resources :teammate_review_questionnaires, controller: :questionnaires
  resources :survey_questionnaires, controller: :questionnaires
  resources :assignment_survey_questionnaires, controller: :questionnaires
  resources :global_survey_questionnaires, controller: :questionnaires
  resources :course_survey_questionnaires, controller: :questionnaires
  resources :bookmarkrating_questionnaires, controller: :questionnaires
  resources :supplementary_review_questionnaire, controller: :questionnaires
</pre>

'''Team members'''
----
Yanchen Zhao

Pingping Chen

Jianshu Zhang

Zirun Han


'''Links'''
----
Expertiza on Github:https://github.com/zyczyh/expertiza

Pull Request:https://github.com/expertiza/expertiza/pull/1325

Expertiza YouTube channel:https://youtu.be/3PUNknSbU-k