Report generation framework(WIP) - Revision history

Asreeku: Created page with "= Report Generation = == Design and Implementation Documentation == === Expertiza Reimplementation Back-End === ---- == Overview == The reporting subsystem was ported from the original Expertiza codebase (referred to as '''Repo X''') into the reimplemented Rails API back-end (referred to as '''Repo Y''') and redesigned in the process. Repo X used a Rails helper module (`ReportFormatterHelper`) that assigned instance variables, such as @reviewers 2026-05-02T21:49:32Z Created page with "= Report Generation = == Design and Implementation Documentation == === Expertiza Reimplementation Back-End === ---- == Overview == The reporting subsystem was ported from the original Expertiza codebase (referred to as '''Repo X''') into the reimplemented Rails API back-end (referred to as '''Repo Y''') and redesigned in the process. Repo X used a Rails helper module (<code>ReportFormatterHelper</code>) that assigned instance variables, such as <code>@reviewers</cod..." New page = Report Generation = == Design and Implementation Documentation == === Expertiza Reimplementation Back-End === ---- == Overview == The reporting subsystem was ported from the original Expertiza codebase (referred to as '''Repo X''') into the reimplemented Rails API back-end (referred to as '''Repo Y''') and redesigned in the process. Repo X used a Rails helper module (<code>ReportFormatterHelper</code>) that assigned instance variables, such as <code>@reviewers</code> and <code>@review_scores</code>, for ERB views. Since Repo Y is a JSON API with no views, instance variables are not applicable. The architecture was therefore redesigned around a '''streaming reduce pipeline''' that returns plain Ruby hashes rendered as JSON. === Design Goals === * Avoid loading entire result sets into memory at once. * Keep domain-specific computation out of the base class. * Make each report type independently composable and testable. * Fix N+1 query patterns present in Repo X. ---- == Anti-Patterns Addressed == The design directly addresses two anti-patterns identified during architectural review of Repo X. === The <code>fetch_responses</code> Anti-Pattern === Repo X loaded all response records into an unnamed, ad-hoc array before processing: <syntaxhighlight lang="ruby"> # WRONG responses = fetch_responses # full memory load grouped = group(responses) metrics = compute_metrics(grouped) </syntaxhighlight> This forces the entire result set into Ruby memory, prevents streaming, and makes the intermediate structure implicit. The fix is to '''never materialise all rows at once'''. Instead, use <code>find_each</code> to stream records in batches, so memory usage scales with the number of '''groups''', not the number of raw rows. === Default Metrics in the Base Class === Repo X placed domain-specific math directly in the base class: <syntaxhighlight lang="ruby"> # WRONG — in BaseReport def compute_metrics(grouped) grouped.transform_values do |responses| { count: responses.size, avg_score: responses.map(&:score).sum / responses.size.to_f } end end </syntaxhighlight> This ties every subclass to one particular shape of computation. The fix is to move all domain math into per-report accumulator logic. The base class contains '''zero''' domain math. ==== What "Domain Math" Means ==== '''Domain math''' refers to the business-logic calculations specific to a particular report type — the actual formulas and aggregations that answer what the report is trying to show. It is called "domain" math because it belongs to the problem domain (peer assessment), not to the generic pipeline machinery. Each report type has its own domain math: {| class="wikitable" ! Report !! Its domain math |- | Review scores | <code>(raw_score / max_score) * 100</code> — percentage score per reviewer per round |- | Avg/ranges | <code>max</code>, <code>min</code>, <code>sum / size</code> — score aggregates across a team's reviewers |- | Feedback | Bucket response IDs into <code>round_1</code>, <code>round_2</code>, <code>round_3</code> arrays |- | Bookmark rating | Collect distinct bookmark IDs into a <code>Set</code> |} Notice that these are completely different in shape: one computes a percentage, another computes min/max/avg, and another just collects IDs. If <code>avg_score</code> lived in <code>BaseReport</code>, every subclass would either inherit math it does not need — for example, a bookmark report has no scores — or be forced to override the method just to suppress it. The rule is therefore: > <code>BaseReport</code> only defines '''how to run the pipeline''' — stream, group, fold, finalize. > The '''what to compute''' belongs entirely inside each report's own <code>accumulate</code> and <code>finalize</code> methods. ---- == System Diagrams [To be updated] == === Overall Request Flow === The diagram below shows how an incoming HTTP request travels through the system from the controller down to the pipeline and back out as JSON. Since MediaWiki does not render TikZ directly, this diagram is represented as a text-based flow. <syntaxhighlight lang="text"> HTTP Client (Front-end) | | GET /reports/response_report?assignment_id=&type= v ReportsController#response_report | | params[:type] v REPORT_CLASSES[type] look up concrete class | | .new(assignment).run v Concrete Report for example, FeedbackReport | | inherits run v BaseReport#run inherited find_each streaming loop | | calls subclass methods v source -> grouper -> accumulate -> finalize | | output hash v render json: { ... } | | JSON response v HTTP Client (Front-end) </syntaxhighlight> ''Figure: End-to-end request flow for report generation'' === Pipeline Internals === This diagram shows the four stages inside <code>BaseReport#run</code>. The stages are defined by each concrete subclass; the pipeline loop itself never changes. <syntaxhighlight lang="text"> +------------------+ +------------------+ | 1. Source | rows | 2. Grouper | | AR relation | -----> | lambda: row->key | | streamed via | | e.g. reviewer_id | | find_each | +------------------+ +------------------+ | | key, row v +------------------+ +------------------+ | 4. Finalize | state | 3. Accumulate | | shape state into | <----- | fold row into | | output hash | | state | +------------------+ | domain math here | | +------------------+ | v Hash -> JSON </syntaxhighlight> Additional notes: * <code>source</code> uses <code>includes(...)</code> where needed to avoid N+1 queries. * <code>accumulate</code> handles scores, deduplication, bucketing, and counting depending on the report type. ''Figure: The four stages every report passes through inside BaseReport#run'' === Class Hierarchy === This diagram shows how concrete report classes relate to <code>BaseReport</code>. Solid inheritance from <code>BaseReport</code> is represented by indentation. Composition from <code>ReviewReport</code> to its inner pipelines is represented under the coordinator. <syntaxhighlight lang="text"> BaseReport | |-- ReviewReport (coordinator) | | | |-- ReviewersPipeline | |-- ScoresPipeline | |-- AvgRangesPipeline | |-- FeedbackReport |-- TeammateReviewReport |-- BookmarkRatingReport |-- BasicReport </syntaxhighlight> Legend: * Direct child under <code>BaseReport</code> = inherits <code>BaseReport</code> * Pipelines under <code>ReviewReport</code> = coordinator runs inner pipeline * The inner pipelines also inherit <code>BaseReport</code> ''Figure: Class hierarchy for the report generation subsystem'' ---- == Architecture: The Streaming Pipeline == === Pipeline Shape === All reports are built on a single pipeline template defined in <code>Reports::BaseReport</code>: <syntaxhighlight lang="ruby"> def run state = initial_state source.find_each(batch_size: 500) do |row| accumulate(state, grouper.call(row), row) end finalize(state) end </syntaxhighlight> The pipeline consists of four concerns: {| class="wikitable" ! Concern !! Responsibility |- | '''Source''' | ActiveRecord relation streamed via <code>find_each</code>. Subclasses use <code>includes(...)</code> to eagerly load associations and prevent N+1 queries. |- | '''Grouper''' | A lambda <code>(row) -> key</code> that determines how rows are bucketed in the accumulator state. |- | '''Accumulate''' | Folds one row into the state in place. Contains all domain-specific math for that report. |- | '''Finalize''' | Post-processes the finished state into the output hash. Default implementation returns state unchanged. |} === Memory Model === State grows proportional to the number of '''groups''', such as the number of distinct reviewers, not the number of raw rows. For example, a dataset with 10,000 responses across 20 reviewers keeps only 20 entries in the accumulator state at any point. === Base Class Definition === <syntaxhighlight lang="ruby"> module Reports class BaseReport def initialize(assignment) @assignment = assignment end def run state = initial_state source.find_each(batch_size: 500) do |row| accumulate(state, grouper.call(row), row) end finalize(state) end private def source = raise NotImplementedError def grouper = raise NotImplementedError def initial_state = raise NotImplementedError def accumulate(_state, _key, _row) raise NotImplementedError end def finalize(state) = state end end </syntaxhighlight> ---- == Controller == === Dispatch Mechanism === The controller uses a constant hash to map type strings to report classes, replacing the <code>send(type)</code> meta-programming pattern used in Repo X. <syntaxhighlight lang="ruby"> REPORT_CLASSES = { 'review_response_map' => Reports::ReviewReport, 'feedback_response_map' => Reports::FeedbackReport, 'teammate_review_response_map' => Reports::TeammateReviewReport, 'bookmark_rating_response_map' => Reports::BookmarkRatingReport, 'basic' => Reports::BasicReport }.freeze def response_report type = params.dig(:report, :type) || params[:type] || 'basic' report_class = REPORT_CLASSES[type] unless report_class return render json: { error: "Unknown report type: #{type}" }, status: :unprocessable_entity end assignment = Assignment.find(params[:assignment_id] || params[:id]) data = report_class.new(assignment).run render json: { type: type, assignment_id: assignment.id }.merge(data) end </syntaxhighlight> === Route === <syntaxhighlight lang="text"> GET /reports/response_report?assignment_id=<id>&type=<type> POST /reports/response_report </syntaxhighlight> ---- == Report Implementations == === Review Report (<code>review_response_map</code>) === The review report is the most complex. It is implemented as a coordinator class (<code>ReviewReport</code>) that runs three independent inner pipelines and merges their results. {| class="wikitable" ! Pipeline !! Source !! Groups by !! Produces |- | <code>ReviewersPipeline</code> | <code>ReviewResponseMap</code> | <code>reviewer_id</code> | Sorted reviewer list |- | <code>ScoresPipeline</code> | <code>Response</code> JOIN map | <code>reviewer_id</code> | Score percentage per round/reviewee |- | <code>AvgRangesPipeline</code> | <code>Response</code> JOIN map | <code>[reviewee_id, round]</code> | Max/min/avg per team/round |} ==== N+1 Fix: Precomputed Max Question Score ==== Repo X called <code>response.maximum_score</code> inside the accumulation loop. This method internally calls: <syntaxhighlight lang="ruby"> response_assignment.assignment_questionnaires .find_by(used_in_round: round) .questionnaire </syntaxhighlight> This resulted in one query per response. In Repo Y, both score pipelines precompute a <code>round -> max_question_score</code> map with a single query before the pipeline runs: <syntaxhighlight lang="ruby"> def precompute_max_q_scores AssignmentQuestionnaire .joins(:questionnaire) .where(assignment_id: @assignment.id) .pluck(:used_in_round, 'questionnaires.max_question_score') .to_h end </syntaxhighlight> The result, for example <code>{nil => 10, 1 => 10, 2 => 5}</code>, is stored in <code>@max_q_score</code> and used as a lookup inside <code>accumulate</code>: <syntaxhighlight lang="ruby"> max_score = total_wt * (@max_q_score[round] || @max_q_score[nil] || 1) </syntaxhighlight> ==== Sample Response ==== <syntaxhighlight lang="json"> { "type": "review_response_map", "assignment_id": 1, "reviewers": [ { "id": 5, "user_id": 2, "name": "alice", "full_name": "Alice Smith", "handle": "alice" } ], "review_scores": { "5": { "1": { "3": 87.5 } } }, "avg_and_ranges": { "3": { "1": { "max": 92.0, "min": 75.0, "avg": 83.5 } } } } </syntaxhighlight> ---- === Feedback Report (<code>feedback_response_map</code>) === Produces the list of authors and the IDs of review responses that received author feedback, bucketed by round for varying-rubric assignments. <syntaxhighlight lang="ruby"> def source Response .joins(:response_map) .where( response_maps: { type: 'ReviewResponseMap', reviewed_object_id: @assignment.id } ) .order(created_at: :desc) end def grouper = ->(r) { [r.map_id, r.round] } def initial_state = { seen: Set.new, round_1: [], round_2: [], round_3: [], all: [] } def accumulate(state, key, response) return if state[:seen].include?(key) state[:seen].add(key) if @assignment.varying_rubrics_by_round? case response.round when 1 then state[:round_1] << response.id when 2 then state[:round_2] << response.id when 3 then state[:round_3] << response.id end else state[:all] << response.id end end </syntaxhighlight> Deduplication uses a <code>Set</code>, which gives O(1) lookup, rather than the array-based <code>seen_map_round_keys.include?</code> from Repo X, which gives O(n) lookup. Authors are fetched once in <code>finalize</code>, not inside the stream. ==== End-to-End Execution Flow ==== The <code>run</code> method is '''inherited from <code>BaseReport</code>'''. <code>FeedbackReport</code> never defines it. Calling <code>FeedbackReport.new(assignment).run</code> triggers the following sequence: <syntaxhighlight lang="text"> ReportsController REPORT_CLASSES['feedback_response_map'].new(assignment).run | | (inherited from BaseReport) v BaseReport#run | |-- Step 1: state = initial_state | => { seen: Set.new, | round_1: [], round_2: [], round_3: [], all: [] } | |-- Step 2: source.find_each(batch_size: 500) | => Response.joins(:response_map) | .where(type: 'ReviewResponseMap', | reviewed_object_id: assignment.id) | .order(created_at: :desc) | streams Response records newest-first, in batches | |-- Step 3: for each Response row: | key = grouper.call(row) | => [row.map_id, row.round] e.g. [42, 1] | | accumulate(state, key, row) | => skip if state[:seen] already has [map_id, round] | (keeps only the latest response per map per round | because source is ordered newest-first) | => otherwise: add key to :seen, then bucket row.id: | round == 1 => state[:round_1] << row.id | round == 2 => state[:round_2] << row.id | round == 3 => state[:round_3] << row.id | (or state[:all] if single-rubric assignment) | |-- Step 4: finalize(state) => fetch_authors (one query: teams -> users -> participants) => if varying_rubrics_by_round? return { authors: [...], review_response_ids: { round_1: [...], round_2: [...], round_3: [...] } } else return { authors: [...], review_response_ids: [...] } </syntaxhighlight> The key point is that <code>FeedbackReport</code> only defines the four pieces the pipeline needs: * <code>source</code> * <code>grouper</code> * <code>initial_state</code> * <code>accumulate</code> * <code>finalize</code> Ruby's inheritance mechanism means calling <code>.run</code> on a <code>FeedbackReport</code> instance automatically executes <code>BaseReport#run</code>, which calls back into <code>FeedbackReport</code>'s implementations of those methods. ==== Sample Response (varying rubrics) ==== <syntaxhighlight lang="json"> { "type": "feedback_response_map", "authors": [{ "id": 7, "name": "bob", "full_name": "Bob Jones" }], "review_response_ids": { "round_1": [12, 15], "round_2": [18], "round_3": [] } } </syntaxhighlight> ---- === Teammate Review Report (<code>teammate_review_response_map</code>) === Streams <code>TeammateReviewResponseMap</code> records grouped by <code>reviewer_id</code>. The first occurrence per reviewer is kept using deduplication via early return if the key already exists in state. Reviewer associations are eagerly loaded. ==== Sample Response ==== <syntaxhighlight lang="json"> { "type": "teammate_review_response_map", "reviewers": [ { "reviewer_id": 5, "user_id": 2, "name": "alice", "full_name": "Alice Smith" } ] } </syntaxhighlight> ---- === Bookmark Rating Report (<code>bookmark_rating_response_map</code>) === Streams <code>BookmarkRatingResponseMap</code> records, accumulating distinct bookmark IDs into a <code>Set</code>. Project topics are fetched once in <code>finalize</code>. ==== Bug Fixed During Port ==== The model's <code>bookmark_response_report</code> in Repo Y was incorrectly calling: <syntaxhighlight lang="ruby"> .pluck(:reviewed_object_id) </syntaxhighlight> This returns assignment IDs, since <code>reviewed_object_id</code> is the foreign key to <code>Assignment</code>. Bookmark IDs are stored in <code>reviewee_id</code>. Therefore, this was fixed to: <syntaxhighlight lang="ruby"> .pluck(:reviewee_id) </syntaxhighlight> ==== Sample Response ==== <syntaxhighlight lang="json"> { "type": "bookmark_rating_response_map", "bookmark_ids": [10, 14, 22], "topics": [{ "id": 3, "topic_name": "Machine Learning" }] } </syntaxhighlight> ---- === Basic Report (<code>basic</code>) === Returns minimal assignment metadata. No streaming is required since all data comes from the already-loaded <code>Assignment</code> object. This report is used as the default when no type parameter is provided. ==== Sample Response ==== <syntaxhighlight lang="json"> { "type": "basic", "assignment_id": 1, "assignment": { "id": 1, "name": "Project 1", "num_review_rounds": 2, "varying_rubrics_by_round": true } } </syntaxhighlight> ---- == File Structure == <syntaxhighlight lang="text"> app/ controllers/ reports_controller.rb Entry point, REPORT_CLASSES dispatch helpers/ report_formatter_helper.rb Empty namespace (logic moved to services) services/ reports/ base_report.rb Abstract pipeline template review_report.rb 3-pipeline coordinator feedback_report.rb Single pipeline, round bucketing teammate_review_report.rb Single pipeline bookmark_rating_report.rb Single pipeline basic_report.rb Simple struct models/ review_response_map.rb .review_response_report class method feedback_response_map.rb .feedback_response_report class method teammate_review_response_map.rb .teammate_response_report class method bookmark_rating_response_map.rb .bookmark_response_report (bug fixed) </syntaxhighlight> ---- == Blocked Report Types == The following report types exist in Repo X but cannot yet be implemented in Repo Y due to missing database tables or models. Each is ready to be added once its dependency is ported. {| class="wikitable" ! Report Type !! Missing Dependency !! Repo X Location |- | <code>calibration</code> | <code>calibrate_to</code> column on <code>response_maps</code> | <code>report_formatter_helper.rb</code> |- | <code>self_review</code> | <code>SelfReviewResponseMap</code> model | <code>self_review_response_map.rb</code> |- | <code>survey</code> | <code>survey_deployments</code> table | <code>survey_response_map.rb</code> |- | <code>quiz</code> | <code>quiz_responses</code> table | <code>quiz_response_map.rb</code> |- | <code>answer_tagging</code> | <code>tag_prompt_deployments</code>, <code>answer_tags</code> tables | <code>tag_prompt_deployment.rb</code> |} To add a blocked report once its dependencies are available: # Create <code>app/services/reports/<name>_report.rb</code> inheriting <code>BaseReport</code>. # Define <code>source</code>, <code>grouper</code>, <code>initial_state</code>, <code>accumulate</code>, and <code>finalize</code>. # Add an entry to <code>ReportsController::REPORT_CLASSES</code>. ---- == Comparison with Repo X == {| class="wikitable" ! Concern !! Repo X !! Repo Y |- | Output format | ERB instance variables, such as <code>@reviewers</code> and <code>@review_scores</code> | JSON hash from <code>report.run</code> |- | Loading strategy | All records loaded into arrays at once | <code>find_each</code> batched streaming |- | Metrics location | <code>compute_metrics</code> in helper base | Each report owns <code>accumulate</code> and <code>finalize</code> |- | Dispatch | <code>send(@type.underscore, params, session)</code> | <code>REPORT_CLASSES[type].new(assignment).run</code> |- | N+1 on scores | <code>response.maximum_score</code> per row — questionnaire lookup each time | Precomputed <code>round->max_score</code> map, one query before pipeline |- | Deduplication | <code>Array#include?</code> — O(n) per check | <code>Set#include?</code> — O(1) per check |- | Additional features | LLM evaluation, CSV export, calibration, self-review, survey, quiz, answer tagging | Not yet ported; blocked on schema |} ---- == Author == {| class="wikitable" ! Name !! Role |- | '''Aanand Sreekumaran Nair Jayakumari''' | Project contributor / developer |}