Set up Canvas to collect student achievement data for outcomes and associated criteria directly from coursework assessment.

1. Courses are identified where assessment will take place.
2. Rubrics are created containing outcomes/criteria.
3. Each assessment task is selected by faculty from coursework and experiences.
4. Students demonstrate their competence in selected courses.
5. Faculty assess the competencies using the scoring criteria in the rubric.
6. Data is collected in Canvas.
7. Data is sent to the enterprise data warehouse.
   1. Data from outcomes will be combined with student data to filter data to requirements. Examples include outcomes based on program/level information or filtering outcomes to specific courses for accreditation vs. program review.
8. Data can be compiled into reports using PowerBI or Cognos with assistance from the Enterprise Data and Analytics (EDA) team.

1. Learning outcome: in the assessment process, it means a specific category of learning or the set of skills or competencies that should be obtained by students through the program.
   • Each time an outcome is assessed that outcome is usually comprised of multiple criteria that provide indicators of achievement.
   • The criteria are often seen as part of a scoring rubric or individual questions on an exam.
   • Each criterion will align with categories for scoring that the program will define.
   • Examples:
     1. Outcome: “Students will be able to communicate effectively in writing”
        1. Criterion used to assess the assignment: Structure is clear, logical, and easy to follow:
           1. 4-Exceeds Mastery
           2. 3-Mastery
           3. 2-Near Mastery
           4. 1-Below Mastery
           5. 0-No Evidence
        2. Criterion used to assess the assignment: Uses correct mechanics such as grammar, spelling, and punctuation:
           1. 4-Exceeds Mastery
           2. 3-Mastery
           3. 2-Near Mastery
           4. 1-Below Mastery
           5. 0-No Evidence
        3. Criterion used to assess the assignment: Effectively incorporates appropriate supporting materials, etc.:
           1. 4-Exceeds Mastery
           2. 3-Mastery
           3. 2-Near Mastery
           4. 1-Below Mastery
           5. 0-No Evidence

1. Create a document outlining the outcomes you need assessed.
   1. This document will be used to copy and paste text into Canvas when adding Outcomes and Criteria.
   2. Include the outcome name and the supporting criteria names and description.
      1. Use a nesting system by using group folders to organize criteria outcomes. An assessment structure can be designed as granular as desired through organizing folders inside of folders in Canvas.
      2. It is important to keep the name of the groups and outcomes short with the descriptive words at the beginning.
         1. If not, it will be challenging when a window truncates the titles, making them all look the same when each begins with “An ability to…”.
         2. A full description can be placed in the outcome statement textbox provided and is available in the data export for use in reporting. 
   3. Use a label and numbering schema to help identify the outcome and related criteria so that when the data is being compiled into a report, the data is easily identified and properly aligned.
      1. The label should reflect the name of the accrediting body or the program name.
      2. When a program is accredited, the number system often comes from the accreditation expectations.
      3. This structure becomes a valuable aspect for aligning the data to accreditation reporting expectations.
      4. Even when a program is not accredited, use a numbering system to make organization easier during analysis.
      5. This document will assist in report creation.
      6. Example of label and numbering schema: Assurance of Learning (AOL)
         1. AOL G1.0 Professional Communication
            1. AOL G1.1 Team Collaboration
               1. AOL G1.1.1 Produced Final Documentation
               2. G1.1.2 Individual Contributions
            2. AOL G1.2 Professional Writing
               1. AOL G1.2.1 Professional Final Document
               2. AOL G1.2.2 Used Supporting Material
               3. AOL G1.2.3 Used Logical Reasoning
            3. AOL G1.3 Professional Speaking
               1. AOL G1.3.1 Organized Information
               2. AOL G1.3.2 Supporting Material
               3. AOL G1.3.3 Confident Delivery 
         2. AOL G2.0 Ethical Behavior
            1. AOL G2.1 Analyze Ethical Behavior
               1. AOL G2.1.1
               2. AOL G2.1.2
               3. AOL G2.1.3
         3. AOL G3.0 Decision Making in a Global Context
            1. AOL G3.1 Brief Description
               1. AOL G3.1.1 Brief Description
            2. AOL G3.2 Brief Description
               1. AOL G3.2.1 Brief Description
               2. AOL G3.2.2 Brief Description
            3. AOL G3.3 Brief Description
               1. AOL G3.3.1 Brief Description
               2. AOL G3.3.2 Brief Description
         4. 0 Use Technology Tools
            1. AOL G4.1 Brief Description
               1. AOL G4.1.1 Brief Description
               2. AOL G4.1.2 Brief Description
               3. AOL G4.1.3 Brief Description
         5. AOL G5.0 Foundational Knowledge of Business Disciplines
            1. AOL G5.1 Brief Description
               1. AOL G5.1.1 Brief Description
      7. Example of label and numbering schema: ACEND DPD-Undergraduate Knowledge Assessment
         1. KRDN 1.0 Scientific and Evidence Base Practice
            1. KRDN 1.1 Demonstrate using professional literature to make decisions
            2. KRDN 1.2 Use info tech to apply guidelines and protocols
         2. KRDN 2.0 Professional Practice Expectations
            1. KRDN 2.1 Oral and Written Communication
            2. KRDN 2.2 Brief Description
            3. KRDN 2.3 Brief Description
            4. KRDN 2.4 Brief Description
            5. KRDN 2.5 Brief Description
            6. KRDN 2.6 Brief Description
            7. KRDN 2.7 Brief Description
            8. KRDN 2.8 Brief Description
            9. KRDN 2.9 Brief Description
         3. KRDN 3.0 Clinical and Client Services
            1. KRDN 3.1 Brief Description
            2. KRDN 3.2 Brief Description
            3. KRDN 3.3 Brief Description
            4. KRDN 3.4 Brief Description
            5. KRDN 3.5 Brief Description
            6. KRDN 3.6 Brief Description
         4. KRND 4.0 Practice Management and Use of Resources
            1. KRDN 4.1 Brief Description
            2. KRDN 4.2 Brief Description
            3. KRDN 4.3 Brief Description
            4. KRDN 4.4 Brief Description
            5. KRDN 4.5 Brief Description
            6. KRDN 4.6 Brief Description
            7. KRDN 4.7 Brief Description
         5. KRND 5.0 Leadership and Career Management
            1. KRDN 5.1 Brief Description
            2. KRDN 5.2 Brief Description
            3. KRDN 5.3 Brief Description
            4. KRDN 5.4 Brief Description
            5. KRDN 5.5 Brief Description
            6. KRDN 5.6 Brief Description
2. Scoring Standard
   1. Illinois State University has adopted a 5-scale scoring standard for rubrics:
      1. 4-Exceeds Mastery
      2. 3-Mastery
      3. 2-Near Mastery
      4. 1-Below Mastery
      5. 0-No Evidence
   2. Example used in COB:
      1. G.1.2.1 Professional Final Document Scoring options:
         1. 4-Exceeds Mastery-Final document followed all professional writing standards of the discipline without error.
         2. 3-Mastery-Minor errors in professional writing standards of the discipline.
         3. 2-Near Mastery-Minor and major errors in some of the professional writing standards of the discipline.
         4. 1-Below Mastery-Major errors in the professional writing standards of the discipline.
         5. 0-No Evidence-No document was submitted.
   3. If your scoring standard is different, you may adjust the number of scores by deleting them when creating the Outcomes.
      1. Example: Accrediting body only requires 2 scores. Use:
         1. 4 points-Exceeds Mastery
         2. 0-No Evidence
         3. Delete the others.
   4. You will need to determine the calculation method when you create the Outcomes.
      1. One of four calculation methods can be used for student mastery: Weighted Average, Decaying Average, n Number of Times, Most Recent Score, and Highest Score.
      2. Descriptions of calculation methods: https://community.canvaslms.com/t5/Admin-Guide/How-do-I-manage-outcome-mastery-calculations-in-an-account/ta-p/429016
   5. Notes about scoring:
      1. Data pulled will only be the outcome scores, not the grades.
      2. The point values are numerical indicators of an achievement level and not necessarily the points used in grading.
      3. Faculty may choose to use the criteria for grading. See Collect Outcome Data in Canvas Courses Job Aids
      4. Failure to use the Illinois State University defined scoring standards will prevent an area’s data from being able to be aggregated or compared with another unit/department.
3. Best practices:
   1. Ensure there is no duplication of outcomes between Institutional, College, or programs.
      1. If so, consider utilizing the higher level (institutional or college level) so that faculty are not forced to provide multiple scores for the similar outcomes.
   2. Review plans with colleagues and academic leadership.
      1. It is important to get them right from the start.
      2. Be sure to share your plans with colleagues and seek the approval of the academic leadership for your unit before publishing anything in Canvas.
      3. Once outcomes or rubrics are created at the account level, they are immediately visible and available for use by instructors in your unit.
      4. If an outcome group has been created and needs to be edited or deleted, you can do so as long as the outcomes in the group have not yet been used to assess a student’s work.
         1. Once an outcome in the group has been used for scoring:
            1. You cannot delete the outcome group.
            2. You can still modify the details of the outcome group.

1. Determine the courses in the curriculum where each outcome is introduced, developed and the student is expected to demonstrate the program completion competency of each outcome.
2. Identify the specific courses.
3. Identify the instructors who teach those courses.
4. Compile in a matrix similar to the one below. This will aid in identifying instructors who must be trained and provided expectations on assessment outcomes in Canvas.
   

1. Request outcome administrator access. After discussing your needs with the Canvas Outcomes User Group and gaining approval to move forward, you may request administrative access to be able to create Outcomes in Canvas.
   1. To request access, send an email to SupportCenter@ilstu.edu with the following information included in the message:
      1. Add the subject line: Request Canvas Outcomes Admin Access
      2. In the body include:
         1. This is a request to gain access to Canvas environment to be able to create outcomes for the following individual:
         2. Who: Reggie Redbird, rredbird
            1. Note: There should only be one individual in this role.
         3. Environments: Canvas Test and Canvas Production
         4. What: Outcome creation access to <program area name, college name, department name>
            1. Outcome level access determines what courses have access to the outcomes. Options include:
               1. Institutional program level: Data can be collected in courses across all colleges at ISU (ex. GenEd or Teacher Ed)
               2. College level: Data can be collected in courses from any department within one college (ex. College of Business)
               3. Department level: Data can be collected in courses from one department (Department of Special Education)
            2. Once the Canvas administrator receives the request, they will obtain approval from the assistant vice-president for academic planning.
            3. This may take a few days to complete, and you will get an email when it has been completed.
2. Select the environment to log into.
   1. About the environments:
      1. The Production environment is where courses are taught, and outcomes collected. (https://canvas.illinoisstate.edu/)
      2. The Test environment is available for practicing and ensuring you understand what to do. (https://illinoisstate.test.instructure.com/)
         1. The Test environment is refreshed the third Saturday of every month. If you are testing at that time, your data will be overwritten with what is in the Production environment. Any changes will be lost and have to be recreated.
         2. After a refresh, if you are unable to access the Test environment, you may need to submit a ticket for your access to be reset.
   2. It is highly recommended that you create a few Outcomes in the Canvas Test environment to become familiar with the interface.
   3. Remember that once an outcome is built in the Canvas Production environment, it is available for instructors to use in their courses.
3. Log into the appropriate Canvas environment.
4. From your Canvas Dashboard, navigate to the Outcomes Area.
   1. Click the Admin shield in the far-left navigation.
   2. Select the name of the college or program in which you wish to create outcomes.
   3. Click Outcomes.
5. Create a new Outcome Group, enter the name, and click the checkmark.
   
6. Enter a group description to the Outcome Group.
   1. Click on the newly created Outcome Group.
   2. Select the 3 vertical dots on the right-hand side of the browser window next to the Outcome Group Name.
   3. Select Edit.
   4. Type in the Group Description.
   5. Select Save.
      
7. Create Subgroups for under the new Group.
   1. These will be the Outcome folders that contain the criteria that align to the Outcome.
   2. With the new group name selected, click Create New Group that is indented under it. 
   3. Enter the first competency group name.
      1. Be sure to use the names created in your label and numbering schema document in order to properly organize and identify the data.
   4. Click the checkmark.
      
      
8. Enter a group description to the new Outcome Group.
   1. Click on the newly created Outcome Group.
   2. Select the 3 vertical dots on the right-hand side of the browser window next to the Outcome Group Name.
   3. Select Edit.
   4. Type in the Group Description.
   5. Select Save. 
      
9. Note: Additional nested subgroups can be created to help manage the criteria as needed. Please use the information in your label and numbering schema document created earlier.
10. Add the criteria to that group. 
    1. With the Outcome Group highlighted, click +Create.
    2. Enter the information in this window:
       1. Name: should be brief.
          1. Use label and numbering schema to keep these in order.
             1. Example: Folder 1. <Name> would have Criterion 1.1 <Name>., Criterion 1.2 <Name>, Criterion 1.3 <Name>, etc.
          2. It will appear on the first line of the outcome.
       2. Friendly name: use entire name.
          1. It will appear in the second line of the outcome.
             
       3. If rating differ from the ISU standards, remove the unneeded points. 
          1. Example: This assessment must be based on 2 scores. Delete the 3, 2, and 1 proficiency rating.
       4. Change the Mastery points to the score that was decided. 
       5. Change Calculation Method as needed.
       6. Click Save. 
          
11. Repeat step 10 until all associated criteria have been added.
    
12. Repeat steps 7-11 until all outcomes and associated criteria have been added.
    
13. Build a common rubric for the new Outcomes.
    1. Ensure that you are still in the Admin area of Canvas (Admin should be selected).
       1. A rubric should be created at the administrator level, not at the course level so that data can be identified across all courses for the assessment.
    2. Click Rubrics.
    3. Select +Add Rubric.
    4. Add Rubric Title.
       1. Keep rubric name and high-level outcome name the same.
       2. Delete the ‘description of criterion’ row or those 5 points will be added to final rubric.
    5. Select Find Outcomes.
       1. Do not use '+ Criterion’ button because what is created will not be an outcome and will not collect the scores.
          
    6. In the Find Outcome window, select the Outcome to build the rubric.
    7. Open the Outcome subgroup.
    8. Select the criteria to include in the rubric.
       1. These are indicated with the bullseye symbol.
    9. Leave the "Use this criterion for scoring" checked. 
    10. Click Import. 
        
    11. Repeat steps e-j until all criteria have been added to the rubric for each outcome within the assessment.
        
    12. Select Create Rubric. 
        
    13. Faculty have the ability to bring rubrics into a course assignment and choose to:
        1. Use in its entirety.
        2. Selectively remove criterion that does not fit their assignment.
        3. Add additional rubric lines to be used when scoring.

1. Communicate clear expectations to faculty whose courses were identified when outcomes were aligned to courses about their assessment responsibilities.
   1. This should include how to locate the rubrics, and when, where, and how they should be used.
   2. See Collect Outcome Data in Canvas Courses Job Aids
2. Communicate with those instructors who may find the rubrics and use them in ways that aren’t consistent with your assessment goals.

After faculty have connected Outcomes to assignments in their courses, as those assignments are assessed, Canvas will store the score assigned.

1. Data from these assessment activities is automatically stored in Canvas and sent to Illinois State University’s enterprise data warehouse.
2. Enterprise Data and Analytics is available to assist in collating the specific assessment data and build reports in Cognos.
   1. Learn more about the reports by sending an email to SupportCenter@ilstu.edu with your request and asking for the ticket to be assigned to EDA.
3. PowerBI reports are another option if you have someone available on staff to create them.
   1. To gain access to the data, send a request to SupportCenter@ilstu.edu and asking for the ticket to be assigned to EDA.
4. Canvas provides an Outcomes Export report. To generate the report:
   1. Navigate to your administrative area in Canvas.
   2. Click the Admin shield in the far-left navigation.
   3. Select the name of the department, college or program.
   4. Select Settings.
   5. Select Reports.
   6. Locate the report named Outcome Results.
   7. Select the requested information in the window.
   8. Click Run Report.
   9. Refresh the "Reports" screen and click the download link to view and/or download the report.
      1. It may take a few minutes for the report to be available.