Rubric for Use in Phil 1338 and CSE 2501 for the Assessment of Program Outcomes

(This is a revised version based on the Sept. 14, '17, discussion in UGSC; the original version is here).

Background: One of the requirements of the BS-CSE (and BS-CIS) curriculum is that students take either CSE 2501 or Phil 1338. The main topic of these courses concerns ethical and social issues related to computing systems and applications. The ethical issues are addressed on the basis of appropriate ethical theories but, given that Phil 1338 is a 4-credit hour course and CSE 2501 is a 1-credit hour course, naturally the depth to which ethical theories are discussed in Phil 1338 is much greater than in CSE 2501; nevertheless, both courses provide sufficient background in ethical theories as well as discussion of the ACM/IEEE Code of Conduct, to enable students to analyze typical cases that may raise ethical dilemmas related to computing concepts and practices.

Both courses require students to write papers analyzing topics involving ethical and social issues related to computing systems and practices; students are also required to make oral presentations on these topics. This is natural, given the nature of these topics. Indeed, these topics also present excellent opportunities for formal debates among students. Although neither course requires students to engage in formal debates, the oral presentations in each course and the q/a period following each presentation offer natural opportunities for students at the presentation to engage actively with each other. Thus these courses make important contributions toward helping students achieve (CAC) outcomes (e), (f), and (g) [respectively, understanding of ethical, social issues, etc.; effective communication; and local and global impact of computing]; as well as EAC outcome (j) [contemporary issues]. The rubric below is designed to assess the extent to which the two courses contribute to these outcomes.

Phil 1338, by the manner in which the oral presentations are organized, also helps contribute, to a small extent, toward achieving effective team skills. Briefly, the one-hour weekly "recitation" periods are reserved for presentations by students. The class of 40 students is grouped into 4 groups of 10 students each. Further, each team of 10 students is split into four teams, two with 3 students each, the other two with 2 students each. Each recitation is devoted to presentations from a 4-person team and a 2-person team from one of the four groups. All students in the group are required to attend the entire recitation periods during which students from the group are presenting and they are expected to participate by asking suitable questions, etc. Each team of students (triple or pair) works as a team during a presentation. In other words, each presentation on a particular topic is by the students in a particular team and all students in the team take turns speaking (and answering questions from the other students in the group and the instructor). The evaluation (by the instructor for grading purposes) of any given student's presentation is determined, in part, by the feedback from the other students in the group. This seems to work well since the group size is small enough to ensure participation by all students and large enough to generate interesting discussions. Because of this, the rubric also includes a dimension related to team skills; this dimension is intended to be used only for Phil 1338.

Details: The rubric includes ten dimensions. The first four have to do with the quality of the oral presentation, specifically having to do respectively with the organization of the presentation, the mechanics (mainly quality of slides), effectiveness of delivery, and how well the speaker relates to the audience. The next two have to do with the quality of written communication, specifically the organization of the paper(s) and the style of presentation. The next three dimensions have to do with understanding of ethical & professional issues; recognition of the local and global impact of the topic under discussion and, more generally, on society at large; and awareness of relevant political, cultural and other contemporary issues. The last dimension, for use only in Phil 1338, deals with the effectiveness of the student's team work during presentations. Each of these dimensions is assigned a score of 1 through 4, these values representing increasing degrees of achievement in the particular dimension, as described in the table below in the rows corresponding to the various dimensions. The last column are the actual scores assigned to this particular student's presentation along the four dimensions.

This rubric is not specifically intended to be used by course instructors to assess individual students in their sections of Phil 1338 or CSE 2501 for purposes of providing formative feedback to the students during the course of the semester and/or for assigning final course grades. Individual instructors will likely use other rubrics, possibly based to a greater or lesser extent on this one, for these purposes. The main purpose of the current rubric is for assessment and continuous improvement of the BS-CSE program. It is expected to be used in one section per year of each of Phil 1338 and CSE 2501. In each case, the section in question should have been taught by an instructor with some experience with the course (rather than a first-time instructor) so that he/she has had a chance to work out any kinks in the specific way he/she approaches the course. The results from one section of each of the two courses will be presented at a regular meeting of the CSE Undergraduate Studies Committee as an important part of the continuous improvement process of the BS-CSE program. The discussion in the committee is expected to assess the extent to which students in the program are enabled by these two courses to achieve the particular outcomes noted above and to identify possible improvements in the program. Of course, other courses, in particular the capstone design courses, contribute to several of the same outcomes and the discussion, in the committee, of the results from the rubrics used to assess the effectiveness of those courses to the relevant outcomes is also a key part of the continuous improvement process.

Name/code of student being evaluated:  __________________________________________
Course and semester:  __________________________________________
Name of evaluator:  __________________________________________

   1 2 3 4 Points
assigned

Oral Communication Skills

Organization Audience cannot understand presentation because of poor organization; introduction is undeveloped or irrelevant; main points and conclusion are unclear; Audience has difficulty following presentation because of some abrupt jumps; some of the main points are unclear or not sufficiently stressed; Satisfactory organization; clear introduction; main points are well stated, even if some transitions are somewhat sudden; clear conclusion; Superb organization; clear introduction; main points well stated and argued, with each leading to the next point of the talk; clear summary and conclusion.

Mechanics Slides seem to have been cut-and pasted together haphazardly at the last minute; numerous mistakes; speaker not always sure what is coming next; Boring slides; no glaring mistakes but no real effort made into creating truly effective slides; Generally good set of slides; conveys the main points well; Very creative slides; carefully thought out to bring out both the main points as well as the subtle issues while keeping the audience interested.

Delivery Mumbles the words, audience members in the back can't hear anything; too many filler words; distracting gestures; Low voice, occasionally inaudible; some distracting filler words and gestures; articulation mostly, but not always, clear; Clear voice, generally effective delivery; minimal distracting gestures, etc., but somewhat monotone; Natural, confident delivery that does not just convey the message but enhances it; excellent use of volume, pace etc.

Relating to audience Reads most of the presentation from the slides or notes with no eye contact with audience members; seems unaware of audience reactions; Occasional eye contact with audience but mostly reads the presentation; some awareness of at least a portion of the audience; only brief responses to audience questions; Generally aware of the audience reactions; maintains good eye contact when speaking and when answering questions; Keeps the audience engaged throughout the presentation; modifies material on-the-fly based on audience questions and comments; keenly aware of audience reactions.

Written Communication Skills

Presentation of ideas and organization of the paper Bland presentation; sequencing and pace of topics seems random; claims are made with little or no justification; Some of the ideas are presented well and justified reasonably; others are lacking; offers plausible conclusion(s); Ideas are well organized and help the reader move along; most of the ideas are well-justified with appeals to relevant ethical considerations; key points are presented but does not always demonstrate in-depth understanding; leads up to convincing conclusion(s); The paper is clear and focused; relevant, quality details give the reader important information; the ideas are cogently justified, with arguments that are well-grounded in ethical theories; where relevant, conflicting views are appropriately evaluated; also where relevant, reference is made to appropriate parts of the ACM/IEEE Code; helps the reader develop insight into the topic.

Style Occasional problems with word choices and sentence structure, leaving the reader unsure of the meaning; often resorts to jargon/ cliches; Words and sentences are adequate in general but lack energy; reader has to struggle to keep reading to the end; Good writing style; sentences flow smoothly and evenly; Compelling writing style; connects strongly with the reader and keeps him or her engaged right to the end.

Ethical/professional issues, local/global impact, contemporary issues
Note: While a student's abilities related to the other dimensions in this rubric will be evidenced primarily during that student's oral presentation(s) and/or paper(s), the student's abilities with respect to the dimensions in this category are likely to be reflected also and, possibly to a greater extent, in the types of questions that he/she raises during presentations by other students and the types of discussions he/she engages in. For example, if there is a presentation that raises questions related to the security of electronic voting machines, that should present an opportunity, for all students at that session, to engage in a serious discussion about the impact on society of real or perceived insecurity of those machines; similarly for presentations that are related to cyber-espionage; etc. This should be kept in mind when arriving at the assessment of student's abilities with respect to the dimensions in this category.

Understanding of ethical and professional issues Little or no understanding of professional/ethical issues even where there are serious questions involved; Some consideration of professional, ethical issues raised directly by the topic under discussion; Good understanding of and reasonable analysis of all the essential relevant issues. Deep understanding of the professional issues involved and the ethical implications of the topic under discussion; careful, convincing analysis of all relevant factors.

Awareness of implications to society at large Little or no understanding of (or interest in?) implications to society related to the topic under discussion; Moderate understanding of the implications to society related to the topic under discussion; Good understanding of the implications to society of the topic, as well as its relation to general societal issues; Deep understanding of the immediate and longterm implications to society of the topic under discussion, and the related potential benefits and risks to society.

Awareness of contemporary issues (political, cultural, ...) Little or no understanding of (or interest in?) contemporary issues directly related to the item under discussion; Moderate understanding of the main relevant contemporary issues directly related to the item; Good understanding of all the relevant contemporary issues directly related to the topic; Deep understanding of all the relevant issues, whether political, cultural or other, related to the topic, as well as of issues that may be only tangetially related; good analysis of the issues and possible impacts on various aspects of society.

Team skills (applies only to Phil 1338)

Contribution as a team member Seems to have no interest in the presentations by the other member(s) of the team; occasionally gets into arguments with the other member(s) during the presentation. Mainly focused on his/her portion of the presentation; responds when another team member asks him/her a direct question but otherwise does not attempt to help other team member(s) address audience questions. Good team player. Is interested in the presentations by the other team member(s); makes a definite effort to ensure success of the entire presentation by occasionally helping the other member(s) respond to audience questions. Excellent team player. Goes out of the way to help the other member(s) in any way possible to address audience questions, get over glitches during their presentations, etc.

Total:

Comments:

	1	2	3	4
Oral Communication Skills
Organization	Audience cannot understand presentation because of poor organization; introduction is undeveloped or irrelevant; main points and conclusion are unclear;	Audience has difficulty following presentation because of some abrupt jumps; some of the main points are unclear or not sufficiently stressed;	Satisfactory organization; clear introduction; main points are well stated, even if some transitions are somewhat sudden; clear conclusion;	Superb organization; clear introduction; main points well stated and argued, with each leading to the next point of the talk; clear summary and conclusion.
Mechanics	Slides seem to have been cut-and pasted together haphazardly at the last minute; numerous mistakes; speaker not always sure what is coming next;	Boring slides; no glaring mistakes but no real effort made into creating truly effective slides;	Generally good set of slides; conveys the main points well;	Very creative slides; carefully thought out to bring out both the main points as well as the subtle issues while keeping the audience interested.
Delivery	Mumbles the words, audience members in the back can't hear anything; too many filler words; distracting gestures;	Low voice, occasionally inaudible; some distracting filler words and gestures; articulation mostly, but not always, clear;	Clear voice, generally effective delivery; minimal distracting gestures, etc., but somewhat monotone;	Natural, confident delivery that does not just convey the message but enhances it; excellent use of volume, pace etc.
Relating to audience	Reads most of the presentation from the slides or notes with no eye contact with audience members; seems unaware of audience reactions;	Occasional eye contact with audience but mostly reads the presentation; some awareness of at least a portion of the audience; only brief responses to audience questions;	Generally aware of the audience reactions; maintains good eye contact when speaking and when answering questions;	Keeps the audience engaged throughout the presentation; modifies material on-the-fly based on audience questions and comments; keenly aware of audience reactions.

Written Communication Skills
Presentation of ideas and organization of the paper	Bland presentation; sequencing and pace of topics seems random; claims are made with little or no justification;	Some of the ideas are presented well and justified reasonably; others are lacking; offers plausible conclusion(s);	Ideas are well organized and help the reader move along; most of the ideas are well-justified with appeals to relevant ethical considerations; key points are presented but does not always demonstrate in-depth understanding; leads up to convincing conclusion(s);	The paper is clear and focused; relevant, quality details give the reader important information; the ideas are cogently justified, with arguments that are well-grounded in ethical theories; where relevant, conflicting views are appropriately evaluated; also where relevant, reference is made to appropriate parts of the ACM/IEEE Code; helps the reader develop insight into the topic.
Style	Occasional problems with word choices and sentence structure, leaving the reader unsure of the meaning; often resorts to jargon/ cliches;	Words and sentences are adequate in general but lack energy; reader has to struggle to keep reading to the end;	Good writing style; sentences flow smoothly and evenly;	Compelling writing style; connects strongly with the reader and keeps him or her engaged right to the end.
Ethical/professional issues, local/global impact, contemporary issues Note: While a student's abilities related to the other dimensions in this rubric will be evidenced primarily during that student's oral presentation(s) and/or paper(s), the student's abilities with respect to the dimensions in this category are likely to be reflected also and, possibly to a greater extent, in the types of questions that he/she raises during presentations by other students and the types of discussions he/she engages in. For example, if there is a presentation that raises questions related to the security of electronic voting machines, that should present an opportunity, for all students at that session, to engage in a serious discussion about the impact on society of real or perceived insecurity of those machines; similarly for presentations that are related to cyber-espionage; etc. This should be kept in mind when arriving at the assessment of student's abilities with respect to the dimensions in this category.
Understanding of ethical and professional issues	Little or no understanding of professional/ethical issues even where there are serious questions involved;	Some consideration of professional, ethical issues raised directly by the topic under discussion;	Good understanding of and reasonable analysis of all the essential relevant issues.	Deep understanding of the professional issues involved and the ethical implications of the topic under discussion; careful, convincing analysis of all relevant factors.
Awareness of implications to society at large	Little or no understanding of (or interest in?) implications to society related to the topic under discussion;	Moderate understanding of the implications to society related to the topic under discussion;	Good understanding of the implications to society of the topic, as well as its relation to general societal issues;	Deep understanding of the immediate and longterm implications to society of the topic under discussion, and the related potential benefits and risks to society.
Awareness of contemporary issues (political, cultural, ...)	Little or no understanding of (or interest in?) contemporary issues directly related to the item under discussion;	Moderate understanding of the main relevant contemporary issues directly related to the item;	Good understanding of all the relevant contemporary issues directly related to the topic;	Deep understanding of all the relevant issues, whether political, cultural or other, related to the topic, as well as of issues that may be only tangetially related; good analysis of the issues and possible impacts on various aspects of society.

Team skills (applies only to Phil 1338)
Contribution as a team member	Seems to have no interest in the presentations by the other member(s) of the team; occasionally gets into arguments with the other member(s) during the presentation.	Mainly focused on his/her portion of the presentation; responds when another team member asks him/her a direct question but otherwise does not attempt to help other team member(s) address audience questions.	Good team player. Is interested in the presentations by the other team member(s); makes a definite effort to ensure success of the entire presentation by occasionally helping the other member(s) respond to audience questions.	Excellent team player. Goes out of the way to help the other member(s) in any way possible to address audience questions, get over glitches during their presentations, etc.
Total:
Comments: