The IAB convenes for a day-long meeting every year on campus, typically in late April. The board gets a detailed update on various recent developments in the department related to research, graduate programs, and undergraduate programs. Once every two years or so, input is sought from the board on the PEOs.
Recent alumni are especially important in the assessment of PEOs since they have intimate knowledge of the program and, at the same time, also have experience in industry. Input from alumni is obtained by means of an alumni survey. The survey is sent to alumni who graduated either two or three years prior to the survey date. Thus the approach lets us gather input from alumni who graduated relatively recently and hence have more or less current knowledge of the program but also have some experience in the job market and hence can comment on how well the program prepared them for the profession. The key portion of the survey that relates to PEOs asks the respondent to rate, on a scale of very unimportant through extremely important, the importance of each of the PEOs. Next, the respondent is asked to rate, on a scale of strongly disagree through strongly agree, the extent to which they agreed with the statement, "the BS-CSE program adequately prepared me to achieve the PEO".
Although all CSE students are required to take the test, the performance on the test does not affect the grades of individual students in any courses, nor are records retained of how individual students performed on the test. When a group of students takes the POCAT, each student receives a unique code that appears on the student's test but only the individual student knows his or her code. Students are instructed not to write their names or other identifying information on their tests. Once the tests have been graded, summary results, organized by this code, are posted on electronic bulletin boards so an interested student can see how well he or she did and how his or her performance compared with that of others who took the test. This was a deliberate decision since we did not want the students to spend a lot of time preparing for the test. The goal of the test is to help assess the program, not individual students. Initially, there was a concern that if the individual students' performance on the test did not affect them in any tangible way, they would not take the test seriously. Our experience with the test since it was institued has eliminated that concern; most students seem to actually enjoy taking the test and take it quite seriously.
The questions on POCAT are based on topics from a number of required courses, many of the core-choice courses, and some popular elective courses related to a variety of key topics such as software engineering, formal languages and automata theory, databases, programming languages, operating systems, computer architecture, AI, etc. Each question on POCAT is a multiple-choice question. The ideal question not only has a specific correct answer but also a number of distractors that correspond to common misconceptions that students might have about the particular concept. It is for this reason that the summary results of a POCAT includes information not only about the percentage of students who answered a given question correctly but also the percentages of students who chose each of the distractors, in other words how many students harbored the particular misconceptions represented by the various distractors about the underlying concept(s). The key goal of the test is to help faculty use the results to identify specific weaknesses in particular courses and help improve the curriculum.
There is one other unusual feature of the test that is worth noting. Each question has, as one of the choices (typically the last one), an answer along the lines of "I don't know". The instructions for the test suggest that the student should pick that answer if he or she has no idea what the correct answer is. Since their performance on the test will have no impact on their record, students who do not know the answer to the question and know that they do not know, pick this answer. This means we do not have to worry about the student trying to make guesses and confounding our attempt to pin down misconceptions that the student may have.
The grading of POCAT and production of summary results is mostly mechanical. The faculty members responsible for each question also provide an estimate of the percentage of students who, in their estimate, ought to be able to answer the question correctly. All of this information is included in the summary results. This allows the Undergrad Studies Committee (UGSC) to have a well-informed discussion about the extent to which the particular group of students had achieved the outcomes related to the various quesions and identify potential problem spots in particular courses, indeed in particular topics, and bring them to the attention of the appropriate group of faculty.
The result page for each POCAT includes three tables. The first one lists, for each question, the particular answer (including the "I don't know" one) each student picked for that question, the percentage of students who answered any given question correctly, as well as the percentages of questions that a given student answered correctly. The second table lists, for each question and each possible answer for the question, the number of students who picked that answer; there is also a summary line that specifies what percentage of students answered that question correctly. The third table lists, for each of the SOs, the average level achievement of that SO. This is computed on the basis of the SOs that each question is related to and the number of students who answered that question correctly. Of these, the second table is perhaps the most valuable since it allows UGSC and relevant faculty to identify which particular misconceptions students most commonly hold concerning a given topic and, hence, arrive at possible improvements to address the problem. From the point of view of the individual student, the first table is more interesting since it allows the student to compare his/her performance on various topics with that of peers.
One important activity of each capstone course is the poster session. Each student team from each of the capstone courses from that semester is expected to present its work at a public poster session which also typically includes demo's of the team's prototype system. The rubric for use at the poster session includes essentially the same dimensions as the previous rubric but with two changes. First, given that the visitors will be looking at the team's poster and interacting with the team members at the same time, it seemed appropriate to combine the two dimensions of that rubric that are related to communication effectiveness into a single dimension for this rubric. Second, and perhaps more substantive, is that rather than specifying distinct levels of achievement for each dimension, we decided to specify a set of characteristics corresponding to a high-level of achievement for the given dimension and have the visitor determine to what extent he/she agreed that the given team and its poster demonstrated those characteristics.
A second feedback mechanism is the departmental advisory board and its annual meeting. As noted earlier, the chair of UGSC makes a presentation about the program during this meeting and seeks feedback on all aspects of the program from the advisory board. Since some members of the board are alums of the program (or the department since they may have obtained their graduate rather than undergrad degree from the department), and several of them serve long terms on the board, they tend to have a reasonably good understanding of both the program as well as the culture in the department. These factors, combined with their long experience, allows them to offer well-considered suggestions for changes and improvements.
The results from the various assessment mechanisms as well as the ideas generated at the annual forum and the advisory board meetings are all discussed at regular UGSC meetings and ideas for possible changes in the program are considered. UGSC includes faculty members, especially those heavily involved with required/popular undergrad courses; members of the Advising Office; and one or more student representatives. Thus the discussions tend to account for a variety of viewpoints.
Last modified: Mon Apr 10 11:33:04 EDT 2017