CSE Undergraduate Studies Committee
Minutes of Meetings (2016-'17)

Spring:	Apr 12; Feb 22; Feb 15; Jan 18;
Fall:	Nov 14; Oct 24; Oct 17; Sept 26; Sept 12;

04/12/'17

1. POCAT results from Sp '17 (available here).
2. Report on Annual Forum.
3. ABET preparations

1. There was an extended discussion on several of the questions on the POCAT, including two of the new questions (one concerning the time complexity of the recursive Fibonacci function and a software-engineering one from 3902 regarding cohesion and coupling). Some main points that were raised:
   • The performance on the SE question was surprisingly poor (with, in the case of one group, only 5% of the students picking the correct answer). A couple of possible explanations were suggested. First, students may not remember some of the terminology from the course (since they may not have seen it elsewhere) even if they understood the underlying concepts. Second, the question may have included references to too many different items and that may have caused some confusion. We decided that it would be helpful, if possible, to split the question into two (or three?) questions with each focusing on one of the items in the current question. Matt Boggus who had come up with the original question agreed to try to come up with these variations and we will try them in future POCATs to see if we can pin down the problem(s) ...
   • The performance on the three versions of question (1) [which we have used many times in the past] was similar to that in the past. Specifically, performance on the first version in which the question was stated in the context of library books was slightly better than the performace on the third version which was otherwise identical to the first version but the question was in the context of finite-state machines. More importantly, performance on the second version of the question (also in terms of finite-state machines) which talked about "digitial encoding" of information, was much worse. The conclusion was that students find this terminology confusing (and it is not clear what it means in the first place!); so we will stop using this version of the question.
   • Following up on an idea that had been suggested at the recent meeting of the department's Advisory Board, Neelam had sent the POCAT to a senior person in industry (someone who, in fact, got his PhD from our dept. some years ago) and asked for his comments. One suggestion that Neelam received in reply was that this first question be replaced (or supplemented) by one that presents a scenario in which a log file doubles in size every two hours or so, with the disk on which it saved being substantially larger than the current size of the log file and ask what, if anything, should be done [ranging from, "ignore it" to "buy the largest disk possible and forget about it" to "I have no idea"]. We will try to come up with a question along these lines and try it in upcoming tests.
   • Mike Fritz had proposed a question that asked about the asymptotic running time of a simple recursive program to compute the Fibonacci function. Here again the performance, especially in two of the four groups, was surprisingly poor. Ross and Viral noted that this specific problem is discussed in some depth in Foundations I or II or both. A couple of conjectures were offered. For example, the fact that Java's int was used as the type of all the variables in the code, especially combined with the fact that there was a statement in the problem that two integers could be added in constant time independent of their sizes, might have caused confusion among some students ... one (perhaps extreme?) version of this was the idea that some students might have decided that the algorithm would take constant time since for large(r) values of the argument, for which the value of the function could not be stored in an int, so the system would crash; while for smaller values of the argument it will terminate in some bounded time. A somewhat surprising number of students also picked the option "linear" as their answer for this question. Another troublesome point was that 3341, the core choice course on programming languages (which many students do take) also talks about how to write efficient versions of the Fibonacci function in a functional language (such as Scheme). Viral noted that Foundations I and II stress the fact that exponential algorithms are, as a rule, horrible choices; given this and given that this algorithm is discussed in several courses may have conceivably led to some students rejecting the "exponential" choice. Mike will try to come up with a version of the question that eliminates as much of the confusing and/or extraneous language as possible so we can try to tease out the problem(s).
   • More generally, it was suggested that perhaps we should include some way for the student to indicate that a particular question was especially confusing or unclear or ambiguous etc. Students would be instructed not to use this option unless the degree of confusion or lack of clarity in a question was excessive ... this might allow us to better understand the results and identify any potential underlying problems in the related courses. We will try such an option in a future test and see if that proves useful.

2. Since the hour was nearly up, there was a very brief discussion of the report on the Annual Forum that was held on March 22, specifically item (5) of the report. Neelam noted that there was considerable interest among students in a course (whose detailed contents may well vary from one offering to the next) that presents a few important and powerful tools/ systems etc. with some essential coverage of the underlying concepts followed by discussion of the technical details of the tool/system and how may be used in practice including, possibly, a detailed assignment/project involving the tool or system. One important consideration for an item to be part of such a course would be that there is a cohesive set of conceptual ideas underlying the tool or system that is not already part of our required curriculum; the reason for this is that if the conceptual ideas underlying the tool or system were already part of the required courses, students should be able to explore the tool on their own once they had completed those course.

   Currently, some students who are interested in such a tool or system talk to an individual faculty member and depending on that person's availability and willingness, register for an independent study with that person; other students try to pick up the material as best as they can on their own; and yet others simply ignore such tools and systems, focusing instead on their course work. A course of this kind would make it simpler and more effective for students to explore some of these systems.

   One possible risk in having such courses is that students may fill up their tech elective hours with a number of such courses with the result that their overall technical foundations may be weak and/or lack coherence. One possible way to address this potential problem is to cap the total number of credits that this course PLUS any independent studies and research hours that a student may include in his/her tech electives at, say, 3 hours. We will explore this option further and, if it seems reasonable, develop a plan for its implementation.

The meeting was adjourned at 12:05.

Next meeting: ??

02/22/'17

1. Rubric results from Au '16 capstone poster session
   (Recall: we recently designed a rubric for use in the capstone poster session that we have at the end of every semester to assess student achievement of several of our outcomes. The rubric is available here.
   Jeremy Morris and Al Cline used the rubric in the poster session at the end of Au '16. We will look at the results.)
2. Rubric results from Au '16 CSE 5911, CSE 5912, using revised rubric
   (Recall: we recently developed a simplified rubric for use by capstone course instructors. The rubric is available here;
   Murthy Narasimhan and Roger Crawfis used the rubric in their capstone courses in the fall. We will look at the results.
3. Rubric for use in the junior project course.
   Paul Sivilotti is planning to use, in his 3901 class, the rubric we recently designed. We will look at the rubric; (and will look at the results from Paul's class at the end of the semester).

1. Assessment of posters at the capstone poster sessions: An important event that the capstone course instructors introduced a few years ago and that has became a regular part of all capstone courses is the poster session. Each capstone project team is expected to prepare a suitable poster documenting the problem that the team worked on; the approach used in the design, implementation, and testing of the system; and main conclusions/lessons learned. Each semester (except the summer), the session is held immediately after the last day of class (during the "reading days"). The poster session at the end of the Spring semester is part of the College of Engineering's capstone poster session and is typically held in the Ohio Union; the session at the end of the Autumn semester is only for CSE projects and is generally held in Dreese 113. Nearly all of the teams in the various capstone course sections participate in the poster session at the end of their capstone course; occasionally, a team misses the session because of work or other conflicts such as job interviews.

   Thus the poster sessions are a good opportunity for doing a high-level assessment of student achievement of some of the intended outcomes of the capstone courses, to complement the instructors' assessment. During summer/fall '16, we designed this rubric for this purpose and piloted it during the poster session at the end of fall '16. Jeremy Morris and Al Cline used the rubric to assess most of the posters at the session. The results are available here.

   There was an extended discussion. The committee concluded the following:

   • Overall, the rubric seems to be satisfactory.

   • The appropriate average "expected level of achievement" for each of the items in this rubric should be "Agree", i.e., 3.0 (with '"Strongly Agree", "Agree", "Disagree", "Strongly Disagree" being converted to 4.0, 3.0, 2.0, and 1.0 respectively in computing the average).

   • One important question has to do with how to ensure that we get a resonable group of people to use this rubric to assess the posters. An important consideration here is that industry practitioners form an especially important constituency for our program and it would be helpful to get feedback from them, especially regarding student performance in the capstone design courses. A number of industry professionals do come to the poster session but most of them do not know much about our program and hence may not be in a position to offer useful feedback.

     At the same time, there is one group of industry practitioners, i.e., industry professionals who teach some of our capstone courses, who are ideally suited for this since not only do they know the program reasonably well, they also usually attend the poster session. Hence, every fall and spring semester (starting with Sp '17), we will have each of the industry professionals who is teaching a section of any of our capstone courses (5911, 5912, 5914, 5915) in that semester to evaluate each of the CSE posters at the poster session at the end of that semester and complete the rubric for each poster. The results will be discussed in a UGSC meeting in either shortly after the poster session or at the start of the next fall or spring semester if a UGSC meeting cannot be scheduled in the days immediately after the poster session.

2. Assessment of Projects by Instructors: (Rubric; Au '16 Results)
   We briefly looked at the results but, given that we had results only from a section of 5911 and a section of 5912, we decided to wait until the end of Spring '17 for a detailed discussion of the rubric and results from it; during the Sp '17 semester, we will have the instructors for a section each of 5914 and 5915, the other two capstone courses, use the rubric in their respective courses. This should give us a better picture of how well the rubric is suited to the various courses and also allow us to compare the results from the instructor rubric with those from the poster session rubric. (The plan is to obtain/discuss results from this rubric for one section per year of each capstone course rather than every section that is offered.)

The meeting was adjourned at 12:05.

Next meeting: ??

02/15/'17

1. Rubric results from Au '16 section of Phil 1338
   Recall: we recently designed a rubric for use in Phil 1338 and CSE 2501 to assess student achievement of outcomes related to communication skills, analyze impact of computing, knowledge of contemporary issues, and understanding of professional, ethical, legal, security, and social issues (outcomes (f, g, h, j)); and, in the case of 1338, team skills (outcome (d)). The rubric is available here
   Dr. Bryan Weaver of Philosophy taught a section of Phil 1338 in Au '16 and used the rubric in his section. We will look at the results from that section.
2. Rubric results from Au '16 capstone poster session
   (Recall: we recently designed a rubric for use in the capstone poster session that we have at the end of every semester to assess student achievement of several of our outcomes. The rubric is available here.
   Jeremy Morris and Al Cline used the rubric in the poster session at the end of Au '16. We will look at the results.)
3. Chris Stewart who works with student groups such as ACM-W would like to briefly discuss the question of suitable space for events that such groups arrange (such as meetings with IT companies, etc.)

1. Facilities for events organized by student groups such as ACM-W:
   Student groups such as ACM-W have been quite active over the last couple of years, organizing numerous events of interest to students. Unfortunately, for a variety of reasons, they have not been able to arrange for suitable room and other (computers, mainly) facilities for these events and, on occasion, have had to cancel the event. A recent example is a one-day workshop, sponsored by Capitol One, that had to be canceled because a suitable room could not be reserved. Part of the problem is the limited availability of resources compared to the demand from student organizations. Another part has to do with the fact that many of these events tend to be on weekends (including the canceled workshop) on after hours and building access, etc., is a problem. Contributing to these is policies concerning food etc. in particular rooms; e.g., the only buildings that were available for the Capitol One event did not allow food and that would not make sense for an all-day workshop.

   Clearly, supporting our student groups in their activities is important. These events not only help students learn from each other in a non-classroom setting, they are also, given the nature of the events, key to preparing our students for their future careers after graduation. Anecdotally, student groups in other departments are able to access facilities for their events. We will try to figure whether similar approaches will work for our groups as well as other alternative approaches; Chris, Paul, Rafe, Nikki, and Neelam will work on this. Suggestions are welcome.

2. Results from outcomes assessment from Phil 1338:
   Phil 1338 is a recently developed course intended to be a combination of Phil 1337 and CSE 2501. Since the main topic of the course is ethical issues related computing, it is very appropriate for our students. The difference with Phil 1337 is that it includes a strong oral presentation component. This was the reason we previously decided to treat Phil 1338 as meeting the requirements of CSE 2501; so students who take that course are considered to have met the Engineering ethics course requirement as well as the requirement of CSE 2501.
   Given the important role that CSE 2501 and now Phil 1338 play in helping our students to achieve several important outcomes (f, g, h, j), we worked with the involved Philosophy faculty to develop a suitable rubric for use by both Phil 1338 and CSE 2501. This is a highly revised version of a rubric that we had been using in CSE 2501. The intent was to pilot the rubric in Phil 1338 in Autumn '16 and in CSE 2501 in Spring '17. Dr. Bryan Weaver of Philosophy recently sent his results from his Au '17 section of 1338 to Neelam and we discussed the results.

   The rubric is available here.
   The last dimension in the rubric is related to team skills. Although CSE 2501 does not include a team-work component (hence this dimension of the rubric will not apply in the case of 2501), Phil 1338 does include such a component. The course, a typical section of which has 40 students, is organized as follows: The course meets thrice a week for 80 minutes each time. The presentations are all held on one day (typically, Friday), the other two days being lectures by the instructor on the philosophy topics. The class of 40 is organized into 4 "Teams" of 10 students each. Each Team is organized into 4 "Groups", with two Groups having 3 students each, the other two having 2 students each. Each Group works on a topic and prepares a presentation on that topic to be delivered jointly by both or all three members of the Group. All ten students in a Team are expected to attend the oral presentations of each of the Groups in the Team and ask suitable questions etc. Each group's presentation is 15 minutes long, followed by 5 minutes for questions. Thus each Group forms a small team that works together closely on its topic and presentation; and each Team forms a loosely-knit team. This structure has worked well in all the sections (at least thus far).

   The results are from Dr. Weaver's Autumn '17 section are available here. For most dimensions, the average achievement was around 3 (on a 4-point scale). Based on the descriptions in the rubric of the various levels for each dimension, this would be classified as "satisfactory" (2 being considered "developing"). This may seem somewhat high, given that these are mostly sophomores; but it must be noted that the instructor assigns these values, keeping that factor very much in mind. In other words, if a student were to demonstrate the same level of achievement in his/her oral presentations in the capstone course, the resulting score for the student would most likely be lower. One figure that was surprisingly high was related to team skills. This may be due to a comibnation of factors including that each Group was quite small and that they worked together on one presentation (of about 10 minutes or so), so the extent of the team work was rather limited. In any case, the results seemed quite satisfactory.

   One question that came up was, what we should consider as an acceptable/satisfactory level of achievement. Given the level of these students, it would seem that an average of around 2.5 (or perhaps slightly higher) would be reasonable with a higher expectation (of, say, 3.0) at the capstone level. That is what we will aim for.

3. The discussion of the results of the assessment of the capstone poster session was postponed to the next meeting since we ran out of time. (But the information is available here:
   • Assessment of Posters: Rubric; Au '16 Results
   • Assessment of Projects by Instructors: Rubric; Au '16 Results

The meeting was adjourned at 12:05.

Next meeting: (Most likely on) 3/1

1/18/'17

1. Preparation for ABET evaluation
   • Assessments
   • Course materials
   • Self-study
2. Feedback from Cisco recruiter (see message to UGSC mailing list of a few days ago)

1. ABET preparations: Neelam summarized the preparations we have to make for the upcoming ABET evaluation:
   • Course materials: Materials, including samples of student work, have been collected from about half of our (undergrad) courses. The plan is to collect the materials from the rest this semester. One major task is organizing all the materials in a somewhat uniform manner so that the evaluators don't have to struggle when they go through the materials during the site-visit. Neelam has been working on this.
   • Assessments: We have been doing this; but, in preparation for the evaluation, we have to reorganize the materials so that they will be easily comprehensible to the evaluation team. [And, in addition, we will, as usual, be spending time this semester on looking at the results, including of the assessments (of the capstone poster session and the rubrics for use in 2501/Phil 1338) that we recently introduced, etc.]
   • Self-study: The other major activity is the preparation of the self-study that has to be submitted to ABET. The self-study will be officially submitted by the college in June; we have to give a final draft to the college by late May. Neelam will work on this and will ask for help from various faculty ...
2. Feedback from recruiter: Engineering Career Services got an email from Junilu Lacar of Cisco about our program. The main point of the message was to convey his impressions about/ideas for our program, especially the SW I, II sequence. The main part of his message read:

   ... one of the main topics of focus in these courses [SW I, II] is Design by Contract. That's not bad in itself; I think DbC and the ideas behind it are important for students to learn. However, nobody that I know in the industry does DbC the way that it's taught in the CSE Software I and II courses, especially not in Java, which is the programming language used in these courses. I've conducted an informal survey with several respondents from all over the world (I posted a question and request for feedback on an online forum where I volunteer as a moderator) saying that they've never done DbC, much less in the way that OSU appears to be teaching it. This is just one example of a serious misalignment of focus between academia and industry and I think it's important to address this misalignment sooner rather than later.

   There was an extended discussion on Lacar's comments and SW I, II as well as the rest of our program. First, and perhaps most importantly, there has always been a tension between the conceptual focus of our program and the immediate, practical needs that industry employers are most concerned about. At the same time, feedback we get from alums who have been out in industry for a few years often suggests that while the conceptual focus of the program may not be what will help new employees of companies such as Cisco "hit the ground running", they tend to be extremely valuable, in the long term, for addressing deep and complex problems that large software systems, especially those that perform critical tasks, often pose. Indeed, before the meeting, Matt Boggus sent a message that read:
   "I won't be able to attend ... but wanted to share a couple of quick comments related to the Cisco recruiter's feedback. During a recent visit from one of our alums currently working at SpaceX, he mentioned they make heavy use of design by contract (C++ and assembly rather than Java though) ..."

   A second key point is that our curriculum is not a uniform monolith, focusing only on conceptual matters. We have numerous project-oriented courses (the junior project course, the capstone courses, and other courses such as the Mobile Computing course and the Info. Security projects course) that have a very practice-oriented focus. Indeed, as Viral noted, even the projects in SW I, II, especially the latter, can be very helpful in preparing students for industry positions; he mentioned that one of the detailed questions he was asked at a recent interview with Amazon was the central theme of one of the projects in SW II!

   In the current instance, it was not exactly clear whether Lacar was mainly concerned about some topics that he felt should have been included in our curriculum but are currently not, or about some topics that should not be included but are included, or some combination of the two or something else. Our goal is to prepare students not just for immediate employment in the computing industry after graduation but also for a lifelong career in computing; indeed, some of our students go on to research careers and our program tries to meet their needs as well. Of course, if there are possible changes to the program that may improve our graduates' preparation to achieve these goals, we will certainly consider them. Toward this end and in order to make sure that industry people are well informed about our program, we will get in touch with Mr. Lacar not only to get more information on any ideas he may have about possible improvements to our program but also to ensure that he is well informed about the program.

3. In late November and early December, Neelam, Jeremy, and Paolo worked on possible changes to the third PEO of the BS-CSE program to account for the suggestion made during the Advisory Board meeting at the end of the Spring semester. During the break, this was sent to the faculty mailing list for possible revisions and then approval; faculty seemed comfortable with the proposed change. We will update the information in the program's website to reflect the revised PEO.

The meeting was adjourned at 12:00 noon.

Next meeting: ??

11/14/'16

1. POCAT results
2. UG Forum report
3. ABET preparations

1. UG Forum report:
   • Some concerns were expressed at the forum concerning the variation, among the various sections of Fnds I (CSE 2321) in the topics and depth to which they are covered since this has a direct impact on how well a student will be prepared for Fnds II (CSE 2331). It is not clear that standardization to the extent that is achieved among the various sections of Software I (2221) is possible in the case of 2321; nevertheless, it may be possible to have a set of slides that can serve as a possible model to clearly illustrate the range of topics and the depth to which each is covered. Rafe will work with Ken Supowit to explore this. Also Cailin noted that, when he took 2331, there was time spent on fairly detailed review of the algorithms-related material from 2321 and that review helped him considerably. This is another avenue worth pursuing (although it would come at the expense of some topic coverage in 2331).
   • One of the comments at the forum concerned that the introduction of something like an online bulletin board on which important announcements are posted. Currently, these are sent by email to the student mailing lists but, given the volume of email that students receive, they can be easily overlooked. Having a simple online board, prominently accessible via the main undergrad website, that contains brief announcement, arranged in (reverse) chronological order would help with this. Nikki will work with Wade in the Advising Office to try to get this implemented soon.
   • There was a question about the impact of the continuing popularity of CS on enrollments in our classes that the forum report briefly mentions; the answer is that, in almost each semester, we are having to scramble to add new sections of various courses and often having to depend much more on adjunct faculty than we have in the past.

2. POCAT results: We discussed the POCAT results briefly. Some key points:
   • For Question 1 (concerning representing information using bits), the performance was extremely weak in Group 2. It was noted that the Group 2 version of the question uses the term "digital encoding" whereas the other two versions simply talk about n-bit labels. It is possible that students thought of "digital encoding" as some complex scheme and were thrown off by that ... It was also noted that italics font was used to stress a portion ("...number of states was doubled...") of the Group 2 version of the question and that may have impacted the performance in this group. We will investigate this further.
   • The performance on (the new) Question 12, related to pipelining, was rather weak in two of the three groups ... we will have to see why this might be. Hopefully, future tests will help with this.
   • Performance on Question 15, related to algorithm analysis, was also rather poor. Rafe pointed out (after the meeting) that the notation used in the question was not the notation usually used in CSE 2331; it was more C-like (whereas the notation usually used in 2331 is more Pascal-like). If that is indeed the reason for the poor performance, we need to explore ways to help students transfer their algorithms' knowledge to such contexts/ syntax; and this ways would, presumably, be in courses beyong 2331. If, on the other hand, the problem is with students' poor grasp of algorithm analysis techniques, the way to address that would be in 2331. In any case, we will try to tease this out in future POCATs by creating versions of this problem that use the C-like syntax on the one hand and a more 2331-like syntax, on the other.
   • Performance on Question 14, related to data bases, in each of the three versions, was extremely poor. Jeremy noted that 3241, the DB I course, covers this topic and uses it in later material in the course. Cailin confirmed that his section did that as well although he seemed to feel that he would have struggled with the question as well. Jeremy and Spyros and other DB faculty will have to discuss this and see how to address it.

The meeting was adjourned at 12:00 noon.

Next meeting: ??

10/24/'16

1. POCAT
2. Rubric for junior project course

1. POCAT (general discussion):
   One question that has come up occasionally is the correlation between a student's performance in a course and his/her performance in POCAT on questions related to that course. Because we put a lot of weight on students' anonymity, there is no easy way to get information about such correlation.
   A somewhat different question has to do with a student's relative performance in different courses; i.e., e.g., does a student's performance in SW I, II say anything about how the student might perform in, say, the junior project course? This information can indeed be arrived at since we can look at, say, the advising reports of all graduating students (after removing their names so anonymity is preserved), and arrive at any correlation that might exist. We will see how feasible this activity is; and, if it is, whether it gives us any useful information that might inform possible improvements in the program.
   (Note related to POCAT questions: Neelam mentioned that Chris Stewart has suggested a conceptual question to replace the more detailed and formula-oriented question on architecture that we have been using in the POCAT. We will use this new question in this semester's POCAT.)

2. Rubric for junior project course:
   Paul, Matt, and Neelam have been working on a rubric for use in the junior project courses. We briefly looked at the current version. The intent of the rubric is to provide additional assessment of students' technical skills related to building software systems of varying complexity, as well as their understanding of social and ethical issues, and their communication and team skills. The plan is to use (a possibly updated version of) this rubric this semester in a section of 3901 (taught by Naeem; Paul will talk to Naeem about this) and see how it can be refined. Comments or suggestions for changes in the rubric are welcome.
   Both assessment and teaching of team skills is, of course, challenging. Currently, what we often do is to simply assign students to team projects and hope they will manage to make it work. In cases where there are extreme problems, such as a student essentially not contributing to the work of the team that he/she is part of, the other students in the team typically bring that to the attention of the instructor and the instructor tries to intervene. Instructors also use peer assessment by students in a team of the other students in the group; but many students tend not to want to blame the other students so this is of somewhat limited value (except perhaps for assigning grades). One problem that Viral noted was that students often drastically underestimate the amount of time/effort that a team member may have put in to get something completed and this gives a distorted view of that member's contribution. The question was whether we could do more to help students develop effective team skills.
   (Viral also noted that his most successful team experience was one in which he was able to hand-pick the team members so that the team included only students whom he knew as possessing the necessary technical knowledge and willing to work hard! While this may have worked in specific team projects, it is clearly not a general recipe since, in practice, CS professionals may not be able to determine who is included in the teams that they are part of.)

3. Assessment of BA-CIS program:
   Our current plan is to use an Exit Survey for assessing the BA program. The question was whether we should also use something like POCAT for the purpose. One problem here is that the BA students are required to take far fewer of the CSE courses than are BS students. The BA core consists of CSE 2221, 2231, 2321, 2421, 2501, junior project course. BA students are also required to take at least 13 hours of additional CSE courses as electives; the most common choices are 2331, 3241, 3521, 3541, 4471. So while it would probably not be appropriate to use the same test as for the BS students, it may be reasonable to have one that has some of the questions from that test. We will work on this.

   A second problem is the relatively small number of students in the BA program. One possible way to address this is to combine the results for an entire year when doing the analysis. We will consider this as well.

The meeting was adjourned at 12:05.

Next meeting: ??

10/17/'16

1. POCAT
2. Rubric for junior project course

1. POCAT (general discussion):
   • Our discussions of POCAT results, naturally, focus on the performance of students in specific questions, especially in cases where student performance is much weaker than expected (as specified by the person setting the question; this information is included in the last row of the relevant table in the results page). This is natural because it allows us to identify possible weaknesses in particular courses and come up with suitable fixes. While this is reasonable, there is a potential problem since ABET is focused on assessment of student outcomes. The POCAT results pages do include a table (the last one in each results page) which provides information about the extent to which student achieve the various outcomes, based on the relation between the various questions and particular outcomes; but there is no information concerning expected performance with respect to these outcomes.

     After some discussion, we decided that we will arrive at this information as follows: for each outcome, we will average together the expected performance in each POCAT question that is related to that outcome and treat that as the expected performance with respect to that outcome. That will allow us to compare the actual student performance with respect to that outcome with the expected performance and identify any substantial differences between the two. Neelam will revise the results-processing script to add this. (Note added: This has now been done so the third table in each POCAT results page, starting from Sp '16, will include an extra row showing the expected achievement percentage for each outcome, immediately below the average achievement percentage.)

   • Separating BS-CSE from BS-CIS: In POCAT, we have not distinguished between BS-CIS and BS-CSE students. This makes sense since students in each program take, essentially, the same CSE courses. But, for the ABET point of view, we should separate them since ABET is only concerned with BS-CSE. So we will start doing that from this semester, reporting the results from these two groups of students separately from each other.

2. POCAT (Spring'16 results):
   • Question 1 (concerning the encoding of information in binary numbers): In one version of this question, we didn't use the term "encoding", instead using the term "label". The student performance for this version of the question was much better than for versions that used the term "encoding". Another important point is that this version didn't mention "finite state machines"; that may also have contributed to the improved performance. We will try this one more time to see if the results repeat.

   • Question 14 (related to CSE 3241): There was a substantial difference in the performance on Q. 14 between version 1 and 2; the latter included a one-sentence reminder of the meaning of "key" and that seems to have improved performance by over a factor of 2. We will try this one too one more time to see if the results repeat. (Another difference, possibly not as significant, was that version 1 included what might be considered as an extra pair of braces; we will remove that in the next POCAT.)

   • Question 12 (concerning 3421): Here again, there was a performance difference in the different tests. Version 2 included the formula for runtime while version 1 did not; the student performance on version 2 was better. Chris offered to look into this more carefully to see what we should do.

3. Junior project rubric: We didn't discuss this.

The meeting was adjourned at 12:05.

Next meeting: 10/24 (to be confirmed)

09/26/'16

1. New/revised rubric for use by capstone course instructors
2. Rubric for use in the capstone poster presntation session
3. New/revised rubric for use by CSE 2501 and Phil 1338 instructors

Minutes:

At the meeting: Spyros Blanas, Paolo Bucci, Jeremy Morris, Kitty Reeves, Neelam Soundarajan, Paul Sivilotti, Nikki Strader, Huamin Wang, Rafe Wenger; Viral Patel, Cailin Pitt.

1. Background: Neelam summarized the background behind these new/revised rubrics. Currently, the instructors of the capstone design courses and CSE 2501 use a number of rubrics, such as one for assessing team work another for assessing team presentations (both of those in the capstone courses), one for assessing oral presentations in 2501, one for assessment by peers of team work (in some capstone courses) etc. While these rubrics do allow the instructors to assess the students' achievement of various outcomes, the variety of rubrics used and the number of instructors involved has made it extremely difficult to have an effective and sustainable process to document and evaluate the assessment results across the program as a whole to help identify possible improvements. In addition, we are currently not obtaining any assessment data from the capstone courses' poster session which is held at the end of fall and spring semesters. Since a fair number of industry professionals attend these sessions, they clearly offer us the opportunity for performing assessment of some important outcomes
2. We discussed all three of the proposed rubrics (which borrow from the existing ones). While the committee liked the rubrics, there was a suggestion that we should try to reduce the amount of text in each one. This applies especially to the one intended for the capstone poster session since this one will be completed by visitors to the poster session who will see it on just that day. (By contrast, the problem is not as critical in the case of the other rubrics since those are intended to be completed by the course instructors.)
3. One other suggestion was that it would be useful to identify/recruit some volunteers among CSE faculty to attend the poster session, check out several of the (CSE) posters, and complete the rubrics. Jeremy has agreed to do this; if we could identify one other person (in addition to the instructors of the capstone courses), that would help. Volunteers welcome!
4. The rubrics were approved and we will use them in each capstone design course, the capstone poster session, CSE 2501, and Phil 1338, starting this semester. (The details of how the assessment data from these rubrics from a section of each course and the poster session will be provided to the Undergraduate Studies Committee so that the results may be discussed in a committee meeting early next semester, and, similarly, for future semesters, remain to be worked out.)

The meeting was adjourned at 12:00 noon.

Next meeting: ??

09/12/'16

At the meeting: Spyros Blanas, Paolo Bucci, Al Cline, Wayne Heym, Jeremy Morris, Neelam Soundarajan, Paul Sivilotti, Chris Stewart, Nikki Strader, Huamin Wang, Yang Wang, Rafe Wenger; Viral Patel.

1. We spent the entire meeting talking about the preparations for the upcoming ABET evaluation of the BS-CSE program. Neelam passed out a handout that served as a basis for the discussion. (The handout summarizes some key points to consider in preparing for the evaluation, includes the full list of ABET (EAC and CAC) Criteria, the BS-CSE program outcomes, etc.)

   Here is a brief summary of the discussion:

   • ABET evaluations tend to place a heavy emphasis on the assessment, evaluation, and continuous improvement processes used by the program. The processes are required to be documented, they are required to be followed as specified, and the results of the assessments are required to be documented. The results must be analyzed appropriately, the results of the analysis must be documented and must inform program improvements.
   • The outcomes of the BS-CSE program are based on the ABET-specified outcomes.
   • Our assessment processes are POCAT, the exit-survey, and the alumni survey with POCAT being the main direct assessment. The results of the POCAT are discussed regularly in UGSC; the documentation of this analysis needs some improvement. One other issue is that, while the relation of this to specific course content is generally clear, the relation to specific outcomes needs to be documented more explicitly.
   • The results of the surveys are also analyzed in discussions in UGSC but, again, the documentation of this analysis needs improvement.

   We will continue this discussion in the next UGSC meeting. In particular, we will consider two newly designed/revised rubrics for the capstone courses (also included in the handout).

2. During the annual meeting of the department's Advisory Board at the end of Spring semester, Neelam made a presentation about our undergraduate programs (as is standard practice each year). Part of the discussion concerned the program educational objectives of the BS-CSE program. Immediately before that part of the discussion, Neelam had talked about CSE 2501 and Phil 1338 and their role in the program, in particular, the discussion concerning the ACM/IEEE Code. One of the board members suggested that we might want to consider revising our third PEO (which currently reads, "Graduates will be informed and involved members of their communities, and responsible engineering and computing professionals") to include a reference to the Code. We discussed this among committee members via email during the summer and agreed that this seems like a sensible idea. Neelam will work on coming up with language for the (third) PEO that will achieve this.

The meeting was adjourned at 12 noon.

Next meeting: 9/26