CSE Undergraduate Studies Committee
Minutes of Meetings (2015-'16)

Spring:	Mar. 8; Mar. 1; Jan. 19;
Fall:	Dec. 4; Oct. 2;

3/8/'16

At the meeting: Spyros Blanas, Matt Boggus, Paolo Bucci, Eric Fosler-Lussier, Wayne Heym, Jeremy Morris, Radu Teodorescu, Neelam Soundarajan, Paul Sivilotti, Nikki Strader, Kitty Reeves, Yang Wang, Rafe Wenger, Glenn Gainer, Cailin Pitt; Al Cline, P. Ramasamy

1. We spent the entire meeting talking about the capstone courses, our program outcomes, which ones are relevant to the capstone courses, how they are assessed, etc. This is a brief summary of the discussion:
   • ABET evaluations tend to place a heavy emphasis on the assessment, evaluation, and continuous improvement processes used by the program. The processes are required to be {\em documented}, they are required to be followed as specified, and the results of the assessments are required to be documented. Also , their evaluat
   • For the most part, our various capstone courses use a set of rubrics which, although not identical from one course to the next, are quite similar to each other.
   • The rubrics are used mostly for assessing students' communication skills and their team-working skills.

The meeting was adjourned at 4:10.

Next meeting: ??

3/1/'16

At the meeting: Spyros Blanas, Matt Boggus, Paolo Bucci, Eric Fosler-Lussier, Wayne Heym, Jeremy Morris, Radu Teodorescu, Neelam Soundarajan, Paul Sivilotti, Nikki Strader, Kitty Reeves, , Rafe Wenger, Glenn Gainer, Cailin Pitt.

1. Annual Forum: After a brief discussion, we decided that the Annual Forum should be held after the Spring break. We decided on Thursday, March 24, at 5:30 pm; room tba.
2. Junior POCAT: We decided, after some discussion, not to pilot a junior version of the POCAT for now. Some concerns were that the nature of POCAT is very different from anything that is considered a required (graded) part of any course. And given that the questions (related to 2221, 2231, 2321, etc.) that we would include in the Junior POCAT are not directly part of the junior project course's content, students might be upset about such a test. So we will think some more about this. (One could argue, for example, that if the results of the POCAT reveal some apparent problem related to, say, 2331, the instructors for that course could ask suitable questions in the final exam of the course to see if the students are not learning the material in the course sufficiently well or they are learning it but forgetting it by the time they are close to finishing the program, etc. It is not clear that the Junior POCAT help provide additional information beyond this.)
3. POCAT: This semester's POCAT will be in the week of March 28. Nikki will try to get one of the labs on a sufficient number of days so that the students can complete the Exit Survey at the same time.
4. POCAT: Roughly 200 students will be taking the POCAT this semester. We will have at least two versions of the test (with questions 1 (binary representations), 14 (databases), and 17 (big-O, omega, etc. having two or more versions). We should have at least one meeting after the POCAT is held to discuss the results.

The meeting was adjourned at 4:00.

Next meeting: 3/8/'16.

1/19/'16

At the meeting: Matt Boggus, Paolo Bucci, Eric Fosler-Lussier, Wayne Heym, Jeremy Morris, Neelam Soundarajan, Paul Sivilotti, Nikki Strader, Ken Supowit, Kitty Reeves, Yang Wang, Rafe Wenger, Glenn Gainer, Cailin Pitt.

1. Online courses: Currently, we have one course, CSE 1110, that is offered as an online course with all class work, except for exams, being online. We have also offered, on a one-time basis, a handful of courses 5241 and 5234. These offerings were for the (relatively small number of) students in the MGEL program. Since we have to apparently(?) continue supporting that program, these courses have to and will be discussed in the Curriculumm Comm. Another course for which an online version is being developed, at the requestt of Fisher College, is CSE 2111. This course will also be discussed in the Curriculum Committee. Since there is considerable overlap in the membership of that committee and UGSC and since, in the long term, we may (have to?) start offering online versions of other courses, including those taken by our majors, it seemed reasonable to discuss this in both committees.

   During the discussion, the following points were noted. One important question relates to budgets. It would seem that if a course were offered online, the administration might assume that you can be taken in any given term by a very large number of students with little additional budget support for additional instructors. Currently, it seems the college/OSU are counting students enrolled in online courses on par with students enrolled in in-person courses; but there has been no assurance this will not change in the (not too distant?) future. Indeed, as Gagan (who was not at the meeting but had shared his concerns before the meeting) noted, it is possible that OSU, in order to meet its promise of affordability might implement a differential tuition structure such as, e.g., students who take a full load of courses in the Spring may take some number of hours of online courses in the Summer without paying additional tuition for the latter. If such schemes were implemented, there is a potential that this will have an impact on the budget support for the depts. offering the online courses. Even in the case of the MGEL program, while the College of Engineering has been directly paying the instructors who teach the courses taken by those students, to our knowledge, there is no support to the dept. for the additional overhead of the courses, nor for the actual development of the courses. In any case, we need to continue to be aware of these possibilities although it is not clear that we will be able to influence any of the policies that the college/OSU might implement/change.

2. Assessment/evaluation: The next ABET evaluation of the BS-CSE program is coming up soon. The actual site visit will be during Fall '17 and much of the preparation, such as collecting materials from courses, writing the self-study document, etc., will happen during '16-'17. Neelam noted that one important concern is that many other programs (both in the college and across the country) have substantially strengthened their assessment activities. As a result, our assessment program does not, unlike during the past evaluation, look especially strong. Indeed, there is a possibility that we might be faulted for having a somewhat inadequate assessment/evaluation program.

   On the other hand, we cannot simply adopt most of the assessment processes implemented by other programs since they tend to be very labor intensive and will be especially so, given the large number of students in our program. Our main direct assessment activity is the POCAT; our indirect assessments are the exit survey and the alumni survey. Another mechanism that we use for obtaining feedback from current students is the Annual Forum. While the documentation of the results of the POCAT and their evaluation and resulting program improvements are reasonable, it would be useful to add at least one other direct assessment instrument.

   After some discussion of these points, one interesting suggestion that was made was the idea of introducing a junior POCAT. This would be modeled on the POCAT and administered in the junior project course (mainly CSE 3901, 3902). This suggestion was well received and Paul Sivilotti (ccordinator for 3901) and Matt Boggus (coordinator of 3902 in Roger Crawfis's absence) seemed to think that although the junior project courses already have a rather full agenda, it should be possible to do this since it should take no more than 20-25 minutes of class time. The intent of the junior POCAT would be to focus on assessing how well students understand the essential ideas from some of the required courses such as CSE 2231, 2321, and 2421.

   We will discuss this again at the next UGSC and try to implement it no later than Au '16 with a possible pilot in Su '16.

The meeting was adjourned at 4:05.

Next meeting: ???

12/4/'15

1. Preparations for ABET evaluation
2. POCAT results
3. Undergrads in 5xy9 courses

1. Feedback from recruiter: ECS forwarded to us feedback that Facebook recruiters sent to ECS following recent on-campus interviews of students (for both internships and full-time positions). In summary, Facebook felt that our students did not have a good grasp of data structures/algorithms (and analysis of algorithms), and did not have a good understanding of object-oriented concepts. Following a discussion in the previous UGSC meeting about this, it was felt that it would be useful to get a feel for the courses these students might have taken, what grades they might have got in those courses, the distribution of grad vs. undergrad students etc. Nikki was able to obtain this information and there was a further discussion:
   • Some of the students that Facebook interviewed were non-majors; i.e., they were students majoring in subjects other than CSE although these students may have taken one or two of our courses. A number of other interviewees were MS-CSE students who have also taken relatively few of our courses. Given this, it is difficult to draw any conclusions about our program/courses based on those students' performance.
   • Many of the CSE-majors who were interviewed seemed to be fairly early in the program and had completed relatively few CSE courses, in some cases were just starting to take CSE 2231 (Software II). So, again, it is difficult to draw any conclusions about our overall program based on those students' performance.
   • Facebook doesn't seem to have any GPA requirement for the students interviews. This may have had an impact on the group of students who interviewed with them.
   • In the future, Facebook may want to consider requiring that students who sign up for interviews have some or all of the following: a specified minimum GPA, a specified minimum average grade in the CSE courses, and a minimum no. of credit hours of completed CSE courses. We believe that, if Facebook were to impose some of these requirements, a more qualified group of students is likely to sign up for the interviews. Indeed, many of the other companies who have interviewed our students (at least some of whom have such requirements) have expressed overall satisfaction with the knowledge and skills of our students.
   Neelam will request ECS to send the above to Facebook.
   Note added: In email discussions following the posting of the minutes, the idea of creating a "feedback form" that we would ask recruiters to complete if they interview more than a handful (4? 5?) of students that tries to get precise feedback (e.g., what exactly did the FB recruiter mean when he/she said that our students didn't have a good understanding of OO concepts?) about our students' knowledge and abilities. This will be a useful addition to our assessment mechanisms. We will work on this.
2. POCAT results: The results of the Au '15 POCAT are available at: http://web.cse.ohio-state.edu/~neelam/abet/DIRASSMNT/POCATRESULTS/index.html. A number of points were noted, including:
   • The hypothesis that we had previously made about poor performance on question 1 (about binary numbers) being because of the mention of "finite state machine" and/or the mention of "digital encoding" in the question seems to be borne out; the performance in the version of the question that was a bit more direct was somewhat better, 50% compared to 30% ... still not good but not quite as terrible.
     Possible change: Maybe the Systems I course could spend a bit more time on this topic?
     Note added: After the minutes were posted, Gagan (who was not at the meeting) pointed out that some (adjunct) lecturers who teach Systems I spend a great deal of time, far more than was intended when the course was designed, on the intricacies of working/hacking with C; and this may cut down on the time spent on more basic ideas (such as binary numbers). We will try to work on this problem.
   • In the previous POCAT, it was suggested that the performance on question 14 (related to keys in Databases) may have been bad because of the noise that started up in the testing room at the time of test, midway through the test. So we didn't make any changes to the question to see what happens when there is no such distraction. Unfortunately, the results were equally bad.
     Possible changes: Add a sentence to the question that reminds students that "key" means *minimal* set of attributes necessary for the purpose. We will try this in the next test.
     Also, given the importance of the topic, it might be worth briefly addressing this topic in the capstone design course, especially 5911, where the projects often relate to databases.
   • The performance in questions 16 and, especially, 17, both related to algorithms, was also cause for concern (and, indeed, may provide added validity to Facebook's comments about our students' poor preparation with respect to algorithms). It was not clear what the problem here might be since the faculty who teach 2321 felt that question 17 was an easy one and should have had a higher performance. The faculty in the area will try to explore this further, possibly asking this same question in the 2321 final exam to see how students do and/or crafting a new version of the question for POCAT and/or proposing other changes in 2321 or other courses.
     Note added later: Mike Fritz tried question 17 in his 2321 finals. The results were as follows: there were 41 students in the class; the options A, B, C, D, E, F were chosen, respectively, by 0, 3, 0, 2, 13, and 23 students. So 23 out of 41 students (i.e., 56 percent) got the correct answer.

The meeting was adjourned at 2:35

Next meeting: Spring ...

10/2/'15

1. Preparations for ABET evaluation
2. POCAT results
3. Undergrads in 5xy9 courses

1. ABET: The site visit for the next ABET evluation will be in Autumn 2017. During the next two years, we will be spending an increasing amount of time on preparations for the evaluation and will be discussing it regularly in UGSC meetings. (The self-study will be due several months before the site visit; materials for the on-site visit will have to be collected during '16-'17.)
2. POCAT results: The results of the Spring/Summer POCAT are available at http://web.cse.ohio-state.edu/~neelam/abet/DIRASSMNT/POCATRESULTS/index.html. The discussion started with the performance of the Group 4 students on question 14, one related to databases. This was a revised version of a question that has been previously used; the idea of the revision was to try to tease out the exact problem that students seem to have with the question. Rather surprisingly, none of the 17 students who took this version of the test (all in the summer) got the correct answer! Various possible explanations were offered. Spyros also noted that while the underlying topic (normal forms) was central to databases at one point, it is no longer very important in practice and, given current architectures and systems, other considerations have become more important. This may suggest that the course needs to be revised accordingly.
   Nevertheless, given that the topic is, in fact, covered in some depth in current offerings of the course, there was considerable puzzlement about the performance of the students. One possible explanation was that this group of students was especially weak. But then Eric and Jeremy pointed out something interesting: that the performance of the students in the questions that preceded this one was quite comparable to those of the students in Group 1. At that point, Nikki recalled that, in fact, on the day that POCAT was administered in the summer, the test was held in DL 298 and about half-way through the test, noisy construction work started next door (in DL 280)! So it seems that it was not the students (or the course) that was the problem, it was the test environment! (This is the first time since we started using POCAT that the test has pointed to an environmental problem!).
   The committee decided it would not be a good idea to give much credence to the results of the summer POCAT and that it would make sense to use this modified version of question 14 in the POCAT this semester and look at those results when they become available. The discussion of the rest of the POCAT results (from Spring) will continue at the next meeting.
3. 5xy9 courses: CSE 5xy9 courses are primarily meant for faculty to recruit PhD students to join their respective research programs. Some well-prepared undergrads, especially those interested in research/grad school, would also benefit from them. But, occasionally, undergrad sign up for one of these courses without understanding the nature of the course, possibly because they are looking for a 2 cr hr tech elective to complete their program requirements; by the time they discover what sort of course it is, it is several weeks into the semester, possibly their planned last semester before graduation, and that clearly poses serious problems.
   To address this, Nikki proposed that the enrollment limit for the undergrad sections of these courses be, by default, set to 0; undergrad students will still find the course when they search and will be able to add to the waitlist; but then they will have to talk to the instructor to get permission to actually enroll and, at that point, the instructor will have an opportunity to ensure that the student understands the nature of the course and has the appropriate background for the course. After a brief discussion, the committee approved the adoption of this approach. Neelam will work with Kitty to take care of the implementation.
4. There was a brief discussion about feedback from a Facebook recruiter. The feedback was that our students seem not very good with data structures and algorithms and also did not understand important object-oriented programming concepts. Neelam requested ECS to get more detailed comments; here is what we got:

     Hello, looping back with you on this. Rebecca (recruiter) preferred to
     not send comments via email so we just got off the phone. So, with my
     limited knowledge of CSE, here goes :)
   
     The issues were mainly about understanding when to use certain data
     structures such as arrays, hash tables, loops and brute force;
     implementation of problem solving; many students just simply couldn't
     finish the problems;  students struggled with even the warm up exercise
     which was moving averages; struggled with coding, turning storing
     interpretation into code;  and asking the right questions during these
     kinds of coding interviews  (ECS is working on a technical interviewing
     handout for students).
   

The meeting was adjourned at 2:35

Next meeting: Oct. 23