CSE Undergraduate Studies Committee
Minutes of Meetings (2016-'17)
Committee Members:
Spyros Blanas, Matt Boggus, Paolo Bucci, Al Cline, Jeremy Morris,
Kitty Reeves,
Neelam Soundarajan (Chair), Paul Sivilotti, Chris Stewart, Nikki Strader,
Huamin Wang,
Rafe Wenger; Viral Patel (CSE student rep), Cailin Pitt (CSE student rep).
(The committee is looking for a CIS student representative.
If you are a BS-CIS major and
are interested in being on the committee, please email Neelam at
soundarajan.1)
Agenda:
- POCAT results from Sp '17
(available
here).
- Report on Annual Forum.
- ABET preparations
At the meeting:
Matt Boggus, Spyros Blanas, Al Cline, Mike Fritz,
Jeremy Morris, Kitty Reeves,
Neelam Soundarajan, Paul Sivilotti, Nikki Strader,
Viral Patel, Ross Vasko.
- There was an extended discussion on several of the questions on the POCAT,
including two of the new questions (one concerning the time complexity of
the recursive Fibonacci function and a software-engineering one from 3902
regarding cohesion and coupling). Some main points that were raised:
- The performance on the SE question was surprisingly poor (with, in the
case of one group, only 5%
of the students picking the correct answer). A couple of
possible explanations were suggested. First, students may not remember some
of the terminology from the course (since they may not have seen it elsewhere)
even if they understood the underlying concepts. Second, the question may
have included
references to too many different items and that may have caused some
confusion. We decided that it would be helpful, if possible, to split the
question into two (or three?) questions with each focusing on one of the
items in the current question. Matt Boggus who had come up with the original
question agreed to try to come up with these variations and we will try them
in future POCATs to see if we can pin down the problem(s) ...
- The performance on the three versions of question (1) [which we
have used many times in the past] was similar to that in the
past. Specifically, performance on the first version in which the
question was stated in the context of library books was slightly better than
the performace on the third version which was otherwise identical to
the first version but the question was in the context of finite-state
machines. More importantly, performance on the second version of the
question (also in terms of finite-state machines) which talked about
"digitial encoding" of information, was much worse. The conclusion was that
students find this terminology confusing (and it is not clear what it means
in the first place!); so we will stop using this version of the question.
- Following up on an idea that had been suggested at the recent
meeting of the department's Advisory Board, Neelam had sent the POCAT
to a senior person in industry (someone who, in fact, got his PhD from
our dept. some years ago) and asked for his comments. One suggestion
that Neelam received in reply was that this first question be replaced
(or supplemented) by one that presents a scenario in which a log
file doubles in size every two hours or so, with the disk on
which it saved being substantially larger than the current size of the
log file and ask what, if anything, should be done [ranging from,
"ignore it" to "buy the largest disk possible and forget about it" to
"I have no idea"]. We will try to come up with a question along these
lines and try it in upcoming tests.
- Mike Fritz had proposed a question that asked about the asymptotic
running time of a simple recursive program to compute the Fibonacci
function. Here again the performance, especially in two of the four
groups, was surprisingly poor. Ross and Viral noted that this
specific problem is discussed in some depth in Foundations I or II or
both. A couple of conjectures were offered. For example, the fact that
Java's int was used as the type of all the variables in the
code, especially combined with the fact that there was a statement in
the problem that two integers could be added in constant time
independent of their sizes, might have caused confusion among some
students ... one (perhaps extreme?) version of this was the idea that some
students might have decided that the algorithm would take constant time
since for large(r) values of the argument, for which the value of the
function could not be stored in an int, so the system would
crash; while for smaller values of the argument it will terminate in
some bounded time. A
somewhat surprising number of students also picked the option "linear"
as their answer for this question. Another troublesome point was that
3341, the core choice course on programming languages (which many students
do take) also talks about how to write efficient versions of the Fibonacci
function in a functional language (such as Scheme). Viral noted that
Foundations I and II stress the fact that exponential algorithms are, as
a rule, horrible choices; given this and given that this algorithm
is discussed in several courses may have conceivably led to
some students rejecting the "exponential" choice.
Mike will try to come up
with a version of the question that eliminates as much of the confusing
and/or extraneous language as possible so we can try to tease out the
problem(s).
- More generally, it was suggested that perhaps we should include
some way for the student to indicate that a particular question was
especially confusing or unclear or ambiguous etc. Students would be
instructed not to use this option unless the degree of confusion or
lack of clarity in a question was excessive ... this might allow us to
better understand the results and identify any potential underlying
problems in the related courses. We will try such an option in a
future test and see if that proves useful.
- Since the hour was nearly up, there was a very brief discussion of the
report on the Annual Forum that was held on March 22, specifically
item (5) of the report. Neelam noted that there was considerable interest
among students in a course (whose detailed contents may well vary from one
offering to the next) that presents a few important and powerful tools/
systems etc. with some essential coverage of the underlying concepts followed
by discussion of the technical details of the tool/system and how may be
used in practice including, possibly, a detailed assignment/project involving
the tool or system. One important consideration for an item to be part of such
a course would be that there is a cohesive set of conceptual ideas
underlying the tool or system that is not already part of our required
curriculum; the reason for this is that if the conceptual ideas underlying
the tool or system were already part of the required courses, students should
be able to explore the tool on their own once they had completed those course.
Currently, some students who are interested in such a tool or system
talk to an individual faculty member and depending on that person's
availability and willingness, register for an independent study with
that person; other students try to pick up the material as best as they
can on their own; and yet others simply ignore such tools and systems, focusing
instead on their course work. A course of this kind would make it simpler
and more effective for students to explore some of these systems.
One possible risk in having such courses is that students may fill up
their tech elective hours with a number of such courses with the result
that their overall technical foundations may be weak and/or lack
coherence. One possible way to address this potential problem is to
cap the total number of credits that this course PLUS any
independent studies and research hours that a student may include in
his/her tech electives at, say, 3 hours. We will explore this option
further and, if it seems reasonable, develop a plan for its
implementation.
The meeting was adjourned at 12:05.
Next meeting: ??
Agenda:
- Rubric results from Au '16 capstone poster session
(Recall: we recently designed a rubric for use in the capstone
poster session that we have at the end of every semester to
assess student achievement of several of our outcomes.
The rubric is available
here.
Jeremy Morris and Al Cline used the rubric in the poster session
at the end of Au '16. We will look at the results.)
- Rubric results from Au '16 CSE 5911, CSE 5912, using revised rubric
(Recall: we recently developed a simplified rubric for use by capstone
course instructors.
The rubric is available
here;
Murthy Narasimhan and Roger Crawfis used the rubric in their capstone
courses in the fall. We will look at the results.
- Rubric for use in the junior project course.
Paul Sivilotti is planning to use, in his 3901 class, the rubric we
recently designed. We will look at the rubric; (and will look at the
results from Paul's class at the end of the semester).
At the meeting:
Spyros Blanas, Paolo Bucci, Al Cline, Jeremy Morris, Kitty Reeves,
Neelam Soundarajan, Paul Sivilotti, Chris Stewart, Nikki Strader,
Huamin Wang, Ross Vasko.
- Assessment of posters at the capstone poster sessions: An important event
that the capstone course instructors introduced a few years ago and that
has became a regular part of all capstone courses is the poster
session. Each capstone project team is expected to prepare a suitable
poster documenting the problem that the team worked on; the approach used
in the design, implementation, and testing of the system; and main
conclusions/lessons learned.
Each semester (except the summer), the session is held immediately after
the last day of class
(during the "reading days"). The poster session at the end of the Spring
semester is part of the College of Engineering's capstone poster session
and is typically held in the Ohio Union; the session at the end of the Autumn
semester is only for CSE projects and is generally held in Dreese 113.
Nearly all of the teams in the various capstone course sections participate
in the poster session at the end of their capstone course; occasionally, a
team misses the session because of work or other conflicts such as job
interviews.
Thus the poster sessions are a good opportunity for doing a
high-level assessment of student achievement of some of the intended
outcomes of the capstone courses, to complement the instructors' assessment.
During summer/fall '16, we designed this rubric for this purpose and
piloted it during the poster session at the end of fall '16.
Jeremy Morris and Al Cline used the rubric to assess most of the
posters at the session. The results are available
here.
There was an extended discussion. The committee concluded the following:
- Overall, the rubric seems to be satisfactory.
- The appropriate average "expected level of achievement" for each of
the items in
this rubric should be "Agree", i.e., 3.0 (with '"Strongly Agree",
"Agree", "Disagree", "Strongly Disagree" being converted to 4.0, 3.0,
2.0, and 1.0 respectively in computing the average).
- One important question has to do with how to ensure that we get a resonable
group of people to use this rubric to assess the posters. An important
consideration here is that industry practitioners form an especially important
constituency for our program and it would be helpful to get feedback from them,
especially regarding student performance in the capstone design courses. A number
of industry professionals do come to the poster session but most of them do not
know much about our program and hence may not be in a position to offer useful
feedback.
At the same time, there is one group of industry practitioners, i.e., industry
professionals who teach some of our capstone courses, who are ideally
suited for this since not only do they know the program reasonably well, they
also usually attend the poster session. Hence, every fall and spring semester
(starting with Sp '17),
we will have each of the industry professionals who is teaching a section of
any of our capstone courses (5911, 5912, 5914, 5915) in that semester to
evaluate each of the CSE posters at the poster session at the end of that semester
and complete the rubric for each poster. The results will be discussed in a
UGSC meeting in either shortly after the poster session or at the
start of the next fall or spring semester if a UGSC meeting cannot be scheduled
in the days immediately after the poster session.
- Assessment of Projects by Instructors:
(Rubric;
Au '16 Results)
We briefly looked at the results but, given that we had results only from
a section of 5911 and a section of 5912, we decided to
wait until the end of Spring '17 for a detailed discussion of the rubric
and results from it; during the Sp '17 semester, we will have the instructors
for a section each of 5914 and 5915, the other two capstone courses, use
the rubric in their respective courses. This should give us a better picture
of how well the rubric is suited to the various courses and also allow us to
compare the results from the instructor rubric with
those from the poster session rubric. (The plan is to obtain/discuss
results from this rubric
for one section per year of each capstone course rather than every section
that is offered.)
The meeting was adjourned at 12:05.
Next meeting: ??
Agenda:
- Rubric results from Au '16 section of Phil 1338
Recall: we recently designed a rubric for use in Phil 1338 and
CSE 2501 to assess student achievement of outcomes related to
communication skills, analyze impact of computing, knowledge of
contemporary issues, and understanding of professional, ethical,
legal, security, and social issues (outcomes (f, g, h, j)); and, in
the case of 1338, team skills (outcome (d)). The rubric is
available
here
Dr. Bryan Weaver of Philosophy taught a section of Phil 1338 in
Au '16 and used
the rubric in his section. We will look at the results from that
section.
- Rubric results from Au '16 capstone poster session
(Recall: we recently designed a rubric for use in the capstone
poster session that we have at the end of every semester to
assess student achievement of several of our outcomes.
The rubric is available
here.
Jeremy Morris and Al Cline used the rubric in the poster session
at the end of Au '16. We will look at the results.)
- Chris Stewart who works with student groups such as ACM-W would
like to briefly discuss the question of suitable space for events that
such groups arrange (such as meetings with IT companies, etc.)
At the meeting:
Spyros Blanas, Paolo Bucci, Al Cline, Jeremy Morris,
Neelam Soundarajan, Paul Sivilotti, Chris Stewart, Nikki Strader,
Huamin Wang,
Rafe Wenger; Ross Vasko, Viral Patel.
- Facilities for events organized by student groups such as ACM-W:
Student groups such as ACM-W have been quite active over the last couple of years,
organizing numerous events of interest to students. Unfortunately, for a variety
of reasons, they have not been able to arrange for suitable room and other
(computers, mainly) facilities for these events and, on occasion, have had to
cancel the event. A recent example is a one-day workshop, sponsored by Capitol One,
that had to be canceled because a suitable room could not be reserved. Part of
the problem is the limited availability of resources compared to the demand from
student organizations. Another part has to do with the fact that many of these
events tend to be on weekends (including the canceled workshop) on after hours and
building access, etc., is a problem. Contributing to these is policies concerning
food etc. in particular rooms; e.g., the only buildings that were available for
the Capitol One event did not allow food and that would not make sense for an
all-day workshop.
Clearly, supporting our student groups in their activities is important. These
events not only help students learn from each other in a non-classroom setting,
they are also, given the nature of the events, key to preparing our students
for their future careers after graduation. Anecdotally, student groups in
other departments are able to access facilities for their events. We will try
to figure whether similar approaches will work for our groups as well as other
alternative approaches; Chris, Paul, Rafe, Nikki, and Neelam will work on
this. Suggestions are welcome.
- Results from outcomes assessment from Phil 1338:
Phil 1338 is a recently developed course intended to be a combination of
Phil 1337 and CSE 2501. Since the main topic of the course is ethical issues related
computing, it is very appropriate for our students. The difference with Phil 1337 is
that it includes a strong oral presentation component. This was the reason we
previously decided to treat Phil 1338 as meeting the requirements of CSE 2501; so
students who take that course are considered to have met the Engineering ethics course
requirement as well as the requirement of CSE 2501.
Given the important role that CSE 2501 and now Phil 1338 play in helping our students
to achieve several important outcomes (f, g, h, j), we worked with the involved
Philosophy faculty to develop a suitable rubric for use by both Phil 1338 and
CSE 2501. This is a highly revised version of a rubric that we had been using in
CSE 2501. The intent was to pilot the rubric in Phil 1338 in Autumn '16 and in
CSE 2501 in Spring '17. Dr. Bryan Weaver of Philosophy recently sent his results
from his Au '17 section of 1338 to Neelam and we discussed the results.
The rubric is available
here.
The last dimension in the rubric is related to team skills. Although CSE 2501
does not include a team-work component (hence this dimension of the rubric will
not apply in the case of 2501), Phil 1338 does include such a component.
The course, a typical
section of which has 40 students, is organized as follows: The course
meets thrice a week for 80 minutes each time. The presentations are all held on
one day (typically, Friday), the other two days being lectures by the instructor
on the philosophy topics. The class of 40 is organized into 4 "Teams" of 10 students
each. Each Team is organized into 4 "Groups", with two Groups having 3 students each, the
other two having 2 students each. Each Group works on a topic and prepares a
presentation on that topic to be delivered jointly by both or all three members
of the Group. All ten students in a Team are expected to attend
the oral presentations of each of the Groups in the Team and ask suitable questions
etc. Each group's presentation is 15 minutes long, followed by 5 minutes for
questions. Thus each Group forms a small team that works together closely on its topic
and presentation;
and each Team forms a loosely-knit team. This structure has worked well in all the
sections (at least thus far).
The results are from Dr. Weaver's Autumn '17 section are available
here. For most dimensions,
the average achievement was around 3 (on a 4-point scale). Based on the descriptions
in the rubric of the various levels for each dimension, this would be classified as
"satisfactory" (2 being considered "developing"). This may seem somewhat high, given
that these are mostly sophomores; but it must be noted that the instructor assigns
these values, keeping that factor very much in mind. In other words, if a student
were to demonstrate the same level of achievement in his/her oral presentations in
the capstone course, the resulting score for the student would most likely be lower.
One figure
that was surprisingly high was related to team skills. This may be due to a
comibnation of factors including that each Group was quite small and that they
worked together on one presentation (of about 10 minutes or so), so the extent
of the team work was rather limited. In any case, the results seemed quite satisfactory.
One question that came up was, what we should consider as an acceptable/satisfactory
level of achievement. Given the level of these students, it would seem that an
average of around 2.5 (or perhaps slightly higher) would be reasonable with a
higher expectation (of, say, 3.0) at the capstone level. That is what we will aim for.
- The discussion of the results of the assessment of the capstone poster session
was postponed to the next meeting since we ran out of time. (But the information
is available here:
The meeting was adjourned at 12:05.
Next meeting: (Most likely on) 3/1
Agenda:
- Preparation for ABET evaluation
- Assessments
- Course materials
- Self-study
- Feedback from Cisco recruiter (see message to UGSC mailing list of a few
days ago)
At the meeting:
Spyros Blanas, Paolo Bucci, Al Cline, Jeremy Morris,
Kitty Reeves, Paul Sivilotti, Neelam Soundarajan, Chris Stewart, Nikki Strader,
Rafe Wenger; Ross Vasko (BS-CSE), Viral Patel (BS-CSE).
- ABET preparations: Neelam summarized the preparations we have to make
for the upcoming ABET evaluation:
- Course materials: Materials, including samples of student work, have been
collected from about half of our (undergrad) courses. The plan is to collect
the materials from the rest this semester. One major task is organizing all the
materials in a somewhat uniform manner so that the evaluators don't have to
struggle when they go through the materials during the site-visit. Neelam has been
working on this.
- Assessments: We have been doing this; but, in preparation for the evaluation,
we have to reorganize the materials so that they will be
easily comprehensible to the evaluation team. [And, in addition, we will,
as usual, be spending
time this semester on looking at the results, including of the assessments (of
the capstone poster session and the rubrics for use in 2501/Phil 1338) that we
recently introduced, etc.]
- Self-study: The other major activity is the preparation of the self-study
that has to be submitted to ABET. The self-study will be officially submitted by the
college in June; we have to give a final draft to the college by late May.
Neelam will work on this and will ask for help from various faculty ...
- Feedback from recruiter: Engineering Career Services got an email from
Junilu Lacar of Cisco about our program. The main point of the message was to
convey his impressions about/ideas for our program,
especially the SW I, II sequence. The main part of his message read:
... one of the main topics of focus in these courses [SW I, II] is
Design by Contract. That's not bad in itself; I think DbC and the
ideas behind it are important for students to learn. However, nobody
that I know in the industry does DbC the way that it's taught in the
CSE Software I and II courses, especially not in Java, which is the
programming language used in these courses. I've conducted an informal
survey with several respondents from all over the world (I posted a
question and request for feedback on an online forum where I volunteer
as a moderator) saying that they've never done DbC, much less in the
way that OSU appears to be teaching it. This is just one example of a
serious misalignment of focus between academia and industry and I
think it's important to address this misalignment sooner rather than
later.
There was an extended discussion on Lacar's comments and SW I, II as well as the rest of
our program. First, and perhaps most
importantly, there has always been a tension between the conceptual focus of
our program and the immediate, practical needs that industry employers are most
concerned about. At the same time, feedback we get from alums who have been
out in industry for a few years often suggests that while the conceptual focus
of the program may not be what will help new employees of companies such as
Cisco "hit the ground running", they tend to be extremely
valuable, in the long term, for addressing deep and complex problems that large software systems,
especially those that perform critical tasks, often pose. Indeed,
before the meeting, Matt Boggus sent a message that read:
"I won't be able to attend ... but
wanted to share a couple of quick comments related to the Cisco
recruiter's feedback. During a recent visit from one of our alums
currently working at SpaceX, he mentioned they make heavy use of
design by contract (C++ and assembly rather than Java though) ..."
A second key point is that our curriculum is not a
uniform monolith, focusing only on conceptual matters. We have
numerous project-oriented courses (the junior project course, the
capstone courses, and other courses such as the Mobile Computing
course and the Info. Security projects course) that have a very
practice-oriented focus. Indeed, as Viral noted, even the projects in
SW I, II, especially the latter, can be very helpful in preparing
students for industry positions; he mentioned that one of the detailed
questions he
was asked at a recent interview with Amazon was the central theme of
one of the projects in SW II!
In the current instance, it was not exactly clear whether Lacar was
mainly concerned about some topics that he felt should have been
included in our curriculum but are currently not, or about some topics
that should not be included but are included, or some combination of
the two or something else. Our goal is to prepare students not just
for immediate employment in the computing industry after graduation
but also for a lifelong career in computing; indeed, some of our
students go on to research careers and our program tries to meet their
needs as well. Of course, if there are possible changes to the program that
may improve our graduates' preparation to achieve these goals, we
will certainly consider them. Toward this end and in order to make
sure that industry people are well informed about our program, we will
get in touch with Mr. Lacar not only to get more
information on any ideas he may have about possible improvements to
our program but also to
ensure that he is well informed about the program.
- In late November and early December, Neelam, Jeremy, and Paolo worked on
possible changes to the third PEO of the BS-CSE program to account for the
suggestion made during the Advisory Board meeting at the end of the Spring
semester. During the break, this was sent to the faculty mailing list for
possible revisions and then approval; faculty seemed comfortable with the
proposed change. We will update the information in the program's
website to reflect the revised PEO.
The meeting was adjourned at 12:00 noon.
Next meeting: ??
Agenda:
- POCAT results
- UG Forum report
- ABET preparations
At the meeting:
Spyros Blanas, Paolo Bucci, Al Cline, Jeremy Morris,
Neelam Soundarajan, Chris Stewart, Nikki Strader,
Huamin Wang,
Rafe Wenger; Cailin Pitt.
- UG Forum report:
- Some concerns were expressed at the forum concerning the variation,
among the various sections of Fnds I (CSE 2321) in the topics and depth to
which they are covered since this has a direct impact on how well a student
will be prepared for Fnds II (CSE 2331).
It is not clear that standardization
to the extent that is achieved among the various sections of Software I (2221)
is possible in the case of 2321; nevertheless, it may be possible to have
a set of slides that can serve as a possible model to clearly illustrate the
range of topics and the depth to which each is covered. Rafe will work with
Ken Supowit to explore this. Also Cailin noted that, when he took
2331, there was time spent on fairly detailed review of the algorithms-related
material from 2321 and that review helped him considerably. This is another
avenue worth pursuing (although it would come at the expense of some topic
coverage in 2331).
- One of the comments at the forum concerned that the introduction of
something like an online bulletin board on which important announcements are
posted. Currently, these are sent by email to the student mailing lists but,
given the volume of email that students receive, they can be easily overlooked.
Having a simple online board, prominently accessible via the main undergrad
website, that contains brief announcement, arranged in (reverse)
chronological order would help with this. Nikki will work with Wade in
the Advising Office to try to get this implemented soon.
- There was a question about the impact of the continuing popularity of
CS on enrollments in our classes that the forum report briefly mentions;
the answer is that, in almost each semester, we are having to scramble to add
new sections of various courses and often having to depend much more on adjunct
faculty than we have in the past.
- POCAT results: We discussed the POCAT results briefly. Some key points:
- For Question 1 (concerning representing information using bits), the
performance was extremely weak in Group 2. It was noted that the Group 2
version of the question uses the term "digital encoding" whereas the other
two versions simply talk about n-bit labels. It is possible that students
thought of "digital encoding" as some complex scheme and were thrown off
by that ... It was also noted that italics font was used to stress a
portion ("...number of states was doubled...") of the Group 2
version of the question and that may have impacted the
performance in this group. We will investigate this further.
- The performance on (the new) Question 12, related to pipelining, was
rather weak in two of the three groups ... we will have to see why this
might be. Hopefully, future tests will help with this.
- Performance on Question 15, related to algorithm analysis, was also
rather poor. Rafe pointed out (after the meeting) that the notation used
in the question was not the notation usually used in CSE 2331; it was more
C-like (whereas the notation usually used in 2331 is more Pascal-like).
If that is indeed the reason for the poor performance, we need to explore
ways to help students transfer their algorithms' knowledge to such contexts/
syntax; and this ways would, presumably, be in courses beyong 2331.
If, on the other hand, the problem is with students' poor grasp of
algorithm analysis techniques, the way to address that would be in 2331.
In any case, we will try to tease this out in future POCATs by creating
versions of this problem that use the C-like syntax on the one hand and a more
2331-like syntax, on the other.
- Performance on Question 14, related to data bases, in each of the
three versions, was extremely poor. Jeremy noted that 3241, the DB I course,
covers this topic and uses it in later material in the course. Cailin confirmed
that his section did that as well although he seemed to feel that he would have
struggled with the question as well. Jeremy and Spyros and other DB faculty
will have to discuss this and see how to address it.
The meeting was adjourned at 12:00 noon.
Next meeting: ??
Agenda:
- POCAT
- Rubric for junior project course
At the meeting:
Spyros Blanas, Paolo Bucci, Jeremy Morris,
Kitty Reeves, Paul Sivilotti,
Neelam Soundarajan, , Nikki Strader,
Viral Patel
- POCAT (general discussion):
One question that has come up occasionally is the correlation between
a student's performance in a course and his/her performance in POCAT on
questions related to that course. Because we put a lot of weight on
students' anonymity, there is no easy way to get information about such
correlation.
A somewhat different question has to do with a student's relative
performance in different courses; i.e., e.g., does a student's performance
in SW I, II say anything about how the student might perform in, say,
the junior project course? This information can indeed be arrived at
since we can look at, say, the advising reports of all graduating students
(after removing their names so anonymity is preserved), and arrive at
any correlation that might exist. We will see how feasible this activity
is; and, if it is, whether it gives us any useful information that might
inform possible improvements in the program.
(Note related to POCAT questions: Neelam mentioned that Chris Stewart has
suggested a conceptual question to replace the
more detailed and formula-oriented question on architecture that we
have been using in the POCAT. We will use this new question in this
semester's POCAT.)
- Rubric for junior project course:
Paul, Matt, and Neelam have been working on a rubric for use in the
junior project courses. We briefly looked at the
current version. The intent of the
rubric is to provide additional assessment of students' technical
skills related to building software systems of varying complexity,
as well as their understanding of social and ethical issues, and their
communication and team skills. The plan is to use (a possibly updated
version of) this rubric this semester in a section of 3901 (taught by
Naeem; Paul will talk to Naeem about this) and see how it can be
refined. Comments or suggestions for changes in the rubric are welcome.
Both assessment and teaching of team skills is, of course, challenging.
Currently, what we often do is to simply assign students to team projects
and hope they will manage to make it work. In cases where there are
extreme problems, such as a student essentially not contributing to the
work of the team that he/she is part of, the other students in the team
typically bring that to the attention of the instructor and the
instructor tries to intervene. Instructors also use peer assessment by
students in a team of the other students in the group; but many students
tend not to want to blame the other students so this is of somewhat
limited value (except perhaps for assigning grades). One problem that
Viral noted was that students often drastically underestimate the amount
of time/effort that a team member may have put in to get something
completed and this gives a distorted view of that member's contribution.
The question was
whether we could do more to help students develop effective team skills.
(Viral also noted that his most successful team experience was one in
which he was able to hand-pick the team members so that the team
included only students whom he knew as possessing the necessary
technical knowledge and willing to work hard! While this may have
worked in specific team projects, it is clearly not a general recipe
since, in practice, CS professionals may not be able to determine who
is included in the teams that they are part of.)
- Assessment of BA-CIS program:
Our current plan is to use an Exit
Survey for assessing the BA program. The question was whether we
should also use something like POCAT for the purpose. One problem here
is that the BA students are required to take far fewer of the CSE
courses than are BS students. The BA core consists of CSE 2221, 2231,
2321, 2421, 2501, junior project course. BA students are also required
to take at least 13 hours of additional CSE courses as electives; the
most common choices are 2331, 3241, 3521, 3541, 4471. So while it would
probably not be appropriate to use the same test as for the BS students,
it may be reasonable to have one that has some of the questions from that
test. We will work on this.
A second problem is the relatively small number of students in the
BA program.
One possible way to address this is to combine the results for an entire
year when doing the analysis. We will consider this as well.
The meeting was adjourned at 12:05.
Next meeting: ??
Agenda:
- POCAT
- Rubric for junior project course
At the meeting:
Paolo Bucci, Al Cline, Jeremy Morris,
Kitty Reeves,
Neelam Soundarajan, Chris Stewart, Nikki Strader,
Huamin Wang, Cailin Pitt.
- POCAT (general discussion):
- Our discussions of POCAT results, naturally,
focus on the performance of students in specific questions, especially in
cases where student performance is much weaker than expected (as specified
by the person setting the question; this information is included in the
last row of the relevant table in the results page). This is natural because
it allows us to identify possible weaknesses in particular courses and
come up with suitable fixes. While this is reasonable, there is a potential
problem since ABET is focused on assessment of student outcomes.
The POCAT results pages do include a table (the last one in each results page)
which provides information about the extent to which student achieve the
various outcomes, based on the relation between the various questions and
particular outcomes; but there is no information concerning expected
performance with respect to these outcomes.
After some discussion, we decided that we will arrive at this
information as follows: for each outcome, we will average together the
expected performance in each POCAT question that is related to that outcome
and treat that as the expected performance with respect to that outcome.
That will allow us to compare the actual student performance with respect
to that outcome with the expected performance and identify any substantial
differences between the two. Neelam will revise the results-processing script
to add this. (Note added: This has now been done so the third table in each
POCAT results page, starting from Sp '16, will include an extra row showing
the expected achievement percentage for each outcome, immediately below the
average achievement percentage.)
- Separating BS-CSE from BS-CIS: In POCAT, we have not distinguished
between BS-CIS and BS-CSE students. This makes sense since students in each
program take, essentially, the same CSE courses. But, for the ABET point
of view, we should separate them since ABET is only concerned with BS-CSE.
So we will start doing that from this semester, reporting the results from
these two groups of students separately from each other.
- POCAT (Spring'16 results):
- Question 1 (concerning the encoding of information in binary numbers):
In one version of this question, we didn't use the term "encoding", instead
using the term "label". The student performance for this version of the
question was much better than for versions that used the term "encoding".
Another important point is that this version didn't mention "finite state machines"; that may also
have contributed to the improved performance. We will try this one more time
to see if the results repeat.
- Question 14 (related to CSE 3241): There was a substantial difference
in the performance on Q. 14 between version 1 and 2; the latter included
a one-sentence reminder of the meaning of "key" and that seems to have
improved performance by over a factor of 2. We will try this one too one more
time to see if the results repeat. (Another difference, possibly not
as significant, was that version 1 included what might be considered as an
extra pair of braces; we will remove that in the next POCAT.)
- Question 12 (concerning 3421): Here again, there was a performance
difference in the different tests. Version 2 included the formula for runtime
while version 1 did not; the student performance on version 2 was better.
Chris offered to look into this more carefully to see what we should do.
- Junior project rubric: We didn't discuss this.
The meeting was adjourned at 12:05.
Next meeting: 10/24 (to be confirmed)
Agenda:
- New/revised rubric for use by capstone course instructors
- Rubric for use in the capstone poster presntation session
- New/revised rubric for use by CSE 2501 and Phil 1338 instructors
The rubrics are available
here.
Minutes:
At the meeting:
Spyros Blanas, Paolo Bucci, Jeremy Morris,
Kitty Reeves,
Neelam Soundarajan, Paul Sivilotti, Nikki Strader,
Huamin Wang,
Rafe Wenger; Viral Patel, Cailin Pitt.
- Background: Neelam summarized the background behind these new/revised
rubrics. Currently, the instructors of the capstone design courses and
CSE 2501 use a number of rubrics, such as one for assessing team work
another for assessing team presentations (both of those in the capstone
courses), one for assessing oral presentations in 2501, one for assessment
by peers of team work (in some capstone courses) etc. While these rubrics
do allow the instructors to assess the students' achievement of various
outcomes, the variety of rubrics used and the number of instructors
involved has made it extremely difficult to have an effective and
sustainable process to document and evaluate the assessment results across the
program as a whole to help identify possible improvements. In addition, we
are currently not obtaining any assessment data from the capstone
courses' poster session which is held at the end of fall and spring
semesters. Since a fair number of industry professionals attend these sessions,
they clearly offer us the opportunity for performing assessment of some
important outcomes
- We discussed all three of the proposed rubrics (which borrow from the
existing ones). While the committee liked the rubrics,
there was a suggestion that we should try to reduce the amount of text
in each one. This applies especially to the one intended for the capstone poster session
since this one will be completed by visitors to the poster session who will
see it on just that day. (By contrast, the problem is not as critical in the
case of the other rubrics since those are intended to be completed by the
course instructors.)
- One other suggestion was that it would be useful to identify/recruit
some volunteers among CSE faculty to attend the poster session, check out
several of the (CSE) posters, and complete the rubrics. Jeremy has agreed
to do this; if we could identify one other person (in addition to the
instructors of the capstone courses), that would help. Volunteers welcome!
- The rubrics were approved and we will use them in each capstone
design course, the capstone poster session, CSE 2501, and Phil 1338,
starting this semester. (The details of how the assessment data from
these rubrics from a section of each course and the poster session will
be provided to the Undergraduate Studies Committee so that the results
may be discussed in a committee meeting early next semester, and, similarly,
for future semesters, remain to be worked out.)
The meeting was adjourned at 12:00 noon.
Next meeting: ??
Agenda: BS-CSE program outcomes; EAC/CAC Criteria; Rubrics for use in
capstone design courses.
At the meeting:
Spyros Blanas, Paolo Bucci, Al Cline, Wayne Heym, Jeremy Morris,
Neelam Soundarajan, Paul Sivilotti, Chris Stewart, Nikki Strader,
Huamin Wang, Yang Wang,
Rafe Wenger; Viral Patel.
- We spent the entire meeting talking about the preparations for the
upcoming ABET evaluation of the BS-CSE program. Neelam passed out a
handout that served as a basis for
the discussion. (The handout summarizes some key points to consider
in preparing for the evaluation, includes the full list of ABET (EAC and CAC)
Criteria, the BS-CSE program outcomes, etc.)
Here is a brief summary of the discussion:
- ABET evaluations tend to place a heavy emphasis on the assessment,
evaluation, and continuous improvement processes used by the program.
The processes are required to be documented, they are required to
be followed as specified, and the results of the assessments are
required to be documented. The results must be analyzed appropriately, the
results of the analysis must be documented and must inform
program improvements.
- The outcomes of the BS-CSE program are based on the ABET-specified
outcomes.
- Our assessment processes are POCAT, the exit-survey, and the alumni
survey with POCAT being the main direct assessment.
The results of the POCAT are discussed regularly in UGSC; the
documentation of this analysis needs some improvement. One other issue
is that, while the relation of this to specific course content is generally
clear, the relation to specific outcomes needs to be documented more explicitly.
- The results of the surveys are also analyzed in discussions in UGSC but,
again, the documentation of this analysis needs improvement.
We will continue this discussion in the next UGSC meeting. In particular,
we will consider two newly designed/revised rubrics for the capstone courses
(also included in the handout).
- During the annual meeting of the department's Advisory Board at the end
of Spring semester, Neelam
made a presentation about our undergraduate programs (as is standard practice
each year). Part of the discussion concerned the program educational
objectives of the BS-CSE program. Immediately before that part of the
discussion, Neelam had talked about CSE 2501 and Phil 1338 and their role in
the program, in particular, the discussion concerning the ACM/IEEE Code.
One of the board members suggested that we might want to consider revising our
third PEO (which currently reads, "Graduates will be informed and involved
members of their communities, and responsible engineering and
computing professionals") to include a reference to the Code. We discussed this
among committee members via email during the summer and agreed that this seems
like a sensible idea. Neelam will work on coming up with language for the
(third) PEO that will achieve this.
The meeting was adjourned at 12 noon.
Next meeting: 9/26