CONSIDER ... the idea

1. The problem

It is widely accepted (see e.g., the extensive literature on active learning, constructionism, etc.) that one of the least effective ways for students to develop deep conceptual understanding of most topics is for them to sit in a classroom, listening to lectures delivered by a teacher. Especially in STEM courses, it often seems to be the case that many students do not get a good grasp of essential concepts related to the topic being presented in the lecture(s) even if, with the aid of homeworks and other activities, they seem to master the mechanics of answering specific questions of detail or solving typical text-book problems and answering exam questions on the topic (see literature by Hestenes, Mazur and many others on conceptual understanding in STEM courses). This applies even if the instructor tries to engage the class in brief discussions by posing appropriate questions during the lecture. Typically, in such discussions, a handful of the same set of students tend to respond while other students, at best, listen passively.

At the K-12 level, various innovative active learning approaches have been implemented but at the college-level, the standard lecture-based approach is very much the dominant mode. Although many STEM faculty may agree that a number of students in their classes do not get as good a grasp of essential concepts as they should, they are reluctant to make substantial changes in the way the courses are taught, given serious --and legitimate-- concerns regarding coverage of course material. Our goal is to address the following questions:

  1. Is it possible to use the affordances of suitable online technologies to engage undergraduate students in STEM courses in activities that will help them develop deep conceptual understanding of key concepts without affecting coverage of course material?; and

  2. If so, is it possible to develop a system/app that is easily usable by STEM faculty, who may not be IT experts, without expending a lot of effort?
As we will argue, the answer to the first question is a "definite yes"; and the answer to the second question is a "qualified yes", the system/app that we are building for the purpose, as part of the CONSIDER approach, being a work in progress. Further, the CONSIDER approach has a number of important additional benefits as described below.

Two points should be noted. First, there have been a few attempts, especially in introductory undergraduate courses on such topics as physics and chemistry, to introduce various learning activities. Indeed, there are some similarities between some of the ideas underlying the CONSIDER approach and peer instruction, the classroom-based approach that Mazur pioneered in introductory physics courses. But these attempts have not carried over to more advanced courses, possibly because of the concern related to topic coverage. Second, we should stress that the question is not about team project activities such as the capstone design projects that are part of most undergraduate engineering programs. Indeed, the engineering accreditation criteria require engineering programs to have an intense team-based capstone design activity/course. But the purpose of these courses is to help prepare students for professional engineering practice, including working effectively as part of project teams, rather than helping them individual students develop deep conceptual understanding which is our focus.

2. The background

Classic work on how children learn by Piaget and others showed that one of the best ways for children to develop deep understanding was by discussing the topic in question with their peers. While that seems obvious, Piaget's work identified two key requirements to make the discussion truly effective. First, the discussion must be among peers and should not involve someone like a teacher. The reason for this is that if a teacher were involved, then most learners in the group will simply accept whatever the teacher says without thinking deeply about the topic. But when it is only peers, each learner has to listen to what the others in the group are saying, try to understand them, and analyze what they are saying because, for all the learner in question knows, the position held by any of the members in the group, including himself or herself, may be right and the others who disagree with that position may all be wrong! Therefore, it is essential to understand and evaluate what everyone in the group is saying in order to identify the --hopefully!-- correct answer; in other words, carefully analyze the position expressed by each of the students in the group and then decide which answer is best. In the process, the learner is likely to develop deep understanding.

Second, the group of peers must include students who have different ideas about the topic/problem being discussed. After all, if they all had the same idea/answer (independent of whether it is right or wrong), there would be nothing for them to discuss. And, perhaps surprisingly, it turns out that even if everyone in a group starts out with a wrong answer (but different wrong answers!), then, by the end of the discussion, several of the students in the group often get to the correct answer! This seems to happen because when a given student analyzes other students' wrong answers to see why they are wrong and, in turn, listens to their analysis of why his/her answer is wrong, that can force the student in question to fundamentally rethink the whole topic and, in that process, deepen his/her conceptual understanding and, incidentally, arrive at the correct answer to the original question.

While Piaget's original work was with very young children, a number of researchers, e.g., Doise and Mugny, have explored similar approaches, mostly in K-12, mostly in-class, and found similar results; i.e., that groups of 4--5 learners with different conceptions, engaging in discussions in which they try to understand and analyze each others' conceptions to identify weaknesses in each, can help all students in the group develop improved conceptual understanding. It may also be worth mentioning that a number of researchers have made the case that the ability to engage in effective scientific argumentation is, by itself, an important skill that students, especially STEM students, ought to acquire. These researchers have investigated various approaches, including online systems, to help students acquire such skills. While our approach will indeed help students hone these skills, our primary motivation is to help students develop deep conceptual understanding in specific STEM topics; improving their scientific argumentation skills is a side-effect rather than a primary focus.

3. The idea

Thus a key part of the thesis underlying our work is that a very effective way for students in undergraduate (and graduate-level) STEM courses to develop deep conceptual understanding of key topics is by engaging in group discussions with each group consisting of 4--5 students who have different conceptions of the topic in question, with each student trying to understand, analyze, and poke holes in each of these conceptions, including his/her own, to arrive at a refined/modified conception; we will call such groups heterogeneous group; it is important to note that the heterogeneity is on the basis of the students in the group having different conceptions of the topic, not such factors as gender, ethnicity, etc. The rationale is that the process of engaging in such careful analysis and discussion of alternative conceptions will enable each student in the group to develop deeper understanding of the topic than would be possible otherwise.

But there are a number of potentially serious difficulties in enabling effective discussions of this kind in STEM classrooms. First, unlike students in elementary and middle schools, or perhaps even college students in social sciences, undergraduate students in STEM courses tend to be unwilling to participate in extended discussions with their peers. This may especially be the case with women students as well as students from other underrepresented groups; compounding this problem is the stereotypical opinions concerning the abilities of students from particular groups that, according to the work of some researchers, are fairly widespread. Further, while some students, not necessarily those with the most thorough understanding of the topic, may be quick with their responses, others, perhaps more thoughtful ones, may prefer to take time to formulate their responses carefully and this type of discussion is not conducive to that.

Second, class sizes tend to be large, thirty students or more being common. So it is difficult to organize students into heterogenous groups of 4--5 each, and have each group engage in deep discussions without the groups disrupting each other. Moreover, class meeting periods are an hour or less and that may be insufficient for these groups to engage in deep discussions; and, of course, this relates to faculty's concern about potential negative impact on topic coverage.

Third, these discussions are ephemeral and a student will have to, perhaps when studying the textbook chapter on the same topic, rely on memory to recall what was said and to relate it to his/her original conception of the topic as well as the textbook material, thereby reducing the long-term value of the discussion.

This brings us to the second key part of the thesis underlying our work which is that all of these problems can be effectively addressed by exploiting the facilities of appropriately designed online systems.

4. CONSIDER

CONSIDER is an approach and an online system that enables instructors of undergraduate or graduate-level STEM courses to have students in the course organized into small heterogenous groups which, for convenience, we will henceforth abbreviate to HGs, with each group consisting of students with different conceptions about the topic under discussion; and have the students in each HG engage in discussions with other students in the group to understand and analyze each other's conceptions, listen to the other students' analysis of their own conception, and appropriately revise/refine their original conception of the topic; and do this not only without compromising topic coverage but also providing a number of additional benefits. The name, CONSIDER, is an acronym for Conflicting Student Ideas Discussed, Evaluated, and Resolved (or Refuted!)

Specifically, the approach works as follows:

  1. Following class discussion on a given topic, the instructor will post, on the CONSIDER web-app, a suitable question or problem related to the topic that will require a two-part answer. The first part will be a multiple-choice component and the second will be a more detailed answer.
  2. Within a specified time, typically 24-36 hours from the time that the question was posted on the app, each student will be required to log into the app and post his/her individual answer to the question. After posting his/her answer, the student may log in again and freely edit it until the deadline.
  3. At the end of the 24-36 hour period, students in the class will be organized into HGs of 4--5 students based on their understanding of the topic, as indicated by their answers. In some cases, depending on the nature of the topic/question and the specific set of multiple-choice options, the formation of the HGs can be done automatically by the app, based on the option chosen by each student in the class; in other cases, the instructor (or possibly teaching assitants) will have to go through the more detailed answers posted by each student to form the HGs; more on this below. For convenience, in the rest of this page, we will assume that each HG will consist of 4 students.
  4. Once the groups are formed, typically within 6--8 hours of the deadline for the individual students to post their answers (or far shorter if the groups are formed automatically), students will receive an email from the app informing them that the discussion phase of the topic had begun and that they should log in and participate in the discussion.
  5. When a student in one of these groups, say, HG5, next logs in, he/she will see the answers submitted, individually (in step 2), by each student in HG5. The student will not know the identities of the other students in HG5. Instead, the students in the group will be referred to as, S1, S2, S3, S4. Thus what this particular student, say, S2, will see will be the answers submitted by each of S1, S2, S3, S4 of the group HG5. Students in each of the other groups will also be referred to as S1, S2, S3, S4; but there will be no confusion since students in a given group will not see the submissions of students in any of the other groups.
  6. The discussion in each HG will be organized as a series of rounds. Each discussion round will (typically) be 24 hours long. In each round, each student in the group will be required to make one post; the student may log in as many times as he/she chooses and freely edit the post until the end of the round; other students in the group will not see the post until the current round ends and the next round begins.
  7. The primary point of each round is for each student in an HG to respond to the posts made, in the previous round, by each of the students in that HG. We will label the various rounds R1, R2, etc.; for convenience, we will also use R0 to refer to the initial "round" in which students posted their initial, individual answers, step (2) above.
  8. During any round, when a student, say, S2 of HG5, logs in, he/she will see the submissions made by each of the four students S1, S2, S3, and S4, of HG5 in the previous round. For each of those submissions, S2 will be required to choose one of agree, disagree, or is neutral/unclear about the submission, by clicking on the corresponding button in the app. Note that S2 is required to do this also for his/her own original submission; indeed, the whole point of the discussion is that, after reading the previous round submissions by the other students in HG5, S2 may no longer agree with he/she posted in that previous round! In addition, S2 will, as part of his/her submission, post a brief text explanation of his/her current position on the question and his/her choices of agree/disagree etc. (Although the focus of S2's post for the current round will be to respond to the posts made in the immediately preceding round, S2 will be able to scroll through the posts made by HG5 students in earlier rounds (if any) and refer to them in his/her textual explanation. Of course, if the current round is R1, then the only preceding round is R0 in which each student posted his/her indvidual initial answers.)
  9. The number of discussion rounds, N, will be determined by the instructor and the information will be included when the instructor posts the question on the CONSIDER app in step (1) at the start of the activity.
  10. Following the last discussion round RN, there will be a final summary round in which each student in each HG will post his/her final answer/position on the question under discussion; and, also, post a brief summary of the discussion in his/her HG and how it influenced (or not, as the case may be!) his/her final answer. The summary round will typically be 36--48 hours long; the duration will be determined by the instructor and also posted on the app at the start of the activity. As in earlier rounds, during the summary round, the student may log in as many times as he/she wishes, until the deadline, and edit his/her submission. During this round, the student will be able to see what each of the students in the group posted in the now-ended final discussion round (and, if necessary, scroll back through the posts made by the students in the group in earlier rounds).
Once the deadline for the summary round has passed, students in the group may log in to the app and will be able to read the final answers/positions adopted by each student in the group; and, if they are interested, scroll through the complete discussion, starting with the initial round posts of all the students in the group.

A central point to note is that the intent is not, unlike is the case with many approaches to collaborative or cooperative learning, for the students in a given HG to arrive at a consensus answer. That may happen but the goal is for each student in the group to refine/revise his/her understanding of the topic by carefully analyzing the diverse positions held by the different students in the group, considering the other students' analysis of his/her own position and of each other's positions, and, on the basis of this, to arrive at a final position with which one or more of the others in the group may agree or disagree to varying extent.

5. The benefits

The CONSIDER approach addresses the various difficulties associated with having in-class discussions in STEM courses that we considered earlier and offers additional benefits.
  1. Clearly no class-time is used for the discussion so there should be no impact on course coverage.
  2. There is no possibility of some students in a group dominating the discussion with the other, possibly shy students, being silent bystanders. The structure ensures that each student in the group participates equally (although, of course, the length of textual explanations submitted in each round by the different students in a group may vary from student to student). Moreover, students who are quick to formulate responses to others' positions as well as those who prefer to take time to construct their responses carefully will be able to participate effectively.
  3. The fact that students can freely edit their posts for a given round until the end of that round gives students the assurance that they will have sufficient opportunity to correct any silly mistakes they might make in their posts, possibly as a result of misunderstanding the previous round's post of another student.
  4. The anonymity of students in each group ensures that any preconceived notions that individual students may harbor about the abilities of other students (based on gender, ethnicity, etc.) in the class does not have any effect on the quality of the discussion or the benefits that students in the group draw from it. Interestingly, another benefit of anonymity, one that we had not anticipated, seems to be that students are willing to explore their ideas more freely without fear of being seen as stupid by their peers
  5. Once the discussion is complete, students in the group will have access to the complete transcript of the discussion and will be able to refer to it as they move on to other topics or when preparing for final exams, etc. Indeed, once the CONSIDER activity on a given topic is complete, the instructor may find it useful to briefly revisit the topic in class, focusing on some of the common issues that may have surfaced in the discussions of several of the HGs. This will be very effective since many of the students have already spent time analyzing the issues carefully and will be eager to contribute to the class discussion. In addition, the instructor may find it useful to make the discussion transcripts of some specific groups that were especially illuminating available to the entire class so that everyone can benefit from them.

6. The potential problems

After using a prototype version of the system in a couple of courses, we have heard a few complaints from students. First, some students felt that the 24-hour duration of each round was too long and prevented effective discussions. This would certainly seem to be an issue, especially when compared to in-class discussions. Of course, the advantage of the 24-hour round is that it allows all students, both those who are quick on their feet as well as ones who are deliberate in formulating responses to participate effectively and the other advantages noted above. Moreover, the 24-hour duration allows students who may have different course, work, and family schedules to participate without impacting their other activities.

On the flip side, there have also been some complaints that expecting students to log into the app, read the previous round submissions of the other students in the group, and post a response every 24 hours is too demanding! This point can indeed be a legitimate concern, especially at certain time points in the semester when students might be working on midterms in other courses or have deadlines for projects in those courses, etc. This is something that the instructor will need to try to account for in scheduling the CONSIDER activities. Moreover, based on our experience, we believe that for most problems/topics that a STEM instructor might want to construct a CONSIDER activity for require only two discussion rounds, so this is not as serious an issue as it might seem. The fact that two discussion rounds is sufficient came as somewhat of a surprise to us because initially we had thought that we would need a minimum of four rounds of discussion for most topics but that turned out not to be the case.

From the instructor's point of view, one serious issue relates to constructing suitable questions with appropriate multiple-choice answers such that students with different conceptions of the topic are likely to choose different answers; if suitable multiple-choice options can be specified, the system can automatically form the HGs. In the absence of this, the instructor and/or teaching assistant will have to go through the initial, individual submissions of the students to form the HGs. This is not problematic in a class of size 30-40 but it is clearly not scalable. We are currently exploring possible alternatives to help with this.

One interesting aspect that we had initially thought might be problematic was that, given the amount of time for each round (including the initial, individual submission and the final summary round), that students will consult on-line resources and the like in formulating their posts. This did, indeed, happen; but, if anything, this is desirable since students are taking the time to explore the topic. Indeed, depending on the topic, in many of the HGs, students often quoted, with attribution, Internet sources with the group discussing the validity of the claims made in those quotes. This is a far cry from the often disinterested, last minute answers that students many students submit for home-work problems.

7. The current status and plans

We implemented the initial version of the CONSIDER system as an app for use on (Android) phones and tablets. But it became clear very soon that a system of this kind should be used on a laptop or desktop. The reason is that the amount and kind of text that students need to enter into the system in formulating and submitting their posts for each round is typically such that it would be quite difficult to do it on a hand-held device. Perhaps even more importantly, the posts that a student sitting at a laptop or desktop makes are likely to be more carefully and thoughtfully written. Hence we have moved to a web app intended to be used on a desktop or a laptop. The current version is accessible at CONSIDER system.

Some of the concerns that students who have used the prototypes have expressed (apart from the issues related to the mobile version) have to do with the naturalness of the user interface (UI). For example, one concern had to do with the fact that when a user made a submission, the system did not provide a suitable acknowledgment; and there were instances where students thought they had made a submission but, given the lack of suitable acknowledgment, were not sure. There were also concerns about the ease with which posts from earlier rounds and even the original question was accessible during later rounds (including, especially, the final summary round) of the activity. We are evaluating these concerns and exploring possible changes.

Instructors have also encountered problems with the (instructor portion of the) UI. Thus, e.g., there is no simple way for the instructor to obtain the transcripts of the discussions in all of the HGs. As another example, the instructor has no way to see what a given problem/activity will look like to a student in one of the HGs. Future iterations of the system will address these.

A more conceptual concern has to do with the problem already mentioned in Section 6; i.e., the difficulty that STEM instructors are likely to face in coming up with suitable multiple-choice options for questions that would be appropriate as CONSIDER activities in their courses. One possible alternative that we are exploring may be summarized as follows. The instructor designs a suitable problem for posting on CONSIDER and, with the problem description, also includes, say, three (declarative) statements concerning the problem; and rather than choose one of a set of multiple-choice options, the student is required, for each of these statements, to pick one of the following:

Note that the three statements are not and do not claim to be complete solutions to the problem being discussed but may be (possibly incorrect) pieces of the solution or aspects directly related or unrelated to the problem/solution. Since the student is required to choose one of the four options listed above for each statement, it is highly likely that different students will choose different combinations of answers, enabling the system to automatically form suitable HGs; moreover, unlike in the case of multiple-choice answers, even if a couple of the statements were directly related to the correct solution to the problem, there would still be pieces of the solution that it would be up to the student to construct. For now, this is at a conceptual phase; it is only after using it on a few problems, which we are currently in the process of planning, that we will be able to judge its effectiveness.

Another conceptual question has to do with assessing the performance of individual students in a CONSIDER activity. One component of this is, naturally, the correctness of the solutions to the original problem that individual students submit, as part of their final submission. That is no different from the way that instructors would assess standard homeworks. So our question here has to do with the assessment of the effectiveness of the student's contribution to the discussion in his/her HG. Systems based on the classic Toulmin framework are most commonly used for evaluating the quality of arguments but such an approach seems inapproporiate or, rather, inadequate for our purposes. The problem is that the framework considers such questions as whether a given argument is based on appropriate data, warrants etc. of the given argument. In our case, though, a key point of a CONSIDER activity is for a student to integrate/refine his/her own position based on the positions of the other students in the group. After all, the submissions of a student who completely ignores important ideas expressed by the other students in the HG could conceivably qualify as a high-quality argument as judged by a system based on the Toulmin framework. An important goal of the CONSIDER project, therefore, is to come up with an assessment scheme that accounts not only for the quality of the student's argument in explaining/defending his/her position but also in suitably integrating/refining/modifying that position based on the positions of the other students in the particular HG. This point may also be phrased in terms of the idea of "arguing to learn" which is the focus of the CONSIDER approach vs. "learning to argue" which is the focus of much other work on argumentation in K-12 and in higher education. Perhaps not surprisingly, the focus in systems based on the Toulmin framework in this other work is on judging how well a student has learned to argue by assessing the quality of the student's argument; the Toulmin-framework-based system that we hope to come up with will instead focus on assessing how well the student is leveraging the argument to learn.

Comments are welcome: neelam@cse.ohio-state.edu.