My university has begun the process of extending our system of assessment into the general education program. Before I began this process I thought I knew the purpose of general education and I was happily ignorant of its size.
Before I launch into thoughts concerning the purpose of general education, let me first comment as to its extent. At my university, each student must have at least a total of 124 credit hours to graduate. Of those 124, at least 46 credit hours must be general education. This is slightly over one-third of the total requirement. Somewhere among the remaining two-thirds of those 124 hours, the student must specialize in something and do some exploring on his own.
In the book Excellence without a Soul, Harry Lewis remarks that the idea that students should know one thing very well but also know a little about everything has long been an ideal at Harvard where he taught. This same ideal permeates higher education throughout the United States and informs our attitude toward general education. In addition, there are other functions performed by general education.
In a setting where many of the students are the first generation of their families to attend college, there is often a lack of knowledge among the students and their support groups of pathways open to them through education. General education provide samples of areas in which students might find a career that they might not otherwise have known about. We all have heard the student stories that begin "I'd never even heard of X before, but I took Professor Y's Introduction to X because my adviser made me, and it changed my life."
There are also aspects of general education that are foundational to later learning. At my school, we have decided these are writing, mathematics, and communication. We refer to these as our Basic Skills.
It is with the Basic Skills that we have chosen to begin as we extend our assessment system to general education. This is being done for a number of reasons. The disciplines involved in this area are almost universally (as near as can be expected on a university campus) accepted as being a part of general education. In addition, Basic Skills only includes courses being taught by three departments, and, therefore, the politics involved are more manageable.
Even though we are only beginning, it has been discovered that expectations for student learning outcomes in these areas can vary widely. We are currently looking at writing, and, without being too surprised, we've learned different disciplines expect different styles of writing. Some expect students to write in a literary style, some in a business style. Some expect essays and some expect text. As we continue to examine mathematics and communications, I expect similar results.
Our purpose, however, is not to bludgeon the faculty into saying there is a single correct vision of any of these areas. We must open lines of communication so that the needs of the students are correctly understood by those whose charge it is to teach them. And, as assessment is our charge, we must do that in such a way that we can measure how well these needs are being met.
Wednesday, July 28, 2010
Tuesday, July 27, 2010
Closing the loop
One phrase that occurs repeatedly in the language of assessment is "closing the loop." It comes from the practice of illustrating processes using graphs or networks. The idea is that one collects data concerning an activity and then reexamines the activity in the light of that data. Some do not like the phrase and prefer to say "using the data." It all amounts to the same thing.
One cannot over-emphasize the importance of closing the loop/using the data. Without this piece, the entire process is pointless. Regardless of how efficient we are, regardless of how simple we keep the process in toto, it still requires energy, i.e. effort and expense. I believe the Second Law of Thermodynamics is at work. Given the expense of taking data, if that data is not used, then money has been wasted. This is a tautology.
Now, having said that, I need to be careful. Sometimes we must monitor things that are in good shape and don't require our action just because they are important. For example, even people who don't have high cholesterol have their cholesterol monitored regularly; they just don't have it monitored as regularly as those who have issues in that area. But we do keep an eye on it because there are consequences if it creeps out of the safety zone. Similarly, there are areas where a particular department might excel and have done so over a historically lengthy interval, and there is no need for change in that area, but it is so important that it should be monitored. In such cases, one can take measurements at longer intervals, such as every three years or every five years.
The point is that data is taken to be used. This is an important point to make, especially in the initial stages of setting up a system of assessment.
In many cases, assessment is rolled-down from above upon groups which are unevenly prepared. In some cases, the message gets through that "Someone up there wants numbers." The response is then to "Come up with some numbers," irrespective of whether or not those numbers are meaningful. This is, of course, not the best possible response.
We don't take data simply for the great joy of taking data. It is done for a purpose.
When I check the outdoor temperature, it is to determine whether I need to wear a parka or gym shorts when I go outside. Data taken in the act of assessment similarly should have a purpose. Measures should be implemented with the idea that particular results will result in particular actions. When I check the outdoor temperature, it measuring the temperature on Mars for which my actions do not matter. I measure it outside my own house. Similarly, whatever departments measure should have some meaning to their mission at the university.
One cannot over-emphasize the importance of closing the loop/using the data. Without this piece, the entire process is pointless. Regardless of how efficient we are, regardless of how simple we keep the process in toto, it still requires energy, i.e. effort and expense. I believe the Second Law of Thermodynamics is at work. Given the expense of taking data, if that data is not used, then money has been wasted. This is a tautology.
Now, having said that, I need to be careful. Sometimes we must monitor things that are in good shape and don't require our action just because they are important. For example, even people who don't have high cholesterol have their cholesterol monitored regularly; they just don't have it monitored as regularly as those who have issues in that area. But we do keep an eye on it because there are consequences if it creeps out of the safety zone. Similarly, there are areas where a particular department might excel and have done so over a historically lengthy interval, and there is no need for change in that area, but it is so important that it should be monitored. In such cases, one can take measurements at longer intervals, such as every three years or every five years.
The point is that data is taken to be used. This is an important point to make, especially in the initial stages of setting up a system of assessment.
In many cases, assessment is rolled-down from above upon groups which are unevenly prepared. In some cases, the message gets through that "Someone up there wants numbers." The response is then to "Come up with some numbers," irrespective of whether or not those numbers are meaningful. This is, of course, not the best possible response.
We don't take data simply for the great joy of taking data. It is done for a purpose.
When I check the outdoor temperature, it is to determine whether I need to wear a parka or gym shorts when I go outside. Data taken in the act of assessment similarly should have a purpose. Measures should be implemented with the idea that particular results will result in particular actions. When I check the outdoor temperature, it measuring the temperature on Mars for which my actions do not matter. I measure it outside my own house. Similarly, whatever departments measure should have some meaning to their mission at the university.
Monday, July 26, 2010
Creating the environment, a culture of continuous improvement
As I've mentioned earlier, I am a mathematics teacher and have been teaching the subject in one way or another with varying levels of intensity since 1983. As a teacher, I've never been able to open a student's head and just pour knowledge in. Lately, I've had fantasies of surgically implanting a USB port in the skull of each student and just plugging him in so that he can slurp up the knowledge. That is unlikely to happen any time soon.
My method of teaching is to structure my classes with a time-line. I set learning goals and embed those goals in a calendar. I then organize the material and try to personalize the presentation of it to the needs of each class collectively and each individual within that class as much as possible by using language that the class understands, using a suitable idiom.
This is a learning environment. Within this environment, the student must apply his own efforts and abilities if he is to learn. There are students who are better at this than others, and I am not necessarily talking about students who are more intelligent. What I mean, is there are students who have cultivated habits which are conductive to the act of learning.
There are teachers as well who have, in an intentional way, acquired habits which improve their teaching. These people will be better teachers on the day they retire than on the day they started. They will be better teachers next year than they are this year.
Part of the business of a healthy educational institution is to create an environment which fosters that sort of teacher. We must create what is called "a culture of continuous improvement."
I use that phrase even though over the course of my career I've seen it misused extensively. Having been a mathematician, I'm not unfamiliar with folks who interpret words in a direct way, and I've seen that phrase taken to mean that there is an expectation that every class will improve its scores every year. This is, of course, idiotic. Because of random variation within each class, it is unreasonable to expect scores that are continually increasing.
It is not, by way of contrast, unreasonable to expect an environment where teachers are encouraged to improve their teaching and given the means, within the available resources, to do so. Indeed, far from being unreasonable, this is a natural expectation.
The creation of an institutional system of assessment is a part of that environment.
My method of teaching is to structure my classes with a time-line. I set learning goals and embed those goals in a calendar. I then organize the material and try to personalize the presentation of it to the needs of each class collectively and each individual within that class as much as possible by using language that the class understands, using a suitable idiom.
This is a learning environment. Within this environment, the student must apply his own efforts and abilities if he is to learn. There are students who are better at this than others, and I am not necessarily talking about students who are more intelligent. What I mean, is there are students who have cultivated habits which are conductive to the act of learning.
There are teachers as well who have, in an intentional way, acquired habits which improve their teaching. These people will be better teachers on the day they retire than on the day they started. They will be better teachers next year than they are this year.
Part of the business of a healthy educational institution is to create an environment which fosters that sort of teacher. We must create what is called "a culture of continuous improvement."
I use that phrase even though over the course of my career I've seen it misused extensively. Having been a mathematician, I'm not unfamiliar with folks who interpret words in a direct way, and I've seen that phrase taken to mean that there is an expectation that every class will improve its scores every year. This is, of course, idiotic. Because of random variation within each class, it is unreasonable to expect scores that are continually increasing.
It is not, by way of contrast, unreasonable to expect an environment where teachers are encouraged to improve their teaching and given the means, within the available resources, to do so. Indeed, far from being unreasonable, this is a natural expectation.
The creation of an institutional system of assessment is a part of that environment.
Thursday, July 22, 2010
Thoughts on sampling
My academic discipline is mathematics. It is natural for someone to say, "Oh you know a lot about statistics," but the truth is more complicated.
I am a topologist, and I've only ever had one course in statistics which was as an undergraduate and was theoretical. However, at a school like ours, we are not allowed to be to picky as to what we teach, so, as a consequence of this, I started teaching our elementary statistics course about fifteen years ago and discovered that I liked the subject. Now, after having taught it for that length of time, I cannot claim expertise, but I can say that I do understand some of the basic ideas.
One of the basic ideas is that of a sample. I would like to explain that to you.
Statistics is about the search for answers in a particular spirit. We begin with the need to know, but with the realization that we can't know exactly. Given that, we proceed to find the approximate answer with the realization that the approximate answer will, in a certain number of cases, not be good enough. It is very practical.
We might start out desiring to know what the average IQ of the students on our campus is. The obvious way to approach this would be to administer an IQ test to every student on campus. There are problems with this, however: Students don't like taking tests, testing is expensive, and so forth. What we would need to do, in this case, would be to take a sample.
The idea is that a sample serves as a representative of the whole. One of the great mathematical miracles that make statistics work is that most samples do serve as representatives of the whole. One of the dangers is that regardless of that, bias can creep in through sampling techniques.
The gold standard of taking a sample of size 100 would be to choose 100 student identification numbers at random and then to administer the test to those 100 individuals. One problem with this is that some of that 100 will not show up, so we would have to over sample by a certain amount. But a more basic problem is to get any so show up at all. It is possible to offer incentives, but then the risk is biasing the sample by the type of reward you offer. What if you offer an iPod as a prize? Will you bias this by attracting students who like to gamble? Will you bias it toward groups that don't already have iPods?
In a similar project, we've dealt with this by choosing to measure particular classes, not for IQ but for other things. In doing this, there is a risk of bias toward students who choose particular classes, but the low cost of this method makes it very attractive and, being aware of the possibility of bias, we can be on the guard against it.
I am a topologist, and I've only ever had one course in statistics which was as an undergraduate and was theoretical. However, at a school like ours, we are not allowed to be to picky as to what we teach, so, as a consequence of this, I started teaching our elementary statistics course about fifteen years ago and discovered that I liked the subject. Now, after having taught it for that length of time, I cannot claim expertise, but I can say that I do understand some of the basic ideas.
One of the basic ideas is that of a sample. I would like to explain that to you.
Statistics is about the search for answers in a particular spirit. We begin with the need to know, but with the realization that we can't know exactly. Given that, we proceed to find the approximate answer with the realization that the approximate answer will, in a certain number of cases, not be good enough. It is very practical.
We might start out desiring to know what the average IQ of the students on our campus is. The obvious way to approach this would be to administer an IQ test to every student on campus. There are problems with this, however: Students don't like taking tests, testing is expensive, and so forth. What we would need to do, in this case, would be to take a sample.
The idea is that a sample serves as a representative of the whole. One of the great mathematical miracles that make statistics work is that most samples do serve as representatives of the whole. One of the dangers is that regardless of that, bias can creep in through sampling techniques.
The gold standard of taking a sample of size 100 would be to choose 100 student identification numbers at random and then to administer the test to those 100 individuals. One problem with this is that some of that 100 will not show up, so we would have to over sample by a certain amount. But a more basic problem is to get any so show up at all. It is possible to offer incentives, but then the risk is biasing the sample by the type of reward you offer. What if you offer an iPod as a prize? Will you bias this by attracting students who like to gamble? Will you bias it toward groups that don't already have iPods?
In a similar project, we've dealt with this by choosing to measure particular classes, not for IQ but for other things. In doing this, there is a risk of bias toward students who choose particular classes, but the low cost of this method makes it very attractive and, being aware of the possibility of bias, we can be on the guard against it.
Tuesday, July 20, 2010
Organization 001
In my last post, I mentioned that assessment, in the context that administration discusses it, is a managerial/political activity. The politics of it are beyond my kin at this point, but I do have some simple comments about management or, to be more precise, organization. The emphasis in the preceding sentence is on the word "simple."
First of all, as I have no formal training in the area of management or organization, anything I tell you is bound to be simple. I don't apologize for this because simplicity has its own beauty and, beyond that, simplicity should be one of our goals. There are, perhaps, a number of reasons for this, but the one that is foremost in my mind can be summarized in a single word: cost. Complication ramps up the cost of effort and in particular it takes the focus away from instruction.
Assessment is, of course, a necessary part of teaching, but the primary part of teaching is student learning. With all other variables being held fixed, the more effort a teacher puts into instruction, the more learning will take place. Since assessment is necessary for teaching, this argues for simple assessment processes.
All of this having been said, what are my simple principles of organization. I have two: Lists and Calendars. Break each activity into a list of tasks to be accomplished and put the items from that list onto a calendar. Each item on the list might be broken further down into a sublist and each of those put onto a calendar. It is what we call a recursive process.
Somewhere in here I also need to add that it is good to have some idea what the final product will look like. There will be a list of calendared events. At some of these events data will be taken, at some the data will be discussed, and at some plans will be devised in order to use the data to inform our teaching.
I might add that simplicity in the managerial portion should have a positive impact on the political portion.
First of all, as I have no formal training in the area of management or organization, anything I tell you is bound to be simple. I don't apologize for this because simplicity has its own beauty and, beyond that, simplicity should be one of our goals. There are, perhaps, a number of reasons for this, but the one that is foremost in my mind can be summarized in a single word: cost. Complication ramps up the cost of effort and in particular it takes the focus away from instruction.
Assessment is, of course, a necessary part of teaching, but the primary part of teaching is student learning. With all other variables being held fixed, the more effort a teacher puts into instruction, the more learning will take place. Since assessment is necessary for teaching, this argues for simple assessment processes.
All of this having been said, what are my simple principles of organization. I have two: Lists and Calendars. Break each activity into a list of tasks to be accomplished and put the items from that list onto a calendar. Each item on the list might be broken further down into a sublist and each of those put onto a calendar. It is what we call a recursive process.
Somewhere in here I also need to add that it is good to have some idea what the final product will look like. There will be a list of calendared events. At some of these events data will be taken, at some the data will be discussed, and at some plans will be devised in order to use the data to inform our teaching.
I might add that simplicity in the managerial portion should have a positive impact on the political portion.
Monday, July 19, 2010
Assessment: A context senstive activity
One of the things that appeals to me about assessment is that is an abstract concept that can be applied in multiple contexts for diverse purposes. In its most abstract form we set a target, take a measure, recalibrate in such a way as to get closer to a target, take the measure again. It is exactly what happens when an archer shoots an arrow. He selects aims at the bulls eye, let's his arrow fly, notes how far the arrow is from the bulls eye, and then corrects for the next shot.
We apply this model at multiple levels at the university. Students do it. In a class, they decide what grade they want, they do assignments, the assignments are scored and handed back, and from that the student can decide what needs to be done in order to meet the goal. The response can be anything from "I got 100 so I can spend a little more time in the bars" to "That was a 59 so I need to hit the books some time BEFORE 3AM."
The students have the advantage that the system for assessment has already been set up for them. At the university, we have something to say about how our system is set up. We, within certain parameters, get to set our own targets and get to determine the means of measuring how well we do within those targets.
When I say "within certain parameters," this is a caveat that, whatever we do, it must be real. We might imagine ourselves as teachers giving the students the assignment of designing their own test. We wouldn't expect the students to put impossible questions on the exam, but we would certainly want the students to include questions that would test the full scope of the material in question.
When "assessment" is talked about by the administration, the context is that of an institution of higher learning which is answerable to multiple publics. This is to say that we have our students, our state, our alumni, our faculty, our accrediting agency, and ourselves to satisfy, just to name a few. Each of these groups has its own, unique point of view. Our aim is to create a system of assessment that will satisfy each of those.
In this context, the abstract model of which I am so fond is the smallest piece. Assessment becomes a managerial/political activity. One must take part in the creation of a shared vision. That is the challenge.
We apply this model at multiple levels at the university. Students do it. In a class, they decide what grade they want, they do assignments, the assignments are scored and handed back, and from that the student can decide what needs to be done in order to meet the goal. The response can be anything from "I got 100 so I can spend a little more time in the bars" to "That was a 59 so I need to hit the books some time BEFORE 3AM."
The students have the advantage that the system for assessment has already been set up for them. At the university, we have something to say about how our system is set up. We, within certain parameters, get to set our own targets and get to determine the means of measuring how well we do within those targets.
When I say "within certain parameters," this is a caveat that, whatever we do, it must be real. We might imagine ourselves as teachers giving the students the assignment of designing their own test. We wouldn't expect the students to put impossible questions on the exam, but we would certainly want the students to include questions that would test the full scope of the material in question.
When "assessment" is talked about by the administration, the context is that of an institution of higher learning which is answerable to multiple publics. This is to say that we have our students, our state, our alumni, our faculty, our accrediting agency, and ourselves to satisfy, just to name a few. Each of these groups has its own, unique point of view. Our aim is to create a system of assessment that will satisfy each of those.
In this context, the abstract model of which I am so fond is the smallest piece. Assessment becomes a managerial/political activity. One must take part in the creation of a shared vision. That is the challenge.
Friday, July 16, 2010
Grades are assessment
Every member of the faculty does assessment. We give tests; we give assignments; we give grades for those tests and assignments. (That's why our students love us so!) This is all part of an assessment process. I say 'part' because there must be more. The grades are a measure, but that measure must be reacted to in some way. The student might look at a 55% on an exam and resolve not to study during "The Office." An instructor might look at a 55.5% average for his class and decide to revisit his teaching techniques.
This makes in confusing when faculty is discouraged from using grades in various assessment reports. While there might be a number of reasons for this, one I see comes from the difficulty of using grades from a class to make comparisons to a national standard. This is to say that while my students all get As that might just be because I am easy and not because I am good, or my students all get Ds just because I have standards and not because I think I am Napoleon.
Grades are a good tool for what we use them for which is measuring the achievement of a particular student learning a particular set of material. Expanding their usefulness beyond that scope is not impossible but would require some sort of a justification, and it usually easier simply to use some other measure.
This makes in confusing when faculty is discouraged from using grades in various assessment reports. While there might be a number of reasons for this, one I see comes from the difficulty of using grades from a class to make comparisons to a national standard. This is to say that while my students all get As that might just be because I am easy and not because I am good, or my students all get Ds just because I have standards and not because I think I am Napoleon.
Grades are a good tool for what we use them for which is measuring the achievement of a particular student learning a particular set of material. Expanding their usefulness beyond that scope is not impossible but would require some sort of a justification, and it usually easier simply to use some other measure.
Thursday, July 15, 2010
I just know...
I am currently reading Malcolm Gladwell's book Blink. In that book, Gladwell discusses the phenomenon of how we know something immediately that might take a lengthy time to discern by reason, and, indeed, reason may never arrive at at all. He call this thin slicing. Thin slicing is how we frequently assess our students and how well they are learning a subject. We look at the class, we talk with them either en masse or as individuals, and we just know how well they are doing.
And, if you have been teaching as long as I have, you know you can still frequently be wrong.
So, in addition, we also employ classroom assessment techniques of a more formal nature like giving exams, etc. As we have to give a written report (i.e. a grade) to the registrar at the end of each semester on the achievements of our students, having a paper trail to back up our assessments is a wise thing to do as our students might have a differing opinion. Indeed, most of the time the grade is constituted of the grades of assignments on which the student received feedback as an opportunity to improve his performance. The wise student looks at his grade, looks at his assignment, and then acts in a way to improve his performance.
This provides a nice model of what we are doing ourselves in what is called the process of assessment. We are teaching ourselves to teach. When we obtained our final degrees, we were set loose in the world with the expectation that we would be independent learners. Assessment should provide us a tool for that. We grade ourselves (honestly!) on how well we have performed a task and apply the data, which is always constituted of more than just the final grade, in order to improve our performance.
Thin slicing has its place in assessment, and, indeed, our "gut" might be our final arbiter, but it has the same shortcomings there as it does in the classroom. Having a simple, natural system for assessment in place is vital in improving our student's educational experience.
And, if you have been teaching as long as I have, you know you can still frequently be wrong.
So, in addition, we also employ classroom assessment techniques of a more formal nature like giving exams, etc. As we have to give a written report (i.e. a grade) to the registrar at the end of each semester on the achievements of our students, having a paper trail to back up our assessments is a wise thing to do as our students might have a differing opinion. Indeed, most of the time the grade is constituted of the grades of assignments on which the student received feedback as an opportunity to improve his performance. The wise student looks at his grade, looks at his assignment, and then acts in a way to improve his performance.
This provides a nice model of what we are doing ourselves in what is called the process of assessment. We are teaching ourselves to teach. When we obtained our final degrees, we were set loose in the world with the expectation that we would be independent learners. Assessment should provide us a tool for that. We grade ourselves (honestly!) on how well we have performed a task and apply the data, which is always constituted of more than just the final grade, in order to improve our performance.
Thin slicing has its place in assessment, and, indeed, our "gut" might be our final arbiter, but it has the same shortcomings there as it does in the classroom. Having a simple, natural system for assessment in place is vital in improving our student's educational experience.
Wednesday, July 14, 2010
In the beginning...
I am the chair of my university's assessment committee. I've been the chair for a couple of years now. In theory, if my goal had been to document that experience, I should've started blogging when I first accepted the position. But I didn't. This is how assessment works in the real world.
The theoretical framework for assessment is as follows. First, you set a goal as to what you want to do. Second, you break down that goal into a list of subtasks that are required to take place in order to achieve that goal. Third, you develop measures in order to help you discern the progress you are making toward achieving those subtasks. Once these measures are put into place, you visit them on a periodic basis in order to monitor your progress and react to the data you gather in an appropriate way.
This is not a new idea. In creating the heaven and the earth, God subdivided the task into six subtasks. At the end of each subtask, the looked at what had been done and noted that it was good. That was assessment.
Few of us are in God's position, however, with the only public we have to please being ourselves. Most of us are in a position where data is required. In addition, few of us have the luxury of knowing with perfection what our goals are. Often our goals evolve over time. We decide we want to monitor our evolving understanding of assessment two years into the process.
Or the focus of the university shifts.
Or an opportunity arises.
All of these things have happened over the last couple of years.
It is my hope--dare I say goal--in this blog to document my evolving understanding of assessment in the area of higher education and to seek input from others. My hope is to create a relaxed atmosphere where readers feel free to comment and to add to my understanding.
The theoretical framework for assessment is as follows. First, you set a goal as to what you want to do. Second, you break down that goal into a list of subtasks that are required to take place in order to achieve that goal. Third, you develop measures in order to help you discern the progress you are making toward achieving those subtasks. Once these measures are put into place, you visit them on a periodic basis in order to monitor your progress and react to the data you gather in an appropriate way.
This is not a new idea. In creating the heaven and the earth, God subdivided the task into six subtasks. At the end of each subtask, the looked at what had been done and noted that it was good. That was assessment.
Few of us are in God's position, however, with the only public we have to please being ourselves. Most of us are in a position where data is required. In addition, few of us have the luxury of knowing with perfection what our goals are. Often our goals evolve over time. We decide we want to monitor our evolving understanding of assessment two years into the process.
Or the focus of the university shifts.
Or an opportunity arises.
All of these things have happened over the last couple of years.
It is my hope--dare I say goal--in this blog to document my evolving understanding of assessment in the area of higher education and to seek input from others. My hope is to create a relaxed atmosphere where readers feel free to comment and to add to my understanding.
Subscribe to:
Posts (Atom)