Assessment Rubric: General Education Mathematics

Assessment Rubric:  General Education Mathematics

	3	2	1	0
Interpretation Ability to articulate a problem and explain information presented in mathematical forms (e.g., equations, graphs, diagrams, tables, words)	Provides accurate explanations of information presented in mathematical forms. For instance, accurately explains the trend data shown in a graph.	Provides somewhat accurate explanations of information presented in mathematical forms, but occasionally makes minor errors related to computations or units. For instance, accurately explains trend data shown in a graph, but may miscalculate the slope of the trend line.	Attempts to explain information presented in mathematical forms, but draws incorrect conclusions about what the information means.  For example, attempts to explain the trend data shown in a graph, but will frequently misinterpret the nature of that trend, perhaps by confusing positive and negative trends.	No meaningful work done.
Representation Ability to convert relevant information into various mathematical forms (e.g., equations, graphs, diagrams, tables, words)	Competently converts relevant information into an appropriate and desired mathematical portrayal.	Completes conversion of information but resulting mathematical portrayal is only partially appropriate or accurate.	Completes conversion of information but resulting mathematical portrayal is inappropriate or inaccurate.	No meaningful work done.
Calculation Use the tools of mathematics.	Calculations attempted are essentially all successful and sufficiently comprehensive to solve the problem. For example, the student decides to use the quadratic formula to solve and problem and substitutes and simplifies appropriately.	Calculations attempted are either unsuccessful or represents only a portion of the calculations required to comprehensively solve the problem. For example, the student decides to use the quadratic formula to solve and problem and substitutes incorrectly or makes a minor simplification error.	Calculations are attempted but are both unsuccessful and are not comprehensive. For example, the student decides to use the quadratic formula to solve and problem and substitutes incorrectly and has simplification errors.	No meaningful work done.
Application Ability to apply mathematical generalizations, principles, theories, or rules to real world problems.	Select and apply the appropriate mathematical principles to correctly solve a real world application problem taking into account important assumptions. (In calculating the area of an irregular polygon, student correctly divides the area into simple shapes and correctly uses known formulae to calculate the areas.)	Chooses appropriate mathematical principles but has errors in applying principles to solve real world problem. (In calculating the area of an irregular polygon, student correctly subdivides the area into simple shapes and improperly uses formulae to calculate the areas.)	Attempts to solve application problem but is unsuccessful. (Student knows some of the technique for subdivision of an irregular polygon but incorrectly divides the area; knows some simple area formulae, but cannot put all the steps together to get a correct result.)	No meaningful work done.
Analysis Ability to make judgments and draw appropriate conclusions based on the quantitative analysis of data, while recognizing the limits of this analysis	Uses the quantitative analysis of data as the basis for competent judgments, drawing reasonable and appropriately qualified conclusions from this work. (Interpolates or extrapolates data from a graph or table to calculate information not specifically given; creates a formula from information to predict results for future events.)	Uses the quantitative analysis of data as the basis for workmanlike (without inspiration or nuance, ordinary) judgments, drawing plausible conclusions from this work. (Given a problem statement, the correct relationship can be identified and known values used to calculate the desired unknown.)	Uses the quantitative analysis of data as the basis for tentative, basic judgments, although is hesitant or uncertain about drawing conclusions from this work. (Given a problem statement, known values can be correctly identified, however, the appropriate relationship is not found or applied correctly and the desired result is not found.)	No meaningful work done.
Communication Expressing quantitative evidence in support of the argument or purpose of the work (in terms of what evidence is used and how it is formatted, presented, and contextualized)	Uses quantitative information in connection with the argument or purpose of the work, though data may be presented in a less than completely effective format or some parts of the explication may be uneven. For instance, effectively uses verbal and/or written skills to explain the quantitative evidence.	Uses quantitative information, but does not effectively connect it to the argument or purpose of the work. For instance, the quantitative evidence may be correct, but verbal and/or written skills are weak.	Presents an argument for which quantitative evidence is pertinent, but does not provide adequate explicit numerical support. (May use quasi-quantitative words such as "many," "few," "increasing," "small," and the like in place of actual quantities.) For instance, does not effectively use verbal and/or written skills to explain quantitative evidence.	No meaningful work done.

A lens for assessment

Recently Stephen J. Dubner in a blog entry entitled Are You Smarter Than an Eighth Grader? Published a copy of document from 1895 that had been published in the Salina (Kansas) Journal. By all appearances, this document looks to be a final exam for eighth graders though Snopes.com casts some doubt on this. As with so many historical questions, we may never know the full truth here, but I would still like to use this as an opportunity to discuss assessment by using this document as a lens.

Whenever I give a test, I am assessing two things. One of these things is how well each of the students understand the material. Their scores reflect how well they have learned. This can either be done as simply an abtract exercise in discerning what percentage of the material has been absorbed or, as is more often the case, against a set of standards. Did the student pass the exam? Did they learn 70 percent of the material and get a gentlemanly C? Did they learn 90 percent and earn an A? The other thing I am assessing is how well I taught the subject. Which questions did they know the answer to? Did they do so well that I could spend less time there and more on an area that they didn't do as well on?

For this exam, we don't have this sort of information. We don't know how many people passed if any or whether anyone even took it. As of this writing, I just don't know.

I recall a conversation at a local mathematics confernence a few years back with a man who had been teaching Intermediate Algebra for many years. He had carefully tracked his how his students had done on an impressive number of student learning outcomes. One of those was arithmetic with imaginary numbers. He'd tried a number of different methods of approaching the topic. He'd spent more time, given more emphasis, but he could never the the student success above a certain threshold. And he had years of data to prove this.

Others there had questions to ask about things he'd tried. The question I wanted to ask was "You are teaching the arithmatic of complex numbers to Intermediate Algebra students? Why?" Intermediate Algebra is taught in college to students who've come to college unprepared to take College Algebra. One need not know much about the arithmetic of complex numbers to meet the challenge of College Algebra, and, indeed, the time might be better spent elsewhere.

The topic is not appropriate for the students in the class. I find the arithmetic of complex numbers to be enjoyable and fascinating; it can be a useful intellectual exercise; and there are areas in which it is actually useful. If I were this teacher, I would remove the topic from the course to spend more time on other topics. I open myself up to the charges of dumbing down the course by doing this, but quite frankly the students who need it will learn about it in Differential Equations or the Theory of Functions of a Complex Variable. The students who are trying to prepare themselves for College Algebra will be able to learn what they need to be ready for that course.

As I look at the document from 1895, I see a lot of things that I don't know about. Putting the other sections aside and concentrating on the math, which is my area of expertise, I see some things there that are good, but I don't see any algebra. We teach a little algebra in the eighth grade now. This has none. There is nothing about computers, of course.

This exam proves that things change. We as teachers can use assessment methods as a tool to attempt to change for the better.

New Year’s and Assessment

During the New Year's season, which stretches from the day after Christmas until the last resolution is broken, i.e. sometime around the Feast of Epiphany, we take time to look at what we've done during the previous year and make plans for what we will do during the coming year. Our plans for the coming year are called resolutions, and as I mentioned above, our resolutions typically have a short self-life. One year I resolved to give up French fries, which lasted approximately to the next time I smelled French fries.

While this is assessment in a sense, it is assessment that is done badly. Before I go any further, I must remind everyone of the importance of context in the practice of assessment. The type of assessment I am talking about here is assessment of personal behavior as opposed to the institutional assessment as it is practiced by universities, but the general principles are the same. Our New Year's resolutions are goals by another name. One of my goals is typically to lose weight. I choose this for a number of reasons one of which would be pressure from my publics, i.e. my wife. Her reason for this pressure is that she desires that I be happy and healthy. These are desires I should, and I do, have for myself, but because of my love of good food I have internal pressures to overeat.

I choose strategies or objectives in order to aid in accomplishing that goal. The particular one I mentioned was cutting out French fries. One advantage of this goal is that it is easy to measure. Either you are successful in cutting out French Fries or you are not.

But, I did say that I had failed in this particular endeavor. Why?

One thing is that I chose a poor strategy, i.e. cutting out a particular food. There were reasons for this. One of which was that French fries are a particular weakness of mine that pull me into overeating. The problem is that the measure is an all or none instrument. Eating a few French fries is measured as a failure in this system, even if it doesn't result in the intake of a large number of calories in the end. The failure is discouraging and results in a premature abandonment of the goal.

It strikes me that the real problem here occurred with the choice of the goal. The goal itself should have been to live a lifestyle conducive to health. Objectives created in the aid of this could have been eating a healthy diet and exercising an appropriate amount. One can then address the healthy diet through means of portion control and a regular exercise schedule. While there is no set of goals that is so good that a human being such as myself can't find a way to wiggle out of it by Epiphany, this goal and these objectives are at least amenable to long to planning, i.e. I can lay myself out a plan for portion control and regular exercise and can put in some more effective measures of house closely I am meeting my objectives.

At the same time, this is a goal which, while pleasing my publics, also is a good in and of itself to me.

This is something we need to keep in mind when we do institution-level assessment at the university. While the impetus is often pressure from a particular public, those applying the pressure are doing so because they care about education…as do we. Our challenge is to articulate the desires of those applying the pressure into an achievable plan. In attempting to do this, it is easy to set up ourselves to fail, so we need to take care in the early stages.

Happy New Year!

Being killed by perfection

I recently talked to one of my cousins, who is a retired engineer. He was interested in what the world of higher education is like these days, and, as a part of our conversation, I talked to him about assessment. When I explained the basic principles to him, his response was, "Ah, TQM."

TQM, for those who don't know, stands for Total Quality Management. My understanding of the history may be faulty, but this was one of those ideas that came into American industry from Japan when we were falling behind them in terms of quality. The idea isn't new, of course. The general idea of set a goal, take an action, and measure how far the outcome is from the goal is as old as civilization. I suppose the novelty of the notion lies in the use of modern analytic tools and institutionalizing it to such a large degree.

My cousin said the process had run afoul in his company when it was transmuted by the slogan "Perfect the first time."

There is no such thing as perfection, so arriving at it the first time is nonsense. And, of course, this is not what a culture of continual improvement about. I don't even like the phrase "continual improvement" as it doesn't seem to allow even for normal statistical variation. The idea is to organize our activities and create structures so that improvement is a natural outcome of the process. We need to recognize that we can improve and to institute processes so this will, over the course of time, happen.

Part of this process might include setting up a "perfect" ideal of what our desired outcomes would be. But, I believe it was Voltaire who said, the perfect is not the enemy of the good. We can be good, but we can always strive to be better. Using the tools of assessment is a natural part of this process.

Math Task Force: On the Road

I look at my previous entries in this blog and see that I'd written about preparing to begin with the Math Task Force on September 16. It is now early November and we've had our second weekly meeting. Those outside of Higher Education might think that there has been some sort of a delay. The truth of the matter is that it has taken this long with everything going as well as possible. A while back I was in a meeting with a faculty member who was in the process of starting a new program. I told her that one I'd been involved in starting took 3 years to begin. She asked me what went wrong. I told her that nothing went wrong; it took that long with everything going perfectly.

There are two constraints that come together that make academic processes go so slowly. The first of these is inclusivity. We operate under a system of shared governance which means that any program or policy that is created will have to be reviewed and approved by representatives of the entire faculty. Including the appropriate representatives from the various sectors of the university is essential. The second of these is scheduling around classes. So many members of the university community teach that meetings must be scheduled around classes; the classes cannot be moved because students come first. Along with this is the fact that most of the folks who you would want on a committee are already on several committees because they are good workers, so we have to schedule around those committees too. The upshot of this is that it takes a while to schedule that first meeting and sometimes all subsequent meetings.

But, as I said earlier, we've already met twice and are meeting on a weekly basis. The conversation has begun. We say that a lot on the university campus, because so much of what we do is based on conversations among thoughtful people who care very much and have their own opinions. Meetings are a good site for these conversations as ours opinions can be given and the opinions of others can be heard. We then leave the meeting and consider what we've heard and whether we want to make those words our words. By this process a group opinion is born which is informed by the knowledge of all those present.

I've been thinking of the issue of mathematics as general education for a considerable time now. It is very affirming to hear many of the thoughts I've had coming independently from the mouths of others. It is also good to hear perspectives I've not considered before. It is a beautiful process.

On standardized tests

This article is from the New York times.

Math Task Force: The Beginning

I am interested in mathematics for a number of reasons. One is that it is my discipline. Another might be that it's simply an inherently interesting subject, but then, given my previous statement, some might consider me biased. The reason that I am interested in mathematics as a subject at this point is that it is listed as a basic skill in my university's general education program and we are currently in the processes of developing a structure of assessment for it.

I speak with no fear of rebuttal when I say that mathematics is a very technical subject, but I will add to this that mathematics has its own beauty. Within its jungle of formulas and theorems, there are nuggets of beauty that await a prepared mind. I've seen them and I know. This sort of beauty is the stuff envisioned by those who have the vision of idealized, liberal-arts-type general education courses. My university doesn't currently offer such a course. Ought we? I don't know, and I don't know whether it's even something we should think about, but I will toss it to the family dog to see if he will chase it.

Beyond that sort of vision of general education math, there is also mathematics as a life skill. It is knowledge that people can actually use in daily life. Mathematics can help you live and it can help you to prosper.

But mathematics is classified as a basic skill in our general education program because it is foundational. There are disciples—the sciences, the social sciences, business, technology, and so forth—that use mathematics. Currently the general education offerings on our campus are structured in order to feed into these various disciplines.

We are currently in the process of putting together a Math Task Force which will explore how well these needs are being met. In the past, the Department of Mathematics has paid attention to these needs without the aid of any larger structure, i.e. it has been done informally which met the needs of the time. As the character of the university changes, regularizing these processes which were previously informal and opportunistic, will help aid the university in demonstrating our commitment to educating our students in this basic skill.

Teaching is a priority

You may have to be a subscriber to the Chronicle to read this. Let me know. In contrast to many of the universities talked about in the article, I am blessed to be at a place that cares about teaching and student learning.

National math woes

Math is an issue that we will be dealing with on our campus this year. This is evidence we are not alone.

The importance of writing

As I may have mentioned in earlier posts, my university is in the process of putting together a system of assessing general education. At the foundation of our general education program are the basic skills. I would argue that the cornerstone of that foundation is writing.

Writing, of course, is a basic means of communication. When a student responds to a test question—at least when beyond the survey classes—the response is in writing, regardless of whether the answer is an essay, a sentence, "text", or simply just a word. Its importance is far beyond that. Writing—good writing—requires that the student have established an internal dialog. We all have multiple voices within ourselves, but having an internal dialog requires that we discipline those voices. We must be able to take an idea and attempt to support it while at the same time honestly looking for valid criticism of that idea. Such an internal dialog enables us to be surer of our opinions and to be able to defend them in a reasoned way.

I would also submit that having learned to write enables us to become better readers. The struggle to convey a particular idea using the writer's toolbox builds the ability to better examine another's writing for nuance. As someone who's played football is better able to appreciate the finer points of the game so is someone who has written better able to understand the subtleties there.

One of the reasons we have chosen to tackle writing first is because of its fundamental importance to learning at level and breadth of the university. Because of its importance, we know that it is thing that we must strive to excel at. As a consequence, the assessment tools we put in place to measure that success must be create carefully and well. These tools, and the spirit of the endeavor, will provide a template to follow in the creation of assessment tools for mathematics and communication, and from there for the rest of general education.

Journey to the core

General education at my university is designed so as to have certain basic skills at its core. These are Writing, Mathematics, and Communication. The vast majority of academics would agree on the importance and foundational nature of these subjects though, of course, each would have view of the value of each.

Communication is obviously important for its utility in helping each of us to communicate what we know. Students gain knowledge over the course of their careers and it is important that they be able to convey what they know at the end of the experience. We need also remember that communication is a two-way street. By learning to communicate one's own ideas, a student is better able to understand the methods teachers use in communicating to them. They are better able to analyze what the teacher is saying.

Writing, of course, is obviously important as a means of communication, but, in addition to this, the process of writing well and employing convincing arguments cannot be separated from critical thinking. Creating an effective argument requires having a robust internal dialog. One must not only be able to create arguments but look for holes in those arguments and find ways to patch those holes or to discern they are un-patchable.

Then my area of specialty, mathematics, enters the fray. That mathematics is basic to the physical sciences no one will dispute. Indeed, given the shift in biology from the field to the lab, few would dispute that a knowledge of mathematics is necessary to the life sciences as well. Certainly few would argue against the utility of statistics in the social sciences and business. As one drifts more deeply into the humanities one meets not only doubt as to its value but outright hostility.

As I said, I am a mathematician and we are used to hostility, so perhaps it is not surprising that I believe mathematics is important to the humanities too, but I would also concede that those in the humanities might be better served by a different sort of mathematics than those in the sciences. I can appreciate music without knowing how to play the guitar, but I can get more out of a musical experience by being taught certain things. An artist need not be taught the technical details of complex variables in order to appreciate the hyperbolic plane.

We are on the verge of appointing a task force to study the mathematical needs on this campus. It grew out of our needs in assessing the mathematical component of general education but its scope is greater than this. Ultimately, it may provide mechanisms for the university to maximize the impact of mathematics on our campus.

Value Added

As a member of my university's faculty, I've been a participant in many conversations concerning assessment. Most (and I say most just to simplify exposition) of the faculty I've known have a deep desire to make their students

1. Love the subject they are teaching;
2. Become as proficient at that subject as the student is capable.

But when faced with the prospect of measuring the student's knowledge in a setting larger than the faculty member's office--like that provided by an institutional assessment system--those faculty become nervous. I speak from personal experience when I say the prospect of having the quality of your work examined in not necessarily comfortable.

For the most part, we as teachers do the best we know how but, as the mathematician in me would put it, there are variables that are beyond our control and they make us feel exposed. One of the concerns frequently voiced is the preparation, or lack thereof, of students when they “come in the front door” and being held accountable for how those students perform when they go out the back door.

There are a number of reasonable answers to this concern. One of these is that all faculty at the university are teaching those same students. Ill-prepared students affect all faculty across the board.

Another answer, which is more germane to the practice of assessment, is the concept of added value.

When a diamond is taken from the mine, it is rough and dull. The diamond has to be cut by and expert hand and polished before it is marketable. When potatoes are dug from the ground, they are covered with soil. They have to be washed, pealed, and cooked before they can be eaten. The diamond and the potato have a natural worth, but value is added to them through processing.

We, in the university, attempt to add value to students in our own special way. Students come in the front door of the class, involve themselves in it activities, and should look different when they go out the back. We can measure the value-added to those students by measuring the change.

There are problems with this.

This is still bothersome because those of us who have taught know that not all students take the fullest advantage of their time under our expert guidance. For this, I retreat to my first answer that everyone teaching at the university is teaching those same students. If they don’t work for me, they are not going to work for you, and if they do work for you, perhaps I should learn from you how you get them to do it.

Another issue could be phrased in this way. Not all diamonds are the same. Some are small and are always going to be small. The master’s hand can maximize their value, but that value will have a small upper bound. Some diamonds, while large and beautiful, come in requiring very little cutting or polishing. They might leave the shop with a very high value, but that might not be much more than when they came in.

The answer to this is not as easy and may require putting an offering before the gods of statistics. It also requires us to learn more about our students and to seek better ways to teach the students we have. This includes learning the strengths and weaknesses they bring with them and learning the ways we can use their strengths and eliminate or compensate for their weaknesses.

Assessing General Education

My university has begun the process of extending our system of assessment into the general education program. Before I began this process I thought I knew the purpose of general education and I was happily ignorant of its size.

Before I launch into thoughts concerning the purpose of general education, let me first comment as to its extent. At my university, each student must have at least a total of 124 credit hours to graduate. Of those 124, at least 46 credit hours must be general education. This is slightly over one-third of the total requirement. Somewhere among the remaining two-thirds of those 124 hours, the student must specialize in something and do some exploring on his own.

In the book Excellence without a Soul, Harry Lewis remarks that the idea that students should know one thing very well but also know a little about everything has long been an ideal at Harvard where he taught. This same ideal permeates higher education throughout the United States and informs our attitude toward general education. In addition, there are other functions performed by general education.

In a setting where many of the students are the first generation of their families to attend college, there is often a lack of knowledge among the students and their support groups of pathways open to them through education. General education provide samples of areas in which students might find a career that they might not otherwise have known about. We all have heard the student stories that begin "I'd never even heard of X before, but I took Professor Y's Introduction to X because my adviser made me, and it changed my life."

There are also aspects of general education that are foundational to later learning. At my school, we have decided these are writing, mathematics, and communication. We refer to these as our Basic Skills.

It is with the Basic Skills that we have chosen to begin as we extend our assessment system to general education. This is being done for a number of reasons. The disciplines involved in this area are almost universally (as near as can be expected on a university campus) accepted as being a part of general education. In addition, Basic Skills only includes courses being taught by three departments, and, therefore, the politics involved are more manageable.

Even though we are only beginning, it has been discovered that expectations for student learning outcomes in these areas can vary widely. We are currently looking at writing, and, without being too surprised, we've learned different disciplines expect different styles of writing. Some expect students to write in a literary style, some in a business style. Some expect essays and some expect text. As we continue to examine mathematics and communications, I expect similar results.

Our purpose, however, is not to bludgeon the faculty into saying there is a single correct vision of any of these areas. We must open lines of communication so that the needs of the students are correctly understood by those whose charge it is to teach them. And, as assessment is our charge, we must do that in such a way that we can measure how well these needs are being met.

Closing the loop

One phrase that occurs repeatedly in the language of assessment is "closing the loop." It comes from the practice of illustrating processes using graphs or networks. The idea is that one collects data concerning an activity and then reexamines the activity in the light of that data. Some do not like the phrase and prefer to say "using the data." It all amounts to the same thing.

One cannot over-emphasize the importance of closing the loop/using the data. Without this piece, the entire process is pointless. Regardless of how efficient we are, regardless of how simple we keep the process in toto, it still requires energy, i.e. effort and expense. I believe the Second Law of Thermodynamics is at work. Given the expense of taking data, if that data is not used, then money has been wasted. This is a tautology.

Now, having said that, I need to be careful. Sometimes we must monitor things that are in good shape and don't require our action just because they are important. For example, even people who don't have high cholesterol have their cholesterol monitored regularly; they just don't have it monitored as regularly as those who have issues in that area. But we do keep an eye on it because there are consequences if it creeps out of the safety zone. Similarly, there are areas where a particular department might excel and have done so over a historically lengthy interval, and there is no need for change in that area, but it is so important that it should be monitored. In such cases, one can take measurements at longer intervals, such as every three years or every five years.

The point is that data is taken to be used. This is an important point to make, especially in the initial stages of setting up a system of assessment.

In many cases, assessment is rolled-down from above upon groups which are unevenly prepared. In some cases, the message gets through that "Someone up there wants numbers." The response is then to "Come up with some numbers," irrespective of whether or not those numbers are meaningful. This is, of course, not the best possible response.

We don't take data simply for the great joy of taking data. It is done for a purpose.

When I check the outdoor temperature, it is to determine whether I need to wear a parka or gym shorts when I go outside. Data taken in the act of assessment similarly should have a purpose. Measures should be implemented with the idea that particular results will result in particular actions. When I check the outdoor temperature, it measuring the temperature on Mars for which my actions do not matter. I measure it outside my own house. Similarly, whatever departments measure should have some meaning to their mission at the university.

Creating the environment, a culture of continuous improvement

As I've mentioned earlier, I am a mathematics teacher and have been teaching the subject in one way or another with varying levels of intensity since 1983. As a teacher, I've never been able to open a student's head and just pour knowledge in. Lately, I've had fantasies of surgically implanting a USB port in the skull of each student and just plugging him in so that he can slurp up the knowledge. That is unlikely to happen any time soon.

My method of teaching is to structure my classes with a time-line. I set learning goals and embed those goals in a calendar. I then organize the material and try to personalize the presentation of it to the needs of each class collectively and each individual within that class as much as possible by using language that the class understands, using a suitable idiom.

This is a learning environment. Within this environment, the student must apply his own efforts and abilities if he is to learn. There are students who are better at this than others, and I am not necessarily talking about students who are more intelligent. What I mean, is there are students who have cultivated habits which are conductive to the act of learning.

There are teachers as well who have, in an intentional way, acquired habits which improve their teaching. These people will be better teachers on the day they retire than on the day they started. They will be better teachers next year than they are this year.

Part of the business of a healthy educational institution is to create an environment which fosters that sort of teacher. We must create what is called "a culture of continuous improvement."

I use that phrase even though over the course of my career I've seen it misused extensively. Having been a mathematician, I'm not unfamiliar with folks who interpret words in a direct way, and I've seen that phrase taken to mean that there is an expectation that every class will improve its scores every year. This is, of course, idiotic. Because of random variation within each class, it is unreasonable to expect scores that are continually increasing.

It is not, by way of contrast, unreasonable to expect an environment where teachers are encouraged to improve their teaching and given the means, within the available resources, to do so. Indeed, far from being unreasonable, this is a natural expectation.

The creation of an institutional system of assessment is a part of that environment.

Thoughts on sampling

My academic discipline is mathematics. It is natural for someone to say, "Oh you know a lot about statistics," but the truth is more complicated.

I am a topologist, and I've only ever had one course in statistics which was as an undergraduate and was theoretical. However, at a school like ours, we are not allowed to be to picky as to what we teach, so, as a consequence of this, I started teaching our elementary statistics course about fifteen years ago and discovered that I liked the subject. Now, after having taught it for that length of time, I cannot claim expertise, but I can say that I do understand some of the basic ideas.

One of the basic ideas is that of a sample. I would like to explain that to you.

Statistics is about the search for answers in a particular spirit. We begin with the need to know, but with the realization that we can't know exactly. Given that, we proceed to find the approximate answer with the realization that the approximate answer will, in a certain number of cases, not be good enough. It is very practical.

We might start out desiring to know what the average IQ of the students on our campus is. The obvious way to approach this would be to administer an IQ test to every student on campus. There are problems with this, however: Students don't like taking tests, testing is expensive, and so forth. What we would need to do, in this case, would be to take a sample.

The idea is that a sample serves as a representative of the whole. One of the great mathematical miracles that make statistics work is that most samples do serve as representatives of the whole. One of the dangers is that regardless of that, bias can creep in through sampling techniques.

The gold standard of taking a sample of size 100 would be to choose 100 student identification numbers at random and then to administer the test to those 100 individuals. One problem with this is that some of that 100 will not show up, so we would have to over sample by a certain amount. But a more basic problem is to get any so show up at all. It is possible to offer incentives, but then the risk is biasing the sample by the type of reward you offer. What if you offer an iPod as a prize? Will you bias this by attracting students who like to gamble? Will you bias it toward groups that don't already have iPods?

In a similar project, we've dealt with this by choosing to measure particular classes, not for IQ but for other things. In doing this, there is a risk of bias toward students who choose particular classes, but the low cost of this method makes it very attractive and, being aware of the possibility of bias, we can be on the guard against it.

Organization 001

In my last post, I mentioned that assessment, in the context that administration discusses it, is a managerial/political activity. The politics of it are beyond my kin at this point, but I do have some simple comments about management or, to be more precise, organization. The emphasis in the preceding sentence is on the word "simple."

First of all, as I have no formal training in the area of management or organization, anything I tell you is bound to be simple. I don't apologize for this because simplicity has its own beauty and, beyond that, simplicity should be one of our goals. There are, perhaps, a number of reasons for this, but the one that is foremost in my mind can be summarized in a single word: cost. Complication ramps up the cost of effort and in particular it takes the focus away from instruction.

Assessment is, of course, a necessary part of teaching, but the primary part of teaching is student learning. With all other variables being held fixed, the more effort a teacher puts into instruction, the more learning will take place. Since assessment is necessary for teaching, this argues for simple assessment processes.

All of this having been said, what are my simple principles of organization. I have two: Lists and Calendars. Break each activity into a list of tasks to be accomplished and put the items from that list onto a calendar. Each item on the list might be broken further down into a sublist and each of those put onto a calendar. It is what we call a recursive process.

Somewhere in here I also need to add that it is good to have some idea what the final product will look like. There will be a list of calendared events. At some of these events data will be taken, at some the data will be discussed, and at some plans will be devised in order to use the data to inform our teaching.

I might add that simplicity in the managerial portion should have a positive impact on the political portion.

Assessment: A context senstive activity

One of the things that appeals to me about assessment is that is an abstract concept that can be applied in multiple contexts for diverse purposes. In its most abstract form we set a target, take a measure, recalibrate in such a way as to get closer to a target, take the measure again. It is exactly what happens when an archer shoots an arrow. He selects aims at the bulls eye, let's his arrow fly, notes how far the arrow is from the bulls eye, and then corrects for the next shot.

We apply this model at multiple levels at the university. Students do it. In a class, they decide what grade they want, they do assignments, the assignments are scored and handed back, and from that the student can decide what needs to be done in order to meet the goal. The response can be anything from "I got 100 so I can spend a little more time in the bars" to "That was a 59 so I need to hit the books some time BEFORE 3AM."

The students have the advantage that the system for assessment has already been set up for them. At the university, we have something to say about how our system is set up. We, within certain parameters, get to set our own targets and get to determine the means of measuring how well we do within those targets.

When I say "within certain parameters," this is a caveat that, whatever we do, it must be real. We might imagine ourselves as teachers giving the students the assignment of designing their own test. We wouldn't expect the students to put impossible questions on the exam, but we would certainly want the students to include questions that would test the full scope of the material in question.

When "assessment" is talked about by the administration, the context is that of an institution of higher learning which is answerable to multiple publics. This is to say that we have our students, our state, our alumni, our faculty, our accrediting agency, and ourselves to satisfy, just to name a few. Each of these groups has its own, unique point of view. Our aim is to create a system of assessment that will satisfy each of those.

In this context, the abstract model of which I am so fond is the smallest piece. Assessment becomes a managerial/political activity. One must take part in the creation of a shared vision. That is the challenge.

Grades are assessment

Every member of the faculty does assessment. We give tests; we give assignments; we give grades for those tests and assignments. (That's why our students love us so!) This is all part of an assessment process. I say 'part' because there must be more. The grades are a measure, but that measure must be reacted to in some way. The student might look at a 55% on an exam and resolve not to study during "The Office." An instructor might look at a 55.5% average for his class and decide to revisit his teaching techniques.

This makes in confusing when faculty is discouraged from using grades in various assessment reports. While there might be a number of reasons for this, one I see comes from the difficulty of using grades from a class to make comparisons to a national standard. This is to say that while my students all get As that might just be because I am easy and not because I am good, or my students all get Ds just because I have standards and not because I think I am Napoleon.

Grades are a good tool for what we use them for which is measuring the achievement of a particular student learning a particular set of material. Expanding their usefulness beyond that scope is not impossible but would require some sort of a justification, and it usually easier simply to use some other measure.

I just know...

I am currently reading Malcolm Gladwell's book Blink. In that book, Gladwell discusses the phenomenon of how we know something immediately that might take a lengthy time to discern by reason, and, indeed, reason may never arrive at at all. He call this thin slicing. Thin slicing is how we frequently assess our students and how well they are learning a subject. We look at the class, we talk with them either en masse or as individuals, and we just know how well they are doing.

And, if you have been teaching as long as I have, you know you can still frequently be wrong.

So, in addition, we also employ classroom assessment techniques of a more formal nature like giving exams, etc. As we have to give a written report (i.e. a grade) to the registrar at the end of each semester on the achievements of our students, having a paper trail to back up our assessments is a wise thing to do as our students might have a differing opinion. Indeed, most of the time the grade is constituted of the grades of assignments on which the student received feedback as an opportunity to improve his performance. The wise student looks at his grade, looks at his assignment, and then acts in a way to improve his performance.

This provides a nice model of what we are doing ourselves in what is called the process of assessment. We are teaching ourselves to teach. When we obtained our final degrees, we were set loose in the world with the expectation that we would be independent learners. Assessment should provide us a tool for that. We grade ourselves (honestly!) on how well we have performed a task and apply the data, which is always constituted of more than just the final grade, in order to improve our performance.

Thin slicing has its place in assessment, and, indeed, our "gut" might be our final arbiter, but it has the same shortcomings there as it does in the classroom. Having a simple, natural system for assessment in place is vital in improving our student's educational experience.