This webbooklet offers students in colleges of education a look at the development of some strongly, but rationally, contested, interrelated disagreements about teaching, about assessment, and about assessing the quality of prospective teachers. The discussion presented here took place from November 18, 1993, to January 14, 1994, on EDPOLYAN, a professional listserv for education (and the predecessor of EDPOLICY), based at Arizona State University. It is hoped that presenting the issues this way will help make them more meaningful to students than essays, articles, or typical textbook material on them normally do. It is my view that until one sees, and appreciates a problem, "answers" to that problem are not often very meaningful. Some problems are definitely displayed here.

Other purposes of this site are also to show:
1) how extremely difficult communication can be without persistent discussion that tries to clarify and resolve differences, some of which are based on, often at first unrecognized, mutual misunderstanding and some of which are based on focusing on different evidence, and
2) how complex (though not necessarily difficult) seemingly simple issues can be.
If you contrast this discussion with a typical journal article or book, I think the difference will be clear.  It is my contention that a discussion of this sort goes much deeper than a typical journal article. I think that too often (education) journal articles are considered definitive when they should instead be discussed and scrutinized.

This particular discussion sprang from questioning what is necessary (and sufficient?) for good teacher training, given that some states were starting to offer alternative progams for teacher certification. This led to the question of quality of teacher education in traditional programs and to questions about how to assess the quality of newly certified teachers. That, in turn, evolved into the issue of evaluating students in the classroom in general.
Rick Garlikov

The "****" symbol in front of a line signifies the line is quoted from a previous post.

The following participated in the discussion:

I bring to the attention of this list certain changes in West Virginia policy that strike me as troubling. I am aware that similar policies have been adopted elsewhere as well. In West Virginia these policies seem, however, to be directed toward the systematic destruction of teacher education. Interestingly, these policies spell out a sort of legislated deregulation of the teacher education process. In this state, policy makers seem to believe that teacher education programs are totally useless and that teacher education ought to take place after people are employed. These policy makers view colleges and departments of education as something worse than wasteful--sort of as parasites on the educational system, draining it of its vitality.

From this sentiment derive policies directed toward downsizing teacher education programs, providing various options for alternative certification, and linking salary increments to in-service rather than college credit. The rationale is that better teachers will be produced for less money if counties are permitted to take people with liberal arts degrees and give them on-the-job training. Moreover, the state has made a serious effort to destroy administrator-training programs, offering the option for anyone with an MA and administrative experience (including all teachers) to receive a 5-dollar certificate that permits them to serve as principals (all levels), supervisors, vocational administrators, or superintendents.

So my questions to the list are these:

1. Where else is this happening and why?

2. Are the policymakers correct in their judgment of teacher education programs?

3. How should colleges and departments of education respond to these policy initiatives?

--Aimee Howley; College of Education; Marshall University

Aimee Howley asked a question that seems interesting to me, though no one has responded yet --about the effort of West Virginia to diminish or eradicate teacher education programs. I consider it to be primarily an empirical question as to whether graduates of teacher education programs make better teachers in general than people with other college training who have become teachers through various alternative means.

Since I believe some people with degrees outside of education make excellent teachers and that some education graduates don't make very good teachers, and that many education graduates do not know enough subject matter content to be able to teach as well as they should, I tend to be partial toward alternative teacher certification, based on demonstrable ability to teach, not on degree earned or knowledge of subject matter. With inservice or extra coursework as needed to learn or polish any missing skills.

I know many knowledgeable people cannot teach their knowledge to others very well, so I do not think a degree in chemistry will automatically make one a good chemistry teacher or that anyone with a degree can teach first grade, etc. But surely there must be some better way than what we have been doing to get knowledgeable, good teachers in more classrooms. Is there any research about any of this, or anecdotal evidence, or any theories..... Will the children of West Virginia end up in ignorance?

Rick Garlikov (DEMS042@UABDPO.DPO.UAB.EDU)

The teacher education question is one that is very troubling to me. As Rick Garlikov said, I too believe that some people with little training can be very good teachers. I went to a Catholic school which did not require certification, and I had some very good teachers, who often didn't even have M.A.'s in their subject areas.

However, I think in general that teacher education is extremely important, especially considering the problems found in public schools today. Someone going into teaching without proper instruction and preparation may be simply over-whelmed.

More personally, I am concerned about my future. I have wanted to become a teacher for as long as I remember. So, I planned my education to get the best preparation as possible. At the end of it all, I should have M.A.'s in education and my subject area. Now I fear that, in light of all the alternative certification routes, I may be seen as over-qualified and too expensive.

Any thoughts on this? Will people who seek advanced teacher education be squeezed out of teaching? Will it knock wages down so that people with any college debt cannot afford to become teachers?

Greg Kirschner yl361c@gwuvm.gwu.edu

I tend to share Rick's uncertainties about the best ways to attract (or perhaps cultivate) teachers who are both intellectually attentive and committed to the work. Coming to West Virginia twenty years ago with an East Coast liberal arts degree and the conviction that I could teach, I was convinced that any school system would want to hire me. But whether they wanted to or not was immaterial. I had two years of undergraduate work still ahead of me if I was to meet WV certification standards. Now--having found a way to be certified at the graduate level, having taught for a considerable time, I find myself deep in the midst of teacher education, mostly committed to a belief in its worth.

But it frustrates me that colleges of education haven't or can't find ways to accelerate or telescope instruction in pedagogy for those who have a good understanding of subject matter and good teaching "instincts." This type of alternative certification is far different from what our state department of education has in mind--but it is not an approach that I've seen advocated by teacher educators either. Are teacher educators simply protecting their turf? Or are they wedded to a socialization process that discourages certain sorts of talent? Or are they correctly upholding the benefits of an educational process that serves all prospective teachers well?

--Aimee Howley

Rick Garlikov said:

****Aimee Howley asked a question that seems interesting to me, though no one
****has responded yet --about the effort of West Virginia to diminish or
****eradicate teacher education programs. I consider it to be primarily an
****empirical question as to whether graduates of teacher education programs
****make better teachers in general than people with other college training
****who have become teachers through various alternative means.

=======================

Yes and no. As Gene has recently said on this list, facts do not change beliefs (at least not very often), beliefs interpret facts. For a reasonably complete overview of the existing empirical evidence, see Linda Darling-Hammond's recent article in the Peabody Journal, I believe. (I don't have the exact citation here, but will post it later.) The major conclusion is that, on the whole, the evidence slightly favors graduates from approved teacher education programs over alternative route teachers (although it also depends on what the "alternate" route consists of).

However, other work shows that there is, as would be expected, a great deal of difference in quality among graduates of DIFFERENT teacher education programs. Shortly after A Nation at Risk some years ago, the "evidence" of SAT scores of high school students who indicated they INTENDED to go into teaching was that they were the worst and the dullest, not the best and the brightest. We did a study of the quality of actual teacher education students across the SUNY system. What we found was that by any measures we had, SAT scores, grade-point averages, grade point averages in general education, grade point averages in the major, graduation rates, etc., the teacher education majors were the equivalent of their counterparts IN THE SAME UNIVERSITY OR COLLEGE. There was, however, a great deal of difference across institutions. Similar studies, I was told, indicated the same thing in California and Washington. All of this is perfectly compatible with the initial low SAT scores when it is combined with Schlecty and Vance's work in the early 80s in North Carolina, I believe. They showed that what had happened over a period of time was that the more selective institutions in that state had largely gotten out of teacher education, leaving the field primarily to the third and fourth rate institutions of higher education, who, in turn, attracted the least qualified students across the board, and prepared proportionately more teachers.

=========================

****Since I believe some people with degrees outside of education make excellent
****teachers and that some education graduates don't make very good teachers,

=========================

Of course. There are even cases of some people without medical degrees making good doctors and fooling lots of people for a good long while, and lots of MDs who make lousy doctors. We also sometimes have shortages of doctors. I wonder why we never hear calls for alternative routes to medicine that could short-circuit all that irrelevant training in parts of medicine that I will never use as a dermatologist?

==========================

****and that many education graduates do not know enough subject matter content
****to be able to teach as well as they should, I tend to be partial toward
****alternative teacher certification, based on demonstrable ability to teach,
****not on degree earned or knowledge of subject matter. With inservice or
****extra coursework as needed to learn or polish any missing skills.

==========================

Given the abdication of responsibility by higher education noted above and the well-known unwillingness of state education departments to take seriously their responsibility for approving only high quality teacher education programs, and coupled with the only recently emerging willingness of the National Council for Accreditation of Teacher Education (NCATE) to impose meaningful standards, there just might be another policy alternative to alternative certification. PUT THE POORLY PERFORMING SCHOOLS AND COLLEGES OF EDUCATION OUT OF THE BUSINESS.

=========================

****I know many knowledgeable people cannot teach their knowledge to others
****very well, so I do not think a degree in chemistry will automatically make
****one a good chemistry teacher or that anyone with a degree can teach first
****grade, etc. But surely there must be some better way than what we have
****been doing to get knowledgeable, good teachers in more classrooms.

=========================

The traditional way that most professions try to solve this problem is by relying on a, roughly, four-fold approach so that people who ought not be in the profession get weeded out under one component or another. These are 1) strong standards for entry into the professional preparation program, 2) a preparation program which has to meet high standards, e.g., state or NCATE approval, 3) a serious internship, 4) an examination of ability to practice.

It will not be surprising that each of these components of professional preparation is less than satisfactory in teacher education, but is it really a wise policy choice to, therefore, throw over the entire system? Actually, what is happening in most states is that 4) is being relied upon to do nearly the whole quality assurance business, with any occasional gesture toward some mentoring by a senior teacher in the first year of a new teacher's job. The problem is that the kind of examination that could actually determine good practice is nowhere in sight. Furthermore, in no other field do we rely solely on an examination of ability to practice without the other checks and balances, no matter how good that examination is. We could, logically, put all the burden of quality assurance on examinations in medicine, the law, accounting, architecture, (and even cosmetology in New York), but we don't. Only in education is such a possibility seriously entertained. Why don't we, instead, get serious about improving all four of the traditional methods for assuring quality in the preparation of professionals? Is it, as Gene says, that the different policy positions reflect different very basic orientations which only allow some folks to see what they want to see?

=======================

****Is there any research about any of this, or anecdotal evidence, or any
****theories..... Will the children of West Virginia end up in ignorance?

=======================

It will depend, won't it, on whether or not they are lucky enough to get an alternatively certified teacher who just happens to take his or her responsibility seriously enough to try to pick up the missing parts of her or his preparation on her or his own. We do, indeed, learn from experience, but if it's lousy experience, it will be lousy learning.

Hugh G. Petrie 716-645-2491 367 Baldy Hall FAX: 716-645-2479 University at Buffalo Buffalo, NY 14260 USA prohugh@ubvms.cc.buffalo.edu

Tom Mauhs-Pugh Cultural Foundations of Education Syracuse University

TJPUGH@MAILBOX.SYR.EDU TJPUGH@SUVM.BITNET

On Thu, 18 Nov 1993, Aimee Howley wrote:

**** of the teacher education process. In this state, policy
**** makers seem to believe that teacher education programs
****are totally useless and that teacher education ought
**** to take place after people are employed. These policy
**** makers view colleges and departments of education as
**** something worse than wasteful--sort of as parasites
**** on the educational system, draining it of its vitality.
****From this sentiment derive policies directed toward
****downsizing teacher education programs, providing
****various options for alternative certification, and
**** linking salary increments to in-service rather than
**** college credit. The rationale is that better teachers
**** will be produced for less money if counties are permitted
**** to take people with liberal arts degrees and give them
**** on-the-job training. Moreover, the state has made a
**** serious effort to destroy administrator-training programs,
**** offering the option for anyone with an MA and administrative
**** experience (including all teachers) to receive a 5-dollar
**** certificate that permits them to serve as principals (all
**** levels), supervisors, vocational administrators, or
**** superintendents.
**** So my questions to the list are these:
****1. Where else is this happening and why?

I don't know where else this is happening, but the why probably has to do with (1) a mistrust of program-specific, as opposed to examination-specific credentialling, (2) a concern with the expense of publicly supported ed. programs, (3) a decade of attacks on education focused on low teacher quality combined with a decade of increasing demands from teacher's unions, (4) a pervasive love affair in this country with anti-intellectualism, business model emphasis on performance over qualifications, and the entrepreneurial spirit, and (5) a clear lack of programmatic defense by schools of education exacerbated by continuous disagreement over core knowledge and commonly accepted practice of the profession.

**** 2. Are the policymakers correct in their judgment of
**** teacher education programs?

I certainly think we could do a lot better in preparing teachers to teach. From the standpoint of State educational policy needs and the staffing concerns of school districts, much of what the average (?) State college or university education program offers is inefficient at best, antagonistic to the State's interest at worst.

**** 3. How should colleges and departments of education respond
**** to these policy initiatives?

A good place to start might be to gain a coherent idea of State ed. policy and staffing concerns and address them explicitly and publicly.

Hugh Petrie has provided a very extensive and well-thought-out reply to the issue raised by Aimee Howley, in his response to my seconding of her question. There are some residual questions I have.

First, however, I thought his presentation about the four step approach to trying to ensure only quality teachers reach, and remain, in classrooms -- along with the analogy he gives in that regard to other professions-- was extremely impressive and accurate. The topic of the (too often ignored) responsibility of schools of education to turn out qualified teachers, as opposed to certified teachers, has arisen here briefly before, with no real response to it when it came up a month or two ago. Hugh's comments are detailed enough that I hope there will be a response to that issue this time, and perhaps even some important policy recommendations coming out of the discussion.

But I do want to respond to the claim that other professions, such as medicine, do not offer alternative certification routes, particularly truncated ones. And I want to talk a bit about the difference between teacher training and medical or architecture training. First, medicine --though not medical schools-- do offer different, often shorter routes to practicing health care delivery. Midwifery, psychology, nursing, EMT training, chiropractics, homeopathy, acupuncture, lab technology, physician assistants, etc. are various entry modes into health care delivery. They meet certain needs (or at least try to or purport to). Further, doctors are often taught by people without medical degrees. Biochemists teach medical biochemistry; anatomists teach anatomy; researchers of whatever sort may teach about their specialties; bioengineers and bioethicists have various roles in teaching medicine and assisting medical practitioners. I would think that teaching would offer a great many similar opportunities -- especially for bringing in specialists periodically to teach areas the teacher is not particularly good at teaching -- for example, many elementary teachers have difficulty teaching certain math concepts: place value, fractions, general sorts of math reasoning. Would it be wrong to have those math specialists who have some real "instincts" for teaching (Howlee's or someone's apt term) be responsible for teaching these kinds of things in those classrooms?

I also do not believe that most undergraduate degrees give enough training in content. A B.A. in math education or even in math, may be insufficient training to be able to really teach math well -- even elementary math. A good teacher is not simply generally just a step or two above their students, but is someone who understands both the subject matter, and students, well enough to be able to make the subject really meaningful and inspiring to them.

Finally, I would like to know what sorts of things you all think an education student with good teaching instincts, who wants to teach in, say grades 6-12, needs to know, and whether he/she could not be taught those in one or two or three courses, one term? A local college gives a "fifth year" certification program, but from what I have heard from students, it seems to be a worthless, self-study, more or less self-directed, literature research kind of program that seems to have little meaningful or practical help. Surely that year could be better spent learning real teaching skills or helpful practices, rationales, etc., no? The 5th year program is an alternative certification program for people with undergraduate degrees outside of education.

These are some of the questions I want to raise at this point. But I want to go back over Hugh's response some more. And I hope most of you will take another look at it and that it can be the genesis of some very useful educational policy recommendations. It is an excellent post that deserves further discussion and recognition, I think.

Rick Garlikov (DEMS042@UABDPO.DPO.UAB.EDU)

Rick asked about the things that a knowledgeable college student with good "instincts" for teaching might need to know about teaching in order to perform competently. I would suggest that foremost among them would be an understanding of what schooling is (has been and could be) all about coupled with an understanding (however imperfect) of what learning might encompass. Focus on the technical skills of teaching strikes me as a distinctly subordinate enterprise.

--Aimee Howley

I tend to agree with Rick Garlikov that alternative routes are important and that good teachers can come from inside or outside of schools of education. I like the graduate schools of ed that are concentrating on school development and teachers in classrooms. When I ran a private school, I always chose BA, BS and Masters degree holders in content areas - people who had those degrees plus the experience or makings of good teachers. In Vermont, there is some discussion about levels of licensure, where one needs the education school more (I think) between level one (just out of college) and level two.

Just a couple of questions on this topic.

I'm all for closing down "weak" schools of education. By what standards do we determine who is to be closed down? Since standards are shunned at both the secondary and college levels, where would we start?

What's wrong with "an emphasis on performance over qualifications?" I think part of the reason we are in the fix we are in is that we WORSHIP the flip-flopped arrangement of priorities.

Cheers,

Kevin Drumm NOVA University 305-424-5758 drummk@Polaris.NOVA.edu

I have now found the reference to the Linda Darling-Hammond literature review article I mentioned in my post of November 20. It is "Teaching and Knowledge: Policy Issues Posed by Alternate Certification for Teachers." _Peabody Journal of Education_ vol. 67(3), pp. 123 - 154.

Rick Garlikov raises some additional questions.

=========================

**** But I do want to respond to the claim that other professions, such as
****medicine, do not offer alternative certification routes, particularly
****truncated ones. And I want to talk a bit about the difference between
****teacher training and medical or architecture training.
**** First, medicine --though not medical schools-- do offer different, often
****shorter routes to practicing health care delivery. Midwifery, psychology,
****nursing, EMT training, chiropractics, homeopathy, acupuncture, lab technology,
****physician assistants, etc. are various entry modes into health care delivery.
****They meet certain needs (or at least try to or purport to).

====================

You point to an extremely important area of policy reform in the system of teaching, not so much teacher education, although it could be used there as well. Other professions have any number of auxiliary roles and specialties defined so that the central practitioner can devote his or her time primarily to what she or he knows best. In education, we have one teacher, one classroom, with little use of teacher aides, interns, or specialists. The few places we do have specialists, they tend to be folks who "pull out" the kids for remedial reading or LD classes, instead of working as part of a team of professionals. The problem here is that this system would likely require more of a differentiated staffing pattern in schools than traditionally teacher unions would like to see.

Where all of this might impact on teacher preparation would be if we did less overall preparation of teachers and did more of it in "professional development schools", the teaching hospitals of the teaching profession. In this way you could get more of a seamless web of expertise, from university professors and graduate students to mentoring from experienced practitioners, all working as a team, which, since we teach as we were taught, might do more for encouraging team teaching than any amount of exhortation.

================ Rick further says

****Further, doctors are often taught by people without medical degrees. Biochemists teach
****medical biochemistry; anatomists teach anatomy; researchers of whatever
****sort may teach about their specialties; bioengineers and bioethicists have
****various roles in teaching medicine and assisting medical practitioners. I would
****think that teaching would offer a great many similar opportunities -- especially
****for bringing in specialists periodically to teach areas the teacher is not particularly
****good at teaching -- for example, many elementary teachers have difficulty
****teaching certain math concepts: place value, fractions, general sorts of math
****reasoning. Would it be wrong to have those math specialists who have some real
****"instincts" for teaching (Howlee's or someone's apt term) be responsible for
****teaching these kinds of things in those classrooms?

================

Again, this is a very plausible suggestion. If we had teams of teachers and teacher aides and instructors and specialists responsible for different aspects of the educational experience, we could, in principle, make better use of their individual talents. However, one needs to realize that this would represent a MAJOR cultural change in our current egg-crate organization of schools.

=================

Rick goes on,
****I also do not believe that most undergraduate degrees give enough training
****in content. A B.A. in math education or even in math, may be insufficient
****training to be able to really teach math well -- even elementary math. A
****good teacher is not simply generally just a step or two above their students,
****but is someone who understands both the subject matter, and students, well
****enough to be able to make the subject really meaningful and inspiring to them.

===================

I would say, ESPECIALLY elementary math. Think for just a moment about the typical collegiate math degree. What parts of it are at all useful for teaching elementary mathematics? The things that might be, e.g., number theory, geometry, and statistics, are often not even part of the collegiate curriculum. We have to have geometry taught at another college here at UB since our math department seldom does so. However, as Denise says, the usual response by state legislators is

==================

****Virginia is trying to remedy the problem of how to educate teachers by
****requiring college students to major in a discipline (content area) and
****get a certification to teach. This is true no matter what grade the
****individual wants to teach (NK-12). There is no longer a major in education.
****What are your thoughts on this approach to teacher education?

==================

Without a major change in the ways in which the non-professional education portions of the major are offered in the typical institution of higher education, this cure will be worse than the problem. I think it was David Berliner about 7-8 years ago who studied the effects of majors on teaching ability and, essentially, found that it made no difference except, perhaps, for those who teach at the advanced placement level. Here, too, I would predict that there would be major differences across the majors and institutions.

There is no question in my mind that a careful approach to offering the major with attention to what Shulman calls pedagogical content knowledge would help a great deal, but, again, this will require a MAJOR change in how, and even whether, our arts and sciences colleagues see themselves as part of teacher education. As Pam Grossman has suggested in a recent _Teachers College Record_ case study on several alternate route teachers, they teach as they were taught. They also learn from the experience of their first job. Those who were concerned that the seminar, abstract styles of their college courses weren't getting across somehow managed to hook up with some people who helped them out. Those who didn't blamed the students for not learning, even though it was painfully apparent that the kind of instruction these alternate route teachers offered was wholly inappropriate.

I would go so far as to predict that the major problems in teaching in secondary schools arise from teachers modeling the teaching they saw in their arts and sciences courses in college. The second major problem in secondary schools probably comes from inadequate content teaching in the arts and sciences. The problem is NOT that secondary teachers don't have a major in their field. Almost all do. The problem is more likely to be that the person who majored in chemistry will also have to teach a biology section and a mathematics section as well as chemistry. The one bright spot here is that the public is beginning to demand more accountability on teaching undergraduates from our institutions of higher education. Maybe that will help.

That brings me to some comments on the value of an "academic" major for elementary education. Implemented mindlessly, as the Virginia system appears to be, this, too, would be a disaster. Think for just a moment about the typical majors in college and their possible usefulness in elementary education, given what we currently ask our elementary teachers to do. If we REALLY want deep understanding of subject matter, combined with a knowledge of the development of kids, then we are, if we don't change our systems drastically, basically asking elementary teachers to attain the level of understanding of FOUR OR FIVE major fields which we don't think even very many of our bachelors or masters students achieve in one field. And just how long is the preparation period for elementary teachers to be?

I would fully grant that if the major problem in secondary education is probably inadequate pedagogy, the major problem in elementary education is probably inadequate content knowledge (and the major problem in middle schools is raging hormones). However, for reasons like those noted above, the answer is unlikely to be to require an academic major of elementary teachers, especially as we currently conceive of academic majors. One solution, compatible with the notion of changing to more team-oriented approaches to schooling outlined above, would be to require elementary teachers to take a seriously and carefully designed minor in one of several areas, e.g., math, reading and literacy, science, sociology. Then that person could be the "expert" on the team in his or her area of concentration.

Rick Garlikov goes on to say

==============

****Finally, I would like to know what sorts of things you all think an education
****student with good teaching instincts, who wants to teach in, say grades 6-12,
****needs to know, and whether he/she could not be taught those in one or two or
****three courses, one term? A local college gives a "fifth year" certification
****program, but from what I have heard from students, it seems to be a worthless,
****self-study, more or less self-directed, literature research kind of program
****that seems to have little meaningful or practical help. Surely that year could
****be better spent learning real teaching skills or helpful practices, rationales,
****etc., no? The 5th year program is an alternative certification program for
****people with undergraduate degrees outside of education.

and Aimee Howley echoes this concern

****But it frustrates me that colleges of education haven't or can't
****find ways to accelerate or telescope instruction in pedagogy
****for those who have a good understanding of subject matter and
****good teaching "instincts." This type of alternative certification
****is far different from what our state department of education has
****in mind--but it is not an approach that I've seen advocated by
****teacher educators either. Are teacher educators simply protecting
****their turf? Or are they wedded to a socialization process that
****discourages certain sorts of talent? Or are they correctly
****upholding the benefits of an educational process that serves
****all prospective teachers well?

====================

I'm not at all sure what "good teaching instincts" are nor how we would tell if someone has them, but the notion of a serious alternative ROUTE or multiple ENTRY POINTS to certification is a good one. The problem is that state departments of education seem to equate "alternate" with emergency and lower standards of certification and colleges of ed seem, for the most part, to believe in "one best way". The challenge is, I think, to design serious fifth year and MAT programs with highly interactive and collaborative clinical work in professional development schools along with significantly new and improved work with our arts and sciences colleagues. Again, however, the problem is a MAJOR cultural difference between what we in higher education value in arts and sciences, in schools of education, and what most schools value. Nevertheless, we must, in my judgment, begin to bridge these cultural divides and think about what the Education Commission of the States is beginning to call the SIMULTANEOUS RENEWAL of schools and higher education.

Sorry to have gone on so long, but maybe this will be helpful.

Hugh G. Petrie 716-645-2491 367 Baldy Hall FAX: 716-645-2479 University at Buffalo Buffalo, NY 14260 USA prohugh@ubvms.cc.buffalo.edu

Hugh Petrie has a wealth of ideas for teacher education and I plan to look back at them from time to time. The ones that stood out on this reading had to do with differential staffing. Some years ago, I was involved in an evaluation of a Follow-Through program that made extensive, but rather mechanical, use of teacher aides. At the same time, I evaluated an innovative program in a wealthy suburban school that was making really interesting use of parent volunteers. Some of us here are now thinking about what we would want to include in a short course for volunteers or aides. Anyone had experience along this line?

bill hunter fax: (403) 282-0083 the university of calgary phone: (403) 220-5691 hunter@acs.ucalgary.ca

Having just finished a rather hectic term instructing prospective Social Studies teachers on the intricacies of methodology and classroom practice (they're now out in the schools for their first practicum round of four weeks before returning on campus for more classes), I believe that I can shed some light on why at least a one or two year program from a school of education is necessary for the development of superior beginning teachers.

The g.p.a. for my methods class was probably in the area of 3.7 to 3.8 out of 4. B.Ed. students are required to have an average of at least 3.5 to enter the program at the end of their first year of general studies at this university. Students who have a first degree are required to have at least a 3.7 average before being admitted into the secondary Social Studies program. Most of my students (19/35) had a first degree in one of the social sciences (primarily history, secondarily sociology); three had M.A.'s; many of the other students are working on simultaneous degrees in the faculties of Arts and Education. Several of these students had extensive experience teaching and instructing in other countries or in fields which did not require formal certification. Most of them are highly motivated, despite the fact that job prospects in our province are worse than dismal (the Alberta government proposes to cut 20 to 30% of the Education budget over the next three years). Yet regardless of their motivation and ability these students have at least one major flaw that was evident as they entered the program: their conception of teaching and learning was based entirely on their experience as successful students in a system that depended primarily on lecture, the replication of textbook answers and the taking of notes as the dominant instructional methodology (I think I see a cross-connection to the "Why Don't Teachers Incorporate Research on Learning" thread here).

So one of the prime tasks that the methods instructors in Social Studies set ourselves was to model a variety of more effective classroom approaches as we persuaded these students that there is more to teaching than "stand and deliver". It would be nice to believe that in their eleven weeks of practicum in the classrooms of this province our students would get that message from practicing teachers, but....as I hear more and more stories coming back from the classrooms I begin to understand the despair of those Curriculum Supervisors who have been trying to improve practice in our jurisdiction over the past ten years. I just ran into one of my students in the library as he was pouring over the microfilmed back issues of newspapers preparing his lesson-plans for tomorrow. As he told me that his students were just finishing studying the issue of NAFTA and he was looking for a concluding activity I blurted out the idea of holding an informal horseshoe debate--allowing students to make use of the concepts and arguments they have been researching for the past week. "Well...I don't know if my co-operating teacher would like it," he replied. "He usually has them sit in straight rows and I don't think they are ready for this kind of group work yet."

Unfortunately that kind of a classroom appears to be the rule rather than the exception as reported by the students who have been bringing back their classroom observations to their practicum and methods instructors. I know that there are different schools out there, I've taught in two over the past twelve years, but it seems clear that the impetus for change and improvement that I have seen owes a great deal to some of the people working within the faculties of Education in this province. I believe this is particularly so in those instances where the instructors or faculty have close or recent connections to the classroom (three of the four Social Studies methods instructors here are graduate students just out of the classroom).

If Faculties and Schools of Education are not the people to provide leadership in education then who are? Some practicing teachers are burned out, some are mired in unproductive teaching methods, some are just focusing on raising their average on the provincial exam and some are counting down the days to early retirement. Principals are kept busy on the rodent-wheel of system and department administrative meetings. The general public seems wedded to the idea of school as it was experienced in the good-old-days of reading, 'riting, 'rithmetic and 25% graduation ratios. As for politicians....well this is a pg rated forum and I don't want to cross-post to alt.scatalogical.comments.

That's it for now, they're shutting down the microlab and I still have 25 unit plans to mark that my students left me before they went out to the schools.

--Noel Jantzie

Date: Fri, 26 Nov 1993 17:49:56 CST Reply-To: Education Policy Analysis Forum <EDPOLYAN@ASUACAD.BITNET> Sender: Education Policy Analysis Forum <EDPOLYAN@ASUACAD.BITNET> From: Rick Garlikov <DEMS042@UABDPO.BITNET> Subject: the nature of good teaching

There was a question about the nature of teaching and learning (Aimee Howley asked it, I believe, or Cindy Cotter). And there has been much comment about teacher directed classes versus student centered courses, along with comments about the problems of teaching the way one was taught, especially the way one was taught in college. (Hugh Petrie, for example, said "I would go so far as to predict that the major problems in teaching in secondary schools arise from teachers modeling the teaching they saw in their arts and science courses.") If Hugh and others mean something other than "lecture, memorize, regurgitate on the exam", please say so. Many of my liberal arts courses were not of that type, so I am wondering if there is something besides that which is also considered NOT good teaching. Hugh also said he was not sure what "good teaching instincts" meant. All this goes together I believe, and I would like to make some general comments for your considerations; and ask some questions.

First, it looks as if the study about the quality of education graduates versus non-education graduates that Hugh reported on was about their quality as students, not their quality as teachers. What I wanted to know is whether there is a difference in TEACHING ABILITY, in general, between ed grads and non-ed grads. Can, for example, philosophy grads or liberal arts grads, teach a given high school topic, or whole subject, or teach elementary school, better than an ed grad? Can a non-ed grad with an MA in English or math teach English or math better than an ed grad, even an ed grad with major in math or English? Comparisons of SAT's and GPA's will not show that. Quite good students often make quite lousy teachers; and I have known some poor students who made excellent teachers in a particular area. Quite knowledgeable professors who are experts in their fields make some of the worst teachers. I have had a few of them. Many of you have also.

What I mean by a good teacher is someone who can get, or help, (1) a student to learn something, (2) to understand it in those cases where understanding is appropriate, (3) in the easiest, sometimes most efficient way (as long as efficiency does not sacrifice learning or understanding), AND (4) make it interesting enough for the student to want to learn more and/or to use what is taught in some meaningful way. The better teacher is not always the more knowledgeable person. I once served as a graduate teaching assistant to a man who knows fifty times more information about the history of philosophy than I will ever know, but his lectures and his topics of course study for philosophy 101 killed the interest of hundreds of students that unknowingly signed up for his course when he was the lecturer. Though I knew far less, I could turn out students who knew far more philosophy, understood it better, liked it better, and wanted to go on in it. He was by far the better scholar, but for introductory philosophy, I was the better teacher, I would argue.

I would argue that a teacher is to be judged by how well they teach a given student or group of students -- where the person doing the judging needs to have a good idea of the degree of difficulty of the subject matter for a given group of students. Judgment is a somewhat subjective enterprise but hardly just a matter of whim. There is some inter-subjectivity to it. If I can teach almost anyone to ride a bicycle in thirty minutes, and they enjoy learning and want to ride all day after they have learned, and if they are not afraid with me teaching them, and don't cry, etc., then I am a better bicycle riding teacher than someone who makes kids cry, makes kids not want to learn, takes weeks to teach them, and makes the experience so terrifying and unhappy that they really don't much care to learn or to ride their bike once they have learned. Something similar could be said for any two teachers in any given subject.

All this has some bearing on tracking, and on Hugh's comments about professional quality control. The rap against tracking, as I understand it, is that, besides taking good student role models away from students who might benefit from their example, generally the "lowest" kids are given the worst teachers, which perpetuates, sometimes forever, their inability and lack of learning, lack of knowledge, lack of understanding, lack of skill. But giving these kids bad teachers is not an essential ingredient in tracking. It might be that kids at different skill levels or different rates of learning a given subject need teachers who understand their needs best. And this might be a boon. I don't know exactly how research on tracking is done, but it sounds suspiciously like there are often factors, apart from merely grouping students of like ability, that have far more influence on the outcome than the grouping itself does. Of course, it is difficult to have proper "control groups" and to eliminate some of these other factors. I tend to suspect sometimes tracking is a wise way to proceed and sometimes it may not be. I am not sure there is sustainable evidence that it is never wise to track. However, I am against giving anyone bad teachers, i.e., teachers who cannot teach them the most they can learn in a way they can most readily learn it, and like it, and want to learn more. And I am particularly against giving kids bad teachers at a stage of development that pretty much makes it impossible for those kids to ever go on in the subject. A bad teacher who does not "ruin" it for a kid is one thing; a bad teacher who does is a far more reprehensible matter. Giving "at risk" kids bad teachers seems to me to be particularly unconscionable because you are virtually consigning them to their educational doom.

And, I am with Hugh, though I may say it more strongly than he does, in believing it is reprehensible for ed schools to turn out bad teachers and for administrators to keep them on and in some cases give them to the most vulnerable children. And I do not think it is that difficult to tell whether somebody can teach something or not. Unless, of course, the judge does not have a clue how to tell whether a student or group of students has learned something, and learned it enthusiastically or not. In terms of POLICY RECOMMENDATIONS, I would love to see school administrators and college faculty and departments able to be fined or sued for teaching bad courses and/or for graduating and certifying bad teachers -- not just inexperienced teachers, but teachers with little or no ability to teach anyone anything. A little personal or professional responsibility with some personal consequences might do wonders for instituting a real four-point quality control program of the sort Hugh has described.

Putting "bad" or inappropriate teachers in courses happens in college as much as anywhere else. I had a graduate course in logic one time taught by a Swedish logician who did not speak English well enough to be able to understand questions, or to answer them in any intelligible way. I had a course that was supposed to be an intermediate graduate level course in Aristotle. The teacher who was hired thought it was supposed to be an advanced level course, and because two of the twelve students in the seminar could read classical Greek, he taught them while the rest of us sat quietly. He would not change the course, nor would the department make him, even though it was not "teaching" ten of us anything much about Aristotle at all. Almost every graduate student has more than one such story about worthless courses they had to take that everyone knew was worthless.

In regard to "teaching instincts", there are a number of things involved that all relate to the above. Does a perspective teacher understand that more than one method may be necessary to teach a given person a given skill or some understanding or knowledge? Does he/she understand that the way things are organized and presented can make a big difference in how or whether it is learned? Does the teacher understand that the failure to teach is not necessarily, or likely, the students' fault, and that it means a different teaching approach is necessary, or a different attempt, or some "block" needs to be overcome somehow, or that some other thing needs to be taught first in order to make the current subject accessible to the student? Is the teacher motivated to keep trying? Is the teacher excited by success of the student's learning, and is the teacher really interested in general in getting or helping students learn things? Does the teacher think knowledge and ability is great to achieve, and to help others achieve? Do they care more about students' learning than about getting a bell-curve grade distribution? Does the teacher understand nothing is taught until a student has learned it; that presentation is not teaching unless the presentation is meaningful to the student in the right way? Does the teacher see the student as a human being or just an empty vessel? Does the teacher try to find out what the student already knows, or is learning as the "lesson" progresses? Is there any dialogue, or any curiosity on the teacher's part as to what sort of impact he/she is actually having on the child? Is the teacher able to appreciate ambiguities, mistakes, and misunderstandings, and treat them as natural occurrences without making the student feel the student must be somehow stupid or inept, or that it is their own fault they cannot understand? Etc., etc., etc.

Aren't these the kinds of things that good teaching is about?

Now, I surely do not understand the stuff about teacher directness versus student centered learning, or however it is called. I do not see these as mutually exclusive nor jointly exhaustive. When Louis Schmier took his camera equipment to the classroom he tells about, he was directing the lesson and controlling its content within certain boundaries; it is just that he is doing it by letting students do certain things actively instead of just taking notes or "listening" while sitting passively. He has set up the environment, and probably says enough things and steers just enough so that the students' explorations are productive. Had he not set any direction or taken any equipment in, or answered any questions at all, little would have happened. I assume that we don't want students having to reinvent by themselves the history of civilization and its achievements. Some sort of "telling", guiding, steering, directing, or whatever has to go on or it will take them thousands of years to learn thousands of years worth of accumulated knowledge. Isn't "teaching" supposed to be a more or less shorter way for students to gain knowledge than by mere exploration and re-invention. Otherwise why have teachers and schools at all! Aren't we interested in the most effective ways to transfer knowledge and ability to the next generation (or whoever comes to us for learning). If a way is the most effective, does it matter whether it is lecture or not? Perhaps some people can lecture very well, like a good story teller. If it is not effective, does it matter whether it is student "driven" or not?

From what has been said, and the examples given, I assume YOU mean by "teacher directed" either of the following two things: 1) teachers just lecturing, especially droning on in ways that stifle children's learning, and 2) teachers not letting students explore and attempt to figure things out, or do, things that would be good learning experiences for them, and which they could do successfully in some fashion or other. Having kids paint by the numbers would be teacher directed, I presume, even though the kids are doing the work. Using the Socratic method to question and challenge students, and make them figure things out for themselves, would, I presume, be child directed even though the teacher would be guiding the discussion to keep it logical and sensible in case a student happened to start going too far astray. Is this a fair description of the dichotomy you have in mind? If so, I think the words are misleading that you use to express the dichotomy; misleading to ed students and to others. I would rather see the emphasis put on when "telling" might be most appropriate, when exploration, when questioning, what sorts of questions are the most productive, how to guide exploration without "killing it" or oversteering it, yet without letting it turn into fruitless, wasted effort.

Let me give two examples of what I consider bad teaching, though they emphasized student involvement, teams, etc. (1) Seventh graders were grouped in fours and were shown objects brought from the teacher's home: a Scotch tape dispenser, a small jewelry box, a melon "baller" (the thing with little "scoops" on each end to make melon balls with), a scissors, a small jar, etc. Each group was given an object and asked to "analyze" it, which, from the best I could judge, meant merely to describe it, since descriptions were what the teacher accepted as good answers. As luck would have it, a group of boys got the melon baller, and none of them had the foggiest notion what the devil it was. But that was the easy part. The difficult part was they were to use the object they were given as a metaphor for any one of a group of "abstract" nouns the teacher had put on the board: love, justice, prejudice, honor, honesty, truth, democracy, etc. One group of girls was able to do it with one item and one noun, but the rest of the kids all sat there essentially waiting for the bell to mercifully ring. The lesson was meant to teach metaphor. Wouldn't these kids have been better off with some artfully and interestingly presented neat examples of metaphor first that they might have been able to relate to?! (2) Another seventh grade teacher was introducing her social studies or English class to debate. She chose the topic and divided the students into two teams. Resolved: that euthanasia be made illegal. Or some such. Notice that the "pro" side was both the negative, in a sense, and the current state of affairs. That alone had half the kids unable to understand what it was they were supposed to be arguing for, since euthanasia already was illegal. Everyone was to do their own "research", meaning things like finding out what Time Magazine said, etc. They were to try to be as persuasive as they could using the "facts" they found. But neither sarcasm nor any other kind of pointed comments were permitted during the debate. The teacher admitted all this was very difficult for students "especially," she said, "when they have to argue for the side they don't believe in". She even KNEW she was making it harder and more confusing than it had to be.... This was their first debate. The students hated it; I am told most of the students each year hated debate at the end of the term, and "thought it was stupid". Yet, for the most part, it was not teacher lecturing. Yet still bad teaching, no?

Aren't there better ways to talk about good teaching than whether it is lecture or not, teacher directed or not, etc.?

Rick Garlikov (DEMS042@UABDPO.DPO.UAB.EDU)

On Tue, 30 Nov 1993 19:19:51 CST Rick Garlikov said:

****What I am really getting at is that it seems to me that schools of education
****should, above all, graduate or not graduate their own students on the basis of
****how well those students can teach. =================================================================
Rick: What would you think of the proposition that we should judge the performance of teachers of ethics by determining the moral qualities of their students? Tom Green

Date: Wed, 1 Dec 1993 19:40:53 CST Reply-To: Education Policy Analysis Forum <EDPOLYAN@ASUACAD.BITNET> Sender: Education Policy Analysis Forum <EDPOLYAN@ASUACAD.BITNET> From: Rick Garlikov <DEMS042@UABDPO.BITNET> Subject: Re: On the Nature of Teaching In-Reply-To: Message of Wed, 1 Dec 1993 09:48:40 LCL from <TFGREEN@SUVM>

Tom Green said:
****On Tue, 30 Nov 1993 19:19:51 CST Rick Garlikov said:
********What I am really getting at is that it seems to me that schools of education
********should, above all, graduate or not graduate their own students on the basis of
********how well those students can teach.
**** Rick: What would you think of the proposition that we should judge the performance of teachers of ****ethics by determining the moral qualities of their students?

I am not asking that education professors be judged on how well their students teach. I am asking that professors and schools of education not certify people to be teachers who cannot teach. Obviously not everyone learns what they are supposed to in various courses, and that is not always the fault of the teacher; but it is the fault of the teacher to pass students who do not learn what they are taught. By the way, an ethics course is not a socialization course, at least as I teach it. What I try to do in part is to help students be able to decide what is right or wrong, good or bad. I cannot make them choose the right, though I believe, as Socrates did, that people with any ethical understanding and sensitivity at all will choose the right. By passing a student I am in part certifying that he has learned to be reasonable about determining what is right or wrong, good or bad. I am not certifying he/she will always be right or will choose the socially acceptable or fashionable course. Nor am I certifying that he/she is a good person.

I don't hold schools of education accountable for teaching all their students how to be good teachers; I do hold them accountable for graduating those that are not good teachers and that the schools ought to know are not good teachers.

By the way, I had a student in one of my ethics classes pull a prank on the college by semi-hacking into their computer system and changing the password. After he did it, he realized they might not have known it was merely a prank and that no records or anything else had been tampered with. So he turned himself in and let them know that was all he had done. They were not amused, and expelled him with a much too harsh and rather stupid penance required for his reinstatement. I found out about it after the penalty had been meted out. He had not really had a fair hearing, and I wrote a long letter pleading his case (noting the irony that perhaps I should not be considered a good ethics teacher). The president of the college wrote back that my arguments had considerable merit but he would not change the punishment the faculty panel had recommended. I wrote him back asking if he would let the faculty panel read my recommendations for what I thought was a more just and helpful penance for re-instatement, to see whether they would change their recommendation. He never answered that letter. I pursued it to no avail, and gave the student a copy of my letter to use for seeking transfer to another college. I thought the kid was a good kid who had made a mistake for which he was being overzealously and irrationally punished. My letter stated that, and why. The student's grade was not all that good in the course because it was extremely difficult for him to articulate his ideas, but he had done enough to pass the course. My grade would not have certified him to be ethical. But my letter did. Education schools by their degrees are certifying people can teach, not just that they can state ed theory, etc. Rick

RICK GARLIKOV (DEMS042@UABDPO.DPO.UAB.EDU)

Date: Thu, 2 Dec 1993 08:28:35 EST From: Aimee Howley <U176C@WVNVM.BITNET> Subject: Re: On the Nature of Teaching To: Multiple recipients of list EDPOLYAN <EDPOLYAN@ASUACAD.BITNET> In-Reply-To: Message of 12/01/93 at 19:40:53 from DEMS042@UABDPO.BITNET

Rick--

Teaching depends a great deal on context, which makes it difficult for teacher education programs to assure that their graduates can "actually" teach. Moreover, the climate of schools often discourages thoughtful teaching, so some of the very best candidates (by which I mean the ones who understand kids, subjects, and how to bring the two together meaningfully) have the worst time demonstrating their ability to teach (as that role is defined by the real schools in which they do their student teaching).

I worked with a number of student teachers whose performance in one setting was exemplary and whose performance in another was poor. Since we can't put student teachers in all the sorts of settings that they might actually find themselves, we have to rely on a few placements. Our judgments about teaching potential are, thus, quite speculative.

--Aimee Howley

bill hunter fax: (403) 282-0083 the university of calgary phone: (403) 220-5691 hunter@acs.ucalgary.ca

Sue, your previous post pointed out a particular thing I left out, in order for my claim to be true: the person doing the judging has to know enough about the topic (usually) in order to be able to ferret out whether the learner has learned from the student teacher. I am assuming, however, that for the most part that is true. It is not necessary in those cases where the learner clearly says he/she does not understand what the student teacher is getting at; it is necessary where the learner thinks he/she understands and the judge has to determine whether the learner really does understand. I think it may be difficult or impossible in some cases for a judge who does not understand the topic to judge whether it was taught or not. And that does happen in some cases -- judges get fooled. (I have just done a theoretical review of the literature on teaching numerical "place-value", and I argue there are a number of cases where the researcher does not properly understand the concept of place-value and therefore is incorrectly judging the effective- ness of teaching it.) The cases I was thinking of when I made the claim, were cases where the judge understood the topic and could fairly easily "pre-test" and "post-test", or could diagnose student learning at the same time the student teacher did, by observing learner's answers or responses to questions and comments.

Moreover, I have only been talking about teaching a topic under reasonable conditions. Reasonable conditions are not always existent. Both you and Cindy Cotter point out some of the things in schools that make teaching almost impossible for even the otherwise best of teachers. Having to teach topics you don't know well; having to teach with interruptions by intercoms and bells; having to write out lesson plans; monitor lunch; deal with parents and unhelpful administrators; not having sufficient materials; etc., etc., etc. But these are all kinds of things that need to be changed wherever possible, not things that good teachers can simply work around. Nobody can work around many of those kinds of things; and I am not faulting people who cannot teach well under those sorts of conditions; nor do I have a cure that does not involve first removing those conditions.


**** > 4) Those with a natural gift for teaching (i.e.,
****those who are the
**** opposite of the above descriptions) ought to be able, it seems, to be
**** taught whatever other aspects of the profession that teachers need to
**** know in a shorter time period, since they have the, or one of the essential,
**** ingredient(s) already. That is why some have asked for a shorter
**** certification process for those with a degree in a content area and
**** "good teaching instincts."
****
****Great. I think you're absolutely right. Now explain to me how I can
****tell, upon entry to a teaching program, who has this "natural gift" and who
****doesn't. I'd just like to be able to tell, upon application, who will be
****harmful to students and who won't, so I can keep the harmful ones out of
****even student teaching.
**** Since I think people can learn how to teach, I do not mean for those without natural ability to be kept away from student teaching. I want them kept away from student teaching if they have not learned how to teach (or had natural ability to teach) by the time they are supposed to be ready to student teach. That being said, at whatever point you want to test YOUR students: 1) Have them teach you something you do not know -- maybe a card game or a board game or some such; or a physics or math principle, if that is their field. See how well they teach it to you. Or 2) have them teach another adult. See how well they teach a math principle to someone whose field is English, or vice verse. Or 3) test them out on a small group of students in a serene environment where the students realize their possible failure to learn in this case is no reflection on them, but that you are testing to see how well this person can teach them something.

Remember, all this is designed to do is to see whether somebody is egregiously bad enough to keep out of a classroom type situation at the present time, or is good enough to skip ahead faster into a classroom situation once they have learned whatever other things are necessary to know about being in a classroom. Success at this is not a guarantee of success in the classroom; but failure at this would, I would think, be virtually a guarantee of failure in the classroom. (I am talking about severe failure here, not just an approach that fails but could be made to succeed with a chance to reflect on the failure and modify the approach. Even REALLY GOOD teachers sometimes come up against a brick wall and need to reflect and revamp in order to achieve success.) If a student of yours cannot teach you or some other adult, or some small "control" group of children something he/she is assigned to teach -- and demonstrates real inability to teach -- I would think you know they are definitely not ready for the classroom yet. Is this a feasible way of looking at this? Is it pragmatically feasible given your workload, etc. Remember, the point is simply to discover those people who really are not ready (yet) to teach, before inflicting them on a bunch of kids and the classroom teacher who is responsible for them. I think there are many people who simply cannot teach (perhaps anything) very well. Those are the ones we need to keep away from kids whom they would intellectually harm. And, at least in Alabama, many of those people are not kept away from classrooms. If you find a number of these kinds of student teachers once you have turned them loose in a classroom, couldn't you just as easily, more quickly, and less detrimentally find them beforehand using something like the ways I just described? Rick

RICK GARLIKOV (DEMS042@UABDPO.DPO.UAB.EDU)

Jack Letarte Taylor University Upland, Indiana 46989 317-998-5153 Internet: jcletarte@tayloru.edu

Rick Garlikov (DEMS042@UABDPO.DPO.UAB.EDU)

----------------------------Original message----------------------------

I have some problems with your way of helping students deal with the controversy, \in ed schools between different theories taught by different profs-RG| but I surely agree with you that we need to do this. Largely it is ducked, I think. A way we have tried here with some success is to team instructors that have different perspectives, and have discussions in the classroom with our students in which the profs also take different stances. (We teamed a C&I, a Spec. Ed., and an Ed Psych prof with a classroom teacher in a middle school seminar for prospective teachers and had some very interesting debates and discussions.)

Another idea a group of us just came up with is to have students design a basic approach to organizing their classroom, using theories of development and individual difference, but within the school they are doing field work in. Sort of: imagine you are in an interview with teachers and the principal of your current school. They ask you to describe how you would set up and manage your classroom, if hired. Given what you know of theory and what you know of the teachers and principals at your school, how would you describe your approach? How can you convince them to accept your ideas? Or something like that. We're still working on it.

Hasn't this been a fun discussion? SN

Susan B. Nolen 322 Miller Hall DQ-12 University of Washington sunolen@u.washington.edu Seattle, WA 98195

Tom Mauhs-Pugh recounts a horror story that surpasses most of our worst nightmares about schools, even though most people see some aspects of his experience in schools from time to time. But to have lived all of them at once..... From this example he says "Single causal theories about improving education or teachers are doomed to failure." I suspect that is true, and that his example demonstrates that pretty well. However, I think it is fair to say that our discussions about teacher certification and testing have not been efforts to determine single causal theories about educational problems. I think we are pointing out a number of specific problems, some of which are exemplified in his account. I don't remember anyone saying that if any one thing was "fixed" all would be well with schools.

I may have come closest to doing that, but I was talking about a particular, fairly well-to-do district where materials, supplies, "good" students from relatively advantaged backgrounds, etc. abounded, and where principals tend to be non-interfering (to a fault in some cases), and where teachers still taught poorly in ways that seemed to miss some real "basics". I pushed in this discussion for why THEY couldn't teach -- weren't they taught certain things about teaching? Is anybody? Etc. From that modest beginning, started if I remember, by Aimee Howley, the discussion has grown fairly logically and well-structured, I think for a discussion of this sort, into the diagnosis of a number of related ills.

Rick
RICK GARLIKOV (DEMS042@UABDPO.DPO.UAB.EDU)

I have since seen many students give terrible answers to exam type questions, BUT IF YOU RESPOND TO THEIR ANSWERS, AND GIVE THEM A CHANCE TO EXPLAIN AND DISCUSS WHAT THEY REALLY THINK, THEY GIVE QUITE DIFFERENT ANSWERS FROM THEIR INITIAL APPARENT ANSWER. They give much better answers as they respond. Exams tend to make that impossible, particularly final exams, graded after students have gone home and the course is over. I never came up with an exam I thought really showed me what students knew or understood. And I have tried all different kinds of questions. So what I do is, take pictures of them the first day of class so I can learn their names by the second day of class, in order to keep straight who says what kinds of things essentially from the beginning. Then I conduct class Socratically so that every day I get to find out what most or all of them are thinking, particularly by asking them to respond to the responses other students give: e. g., How many agree with Greg? Why? etc. You can get a lot of "mileage" out of one question, even after a bunch of people have answered it. The class goes on like this pretty much all term, though usually they start to respond more and more to each other without my having to ask. After weeks and weeks of this, you get a pretty good idea of what each student can do. Any you are not certain about, you can ask some questions and engage in a particular dialogue.

With at least two weeks to go in the term (or sooner if anyone asks me) I tell them what grade I am going to give them and why, and they can argue with me if they wish. Almost none argue. Two have argued I was grading them too high. (Since I grade partially on how well they can argue evidence, they were in a catch-22 since if they could argue well enough to convince me they deserved a lower grade than I thought, they probably didn't....) I have had three students whom I could not convince their grade was fair; that is when I give a written final exam, to give them a chance to improve their grade. None of the three did improve their grade; and none of them even asked to see why not. Something about giving me answers on a test made them more accepting of my (SAME) ideas about their answers than did discussing their oral answers with me. I don't understand that, but.... I assign no papers and give no quizzes. I know what intro philosophy papers look like. They are almost all no good. They are excruciating for them to write, and excruciating for teachers to read. If a student wants to write something, usually about a class topic he/she feels still in disagreement about, I will read it and respond, but without giving it a grade. Anything they say, in or out of class, and anything they write goes into my evaluation of them, but in a kind of fuzzy way, since I don't exactly know what constitutes a low A in philosophy as opposed to a high B. My first philosophy teacher used to grade essays with number grades. The kid behind me one time got an 88 that had been scratched out and replaced with an 87. I have always wondered how the teacher could be so precise on that sort of thing. There were no marks on the paper otherwise. (I don't know whether it was technical merit or artistic style that caused the one point change of mind....)

I am working through a mini-ethics course via e-mail with someone on this list who is quite intelligent and reflective and who gives very interesting answers that in many cases I disagree with, but see some merit and point in. If I had to give a grade to those answers I could not. Why should I treat students any differently. I simply respond to the answers and we go on from there. Any assessments I make are for purposes of trying to decide how to better teach them, not simply give them a label. Kevin said he thought exams could be quite instructive for students. I used to think that, but never could design one I thought really taught anyone anything just by his/her taking it. Maybe thinking about it afterward, but not according to the answers written on it. Also, as a student, I have, on two occasions, had insights the night before an exam that let me ace the exam. Otherwise I would have done poorly. Suppose those insights came the night after the exam; is the difference between an A student and a C students two nights worth of knowing something! I doubt it. Finally, I think dialogue is far more important than exams, and that as Gene or someone said, every minute giving tests is a minute you could have been "teaching" instead. And testing gives a kind of false sense of authority about answers, which I think is unfair to students. My view is I should be able to convince students of my ideas by reason, or they should be able to remain unconvinced without penalty -- assuming they are really trying to learn (showing up, listening, addressing the reasons given, asking questions, giving rational objections, etc.).

Sometimes interesting things come out of that. When talking about sexual ethics in class one time, I had argued for a number of reasons that discussion about certain aspects of sex is extremely important so that you might decrease the chances of doing something you would later regret. One handsome guy who was quite the ladies' man, if his own accounts were accurate, challenged with "What could a woman possibly say who was about to have sex with me that would make me not want to have sex with her?" I asked "How about, 'I am really glad we are doing this because since I tested AIDS positive, other men have been afraid to be with me.' Or, 'I have been wanting to get pregnant for a long time but my husband refuses to have children as long as he is still playing linebacker for the Packers; he wants to wait until he no longer has to travel; but I don't want to wait.'" His mouth dropped open, and he said: "Oh, okay; those would work." Another time (before AIDS) a male student swore up and down he couldn't talk about the meaning of kissing before kissing a girl because "that would kill the mood; and I wouldn't get kissed." After much discussion, he was still adamant. The following Monday he came to class and couldn't wait to discuss this further: "Tell me again about what you said about asking someone why they wanted to kiss, before you kissed them." I said "Why? Did you ask some girl?" He said, "No, I was out on a date with this girl Saturday night, and just as we were starting to get romantic, SHE ASKED ME." He had this kind of stricken look on his face, and I couldn't help asking him to tell us all what he had said. He looked around the room a minute and said "Well, I was so surprised she asked me that, the only thing I could do was to ask her if she had taken your course." The class cracked up. I asked whether she had taken my course, and he said "She said she hadn't." He seemed relieved until I asked: "What's her name, maybe I DATED her."

If you only lecture, and you only give tests, I think, besides its being ineffective, you rob your students and yourself of some interesting human verbal exchanges that can be more meaningful to the students and that can be fun for all in a way that does encourage thinking. I know that when students challenge me, I learn more by having to figure out answers for them. One student and I argued to an impasse one day and it was not until a year later that I thought of something I thought would clinch my view for him. So I tracked him down and called him to tell him. I think it is important for students to see you are more interested in ideas than in simply testing and grading them. The students I had this past term all were black and we were discussing racism and "being black" in America. We were into a pretty intense discussion with me challenging some of their conclusions, when one of them said (and the others agreed) that "one of the problems with being black is that you are always outnumbered by white people if you are around any white people at all. You NEVER see a white person outnumbered by black people, so that the white person is the 'odd' person." I looked at them for a minute and then looked at the backs of my hands, held out my hands and said "What do you call this class?" Their mouths dropped, and one of them said "But you're not white...you're Rick". I don't think they would have felt that way if I had only lectured and tested. Then I would have been very white. And I don't think they would have learned as much about ethics.
Rick
RICK GARLIKOV (DEMS042@UABDPO.DPO.UAB.EDU)

I would love to engage in a lengthy dialogue with Rick Garlikov on any topic and especially on the merits/problems of testing. However, for the moment, I'll settle for this: Rick--you say you don't do any testing; I say you are testing constantly. At the very least, you are measuring constantly. I share your concerns (and have others) about grades, but apparently you and I are both constrained to give them.

bill hunter fax: (403) 282-0083 the university of calgary phone: (403) 220-5691 hunter@acs.ucalgary.ca

Bill Hunter:
****Rick -- you say you
****don't do any testing; I say you are testing constantly. At the
****very least, you are measuring constantly.

TRUE; hence, the "sort of" in the topic title "teaching without testing, sort of". It just is not testing in the usual sense for the usual reasons and in the usual way. Good observation.

Rick
Rick Garlikov (DEMS042@UABDPO.DPO.UAB.EDU)

In a recent series of exchanges stemming from Cindy Cotter's request that Susan Nolen and I share our views about assessment, the following discussion ensued. I believe my comment was that the content of a course on testing which might cover concepts of validity and reliability might not be so relevant to a classroom teacher. To which Cindy Cotter asked"...how can you dismiss validity and reliability of testing as not being the concern of the teacher and the student?

Validity simply means testing what you meant to test." I did not mean to imply that a classroom teacher would have no use for these concepts at all, but only those concepts as applied in large scale testing as we have known it up to this point. In addition, the terms might mean one thing to a classroom teacher and something to a large scale test maker. Gene Glass wrote "validity is not what the psychometricians think it is" I probably agree. I at least would like to expand the definition to: How well a series of assessments (more broadly than a test or set of tests) measures what students are expected to have learned. Cindy Cotter "...And reliability just means that the scores will not be thrown off by irrelevant factors like whether the teacher is in a good mood when he scores the test or whether it's scored by the teacher or the aide." In Vermont's large scale program of writing and math portfolios, "reliability" has been difficult to define and achieve. It seems that when the issue of scoring assessment changes from "answers" being "right or wrong" to "exhibiting" various degrees of agreed up on qualities, teachers cannot reliably put a work into the same bin as other teachers...even when there are only a few criteria.

The statewide experiment and adjustment is continuing. In the first year, changing the scoring process during "moderation" was supposed to improve things for this year. Next year, crisper definitions of the bins may help improve the process further. In the state's rush to get large scale reliability, some teachers have lost the sense that the project is about classroom improvement, because the "testing" issues seem to drive some of the state's processes of training. Others just see those folks as whiners who don't understand the importance of proving that a portfolio assessment system can have the same testing statistics as the methods it is seeking to replace. I'm not sure that we WANT it to. Assessment as it is practiced in the ArtsPROPEL project in Pittsburgh apparently has no raters other than the teacher and student, and differences in ratings are considered opportunities for reflection and teaching. I suppose one could say that at a statewide level, the same thing is happening, the absence of sufficient reliability is an opportunity for the entire profession to think about what it is doing. Gene Glass "reliability is irrelevant (nobody really knows what the heck it is anyway)." I'll pass along your comment to the Commissioner. If this could only be entirely true at the statewide level, I know a few people who could take a vacation. It may indeed be true in the classroom. I think the entire role of these terms changes when we think of the scoring of assessments as highly specific indications for next steps in instruction, including self-instruction.

That is how I think assessment looks when it is fully embedded in teaching-as-coaching. Reliability might then be thought of as a measure of how useful a feedback/scoring was to continued instruction. In large scale assessment, where the criteria is clumped into a very few bins, reliability would be expected to be low, because the feedback to the learner is relatively uninformative of "next steps." In other words, by the time a state system forms a highly reliable performance-based assessment, even a relatively uninformed person might be able to score pieces of work as well as an expert. We will have achieved reliability, but what will be the qualities of that information?

Cindy Cotter: "Well, I suppose [fitting grades to a curve] might be helpful sometimes to discover that you don't understand something when you thought you did. I remember the horror of being completely lost on my first computer programming assignment when I'd thought I'd understood all the lectures. It could also be helpful to know, if you're considering career choices or college choices, how you stand in relation to your competitors."

The interesting issue in your second comment, Cindy, is that you may very well have understood all the lectures, and even scored well on valid and reliable tests, and STILL not had a relevant, authentic, learning experience that prepared you for programming. I think this is similar to my critical reflection of the course on testing. Future teachers may take the courses, and even pay attention, like them and do well on their tests...and still be completely lost in their first real world teaching/assessing assignment. On your second point, I am personally not interested in testing or assessing for social/competitive comparison purposes. I suppose there may even be a link to motivation (albeit a selfish, "get ahead" goal) for school achievement...but I think of learning as something apart from school achievement in that sense, and I would rather have assessment serve the purposes I have in mind, simply getting better at knowing and doing things.

Gene Glass: "ALL grading is curve grading; it could be no other way Corollary: "mastery" is a chimera; it doesn't exist."

I'd be interested in Gene's reasons why absolute grading schemes also curve things. I fully agree that summative scores don't say much and do collapse most truly useful information. Perhaps that is why I think statewide performance assessment reliability mentioned above may also be gained with a loss of information. The "mastery" ideas seem to be like the benchmarks of performance rating systems. They represent steps toward expertise, not "mastery" in any global sense. And if learners are like the future teachers we've discussing elsewhere on EDPOLYAN, we should expect that they are not "done" when they leave the K-12 system either. So, in that sense "mastery" does not exist. Perhaps we could talk about a system of standards however, that contains a vision of mastery which is never fully reached. The image itself serves as the stars did to a sea going traveler of past days. In Vermont discussions, we've been going around on whether the word "standards" means something that can be reached or not. Perhaps they are like Gene's "mastery."

Bill Hunter said, "I not only share Cindy's assumption that teachers are likely to continue to use tests and should therefore know some things about them, I also think the potential abuses of tests are so enormous that it is essential that teachers be the first line of (critical) defense against testing abuses....To some extent, one might well argue that the continued use of grade equivalent scores is prima facie evidence of the failure of courses in educational testing, but I don't think that battle will be won by taking teachers off the battlefield." I absolutely agree, and meant to originally recommend a rethinking of HOW teacher s learn about testing, not WHETHER they do learn something or not. Aimee Howley then posted that ""Authentic" assessment is considered better than multiple choice testing and grading on the curve because it gives more and better information about individuals. The question of what we do with that information and who benefits from its accumulation are seldom asked." I think Aimee has a key idea of the discussion so far. No single system of assessment seems likely to serve all of the purposes of education. And no individual system of assessment will serve education at all if we don't know what we would LIKE it to do, and what it is SUPPOSED to do. I would just like to vote for putting most of our energy into systems that improve learning and learners first...and make teaching and the other parts of the education system accountable on that basis.

David Gibson dgibson@ssi.edc.org

David Gibson generously read my comments on assessment as reflecting something like hope; I intended them (much less generously) as conveying something more like despair. Assessment is a technology whose purposes determine the form it takes. If the purpose of education really were to help children learn a lot of things that we and they thought were important, assessment would be indistinguishable from the dialogue and demonstration that constitutes teaching. The extent to which it is construed apart from and above the occasions of learning and teaching reflects the extent to which it is a technology directed at a much different--and much less worthy-- purpose. My despair comes from hearing in much of the public discussion about assessment themes and variations of this less worthy-- indeed fully suspect--purpose. --Aimee Howley

Rick G. responded to my comment:
****  Rick -- you say you
****  don't do any testing; I say you are testing constantly. At the
****  very least, you are measuring constantly. by answering:
**** "TRUE; hence, the 'sort of' in the topic title 'teaching without
**** testing, sort of'. It just is not testing in the usual sense for
**** the usual reasons and in the usual way. Good observation."

Good. Let's see if we can advance this a bit. In the testing embodied in your ongoing give-and-take with students, what are you measuring against? You talked about the students' levels of understanding, but can you say what constitutes evidence of understanding (as opposed to, say, parroting). Since you do indeed give marks that, I presume, differ from one student to another, do you have reason for having confidence that the better marks given to one student over another indeed represent "better" levels of understanding or would it be "more" improvement from some initial state? If the initial states vary, as I am sure they must, do you have some rationale that allows you to feel comfortable with the fact that students who start high and gain little may end up with higher marks (under the "better level" model) or with the fact that students who start low, gain large, yet finish with lesser understandings than some others may indeed receive better marks (under the "more improvement" model)? I expect you have good answers to these questions.

If not, you could probably not ethically continue to teach as you do (let me reiterate that it sounds as if you are a dynamic teacher who really is a catalyst for growth in reasoning for your students). My point is not to get you to defend your manner of teaching and testing, but rather to call attention to the kinds of questions that more formalized approaches to testing attempt to raise. I do not expect that a testing course will cause teachers to become test construction geniuses (I doubt that they exist); I expect only that it will provide them with tools for thinking critically about how they assess student progress. Concepts like reliability, validity, utility, etc., etc., provide some common ground for discussing assessment issues.

If Gene G. is arguing that the rigid application of measures of reliability and validity has little merit in the classroom, I am in full agreement; however, I believe strongly that there is merit in having prospective teachers think about how to make judgements about the kinds of assessments that are appropriate for their aims and of what qualities are desirable in classroom measurements. My earlier reference to item analysis is a case in point--you engage in assessing the quality of your questions as you hear them answered, you may rephrase them or abandon them altogether in favor of new questions. In a context in which a teacher uses a paper and pencil measure, item analysis may help them to make these same judgements (based on responses from many students rather than one--a characteristic that has both virtues and faults).

bill hunter fax: (403) 282-0083 the university of calgary phone: (403) 220-5691 hunter@acs.ucalgary.ca

As a newcomer to the list (and the Net, for that matter) I am delighted to find this discussion in progress and hope that the following comments do not re-plow old ground. Unfortunately I missed the initial remarks.

If a recommendation was made that teachers receive formal training in testing, I agree and would extend the recommendation to include all college graduates (and all state and federal legislators, if that were possible). Unfortunately our expectations of tests tend to derive from our personal experiences as test subjects and from our attitudes toward the claims of test publishers and wishful thinkers who promote tests as quick fixes for all sorts of threatened or perceived lapses in standards.

We would do better to draw our concept of testing from the realm of sports where a public demand for fairness has led to more reasonable con*tests*: Individuals or teams typically have repeated opportunities to show their stuff under uniform (in testing we say standardized) rules and criteria for winning (or passing to the next level). When judgment is involved, the really serious contests use a panel of judges to counterbalance inevitable personal bias. Furthermore, the athletes know the expectations and criteria for scoring in advance and have plenty of opportunities for practice. Athletic contests, then, have something in common with "authentic assessments," where subjects are evaluated upon their demonstration of important knowledge or performance at meaningful tasks. The limitations of traditional standardized multiple choice tests are intuitively clear if we think about using a multiple choice test ABOUT football to determine who will go the superbowl or who will be selected for a team. The limitations of free form assessment are also clear if we think about attempting to judge an athletic contest without clear rules about playing, scoring, or winning. Efforts to avoid the pitfalls of multiple choice tests will simply result in other, perhaps more insidious, injustices if alternative assessments are not carefully structured. Standardized paper and pencil tests at least have the value of recorded questions and objective rules for scoring. The move to authentic assessment should continue to allow scrutiny and reflection upon what is expected and how performance is evaluated. Your comments would be most welcome.

Joan Gipson-Fredin joangf@fallbrook.csusm.edu

Bill Hunter:
****Good. Let's see if we can advance this a bit. In the testing
****embodied in your ongoing give-and-take with students, what are
****you measuring against?

First of all, my questions are not so much meant to elicit answers for purposes of evaluating students in terms of a grade; but I want to know what they believe about particular topics, and why they believe it -- and whether they are articulating what they really believe (in the sense of 'is what they are saying what they mean to say, or what they think they are saying?'). So I might ask "When is it right to break a date or an appointment? And why?" You get a whole bunch of different answers to that at the beginning of the term. One girl one time said she would not date someone who did not have a car. So I said that meant she would not go out with Ralph Nader who might fly her to Paris for dinner. She said "Well, I meant I would not date someone who had no transportation." She had not said what she meant because she wasn't thinking about such a possibility, so car and transportation were assumed synonymous until the example came up where they weren't. At this point I am evaluating her answers but I am not grading them or her. I am evaluating them in terms of what I need to respond. I may need to ask her further questions.

When I do something like the date question, I take every answer called out and write it on the board. I may ask a question to get them to clarify what they mean -- so in the above case, I would cross out car and write 'transportation'. Then when all the answers anyone can think of are written down, I let them argue with any they think wrong. Then if there are any left I believe wrong, or any I believe left out, I will ask questions or give examples of situations to see whether they can deduce what I am driving at. If all else fails, I will spell out what I am driving at. If I have to do that, they usually are 'primed' for my answer by then. What I am looking for in any answer is clarity and reasonableness and/or truth insofar as I can recognize it. I am also looking for insight or perspective that might be helpful. Students will give better and worse comments in regard to these things; and some students may give a brilliant answer one minute, and a really faulty answer the next. I try to remember who gave good answers and what they were.

What I am "measuring against" for grading is how clear, precise, and reasonable their comments are, and --as the term goes on-- their not saying things we have already shown inadequate, unless they have additional support we have not considered. But grading is not my normal consideration in any of this. I am trying to get them to derive for themselves much of the history of ethics by asking them questions and then pursuing what they say -- and having them pursue what each other says, and what I say. And I particularly am trying to get them to be able to be much more precise, specific, and reflective as we go.

**** You talked about the students' levels of
****understanding, but can you say what constitutes evidence of
****understanding (as opposed to, say, parroting).

As in real life, this is never a sure thing, but the point is to ask questions that require answers that are DERIVED from material that is taught, rather than in asking questions whose specific answers have been given already. This rules out parroting though it does not rule out a lucky guess or an answer that comes from something else than what I have "taught" or drawn out of them. For example, in teaching young children how to do representational "place-value" "regrouping" subtraction using poker chips, a question that tests for understanding is to ask them to take one red chip (which represents '100') and subtract '37' from it -- when you have not worked with hundreds before and when you have not done anything with 2 zeroes in the number from which you are subtracting. If they can extrapolate from the 10's and 1's work we have done, I take that as a sign of understanding UNTIL AND UNLESS some other evidence arises to show me otherwise. In philosophy classes I try to find questions or comments that evoke responses which require deductions from previous material. And I particularly relish trying to trick them into making faulty deductions, to see if they can avoid the tricks.

For example, I explain to them during the course why the normal understanding of the "Golden Rule" is a faulty ethical principle. (In short, other people don't always want or need what you want; and sometimes you and they may both want things which are bad or wrong anyway.) To see how well they understand this after I have given a series of memorable examples, I don't just ask them whether the Golden Rule is a good rule or not, because they will say it is not and give me back the examples. Instead I say something like "I know the Golden Rule does not work in some cases; but in what cases DOES it work as a good principle for deciding what is the right way to treat someone else?" Well, the answer is NEVER; but they don't usually see that because they are trying to think of cases where other people want what you want and where what you each want is a good (right) thing. But this latter part begs the question, so the Golden Rule always fails to be the guide, by itself, to what is right. If they can explain this, and why, then they have not parroted but have had to understand.

Or I can give them cases we have not discussed and see how they can use what we have discussed to elucidate these cases. I doubt I ever directly ask anything we have discussed in the way we discussed it. So there is nothing for them to parrot or merely repeat. Not answering satisfactorily does not show lack of understanding, and answering successfully does not guarantee understanding. But answering successfully gives evidence that 'confirms' understanding in the scientific sense of 'confirms'. (I can explain that if I have to, but I assume most of you have some familiarity with the notion of scientific confirmation and the difference between it and 'proof'. If not, just tell me.)

****Since you do
****indeed give marks that, I presume, differ from one student to
****another, do you have reason for having confidence that the better
****marks given to one student over another indeed represent "better"
****levels of understanding or would it be "more" improvement from
****some initial state? If the initial states vary, as I am sure they must,

Actually they vary very little, if at all. I have taught senior citizens and fifth graders, blacks, whites, wealthy, and poor, and sometimes people with very good educations outside of philosophy; and almost all start out making the same kinds of errors on the same issues, though the specific answers tend to vary a bit. The problem I have with grades is not distinguishing between students' abilities, but trying to figure out how to correlate different abilities with some particular single letter grade. I don't know how to do that, not with only three real grades to choose from (nobody who shows up and who gives this any effort at all is likely to fail; and I have no earthly idea what a "D" can possibly mean in a course. I once saw a student paper which a teaching assistant had marked with a "D+". I figured the "+" was sarcastically gratuitous; what the devil could a D+ mean!). I give A's to students who demonstrate generally the ability to make conceptual and logical distinctions and to be clear, precise, and logical about the subject, and who show a good grasp of the material. I give C's to those who have some understanding but who have not learned as much as I think they should have -- for reasons I give them -- in all these areas. And I give B's to those in between somewhere. My B range tends to be widest and I sometimes have students in it I wish I could separate somehow grade-wise, but not at the expense of giving some of them C's -- which I consider a fairly low grade in a course like this, where no one is a philosophy major and many are only taking it either out of curiosity or as a way to meet some sort of humanities requirement. I am not going to punish people or mess up their college or professional careers because they may not yet have become as facile at making distinctions, etc. as others. I don't want someone kept out of med school because he doesn't quite see why the Golden Rule doesn't ever work, though he sees why it sometimes doesn't work and knows what at least those times are.

**** do you have some rationale that allows you to feel
****comfortable with the fact that students who start high and gain
****little may end up with higher marks (under the "better level" model)
****or with the fact that students who start low, gain large, yet
****finish with lesser understandings than some others may indeed
****receive better marks (under the "more improvement" model)?

No. I have no rationale that really lets me feel comfortable with grades at all, in part because grades do not by themselves reflect specific knowledge, ability, or amount of improvement, or percentage of suspected potential actualized. I had a really bright student this past term who had an "attitude" which softened somewhat by the end of the term, but who missed some classes (for no good reason) that might have helped her learn some things that were important. In terms of logic and insight, she was better than almost everyone else in the class; and she had a better grasp of most of the material than most of the students. I wanted to give her an A for the things she did well and a B for the things SHE should have been able to do but couldn't do because of her absences. I finally just essentially threw up my hands and gave her the B. I told her why.

****I expect you have good answers to these questions. If not, you
****could probably not ethically continue to teach as you do

ONLY IF YOU ASSUME THAT GRADING --in the sense of giving A, B, C, D, F (or E) letter grades -- HAS ANYTHING EVEN REMOTELY TO DO WITH TEACHING! Or even remotely to do with assessing a student's learning. Until you can standardize teachers' expectations and assessment methods and the standards by which they are judging, you cannot tell me comparing a junior college student's B in a course with a Harvard student's B in a course has any significance for comparing either student with the other. I had a student last year who was really good in certain ways. He wanted to be a teacher. I thought he would make a great teacher. I sent him over to a local highly prestigious, expensive, private university for an interview. The ed prof that interviewed him called me back and said he was the most impressive student she had ever met. Yet he had a C average from this community college because he was not good at learning things that made no sense just so he could get a good grade. He despised many of the teachers he had had.

He got in to the university I sent him to, with a scholarship; and I have reports he is doing well, particularly given his academically disadvantaged background and the fact that he has a grueling job outside of school. A prof who I spoke with says he is a tremendous student and a tremendous asset to the class, though he is carrying a low B in her course --based on her exams-- for reasons she perfectly well understands. I don't know if she will make a non-exam kind of adjustment in his grade or not. I didn't even ask her. What do you think his grade will mean -- whatever it is-- if it is based primarily on exams and other formal means of grading?!

When I was in school at Michigan, I invariably opted for every honors course I could take. You had at least a B in most honors courses just for doing normal work. They were often better taught. It was easier to get an A or B in an honors course or section of a course than in a non-honors one.

When I taught as a grad student at Michigan, one girl brought in her boyfriend to visit the class. He was from Princeton and he had taken intro philosophy there; and he had "know-it-all, holier-than-my-students" written all over him. I invited him in and said he could participate. He started with some smug comments that my students pointed out to him why he could not even begin to justify. He ended up quite self-chastened. What did his Princeton A mean in his philosophy class? And one last example in this regard, I had a cousin at the U. of Oklahoma who transferred to Illinois after her freshmen year because she had a 4.0 at Oklahoma and did not feel like she was learning anything. She was majoring in chemistry at Illinois. She knew I loved philosophy so she took an intro course in it. She got a young guy who had a year ago received his PhD from somewhere based on his dissertation on St. Anselm, an eleventh or twelfth century (I for- get which) monk who has one famous strange argument that periodically shows up in intro philo courses for one class period or so. This guy taught the whole intro philo course using only Anselm's works. My cousin (and God knows who else) got a C, and an enduring dislike for "philosophy". What did her C mean? Maybe his tests were quite valid, reliable, etc. for testing what HE wanted to test.

****My point is not to get you to defend your manner of teaching and
****testing, but rather to call attention to the kinds of questions
****that more formalized approaches to testing attempt to raise.

And in certain kinds of cases, particularly those involving reason and understanding and conceptualization, etc., I would bet almost always fail to even possibly BE ABLE to answer. In such cases, nothing can take the place of a give-and-take dialogue where the student has every opportunity to express him/herself more fully and completely than in a "simple" test situation, which might even be oral, but is not give-and-take. When I took my oral comprehensive exam, I took it in philo of religion. I had been pre-med and had taken lots of science, so when I was asked whether anything I had studied outside of philosophy had anything to do with anything in philosophy of religion, I went into an explanation of how the biochemistry of the hemoglobin molecule showed that the theory of evolution had some gaps in it. This was an original insight of my own while I was taking biochemistry. I passed the comprehensive but was given a low grade because I had not talked about Freud or Marx instead of Darwin. The senior of the two profs did not know anything or care anything about biochemistry, so my answer did not meet with his approval -- but he said nothing during the oral, other than with his blank eyes, about my answer. He merely graded it. It was not the kind of oral testing I had done with the student I described in my previous post who thought she understood nothing in my course, but who upon two hours of intense discussion showed me she had understood everything perfectly.

****I do not expect that a testing course will cause teachers to become
****test construction geniuses (I doubt that they exist); I expect
****only that it will provide them with tools for thinking critically
****about how they assess student progress. Concepts like
****reliability, validity, utility, etc., etc., provide some common
****ground for discussing assessment issues.
****
****If Gene G. is arguing that the rigid application of measures of
****reliability and validity has little merit in the classroom, I am
****in full agreement; however, I believe strongly that there is
****merit in having prospective teachers think about how to make
****judgements about the kinds of assessments that are appropriate for
****their aims and of what qualities are desirable in classroom
****measurements.

I might agree with this, IF they can really learn this by studying FORMAL methods about this sort of thing. I have strong suspicions, however, against formal methods being able to teach this properly. And, I am also worried about what appropriate aims are to be tested -- e.g., the Anselm teacher probably did not have appropriate aims to begin with.

**** My earlier reference to item analysis is a case in
****point--you engage in assessing the quality of your questions as
****you hear them answered, you may rephrase them or abandon them
****altogether in favor of new questions. In a context in which a
****teacher uses a paper and pencil measure, item analysis may help
****them to make these same judgements (based on responses from many
****students rather than one--a characteristic that has both virtues
****and faults).

I don't think item analysis by itself can be anywhere near as powerful as "follow-up" questioning, for particular students. I am not content to "flush" students who give unique apparently bad answers to questions everyone else does ok on, just because that student has an insight they don't express very well in their "initial" written answer -- or does not express it in a way I am wise enough to understand it (or know I don't understand it) on his written answer.

Rick
RICK GARLIKOV (DEMS042@UABDPO.DPO.UAB.EDU)

Joan, I am glad you are enjoying this exchange. You did not rehash anything already said, but unfortunately I think if we follow your sports analogy with testing, sports will be as problematic as testing students is. Superior athletes and teams often lose, and yet still arguably are the superior ones. Sometimes technicalities occur in sports that skew performances in ways pundits argue about for decades. The playing fields change; the games change in strategy and abilities of opposing players. It is impossible to compare players of different eras because of all kinds of changes from the above to equipment, etc. occur. What if Bobby Jones had had graphite shafts? What if Babe Ruth had to play at night? and travel to the West Coast 4 to 8 times a season? What if a player wins a zillion minor golf tournaments and no majors? Is he great? Was Colorado really national co-champ in football a few years back even though it took them five downs to beat Missouri when the refs screwed up? Didn't the NCAA shaft North Carolina by putting in the shot clock and taking away their 4-corners game? Didn't the NFL shaft teams with good kickers when they moved goal posts back 10 yards and changed the possession point after missed field goals? Does the ATP computer really tell us who the number one tennis player is each week? Is the NBA's draft procedure really fair? Fairer than any other sports? Why can't the NFL or the NCAA figure out the fair and obvious, reasonable way to use the video replay during football games -- changing those egregious missed calls that all America except for the refs on the field saw clearly and accurately; those and only those calls? Does a poor figure skating performance on a given night mean you don't deserve to go to the Olympics or to have the gold medal there? Does the running back with the best blockers or the quarterback with the best receivers deserve the Heisman trophy? Etc., etc., etc.

Even though it is big business, sports is just winning and losing games played in as fair a way as we know how to play and officiate them. Nobody involved thinks it is perfect or that there are not problems. Sports is an artificial, contrived contest. Judging students is not a game, should not be a sport, and does not need to have artificial, contrived single-letter results, such as A, B, C, D, or F, that represents 10 to 15 weeks of work and effort, and learning. As long as classroom testing boils down to giving a single grade, it is a bad enterprise that cannot be done well, at least not in certain kinds of courses. Bad assessments should be more tolerable in sports than in academics; bad breaks may sometimes even out in sports; and if they don't, that is just too bad, and often cannot be helped. We can do better in regard to genuinely assessing students.

Rick
RICK GARLIKOV (DEMS042@UABDPO.DPO.UAB.EDU)

In further response to Bill Hunter's questions about assessing students in the way I do: apart from issues he raises of validity, reliability, etc. of testing (i.e., giving specific exams), there is often a "hidden" problem with tests in that they encourage anti-intellectualism under certain conditions. I do not know exactly what those conditions have to be, but I can describe the phenomenon. The phenomenon is simply that getting a good grade on the test becomes the chief priority of (a) many, (b) most, (c) all students; and learning is only a means to that end. And as someone posted somewhere a few months back, this makes learning more than you have to for the exam, or working harder than you need to for doing satisfactorily on the exam, a wasted effort, somewhat tantamount to paying more than you need to for a purchase; and learning something after the exam is really terrible because then it 'doesn't count'.

One classic, though fictitious case, is in the movie "Dead Poets' Society", the Carpe Diem speech scene, where after Robin Williams has gathered the boys around the photos and trophies portraying students, long dead, in the glory of their vitality and youth, and articulately and passionately explaining to his boys how they too, just like these lads, will finish their existence as worm food and so they need to live life to the fullest, etc., the first question one kid asks is something like "do we need to know this for the exam?"

I sat in on a friend of mine's college English Lit course for two consecutive class periods one time. The teacher was, I'd guess, a 28 or 30 year old woman, who would ask typical questions about the literature that had been assigned, and would get no response, or minimal response, from the class. After sufficient embarrassing silent, 'dead time', she would give her own answer to the question herself. Then she would ask another question, and the process would be repeated. The few times any students answered, (usually timidly, self-consciously, haltingly, and in obvious need of encouragement or cultivation) no one else responded, seemed in any way to be interested, or wrote anything down. She herself did not tend to challenge answers or in any meaningful way cultivate them further. She tended to either agree or disagree and then move on, even though the answers were usually quite weak and simplistic 'starter' answers. When she answered her own questions, they all wrote down what she said. This went on repeatedly each class period until, near the end of the second class I happened to attend, she exasperatedly asked them all why they never spoke up in class. They wouldn't answer that, of course. I asked whether she might want my observation; and she, being desperate, agreed to listen. It was my view that as long as she answered her own questions about the material, and as long as she gave tests covering that material and her views on it (or as long as students believed or knew she would test that way) they were not about to "waste" their own time or each others' time by giving their views. Their views were of no consequence for the exam; her views were; and if they talked too much, it might keep her from giving her views, and/or waste a lot of their own effort doing unnecessary work. I told her she needed to not give exams or to re-design them (she admitted that she did test on what she said in class) or to not let them know what she thought, in the way she was doing it. Her exams and class style, along with these students' mindsets about college, were counterproductive to how she wanted the class to really work.

I don't know what gives students that mindset to begin with, since many students will enjoy discussion and learning. And my explanation does not explain why many people will be passive in classes, like adult Sunday school classes, where there are no exams anyway, except that they don't want to "waste" people's time with their views instead of hearing the teacher's (i.e., the supposed expert's supposed correct) views. But the point is that whatever promotes or causes such lack of student effort and lack of intellectually active participation in the first place, lecturing and testing, even in the way she did it, by beginning with questions, do not serve in any way to counteract the phenomenon. You have to do something to (1) more aggressively provoke excited response (usually by asking more interesting and challenging questions, I suspect, or by setting up the questions to make them have a dilemma or psychologically problematic "bite" to them), (2) get their minds off the tests somehow, either by not giving tests or by not letting them know what will be tested or by demonstrating to them that it is their active intellectual efforts that will stand them in much better stead on the exams than will anything they memorize or merely repeat.

Rick
Rick Garlikov (DEMS042@UABDPO.DPO.UAB.EDU)

Right you are Aimee. Last year sometime the more, or less regular newsletter from ETS featured an article on portfolio assessment. In the article the woman in charge of the ETS project stated that one major problem with portfolio assessment was that it did not allow cross school comparisons important for policymakers. I did not save the newsletter so I can not attribute accurately, but that was the sense of the expression. This makes David Gibson's perceptive comment about losing information by establishing reliability for portfolio assessment also poignant. Its a shame that the concerns of ETS seem to lie more with making school-wide comparisons than with helping teachers make better judgments of student learning.

Eugene Bartoo said:
****On Sat, 11 Dec 1993 11:25:49 EST Aimee Howley said:
**** ****David Gibson generously read my comments on assessment as reflecting
**** ****something like hope; I intended them (much less generously) as
**** ****conveying something more like despair. Assessment is a technology
**** ****whose purposes determine the form it takes. If the purpose of education
**** ****really were to help children learn a lot of things that we and
**** ****they thought were important, assessment would be indistinguishable
**** ****from the dialogue and demonstration that constitutes teaching.
**** ****The extent to which it is construed apart from and above the
**** ****occasions of learning and teaching reflects the extent to which it
**** ****is a technology directed at a much different--and much less worthy--
**** ****purpose. My despair comes from hearing in much of the public discussion
**** ****about assessment themes and variations of this less worthy--
**** ****indeed fully suspect--purpose.
**** Right you are Aimee. Last year sometime the more, or less regular newsletter
****from ETS featured an article on portfolio assessment. In the article the
****woman in charge of the ETS project stated that one major problem with portfolio
****assessment was that it did not allow cross school comparisons important for
****policymakers. I did not save the newsletter so I can not attribute accurately,
****but that was the sense of the expression. This makes David Gibson's perceptive
****comment about losing information by establishing reliability for portfolio
****assessment also poignant. Its a shame that the concerns of ETS seem to lie
****more with making school-wide comparisons than with helping teachers make better
****judgments of student learning.

This is a beautiful illustration of a problem I mentioned earlier -- one in which we are all caught, but seldom make explicit. It is that education necessarily is something that occurs at a very low level of aggregation, but that policy necessarily is something that appears, and can appear only at a very high level of aggregation, and further that assessment [as Rick Garlikov uses it] is something that is engaged IN teaching, whereas assessment [as it occurs at ETS] is something that necessarily is bent to the problems of policy.

THOMAS F. GREEN (TFGREEN@SUVM.BITNET) + + EMERITUS FROM SYRACUSE UNIVERSITY PHILOSOPHER IN RESIDENCE ON POMPEY HILL + + Box 100 Pompey, NY 13138 (315) 677-9935

In response to Rick's post on why students do not respond in class, I have not kept up with research in this area, but do remember some findings from several years back: 1) Students establish pretty early which of their classmates can be depended upon to carry the load of class discussion. After a bit, they relax and let things continue that way. 2) There's some fear of being "put down" by the instructor with his superior knowledge. 3) Students see themselves as consumers of knowledge, and the teacher as a dispenser. 4) Teachers see the classroom as a "focused interaction", where responding to one another is called for; students see it as any other public gathering where one may choose to be anonymous. Just found the citation for this, and it is old: Karp and Yoels, The College Classroom: Some Observations on the Meanings of Student Participation, in *Sociology and Social Research, Vol. 60-4, July 1976, pp. 421-437.

Jack Letarte Taylor University Upland, IN 46989 317-9985153 Internet: jcletarte@tayloru.edu

Here are Bob Linn's recommendations, followed by the citation. (He refers also to a survey in J. Ed. Msmt. (1986) by Arlen Gullickson which shows a mismatch between what is taught in ed msmt classes for teachers and what teachers think needs to be taught. Some of you may be interested in that as well.) Linn's recommendations for assessment classes for teachers & teachers-to-be: 1. Planning & constructing classroom tests. (Gullickson found this was the sole area of match between teachers' perceived needs and profs reports of instructional content, by the way.) 2. The use of non-test evaluation procedures. (Currently tends to be called "alternative assessment.") "Knowledge about assessment principles needs to be integrated with an understanding of the subject matter, pedagogy, principles of learning, and children." 3. Use of assessment results for instructional planning and formative evaluation. 4. Use of assessment results for summative evaluation. (Including fair grading practices, and how grading communicates what is important to learn within a discipline.) 5. Administration and scoring of tests. (Linn actually discusses non-test evaluation here as well.) 6. General assessment information regarding the selection and use of tests. Includes ethical issues of assessment. 7. Principles of measurement. Linn recommends reliability and validity, taught as guiding principles. Linn, R. L. (1990). Essentials of student assessment: From accountability to instructional aid. *Teachers College Record*, 91(3), 422-436. The text we use in our preservice assessment class is the only one I've seen that is built around the kinds of assessment teachers actually need to do. Those of you who teach such courses or are involved in teacher ed reform may want to check it out: Peter Airasian (1993) Classroom Assessment, 2nd Ed. McGraw Hill.

Susan B. Nolen 322 Miller Hall DQ-12 University of Washington sunolen@u.washington.edu Seattle, WA 98195

Rick Garlikov wrote, "I do not know exactly what those conditions have to be, but I can describe the phenomenon. The phenomenon is simply that getting a good grade on the test becomes the chief priority of (a) many, (b) most, (c) all students; and learning is only a means to that end."

Schooling is not currently a voluntary activity. It is made clear to virtually all children that they are expected to attend school for the prescribed number of hours, days, and weeks each year. Students often fail to see the benefit of learning the subject matter in many of their courses. Even at the college level, when a course is required of the students' program they will take it, because they have to, even if they do not see what it will gain them. Though the value of the material taught in a given class may escape a student, the value of the grade rarely does. The importance of grades is readily apparent to anyone wishing to move higher up the academic ladder. We thus have a situation in which people are forced to do something they do not necessarily value, in order to gain something they do.

It seems clear that within the bounds of their own ethics they will try to subvert the system as necessary to get what they want. This might just mean studying only for tests, but it often means copying off other students or stealing the test in advance. This is not to say that if schools were entirely voluntary they would immediately empty out. It seems that people, particularly children, have a habit of balking at things they are forced to do even if they don't really object to the things themselves. There are a number of approaches to easing this problem: children could be given more flexibility in what they learn but, being children, they might not think far enough ahead to what they will want or need in the years ahead; children could be offered more freedom as to how and when they learn, though this would be difficult to orchestrate within our current system; a greater effort could be made to explain the value of the various fields of study (answering the "what do I need this for", "why do I have to study this" questions), so that they would not seem so distant from the children's lives; evaluations could be done, as has been suggested by Rick and others, in such a way that it is not so easy to cheat or to do well by simply memorizing a tiny portion of the material, though this will make the job of college admissions more complex.

I would love to hear other suggestions, and I apologize if these repeat some of the last month's discussions as I've been too busy to follow them. In studying some of the great figures of the Renaissance I've found, unsurprisingly, that they drove their own course of study from a very young age. Rubens, for instance, was forced to leave school and go into the service of a wealthy lady as her page, because his mother was unable to support him. He soon left this service, however, at the age of thirteen, after convincing his mother that he would quickly be able to earn his keep as an artist. This he did, and despite his lack of formal education he went on to negotiate several peace treaties between Western European nations (in addition to painting a lot of fleshy bodies). This is not to say that all children would benefit from self-directed study, but simply that there are many precedents, Rubens among them, for a freer approach to education than the one that has grown up in the recent history of the United States.

Andrew Coulson

I much appreciate your emphasis on teaching rather than testing. When I was 'trained' as an elementary school teacher in New Zealand we were often told, "testing is not teaching." I think much of my life has been an attempt to interpret this. I now commonly ask students in my educational psychology courses to evaluate their own work and grade it. I only felt able to do this when I became very clear that what they should be doing was developing their own ethical stances on what type of teachers they hoped to be, and not collecting truths delivered by researchers.

I can, I think, only do this when I establish that this is what they are doing or should be doing. I do this by pointing out that the experts disagree with one another and that much more than facts are involved in these disagreements. I also have, or feel I have to directly assault student's notions that they are here to collect noncontroversial facts or truths which I can grade fairly. While they think this way, they destroy my chances of having discussions with them that might teach me something about their world views and that will get them to elaborate and challenge their own views. So I give then material to read that describes the deadly life in classrooms where students seek such answers and, yet, see it as fair.

Sometimes the first assignment, on day two of class, is to discuss such an article and, if bothered by it suggest remedies that will make sure it doesn't happen in this class. This barely gets the process of re-construing class as a collaborative exploration of different possible ethics of teaching in which it is assumed that we will leave disagreeing with one another but doing one another a lot of good by listening carefully and challenging openly and respectfully. Along the way the criteria for grading have to be discussed. Trust is essential and I think partly established by showing that people can, indeed must disagree with the instructor and survive with colors flying. But more positively, I generally advance the notion that every significant paper will be unique and be of quality if it involves some significant dialogue between personal and school knowledge. These concepts are discussed as part of the course -- they come up in what I use as first reading -- so a course on education offers special advantages I think.

With grad classes, which are usually intro classes, I am ready to allow that everyone can get As. But I find students generally very tough on themselves and I more often raise than lower grades. But grades are a small part of the process. they have to write detailed evaluations of their papers and I write evaluations of these, often elaborating in detail why I disagree with the substance of their paper or their evaluation, but still allowing an A if they have evidence of a significant dialogue between school and personal knowledge. When this comes of, I find it much more liberating for me than teaching used to be. I used to delight in setting tricky questions that would reveal flaws, if they existed, in students' knowledge of majortexts. Real thinking was, I used to think, what I was testing. I now see this as a trivial exercise. All knowledge of consequence in this business is controversial and it is so because it involves ethics and politics (not science that is distinct from ethics). My role is to keep a lively, strenuous, conversation going on the topics. I am delighted when students who disagree with my positions later drop by to talk and ask advice on careers or ask me to write on their behalf. These things suggest to me that, for these students, I succeeded. It is a more satisfying vision of success than my old one wherein the successful student was the one who got the various texts right and applied or used them "intelligently."

John Nicholls.

I'll take this off list else I will seem to be flattering you.

[But he sent it accidentally to the list, and so it appears here.]

I agree with nearly all of what you are saying. I personally like to get test data for moderate sized classes (20-50) as a perception check, especially if I consider that part of the purpose of the course has involved mastery of some body of information (not a profound objective, but a common one--suppose, e.g., you were teaching the history of philosophy--with a large class, would you not want some assurance that people completing the course could associate some important names with their achievements?). For large classes, I cannot imagine getting to know students as well as you describe (here I am talking about 100 or more students). For teachers in schools, neither of these concerns is likely to hold since they work with smaller numbers for longer time periods. They do, however, bear a burden of documentation for both administration and parents--e.g., are you prepared to accept the judgement of the teacher you described without some evidence of the basis of her judgement (clearly, you do not trust her questioning skills). Anyhow, what I wanted to get at it is this: again, I think the extent to which what you do is natural for you makes it difficult to see how unnatural it may be for others. Specifically, the formal nature of learning about testing _may be_ a reasonable way of getting folk toward what you do with such facility. We also teach people about lesson plans because some of them would not have another way of thinking ahead. Remember that we are not expecting to produce polished teachers. In truth, we have to recognize that "average teachers" is what we can hope for, but we can keep trying to push up the average (and I think we are). You seem to be a gifted teacher and we do get some of those through our doors to, but we can't take credit for them. We can only hope to inspire them and support them.

bill hunter fax: (403) 282-0083 the university of calgary phone: (403) 220-5691 hunter@acs.ucalgary.ca

Bill Hunter:
****I personally like
****to get test data for moderate sized classes (20-50) as a
****perception check, especially if I consider that part of the
****purpose of the course has involved mastery of some body of
****information (not a profound objective, but a common one I think this is right.

I only worry that course material is more often thought to be of this sort than it ought to be. E.g., where history is thought to be primarily facts or math thought to be primarily algorithms. And as you describe these, you are using it more for seeing what you need to teach ("over again") than just for 'grading' students. That is quite fair and important, I think.

****For large classes, I cannot imagine getting to
****know students as well as you describe (here I am talking about
****100 or more students).

It can be done if you don't have too many such classes at one time -- though it is done better for teaching purposes than for grading purposes. It is difficult to have much confidence in grading this many, particularly some of the ones who are reticent much of the time, or who, when called on, seem to have a hard time (psychologically) saying what they think.

****They [school teachers] do, however, bear a burden of
****documentation for both administration and parents--e.g., are you
****prepared to accept the judgement of the teacher you described
****without some evidence of the basis of her judgement (clearly, you
****do not trust her questioning skills).

The problem is making the documentation be meaningful, which, I guess encompasses validity, etc., but also significance of material tested, etc.

****Anyhow, what I wanted to get at it is this: again, I think the
****extent to which what you do is natural for you makes it difficult
****to see how unnatural it may be for others.

Except that it is 'natural' for me only after getting certain perspectives, perspectives which I think could be taught to others perhaps without too much difficulty. In fact, perhaps, taught more easily than the 'hard' way I got them. When I ask for certain techniques to be used in schools, or certain things taught, teachers and principles tell me they won't work with kids; and I say they worked with my kids; and they say 'that is because your kids are so bright'. I honestly reply 'They just seem bright because they learned these things taught this way; if you taught this way to all kids, most of them would seem bright too. I don't think my kids have any more than average intellectual ability -- my wife and I have just been trying to develop what they have.' I think perhaps there are techniques -- some of which you and Sue and John and others may use (or you may have more and better ones-- that can be taught to ed students to make them also seem to be natural or to have teaching come easily.

****Specifically, the
****formal nature of learning about testing _may be_ a reasonable way
****of getting folk toward what you do with such facility. We also
****teach people about lesson plans because some of them would not
****have another way of thinking ahead.

My concern about these sorts of things was (is) that (1) they become ends for teachers instead of means -- e.g., teachers may feel bound to stick to lesson plans regardless of student academic needs and forget their lesson plan was only a "plan"; and more importantly (2) that ed students, as any students, won't see the real significance behind the formal descriptions -- so that they develop a kind of surface understanding of validity, reliability, etc., but don't have it in some sense 'internalized' -- like the engineers who understood physics principles in some sense but did not understand about the need to air condition refrigerated testing rooms. Perhaps this is a distinction between 'book knowledge' and 'practical understanding'. It took me a long time before I realized that what philosophers call 'validity' and 'soundness' have a meaningfulness outside of intellectual debate or doing philosophy. And it is still very difficult to translate their formal meanings into ordinary or useful explanations without having to ignore some difficulties, though these difficulties may be only of an esoteric nature if they are difficulties at all. But even when teaching these concepts in a less formal way, it is really difficult to have discussions with students and at the same time point out where these things come into play. For every time I do it, I feel there may three or five or ten times I don't because (1) it would disrupt the flow or take too long, or (2) because even I don't view what I am doing in those terms at the time I am doing it. Since, in a sense, the formal description is only a characterization of an idea or process (or whatever), it often is not the way we think about what we are doing -- perhaps just as basketball players are not thinking about geometric angles or the coaches explanations about a given passing pattern when they shoot off the glass or break down the floor.

****Remember that we are not
****expecting to produce polished teachers. In truth, we have to
****recognize that "average teachers" is what we can hope for, but we
****can keep trying to push up the average (and I think we are).

That is really what I wanted.

****You
****seem to be a gifted teacher

Of course I have only been describing my 'successes', and those as seen through my own eyes. Many days I don't feel I did very well.

****and we do get some of those through
****our doors to, but we can't take credit for them. We can only
****hope to inspire them and support them.

That is an extremely important and worthy goal in itself.

Rick
RICK GARLIKOV (DEMS042@UABDPO.DPO.UAB.EDU)

I am not embarrassed about the content, but about the error itself. However, it is small potatoes. Tom Green sent me a little note saying he was glad it appeared on the list and asking if he could be included in some invitation he thought I was issuing to you. I don't know what he meant, so I've told him to let me know what it was and I'll deal him in.

I am not responding to your last post because it left me with nothing to say. I have saved much of this conversation and hope to use it in my testing course next term. I really appreciate the thought you have given these issues and the time you have put into writing for the list. I am quite certain that I would have seriously considered a philosophy major had you been one of my instructors. My only undergrad course in philosophy was taught as a lecture to 1,500 students. Needless to say, there was not much interaction.

One small comment. You acknowledge the difficulty of reticent students. My wife was very gifted student and I am sure that she would have been both vocal and articulate in your class, but she took it as a matter of principle in most courses to stay quiet (for some of the reasons you mentioned). A teacher less gifted in framing evocative questions and responses could easily underestimate someone like her. There WERE courses in which she would talk frequently (part of our courtship took place in a Grade 12 English class which had weekly debate/discussion days--we were so often obviously and vocally on opposite sides of issues that only the two of us could understand how we managed to stay together and I am sure some folks still wonder nearly 30 years later).

You may understand better than I why I always feel compelled to go on and explain my point--here it is: I share your perception of the dangers of teachers accepting as normal, expected, or desirable, those techniques that we encourage them to use as heuristics in becoming skilled teachers, but I think that is balanced by the dangers of people THINKING they can do what you do when they are not really able to adequately inspire participation (your recent story was a good case in point).

bill hunter fax: (403) 282-0083 the university of calgary phone: (403) 220-5691 hunter@acs.ucalgary.ca

"Does Spending Money on Education Help?" asks Howard Wainer in the 12/93 Educational Researcher. He's talking about a table prepared by the Heritage Foundation and published last June by the Wall Street Journal that compares state SAT rank with average per pupil expenditure. Wainer observes, "It appears that a state's SAT ranking worsens by almost 6 places for every thousand dollars that was spent on the average pupil." More interesting than Wainer's refutation of the inference (the sample of SAT takers is nowhere nearly random or representative) is the fact that that the Heritage Foundation does this kind of study and the Wall Street Journal publishes it. And, it seems that some prominent group or person resurrects similar SAT findings almost every year. In many schools, districts, and state agencies there is a palpable tension just before the College Board releases the test results. Jobs, elections, and funding often depend in part on relatively insignificant swings in the average. Why?

Considered rationally, it is easy to understand how self-selection makes the SAT a wildly inaccurate barometer of school, district, state, or national achievement. We're talking common sense, pre introductory statistics. Of course, the Heritage Foundation senses a political lever, an argument for schools to "do more with less." On the other hand, more liberal groups manage to find evidence of gender or ethnic bias in the results, in order to support equally political agendas, i.e., to do more with more. I don't say this to diminish the value of political dialogue, but only to point out that the espoused rational claims for the SAT don't really support the actual political agendas of these groups. Interesting is that the political claims made with the SAT generally do carry weight with the public. Why?

The answer, I think, is that the SAT has become a symbol of the condition of public education. The success of the College Board and ETS has made the SAT highly visible. Its tough to make any simple generalizations about public education. Governance, funding, mission, structure, etc. vary enormously within and across states. The SAT score or rank is one apparently simple number that sums it all up and helps people to make sense of schools. The credibility of the SAT is boosted by the seeming rationality of using a test score, a number, to describe performance.

This wave of enthusiasm in testing probably had its origins in federal evaluation requirements for categorical programs. Many states followed suit. At the same time individual decisions about selection and placement of individuals needed to be defended against possible litigation. What could be more fair and objective than a test? While the quality of individual assessments and program evaluations varies enormously, the overall result has been to heighten the visibility and importance of test scores to the general public.

Now testing and program evaluation are the expected routine in public education. The catch is that good assessment tends to be expensive. So we find expedients that have strengths and weaknesses. Good commercial tests may meet technical standards, but aren't relevant to the curriculum. Teacher devised tests may be relevant, but are technically weak. Mandated assessments are irrelevant to the classroom and therefore may be poorly administered, scored, and reported. There are many ways that the quality of a score can be degraded. And yet, it is almost unthinkable that schools reduce the amount of testing. Likely, portfolios will be an add on, and then we'll see numerical scores assigned to the portfolios!? Tests have become symbols and part of the ritual of public education. And as most leaders know, one does not casually dispense with established symbols and rituals.

Rick: Thought this might amuse in re your reference to that scene from Dead Poets in which the same question is raised.
          Tom Green

A colleague gave me the following item; I pass it on to you now, hoping you'll enjoy it.

Then Jesus took His disciples up the mountain, and gathering them around Him, He taught them saying: "Blessed are the poor in spirit, for theirs is the kingdom of heaven. Blessed are the meek. Blessed are they that mourn. Blessed are the merciful. Blessed are the they who search for justice. Blessed are you who suffer. Be glad and rejoice, for your reward is great in heaven." "Remember what I am telling you."

Then Simon Peter said, "Do we have to write this down?"

And Andrew said, "Are we supposed to know this stuff?"

And James said, "Will we have a test on it?"

And Phillip said, "Can I borrow a pencil?"

And Bartholomew said, "Do we have to turn this in?"

And John said, "The other disciples didn't have to learn this!"

And Matthew said, "When do we get out of here?"

And Judas said, "What does this have to do with the real world?"

And the other disciples likewise. Then one of the pharisees who were present asked to see Jesus' lesson plan and inquired of Jesus His terminal objectives in the cognitive domain.

And Jesus wept.

My students in an introductory measurement class have just completed an assignment in constructing a short multiple choice test. This consists of identifying a construct; developing a blueprint; writing, critiquing, and piloting items; administering the test and performing an item analysis; revisiting the test items and options; and finally determining whether anything has been learned. Why do this kind of thing? It's almost impossible to understand the strengths and weaknesses of multiple choice tests unless you've gone to the trouble of constructing one. The same is true for constructed response and essay exams. What my students learn is that individual questions, no matter how carefully crafted, sometimes fail to be good indicators of a student's knowledge. In fact, that is why a test is composed of multiple questions whether it is written, oral or observational; hence the concern with reliability. A really bad test, reliable or not, is one in which most or all of the items miss the mark. We enter the realm of validity (when we speak of a test missing the mark) which a lot of psychometricians have worried about including Anastasi, Cronbach and Messick.

Cronbach in the 1988 book TEST VALIDITY described five perspectives on validity: functional, political, operationist, economic and explanatory. Further, he wrote that "Validation speaks to a diverse and potentially critical audience; therefore, the [validity] argument must link concepts, evidence, social and personal consequences, and values." It seems like he could have been addressing (broadly) many of the concerns with testing that have recently been posted. My point here is that many persons in the field of educational measurement are not ignorant of issues concerning test validation. Rick has advocated the idea that whether a student demonstrates competence can only assessed through verbal "give-and-take" interaction. While this may be true of philosophy, it is clearly not true for other subjects such as writing and math classes. This is not to say that classroom discussion is not important, only that it cannot be central to assessment. Classroom tests, if well-made, can be useful for assessing particular skills which can involve higher order reasoning. I note that it is possible for a teacher to make a mistake by giving too much credence to a test score. However, to argue that tests should not be used because they are fallible implies that only infallible methods should be used for assessment. This is the force of many of Rick's arguments; they are emotional appeals to anomalous decisions based on various types of tests.

In the hands of a less dedicated teacher than Rick, I dare say that verbal give-and-take is as equally fallible as a classroom test. Other arguments against testing have been based on the premise that any valid method of assessment must directly concern instruction, that is, must be indistinguishable from classroom instruction. If it isn't, this just shows that we don't really care about children or students. I don't understand what makes this assertion obviously or necessarily true. I held a discussion on grading during my last class session, but first I had students give assessments of all member of their small groups (including themselves). First I had students give a global rating on a scale from 1 (low) to 5 (high) of each members contribution. Then they turned over this sheet and ranked each member in term of their contribution. I then allowed them time to give written comments, including a critique of this assessment method. As you might imagine some students bristled at the activity of "assessment," AKA grading.

After this activity, to make a long story short, I asked them whether they would prefer an educational system in which grades were not assigned. The result wasn't even close: for many different reasons, most preferred a graded system. One student remarked that in college it should be the professor's responsibility to establish, maintain and communicate standards, while it should be the student's prerogative to take the professor's class, i.e., to accept the standards.

Testing is only a symbol and ritual of public education in the mythical school house depicted in many posts on this lists. Teaching in this mythical school house is a holy calling carried out by high priests, the professional teachers, who always knows best, and are constantly struggling to break through the bonds of conventionality that stand in the way of brilliant communication with students. The reality at least in public schools in New York City, is far more complicated, and sometimes bleak. While there are many solid old fashioned schools, and every teaching innovation ever dreamed up by man has found a home in one school or another, there are scores or perhaps hundreds of schools that are failing. For whatever reason, poor local leadership, political corruption, bureaucratic bungling, administrative grid lock, inferior professional development, counterproductive contract requirements, demoralized teachers, some of these schools have been failing for years -- and blaming their students for the failures. (And these schools do poorly not only compared to suburban schools, but also compared to other high poverty schools.) It may be that it is only through the use of valid, credible and easily understood measures of school performance, from dropout rates, attendance, to standardized testing, that these schools be identified, and public pressure can begin to make a case for change.

We are far from the day when a portfolio evaluation will convince parents in an impoverished neighborhood that it was time to try to depose a failed tenured principal and reorganize the school. Last week, the New York City Board of Education reported that one out of four intermediate schools, had failed to meet any one of nine city and state performance standards, from MINIMUM competency tests in math and reading, to attendance standards, and measures of student gains. These schools are undoubtedly stocked with qualified, and often caring teachers and administrators. Yet for whatever reason, many school people say, these enterprises are failing, and need to be retooled. (And by the way, while school spending in New York City, is below average for the state, It spends upwards of $7,200 a student, more than the median spending in all but a handful of states.) As a reporter who writes about schools in New York City, but does not work in a school, testing -- and you are not going to get me to defend bad testing, or misinterpreted tests -- is one of the few tools capable of providing an independent view, even if a distorted one, of what goes on in a school.

Josh Barbanel The New York Times jobarb@panix.com

On Wed, 15 Dec 1993, Mark Fetler wrote:

**** ... it is almost unthinkable
**** that schools reduce the amount of testing. Likely,
**** portfolios will be an add on, and then we'll see numerical
**** scores assigned to the portfolios!? Tests have become
**** symbols and part of the ritual of public education. And as
**** most leaders know, one does not casually dispense with
**** established symbols and rituals.
****

**** I don't think lone intuitive judgment is a fair basis for GRADING, but I am
**** not sure other assessment methods that seem objective don't often boil down
**** to lone intuitive judgment, though not in an obvious way.

Sports boils down to that though not in an obvious way. Much traditional grading also does....

**** An important point. Biases may not be at all obvious, but they are there. Stephen Jay Gould (_Mismeasure of Man_) would agree. So where does that leave us? Still with a responsibility to make decisions about who goes forward and who does not, still with a desire to do that as fairly as possible. We will never achieve a "once and for all" perfect standard, but we must be as fair as we can. The principles of fair academic assessment (like the principles for fair officiating in sports) have been hashed over, represent our current collective best effort to prevent injustice, and are nonetheless fallible. Our best recourse is to know them, apply them, and challenge them when needs be. Unfortunately many teachers, who could be among the most articulate spokespeople for improvement, know intuitively when something is awry, but leave college without the training to help correct the problems. The injustices in assessment systems will respond to public pressure when there are enough caring and well-educated people who are also well-equipped to take them on.

Joan Gipson-Fredin

joangf@fallbrook.csusm.edu

Judgments and judging processes always need to be monitored and, where necessary, refined. And reflective, reasoned judgment always, or almost always, needs to be the guide. If you notice that autopilot is flying you into the ground in a particular case, it is time to suspect there is something wrong with the autopilot, even if in general the autopilot flies much better than you do. I am taking formal assessment procedures to be something like autopilot assessment procedures. There are three problems with formal assessment procedures of the sort we have been discussing: (1) they are subject to new discoveries that might show them in need of amendment, (2) they may not be properly understood or used by practicing teachers (which may be what Gene meant when he said that 'nobody' understands validity (or whatever) anyway, and which is what I was worried about with regard to students trying to learn these concepts in ways that were meaningful to them outside an ed school assessments course, and (3) they are subject simply to accidental mistakes, which judgment is necessary to detect.

Examples of the latter case are recipes typed up with obviously wrong ingredients or quantities, or math calculations that give very erroneous answers. (In teaching math, one of the problems is getting kids to understand how to use 'rough judgment' to help them spot egregious mistakes they make in applying formal procedures. If you don't use judgment, there is no reason to suspect any answer you have double checked merely procedurally (making the same mistake each time) is incorrect. Formal procedures merely crank out answers, not necessarily right or reasonable answers.) At any rate, the point about fallibility I was trying to make is that it is more risky than necessary to relinquish reasoned judgment to merely formal or procedural judgment IN IMPORTANT KINDS OF CASES (like grading) because there are too many possibilities for error in merely following, or trying to follow, formal procedures.

My point is not that formal assessment procedures are wrong to use, but that they are wrong to be totally and blindly relied upon without continuous checking and monitoring by reflective and reasoned judgment in those kinds of cases where the extra effort is warranted by the importance of the consequences. (In cases where the consequences are not that important, the extra effort is not so necessary. E.g., our bathroom scale sometimes appears to be five pounds off. That is not worth getting a new scale to correct.)

The general point about not relying solely and blindly on formal procedures is important not only in education but in all other areas of life also. For example, law tends to be a fairly formal enterprise; and therefore it often leads to the obviously screwy sorts of judgments and results that either get us to change the law or to shake our heads at the lack of wisdom or concern of legislators and/or judges. Or following a doctor's orders blindly can lead to problems. An older man I know was released from the hospital after a bout with pneumonia, but at home was having really terrible difficulties which he and his wife thought were just the normal consequences from pneumonia coupled with age. The symptoms sounded suspiciously to me like he was having a medication reaction and I had them check with a pharmacist, who confirmed the symptoms were the likely result of a dangerous drug combination. When the medications were changed, the man recovered almost immediately. It was not easy talking the couple into calling the pharmacist; they assumed the doctor and hospital knew more about health than they or I or the pharmacist would. GENERALLY THAT WOULD BE TRUE, but you cannot let what is generally better than your own judgment simply totally replace your own judgment. (Autopilot into the ground....)

My favorite experience with formal procedure replacing judgment is the case where I bought a chocolate shake at a brand new McDonald's. The shake was slightly off-white in color, and it tasted like a not very good vanilla shake. I thought the counter high school girl had made a mistake. She and another girl with her explained that is just the way their chocolate shakes are. I figured the manager needed to know there was a problem with the shake machine, and asked to speak to the manager. She came out and assured me the shake machine had just last week been installed by McDonald's experts to McDonald's specifications and there was no problem with my shake. "That is the way a McDonald's chocolate shake is supposed to be!" I said I had had many different McDonald's chocolate shakes from all over, and this was not like any of them ever. She said "Well, this machine is the way it is supposed to be so there is nothing wrong with that shake." After she went back to her office, I thought of what I should have said to her, and said to the girls, "Gee, I should have had her taste a vanilla shake and a chocolate shake and show her she wouldn't be able to tell the difference." The girls looked at me and said, "No, she would have been able to tell the difference easily. Our vanilla shakes taste like chalk." THEY knew something was wrong.

The point of all this is simply that neither initial "intuitive" judgment alone, nor formalized procedures alone, nor the two in combination give the most reasonable and reasonably justifiable assessment procedures. What is needed is reflective judgment that looks at both intuitive judgment and formalize refinements AND at whatever else seems relevant to make a determination that is the BEST one can do at the time. This judgment itself may lead to further refinements, and the process continues. Joan, in her post on this, spoke of checks and balances, but she was speaking of formal checks and balances --validity, reliability, etc. I am arguing that we also have to always allow "informal" checks and balances as well -- in this case, the kind of give-and-take discussions I previously described.

Finally, Greg said he did not think my conferring with my students was sufficient to make my grades reliable or maybe "fair" was the word he used. I had mentioned that not as a way of making them necessarily either, but as a response to Joan's concern that my lone intuitive judgment was not a very good way of assigning grades. I don't use just that. And I responded that I think teachers do rely on lone individual intuition more than we might realize -- when they grade essays or discount "objective" answers students have good reason for believing ought to be counted right. If Greg or anyone wants to point out the grading problems with the kind of collaboration I do with students about their grades, I'll read and consider them. I collaborate with students (1) as a kind of check and balance, to hear evidence they might have that I might not have noticed or properly considered or remembered, (2) to explain to them WHY I think their grade ought to be such-and-such, and (3) to try to settle disagreements about grades before the course is over, rather than after. Not seeking their views and considering them seems to me to ignore potentially relevant and useful information about what their grade ought to be. And not seeking their input and their agreement also seems to me to make grading more dictatorial and less decent an enterprise than it has to be, regardless of how "accurate" (whatever that might mean) the grade is in any case. Rick

RICK GARLIKOV (DEMS042@UABDPO.DPO.UAB.EDU)

[At this point, with the discussion pretty much having run its course, another major discussion on a totally different topic began. Hence, the gap in dates before the next entry.]

Date: Wed, 5 Jan 1994 22:33:03 -800 Reply-To: Education Policy Analysis Forum <EDPOLYAN@ASUACAD.BITNET> Sender: Education Policy Analysis Forum <EDPOLYAN@ASUACAD.BITNET> From: Joan Gipson-Fredin <joangf@FALLBROOK.CSUSM.EDU> Subject: Re: Infallible Assessment In-Reply-To: <9312181337.AA11585@fallbrook.csusm.edu>

Happy New Year, folks.

It was my pleasure to conclude the old year with an introduction to the internet and to an engaging conversation about "teaching without testing, sort of." While Rick Garlikov argued the pitfalls of formal assessment and the merits of his classroom assessment system for philosophy students, I argued that society would benefit if a larger segment had more instruction in formal assessment.

Rick has since kindly shared his original post with me and has commented further. I see that we are not so very far apart. . .

On Sat, 18 Dec 1993, Rick Garlikov wrote:

**** Formal judging procedures are attempts to eliminate the problems with
**** intuitive judgments -- that is, the KNOWN problems with intuitive judgments.
**** But new problems are discovered periodically. Discovering problems with
**** formal judging procedures requires judgment and insight itself. It would be
**** folly to ever assume we have perfected a judgment procedure that itself never
**** needs to be monitored and judged in the light of new information, new circum-
**** stances, or new perspectives.... Judgments and judging processes always
****need to be monitored and, where necessary, refined. And reflective, reasoned
**** judgment always, or almost always, needs to be the guide.

To this I say, "Amen." Oscar Krisen Buros would surely applaud. But Rick goes on to say:

**** If you notice that autopilot is flying you into the ground in a particular
****case, it is time to suspect there is something wrong with the autopilot, even
****if in general the autopilot flies much better than you do. I am taking
****formal assessment procedures to be something like autopilot assessment
****procedures.

This seems to be the crux of our differences and largely a matter of a need to define terms. To me "formal assessment" is a system properly *based upon* reflective, reasoned judgment and intended to assure reflective, reasoned monitoring. Tests of all sorts originate with reflectionupon what is worth assessing and at best conclude with an evaluation of how well the test served its intended purpose (followed, ideally, with improvements before the test is reused). Rick's objections seem to me to zero in on *violations* of good formal assessment--tests which miss the mark, fallacious numeration (remember his great example of the essay returned unmarked but mysteriously scored as an"86" with a score of "87" crossed out), and overreliance on the results.

But, Rick, let's not throw out the baby with the bathwater. Good formal assessment practices attempt, among other things, to ensure that all of the students get asked the most important questions (or are assigned the most telling tasks), that all students have a fair opportunity to show their stuff, and that they are impartially evaluated according to consistent standards. One of the more important features of formal written assessment is that all of its stages are usually recorded, allowing for analysis and reflective judgment about its value by independent judges.

As you have so far described your approach, Rick, it seems to rely very heavily on your skillful application of your good intentions to teach all of the main ideas, involve everyone to the best of their ability, and to remember how well they did. Keeping track of it all--while engaged in Socratic dialogue, no less :) -- sounds like a real challenge. Chances are there's some as yet unmentioned underlying formality to your method. (You seem to be bent on discovering all of the principles of fair assessment through philosophizing.) Please tell me more.

Joan

joangf@fallbrook.csusm.edu

Date: Sat, 8 Jan 1994 19:34:08 CST From: Rick Garlikov <DEMS042@UABDPO.BITNET> Subject: Re: Infallible Assessment To: Multiple recipients of list EDPOLYAN <EDPOLYAN@ASUACAD.BITNET> In-Reply-To: Message of Wed, 5 Jan 1994 22:33:03 -800 from <joangf@FALLBROOK.CSUSM.EDU>

Joan Gipson-Fredin:
**** Rick Garlikov wrote:
****
**** Formal judging procedures are attempts to eliminate the problems with
**** intuitive judgments -- that is, the KNOWN problems with intuitive judgments.
**** But new problems are discovered periodically. Discovering problems with
**** formal judging procedures requires judgment and insight itself. It would be
**** folly to ever assume we have perfected a judgment procedure that itself never
**** needs to be monitored and judged in the light of new information, new circum-
**** stances, or new perspectives.... Judgments and judging processes always
****need to be monitored and, where necessary, refined. And reflective, reasoned
**** judgment always, or almost always, needs to be the guide.
****
****To this I say, "Amen." But Rick
****goes on to say:
****
**** If you notice that autopilot is flying you into the ground in a particular
****case, it is time to suspect there is something wrong with the autopilot, even
****if in general the autopilot flies much better than you do. I am taking
****formal assessment procedures to be something like autopilot assessment
****procedures.
****
****This seems to be the crux of our differences and largely a matter of a
****need to define terms. To me "formal assessment" is a system properly *based
****upon* reflective, reasoned judgment and intended to assure reflective,
****reasoned monitoring. Tests of all sorts originate with reflection upon
****what is worth assessing and at best conclude with an evaluation of how well
****the test served its intended purpose (followed, ideally, with improvements
****before the test is reused). Rick's objections seem to me to zero in on
*****violations* of good formal assessment--tests which miss the mark,
****fallacious numeration (remember his great example of the essay returned
****unmarked but mysteriously scored as an"86" with a score of "87" crossed
****out), and overreliance on the results.
****

There are all kinds of possible "violations" of "good formal assessment". What I am saying is that I have seen in my own dealing with students and in watching other teachers as well, that quite often what a teacher thinks a student is answering is not really indicative of what the student is actually answering, or what the student understands. Students themselves don't always say what they mean or know what they understand. It usually takes some sort of extended dialogue to understand what people mean by assertions about logical or conceptual matters -- extended dialogue of the sort we often see on this list, such as Glass's asking Gintis what he means and why he says certain things that seem either false or contradictory to his own previous writings. Then Gintis responds in a way that makes a distinction either he did not state before or that Glass et al. did not realize. I don't know how to condense that sort of thing into "tests" where, say, Gintis has one chance to say exactly what he means, and Glass grades it based on his own ideas of what he thinks it means and his own ideas of whether (and why) he thinks it is right or wrong. If we operated that way on this list, we would all give each other bad grades most of the time. It is almost impossible to say something in a way that cannot be misunderstood. E.g., you and Alan Davis think I am somehow grading without stated standards or just by intuition or whim. I will repeat shortly why that is not so. But in the meantime, if I were your student, you would have given me a low grade for my previous comments. Instead you are talking with me -- as I talk with my students.

Now, I am not certain about how you or other test experts "evaluate how well the test served its intended purposes", but it is this aspect that I am most concerned with; and I would think there is no way to judge that totally by merely examining the tests through some sort of formal analysis. You cannot find that some people did not mispeak themselves or misunderstand the question or have assumptions you need to know, etc., all in quite rational, legitimate, intelligent ways. And my informal observations show me that teachers and students (like many other people) frequently misunderstand each other without realizing they are doing so, but which may be obvious to a third party. Or which may not be obvious to anyone until some "crucial" evidence is stumbled upon that demonstrates there is a problem or some unsuspected dichotomy. Look- ing at the tests will not demonstrate whether the tests are reliable each time; You still have to talk with the students.

Further, if you are teaching, and a student misses something on a test, I presume a good teacher wants to help the student understand what he/she missed. Now, it frequently does no good in that regard just to tell the student the correct answer. Suppose the teacher had merely told the students (as she did) that "Tom is sleeping" IS a sentence. They might get that right on a re-test, yet have no clue why something that is an "inaction" is an "action". (Remember, she had told the class sentences have to name a 'thing' and an 'action', and they had all thought sleeping was not an 'action'.) Often when students miss a test item, or when much of the class misses a test item, the teacher will launch into another (or repeat the same) lecture about what the answer ought to be -- again without trying to see how the students were thinking about the problem and how to solve it. So again the lecture will often be without relevance to how the students are thinking about the material.

But moreover, students can often get things right in some mechanical or lucky way without any real understanding of the process. That is not uncommon. And talking with them is the only way I know of to try to discover this. For example it was a common practice on college board exams for a long time (and may still be) to put a likely incorrect answer in front of the correct answer. So if a student wasn't sure whether to do a problem one way or the other, picking the second answer was more likely to be correct. Or, as I pointed out, the third grade girl who was able to do subtraction problems of certain sorts right by using unreliable methods that simply happened to work for the kinds of problems they were working. In calculus, we often had to use the chain rule to get certain kinds of derivatives. On one exam we had to use a "double" chain rule in an abstract problem. I not only missed it; I never understood how to that particular problem at all, and could not understand the explanation. I now believe I never really understood the chain rule at all, but could use my version of it to work "normal" problems. Yet, the missing of that one problem did not prevent me from getting A's in calculus, even though I think I had no real understanding of a very major, important element of calculus. I could work the 'normal' test problems the teachers thought showed a mastery of the subject. If they weren't going to talk with me, how would they determine otherwise?

****But, Rick, let's not throw out the baby with the bathwater. Good formal
****assessment practices attempt, among other things, to ensure that all of the
****students get asked the most important questions (or are assigned the most
****telling tasks), that all students have a fair opportunity to show their
****stuff, and that they are impartially evaluated according to consistent
****standards.

Ahhh, but there's the rub. How do you know which questions or tasks are the most "telling"? I agree that IF you knew what the definitive questions to ask were, that would be a good start. But I am saying you cannot always know you know them.

Further, you cannot tell whether a student is being given a fair opportunity "to show his/her stuff". You are seeking something analogous to definitive experiments in science. And those are hard to come by even in science, and are often hotly contested by those whose theories seem to "lose" out.

Then you think that standards' being consistent somehow makes them also accurate. That is hardly necessarily true. Teachers can have faulty standards and thus award the best grades to kids who parrot them, rather than who give actually better or more intelligent answers. Or a teacher can have a grade averaging system that messes up a good student who happened to have a bad day or who misunderstood one thing at test time that skewed his average badly. Yet in many other ways, it might be demonstrable that student was the 'best' student in the class.

When I was an academic advisor to freshmen and sophomores at Michigan while I was a grad student, incoming freshmen were given Benno Fricke's (if I remember his name right) Opinions, Attitudes, and Interests Survey (OAIS) or "cooked carrots" test, after one of the questions on the survey. In the five years I counseled students, I never saw an occasion when that test gave mistaken information about a student -- yet I would never presume to rely on that test information alone to help a student choose courses. I used that test as an indicator for what kinds of questions to ask students who seemed to have interests, and who wanted to choose courses, that the OAIS test showed could be problematic for them. Sometimes you got answers from students that showed they had mistaken understanding or expectations of what a field was about. If you explained it to them, they would say they had made a mistake and did not really want to choose that course. Other times they were sure they were right, and I allowed the choice, but built in certain safety factors in various ways, including warning them what the OAIS test said might happen to them while they were in the course, and that if they found such things happening, to take certain steps before it was too late. It virtually always worked out the way the OAIS test predicted. But the test alone would not have been either proper to rely on or helpful in telling you why its results were the way they were.

Finally in this regard, after my freshman year I was accepted into an an experimental joint undergrad/medical school program that combined med courses and non-med courses in various ways over an eight year span. At the end of my 'senior' undergrad year, they told us we had to take the MEDCAT exam. I assumed it was a mere formality. The night before the exam I met a girl I found interesting, and ended up talking with till after 2 a.m. I finally went to sleep around 3 a.m. and had to get up a few hours later to go take the 8 a.m. interminable test. I was relaxed because I really did not care about the test. I was a philosophy major, but the philosophy questions they asked had to do with labels of theories, and I had learned the theories but not the labels. The physics questions seemed to deal mostly with hydraulics, and both my high school physics, and my college physics courses, had left hydraulics out of the sequence. Strike 2. The rest seemed about average. But I was really tired and just simply did whatever I did and didn't really care about it. At my med school interview I was told I had one of the highest MEDCAT scores they had received. I told them I thought that must have meant there was something wrong with the test. I still think that. Maybe by now, they think that too. My score had to be mostly luck. When I and later others dropped out of the program and out of medicine, they began trying to revise their program and their "auditioning" procedures for it and for medical school in general.

****One of the more important features of formal written assessment is
****that all of its stages are usually recorded, allowing for analysis
****and reflective judgment about its value by independent judges.

Great except (1) that almost never happens anyway; other profs don't regrade student exams (even the BAR exam I told about); (2) the problems I have been discussing won't show up on re-grading any more than grading, and won't be any more apparent to other graders than the first.

****As you have so far described your approach, Rick, it seems to rely very
****heavily on your skillful application of your good intentions to teach all
****of the main ideas, involve everyone to the best of their ability, and to
****remember how well they did. Keeping track of it all--while engaged in
****Socratic dialogue, no less :) -- sounds like a real challenge.

Ahh, but this, and Alan Davis's comment show you both miss the point of what I was trying to say. 1) I am teaching to teach, not to grade; grading is an aftermath and is based on the best they can do, not some kind of average. I don't have to keep track of all the bad things they do or the wrong things they say -- as long as by the end of the term I know what the best they can do is. And they have zillions of opportunities to show me their best.

2) The Socratic method --where I really listen to their answers-- makes what they say tend to stick with me, because I have to think about it, and sometimes think up arguments against it, or think up ways to strengthen an idea that they only have the kernel of. Almost everything they say is interesting and memorable in some way or other. (I have written papers to them in response to class discussions when I felt the need. I even have written a 600 page manuscript that grew out of a three page paper I wrote a class in response to an issue where I disagreed with them -- and they still disagreed with me, for challenging reasons, and I had to write more. So I wrote six more pages and we discussed them. Then we moved on, but by then I was hooked on this topic, and kept writing long after the course was over. I take what they say seriously; so it is not hard to remember it.)

And I write down who said some things, if I think I will have difficulty remembering who said what. But the "what" is easy to remember. In order to teach them, I --as I have maintained all along anyone does-- need to know what they are thinking that I need to change or add to. I don't use the Socratic method as a game; I use it to find out what they think and why they think it. The whole course is a kind of dialogue back and forth between them and me, and between them and their classmates. We try to resolve every issue we discuss. We take each issue at least as far as we can. I am deeply involved in that discussion, so I know where we are and where most of them are in each issue. I am not having to keep track of every thing they say and then give them a 48/70 ratio or some such. I care where they end up; not how hard it was for them to get there or how many errors they made along the way. Same as when I teach a kid to ride a bike. I can remember most of the riding problems of most of the kids I taught (like when my sister hit a tree AFTER she realized she was riding by herself and started looking around for where I was), but I don't grade them on their problems. I just want them to learn to ride. They all do. What more do you want of them? (Besides safety or maybe bike maintenance, etc. -- but I mean only with regard to basic riding ability. They can start, stop, turn, go where they want. That is all that is required with regard to "learning to ride the bike".)

3) When I tell students grades, I tell them specifically WHY they got that grade -- what they can and cannot do; i.e., what skills and understanding they do and do not have. They can argue with me or refresh my memory that they did do something I say they cannot; or they can demonstrate they can. Or they can demonstrate up until I turn in grades that they can. As a last resort they can take a written exam; or they can write to me about any issues they feel they have not shown "their stuff" properly to me. My grades are based on reasons and standards, not whims; though I admitted I am not sure how to set standards for grades exactly (what should count as a D? What distinguishes a very high C from a very low B? Etc.) If Joan and Alan, say, give intricate multiple choice tests, and have specific, uncompromising standards for each letter grade, say 92% for an A, 85% for a B, etc., what do their grades mean about what students actually think and can do? What if they teach poorly? What if they use a book that doesn't explain things very well to a given group of students? They have set standards and have "fairly" tested the students, but what if the standards are meaningless in any real sense? Weren't the witch tests in Salem objective, uncompromising, and the results duly recorded? The same with trial by combat? I met a guy once at a party during which we had a long discussion about a number of things. He did not seem particularly astute about most of them. Yet it turned out he had a degree in philosophy from Vanderbilt and had finished second in his class at law school from a reasonably respected law school. I was real concerned about all the students who finished lower than second. Yet I assume his class ranking was based on much written evidence and strict standards of some sort or other. Unless he had some esoteric specialty, I would not want him for a lawyer for any sorts of general purposes, nor as the company philosopher either. Or have you never met anyone with impeccable credentials based on written compliance with strict standards who just could not do the job very well at which he is supposedly certified as competent?

**** I suspect
****there's some as yet unmentioned underlying formality to your method. (You
****seem to be bent on discovering all of the principles of fair assessment
****through philosophizing.)

Maybe there is some formal mechanism to my methods. I don't know. But its being formal isn't what will make it right, even if it is right. And, no, I am not seeking the principles of fair assessment; I am only seeking to make particular arguably fair assessments. Before you can have principles of fair assessment, you have to be able to identify what your principles need to distinguish between. Otherwise you run the risk of having a formal structure that has no "real" significance for what it distinguishes. Trial by combat is a formal structure; but it does not likely decide criminal guilt or innocence, does it? Nor does looking at the procedure and making modifications in it, so as to include "stand-ins" or "handicaps", etc.

Rick

RICK GARLIKOV (DEMS042@UABDPO.DPO.UAB.EDU)

Amen to that.... I can see a certain amount of formalization in the assigning of grades, though it is, of course, not by recipe or algorithm; but would you say there is any formalization in my analyzing or evaluating student comments and responding to them? I would think not. Unless you would count any sort of trial and error, poking and prodding, etc. as "systematic" attempts to counter sources of ... error. And, of course, I am more interested in the analyzing and using it to teach than in grading to give grades.

****I think that Rick has been an exceptionally good sport to share his
****approach with this list.

No; this group makes it very easy to share things with. Everyone who responds tends to have both curiosity and civility. This is probably the finest group of people to communicate with intellectually assembled (or unassembled, as it were) any place on the planet. And if anyone shows one of your beliefs to be wrong, they generally do it in such a way as to make you grateful to find out, and to have the new information or perspective. The amount of knowledge, experience, enthusiasm, and commitment to education and to reflection in general by people that contribute to this list is truly remarkable.

****While I understand that his goal is good teaching
****and that he would eschew his role as grader if he could, the question any
****teacher faces is how to employ assessment in the interests of instruction
****and fair treatment of students.

True!

****1) I would LOVE a system where I was graded on oral remarks. I am highly
****verbal, don't hesitate to speak out in class, and almost always hit it off
****with my teachers. But what about the students who are shy or who are
****reflective learners whose best ideas occur to them later, after the class
****discussion?

Since I grade on the QUALITY (i.e., truth, reasonableness, sign of having been attentive to previous discussion, etc.), quantity of comments is not necessarily in anyone's favor, and can be (and has been) a detriment if they talk without thinking. Also, these things and serious endeavor are what influence me more than "hitting it off with me". There are students I have genuinely liked as people who did not get A's from me, since I was not grading them on likability. There is no time limit on when a student can contribute, or when I can. As I said previously, I even called a student a year later when I thought up new grounds to convince him of something. Students can say or write anything they want at any time. It does not matter if we have gone "past" the subject. We don't really "go past" any subjects. Some things come up repeatedly as new questions or objections "dawn" on different students at different times. But also, I tend to phrase questions and comments, etc. in ways that evoke comment and reflection. I ask specific, pointed kinds of questions -- often with some introductory info first. Then I press students who seem to be thinking but not quite on the mark. The Socratic method (http://www.Garlikov.com/Soc_Meth.html) is a way of asking leading questions. And with the right attitude on the part of the teacher (which I hope to have) that indicates non-threatening intensity, even the most shy students participate. One woman last term told me I had drawn her out in a way that really pleased her because she never spoke up in other courses for fear she was not very good. She participated from the beginning in my class; and I had no idea it was not natural to her.

****2) I'd be uncomfortable not doing any writing. I may sound pretty good,
****but often my ideas don't take shape till I write them down. Writing helps
****me learn, and a record helps me review my thinking. A record also helps if
****there's a disagreement with a teacher over a grade.

So write to me or us then. I tell that to students at the beginning. Nothing says your comments to me have to be oral. Write me; I write back. Sometimes I read or mention what you wrote in class, and respond to it then. My own comments often don't take shape until I try to articulate them orally or in writing. Few people's do. It is difficult to know what you think sometimes until you get it out or down in words and see whether they make sense and seem to be true. Most papers I write come out quite different from the way I thought they would when I sat down to write them. Often when I am talking in class, I have to backtrack and say, "No, I take that back; I just thought of a problem with that." or "with the way I said that." I think it is extremely important to articulate your ideas in some way though, since almost no students who say or write nothing learn much in the course. When they finally have to say or write something, they sound just like people who were never there; pitiful; they have things all mixed up. Having stuff in writing does not generally help in a grade dispute. Half the time nobody else will bother to read it. If they do, it will probably seem worse to them than it did to the teacher. Generally, if you want a second opinion I can ask you the questions again in front of another person and let you state the answers -- or they can ask you the questions. After all, if you know the stuff, you should still know it. Right?

****3) There's an imbalance in power which prevents students from challenging
****a prof's grading methods. I might not like a grading system but probably
****wouldn't say so if I needed the class.

I do everything I can to neither have, nor appear to have, power other than the power of reason. If YOU still will not speak up under even these conditions, I am not to blame for your timidity. No one is but you. What exactly are you afraid of? Maybe you need to buy one of the 10,000 assertiveness books on the market. All of life makes it hard to speak up if you are always afraid of the consequences of someone's not liking what you say and having some power over you. I probably will be out of a job one of these days, since I challenged the VP for academic affairs in front of my students one day when he tried to assert unfair and irrational power over two of them in my classroom. I asked him to justify his demands of them. He said he was the chief academic officer of the college. I said that did not justify his demands. He said we could discuss it privately. I said I had to justify my actions to the students, and they had to justify their beliefs to me; if he had any justification for his demands, we would all like to hear them. After a bit more of this, where he dug himself deeper and deeper into a hole, he left in anger. He has told my chairman I am not to teach there any more. She hires me anyway. We will see who outlasts whom. I used the session to make a point to my students, since this administrator was black and since he had picked on two black students -- one of whom had insisted for a month that if blacks had their own institutions run by black people they would be better off. I asked him if the VP had been white, what he would have called that attack on him. He answered "discrimination". "Since he was black, what do you call it?" He had no answer. I asked whether he thought he would be better off in a college run by someone like me though I am white or in a college run by this VP. He preferred me. Maybe that made him think a bit more about his separatist utopia. When I was in ninth grade, the teacher and I disagreed over a grammar issue in the text book, Warriner's Handbook of English Grammar. I wrote Warriner to get his view. He wrote back that she was right but that he had not explained the point at issue in that book, but had covered it in the tenth grade book. He said the question should not have been in the book and he would change it in the next edition. He did. I told my teacher I had an apology to make to her. (She looked surprised and said "Just ONE?") I showed her the letter, assuming she would be ticked off that I had pursued it but glad to know she had "won". Instead it impressed the stew out of her that I had written and got (such) a reply. You never know where assertiveness might get you.

Also in this regard, when I was an academic advisor I had a student who was closed out of an English course she needed. I was a grad student. I called the English Dept Chairman (this was at the U of Michigan). He said the class was full, and closed. He agreed she had good reason to be let in, but... the class was closed. I asked him to re-open it, since he had the authority to do that; I already knew the class was closed; that was WHY I was calling him! He said but if he opened it for her he would have to open it for others. I said he would only have to open it for others with good reasons. And if they were turning away students with good reasons, perhaps they had too few sections of the course and might need to open another one or two. He said since they had already rejected other students who were closed out, it would not be fair to them. I asked if he thought they should not give penicillin to people after they discovered it because people had died before they discovered it. He let her in the course. I told the director of the counseling center and she just about had a stroke, saying "You can't call him. Nobody wins an argument from him; he gets upset when you argue with him." Wrong. He seemed to love it.

Finally, a great many Jews got in the trains the Nazis put them in because they were afraid to anger the soldiers they outnumbered and get themselves shot. So instead of a few dying, six million died. Jews today say "never again" will they be passive out of fear. I would hope you could speak up about a measly little case of unfair grading practice. If you are going to live your life in fear of consequences for stating your beliefs (even tactfully when tact is called for) then even if you have a longer life, it will seem much, much longer than it actually is.

****Also a written essay after a grade
****protest to be graded by the person whose grade you protested doesn't sound
****like much of a challenge procedure.

True. Except that sort of thing is only for people who challenge the fact that the grade was determined by oral answers instead of written ones. It is not about challenges to my assessment of their answers, unless they want a third party review. Then they could submit the written exam (not essay) to someone else. But that would probably do them little good (see above). Under my system, however, if you don't like that method of deciding grades, suggest something else, or do something else, or write the president or something. I'm open to any way you want to have your grade decided, but if I am the one deciding it, I will decide it on how reasonable your answer is based on material we have covered in class -- written, oral, or what have you. What have you? One big guy one day told me he would throw me off the building if I did not give him an A. I am not very big. (Grading in trial by combat had not ever occurred to me before. Hmmmm.) The next class period I took in this stupid stretch-across-your-chest body-building thing I had worked with for years. (It developed muscles that, as far as I could tell, you only used when stretching this thing. Huge Michigan football players I had taught were unable to stretch it.) I asked the guy to try to pull the apparatus apart. He couldn't do it. I stood there and did it eight times for him, then asked if he still wanted to determine his grade out on the balcony of the building with me. He didn't. I said, "see, that is another mistake in logic -- you assume because I have the muscles to stretch this thing I have other muscles too." And you still say the imbalance of POWER is in the hands of the teacher. I don't think so.

**** It is almost impossible to say something in a way that cannot be
**** misunderstood.

Agreed. So we try hard and ask for feedback.

**** **** E.g., you and Alan Davis think I am somehow grading without
**** ****stated standards or just by intuition or whim.

**** Now, now. When did you begin to believe that I thought that? You have
****already told us that you explain your criteria to students and that you
****rely on *reflective* not *intuitive* judgment, and I have no reason to
****disbelieve you.

I'll look up the passage from Alan and the one from you that made me think that, and post them in a minute. Sorry, if I misunderstood what you were getting at. Rick

RICK GARLIKOV (DEMS042@UABDPO.DPO.UAB.EDU)

Joan had written, previous to that: "As you have so far described your approach, Rick, it seems to rely very heavily on your skillful application of your good intentions to teach..., involve everyone ..., and to remember how well they did. Keeping track of it all -- while engaged in Socratic dialogue, no less :) -- sounds like a real challenge." Particularly after reading Alan's post, I took this passage to be a mildly sarcastic claim that this could not be done, and that my grading was not based after all on what I thought it was, and was merely my intuitions rationalized and projected (if that is the right word) by this illusion.

By the way, I deleted a line in the last post that indicated the numbered questions in your previous post were from students of yours, not from you. I hope lurkers following this thread will not have been mislead, and will have remembered those were student questions. If not, this is a reminder.

Rick
RICK GARLIKOV (DEMS042@UABDPO.DPO.UAB.EDU)

Rick
Rick Garlikov (DEMS042@UABDPO.DPO.UAB.EDU)

I just don't see the avid pursuit of truth, beauty, and goodness as being a peculiar province of any particular gender, race, culture, or age. In the fifth grade group I taught for a short time, there was a student who, the first day, when the classroom teacher asked everyone how they were, said he was not very good since it was Monday and he was in school having to think. "I hate having to think," was his comment. I interrupted that part of the class (before she introduced me) and asked him what he meant by "thinking". He said "You know, doing assigned math problems that are all the same; having to do projects; and write book reports; and memorize things for tests....." I asked if he like sports. And he said he did. I asked him if he liked doing drills and exercises in P.E. class. He didn't. I asked if he considered what they did in P.E. classes like that doing sports. No. I pointed out that he shouldn't consider those things he mentioned about academics as thinking either -- in about the same way. Once we started doing the kinds of questions I had brought for them, we got into a really heated debate in the class, and at one point that boy who "didn't like to think" came up to the board, grabbed a piece of chalk, started drawing and explaining about as excitedly as anyone could "No, it's like this! Look, let me show you....." He went on for some time about the issue in question, pointing out other students' mistakes (he thought) and things everyone had missed, etc., etc. After he spelled it all out, I stood there smiling at him and said "For somebody who hates thinking, you seem to be having a really good time doing it." He grinned kind of sheepishly and said "Yeah, I guess." The girls in the class were as animated and involved as the boys. And it was one of the same issues that got that reserved assistant superintendent animated as well, and that gets black students involved in the same way.

My black students see discrimination as a MAJOR ethical issue, whereas a white class won't even think about that much, other than in regard to affirmative action. Other than that, views about sex, guns, violence, honesty, fairness, learning, working, even abortion, tend to be pretty much the same from group to group and class to class. In some cases men's and women's ideas about sex and/or dating or marriage tend to differ, but when they do, nobody seems hesitant to speak up at all to make certain the other side hears what they need to know. Well, ALMOST nobody. There have been a few people who thought sex not a proper thing to be discussed in a classroom (even in a course called ETHICS AND SOCIETY), but they were relatively rare; and were of both genders, or as Gene would have it, of both sexes. Rick

Rick
RICK GARLIKOV (DEMS042@UABDPO.DPO.UAB.EDU)

Rick
RICK GARLIKOV (DEMS042@UABDPO.DPO.UAB.EDU)