Tuesday, May 31, 2016

What is the fundamental Basis of Scientific Thought/Discovery? Implications for the EdD

My work on research methodology talks a lot about the fundamental nature of science and scientific discovery. This is because much of what I read in education about the role of theory and how to interpret the practical implications of research results seems to me to be idealized, narrow, and artificial. Education conceptions of the role of theory and the applications of research outcomes to practice appears to me to have become inbred in our wonderful profession with a rhetoric and philosophy that appears to have evolved in a fashion that is increasingly disjointed from the actual fundamental nature of science. This disjointedness is true for education in general and carries over to how theory, research, and science is presented in EdD programs.

Therefore, in order to understand the real role of theory and quantitative analysis I read a great deal about the fundamental nature of science and scientific discovery as conceived and practiced in the physical and medical sciences—as opposed to relying on prior education research texts and existing discussions about the role of theory in education. As a result, my text on Authentic Quantitative Analysis... has a great deal of original perspective on the role of theory and how to apply quantitative results to leadership practice that differs from existing conventional wisdom in education and in EdD programs.

Right now I am reading one of the most fascinating and unique books on the latest research findings in physics. The book, Seven Brief Lessons on Physics by Carlo Rovelli, is unique in that it is short, poetic, and non-technical—and still manages to explain in easy to understand fashion the most important events and ideas in physics about the nature of the universe we exist in. If you want to understand quantum mechanics, Einstein' thought experiments, etc. in a quick and interesting read this is the book for you.

The insight from this book that sparked this post is what the author claims is the fundamental essence that drives scientific discovery. Is it theory? Is it evidence? NO! The author indicates that:

...before experiments, measurements, mathematics, and rigorous deductions, science is above all about visions. Science begins with a vision. Scientific thought is fed by a capacity to "see" things differently differently than they have previously been seen.

This is why Chapter 4 of my text emphasizes the creative and metaphorical nature of scientific discovery and Part II talks about Design-Based Dissertations as an important option in EdD programs. The latter focuses of envisioning and trying new and unique approaches to solving problems regardless of whether they are grounded in existing academic theory or research.  

Wednesday, May 18, 2016

A Reassessment of the Role of Theory by a Leading Researcher: Implications for EdD Dissertations


David Berliner is one of the top education researchers and helped define the field of educational psychology. He is a former president of the American Educational Research Association, and co-author of the books (among others) of  Educational Psychology, Handbook of Educational Psychology, and Manufactured Crisis: Myth, Fraud, and the Attack on America's Public Schools. Dr. Berliner typically extols the tradition perspective of education research. This is why I was surprised to see him describe the role of theory in a very non-traditional fashion in a recent autobiography (in Education Review) summarizing what he has learned in his long and distinguished career. Dr. Berliner concludes that:
"Theory may be overrated. The journals and the scholarly community value “theory.” But I have done a lot of research on teachers and teaching without much theory to guide me. We deal with the practical in education, and the practical is filled with complexity, some of which is hard to fit into psychological or any other social science theory...
A good question is a good question, and should be pursued. Working from a Piagetian or Vygotskian theory is nice, and thinking about the world from a Freirean position, or asking what would Derrida say, is also to be lauded. But in my research career, a good question sensibly answered is worth its weight in gold. So I have come to believe that dust-bowl empiricism is too often dismissed as inadequate, and that theory in education research is too often over emphasized. I am more impressed with the quality of the question asked and the attempt to answer it, and less impressed with the quality of the theory."  

Implications for EdD Dissertations

Dr. Berliner's statements point to the need to rethink the traditional perspective of the role of theory in an applied program such as the EdD. The question is how to do that. It is clearly appropriate for EdD programs to require/expect students to exhibit knowledge of theory and to demonstrate the ability to apply theory to problems in class assignments and exams. However, should students be required to provide a theoretical rational for the questions they ask in an EdD dissertation or the approach chosen to study the problem? Dr. Berliner's perspective and Chapters 4,5 of the above alternative quantitative methodology text for EdD programs suggest that theory and theoretical justifications should not be required as a basis for EdD dissertation work; though students should have the option for using it as a basis. Rather the focus should be on asking good questions, i.e., ones that have clearly empirical evidence of their importance and/or that they have not been previously asked in adequate fashion. Methodology and approach should be demonstrably unique—unless you are trying to replicate or extend the results from a prior study.     

The URL for this autobiography is 

Friday, May 6, 2016

Are the Recommendations of the What Works Clearinghouse Useful/Reliable for Practitioners?

In 2002 the US Congress established the What Works Clearinghouse in the US Department of Education’s Institute of Education Science. What Works Clearinghouse’s role is to set rigorous quantitative standards based on the best science to provide guidance for practitioners as to what works by assessing the quality of research evidence supporting a given intervention. The What Works Clearinghouse then issues a rating on whether the evidence behind an intervention meets its standard of evidence or does not. The What Works Clearinghouse performs the same function in education that the Food and Drug Administration (FDA) does in medicine.

Can leaders trust the recommendations of the What Works Clearinghouse? Are they valid?  

The above text on quantitative analysis raises questions about the quality of the WWC's reviews (see Chapter 3). Specifically the text criticizes WWC for validating the research behind 'Success for All' when there was lots of published contrary evidence, and for concluding that 'charter schools' are more effective than 'traditional public schools' despite the tiny Effect Sizes (ESs). Are these problems anomalies—or are there deep seated problems with WWC's recommendations?


An amazing recent article suggests that the problems with the recommendations of the WWC are deep seated. Ginsburg and Smith (2016) examined the evidence for all the math programs certified by the What Works Clearinghouse as having evidence of effectiveness as provided by the most rigorous “gold standard” research design—Randomized Controlled Trials (RCT). They reviewed all 18 math programs that had been certified by WCC which contained 27 approved RCT studies. They found 12 potential threats to the usefulness of these studies, and concluded that “…none of the RCT’s provides useful information for consumers wishing to make informed judgments about what mathematics curriculum to purchase." 


One of the key problems across the What Works Clearinghouse math studies was that where Ginsburg and Smith (2016) were able to determine the error effects of a threat, the error generated by even one of those threats was at least as great as the Effect Size favoring the treatment group. In others words, the 'gold-standard' of research is not so golden.


BOTTOM LINES

  • Leaders cannot not trust the recommendations of the What Works Clearinghouse (WWC). Leaders need to conduct their own due diligence using the techniques in the book—both on research in general and on the recommendations of the WWC. 
  • The fact that the WWC requires the most rigorous research methodology and statistical evidence, and there are still a dozen potential "threats" to the results, suggests that he real world of practice is too complex for the traditional experimental approach and its reliance on relative measures of performance.  In other words, it is virtually impossible to establish full internal validity in applied experimental research regardless of how rigorous the research standards are. (Simpler alternatives for assessing the effectiveness of interventions are discussed in Chapter 5 of my methodology book.)   

Ginsburg, A., & Smith, M.S., (2016). Do randomized control trials meet the “Gold Standard”? A study of the usefulness of RCTs in the What Works Clearinghouse. The URL to this article is:  
            
            http://www.aei.org/wp-content/uploads/2016/03/Do-randomized-controlled-trials-meet-the-gold-standard.pdf 

Monday, May 2, 2016

Can Theory Emerge From Practice? Does Scientific Discovery Only Occur at a Young Age?

The book Authentic Quantitative Analysis for Leadership Decision-Making starts with discussions about the nature of science and then uses that as the basis of discussing the relationship between theory and application in science in Chapters 1, 2. The discussions about the relationship between theory and practice in the book are unique and counter-intuitive in terms of how they are usually discussed in EdD programs. Some of the innovative nature of the discussion talks about:

  • the importance of personal theories of action along with academic theories, and 
  • the application of theory to practice does not necessarily mean that the practice will be more effective than one that is not based on academic theory. 

Chapter 4 goes a step further and argues that while the importance of theory is universally discussed as a key to improving practice, this ignores the fact that important academic theory can emerge from a successful practice that was based on an alternative mode of scientific discovery; e.g., accident or metaphor.

This is why I was glad to see a recent article about Iva Babushka, a 90-year old mathematician. This article noted that while applications such as cryptography have emerged from highly abstract number theory, "Conversely, many elegant and aesthetically pleasing mathematical theories have emerged from the most utilitarian applications." In other words, even abstract mathematical theory can emerge from the experience of successful practice.

The bottom line is that EdD programs should encourage students to explore innovative approaches to improving practice even if there is no theoretical justification.

And, by the way, one other take away from this article for us older faculty and students in EdD programs, is that the most important work of this mathematician occurred when he was 70. This contradicts the often cited notion that science and math is a young "Man's" game and that folks are over the hill by 30 or 40. In other words, faculty and students over the age of 40, in any field, still make important discoveries. Science is also finally acknowledging and honoring the important contributions of women. So let's embrace as a positive the fact that EdD programs tend to work with more experienced, older students than PhD programs, and also view as a plus the highly diverse nature of our students.