Working Group D
Evidence and Quality of Efficacy in Research Approaches

Of the efficacy research being conducted on EdTech products, who is doing the research, what kind of research is it and what is the quality? For example, how many EdTech companies are doing research of any type? What type of research are they doing and what type of research approaches and criteria/metrics should they be using to evaluate product efficacy? How reliable, unbiased and rigorous are these efficacy studies and how can we better capture the context in which EdTech products are tested?

Working Group D: Blog
Evidence-Based or Evidence-Demonstrated? Big Difference
October 5, 2016

The term evidence-based is ubiquitous across within many circles in the US. However, evidence-based, in and of itself, does not imply anything except a logical proposition that a product or strategy SHOULD work based upon related, but possibly distant, findings. Research on rats, for example, is often cited as evidence for the purchase of weight loss supplements in humans. It is important to note that the strategies IMPLIED by a particular set of findings are hypotheses that need to themselves be tested for the desired effect, as well as for the presence of unintended consequences (side-effects). Thus, evidence-based implications merely provide guidance for technologies that themselves need to be tested and, if successful, evolve into evidence-demonstrated products. The evaluation of research-based implications can lead to evidence-demonstrated interventions, just as it can lead to misapplications that have the potential to subvert learning.

This Shark Tank clip is just a snippet of an episode where a technology developer argues that his product works because it was based upon the conclusions of a number of scientific studies. Note that his product itself has not been tested (later in the actual show, Mark Cuban slams him for this!). He merely points out that the CONCEPT is supported, not this instantiation of it. He then presents user self-reports of satisfaction as evidence for effectiveness, but is put in his place when it is pointed out that the sample was biased (“How many people did it NOT work on?”). In this case, the use of science and the system of proof was deeply flawed. Yet, the owner seems impervious to feedback. It remains an open question as to whether this product is a true operationalization of the science upon which it is based or whether it actually works in the manner in which it was designed. More credible evidence that it does would greatly enhance the marketing and success of this product. Evidence that it doesn’t would allow users to invest their limited resources appropriately…and it could lead to product refinement until it realizes its evidence-based potential. As you can see, both developers and users benefit from properly done efficacy studies.

Our group hopes to document what evidence those who develop and disseminate educational technology rely upon to have confidence in the effectiveness of their products. We will also examine how that evidence is cultivated and used in the development and marketing of their products. Possible outcomes of this investigation include the procurement of resources and models to encourage more useful and efficient efficacy studies, as well as strategies to conduct such research that benefit both product evolution and inform product evaluation by policy makers, target users, and other stakeholders.



Post a Comment

Your email address will not be published. Required fields are marked *

Working Group Leader
David Daniel
Professor
Dept of Psychology, James Madison University
Mahnaz R. Charania
Director, Strategic Planning & Evaluation
Fulton County Schools (GA)
Research Lead
Chris Hulleman
Research Associate Professor
University of Virginia, Curry School of Education
Working Group Members
Andrew Rotherham
Co-Founder & Partner
Bellwether Education Partners
Britte Chang
Senior Education Researcher
SRI International
David Wiley
Chief Academic Officer
Lumen Learning
Eva Dundas
Chief Learning Officer
Branching Minds
Jason Lange
CEO and Co-Founder
BloomBoard
Jill Buban
Senior Director of Research & Innovation
Online Learning Consortium
Julie Young
CEO
Global Personalized Academics
Mark Schneiderman
Vice President of Government Affairs
West / SchoolMessenger Solutions
Michael Staton
Partner
Learn Capital
Pete Wheelan
Chief Executive Officer
InsideTrack Inc.
Logistics Lead
Isabel Callaway
Outreach Coordinator
Jefferson Education
David Daniel
Professor
Dept of Psychology, James Madison University
David B. Daniel is an award-winning teacher and researcher with over 25 years of classroom experience and almost two decades consulting on educational technology development, design, and implementation across a variety of companies and stakeholder groups. He is an established leader in translating psychological science to useable knowledge for educational practice in K-12, higher education and corporate education.
Andrew Rotherham
Co-Founder & Partner
Bellwether Education Partners
Andrew is also the executive editor of Real Clear Education, writes the blog Eduwonk.com, teaches courses on education policy at universities including The University of Virginia, and is co-publisher of “Education Insider,” a federal policy analysis tool produced by Whiteboard Advisors. Andrew previously served at The White House as Special Assistant to the President for Domestic Policy and as a former member of the Virginia Board of Education.
Britte Chang
Senior Education Researcher
SRI International
Britte’s research focuses on the design and impact of learning technologies, instruction and assessment in K-12 math and science. Her work considers systemic issues of educational practice and policy, including the investigation of processes and designs that cross learning contexts and settings.
Chris Hulleman
Research Associate Professor
University of Virginia, Curry School of Education
Chris's research focuses on social psychological theories and interventions to increase student motivation. He was a former high school teacher’s aide, coach, and social worker, and has a passion for using psychological research to enhance educational outcomes.
David Daniel
Professor
Dept of Psychology, James Madison University
David B. Daniel is an award-winning teacher and researcher with over 25 years of classroom experience and almost two decades consulting on educational technology development, design, and implementation across a variety of companies and stakeholder groups. He is an established leader in translating psychological science to useable knowledge for educational practice in K-12, higher education and corporate education.
David Wiley
Chief Academic Officer
Lumen Learning
David is currently the Education Fellow at Creative Commons and adjunct faculty in Brigham Young University's graduate program in Instructional Psychology and Technology. He has founded or co-founded numerous entities including Lumen Learning, Degreed, and the Open High School of Utah.
Eva Dundas
Chief Learning Officer
Branching Minds
Eva is a developmental cognitive psychologist whose work focuses on levering the learning sciences to help K12 schools better support struggling students. At Branching Minds, she leads the efficacy testing and R&D on a web application that streamlines and focuses K12 intervention practice at the teacher and school level.
Isabel Callaway
Outreach Coordinator
Jefferson Education
Isabel is a senior at Duke University studying English and education. Currently she is the Outreach Coordinator at JEA. She is excited to be a part of the Efficacy Symposium as Group D's Logistics Lead.
Jason Lange
CEO and Co-Founder
BloomBoard
Jason previously worked in finance as an investment banking analyst and private equity associate. While in graduate school, he worked as a summer associate at NewSchools Venture Fund, where he supported the development of the initial edtech investment thesis for the NewSchools Seed Fund.
Jill Buban
Senior Director of Research & Innovation
Online Learning Consortium
Dr. Jill Buban serves as Senior Director of Research & Innovation for the Online Learning Consortium. In this role, she works with institutions to disperse online and blended learning research as well as conducts research projects commissioned by organizations and institutions. Prior to joining the Online Learning Consortium, Dr. Buban was the Assistant Provost for Research & Innovation at Post University. In this role, Buban instituted university-wide initiatives with a forward thinking, student-centered focus. These initiatives included, but were not limited to, the university’s transition to digital course materials, the creation of an online academy for high school students, competency-based learning initiatives, credentialing, articulation agreements, enrollment management, oversight of all academic publications, as well as a variety of teaching and learning initiatives. Prior to joining Post University, Dr. Buban worked in Academic Affairs at SUNY Empire State College. She collaborated on a variety of online learning initiatives including the implementation of ePortfolios, open learning access initiatives, and prior learning assessment practices.
Julie Young
CEO
Global Personalized Academics
Julie oversees the company’s plans to offer international students the opportunity to earn a dual diploma from their native country in addition to one from a U.S. accredited high school through virtual learning. She is the founder and former President and CEO of Florida Virtual School and served as a teacher trainer for a partnership between the Lee County school district and IBM.
Mahnaz R. Charania
Director, Strategic Planning & Evaluation
Fulton County Schools (GA)
Mahnaz is a Research Psychologist and K-12 strategic leader with expertise in impact evaluations of organizational investments. At Fulton County Schools, she conducts program evaluations of district- and school-level investments/innovations and ensures appropriate alignment between evaluation findings and strategic initiative priorities.
Mark Schneiderman
Vice President of Government Affairs
West / SchoolMessenger Solutions
Mark represents West’s Education group on education-related issues before federal and state governments. He previously served as Senior Director of Education Policy for the Software & Information Industry Association (SIIA) and as Legislative Associate for the Council of Chief State School Officers.
Michael Staton
Partner
Learn Capital
Michael is a Partner at Learn Capital, where he has helped lead investments into Coursera, Minerva, BrightBytes, AltSchool and Brilliant. He is the founder and former CEO of Uversity. Starting his career as a public school teacher, Michael served as a Venture Partner and Community Advisor to NewSchools Venture Fund’s Seed Fund, now Reach Capital.
Pete Wheelan
Chief Executive Officer
InsideTrack Inc.
Pete has dedicated his career to leading mission-driven, high growth companies focused on helping individuals live up to their full potential. Prior to InsideTrack, he served in executive positions at Blurb and Lonely Planet Publications.
In The News
Click on the stories below to read what people are saying about our Symposium and the importance of efficacy in EdTech.