recorded on December
12, 2014
Dr Ellen Wagner joins vTapestry to continue the conversation from Online
Educa Berlin 2014 about whether or not big data is corrupting education.
Learn how big data can be used - in context - to drive evidence-based
decisions at home, at play, and at work. You might be surprised at who is
rating/grading you - and why!
(vTap beat and
beeps)
REBEKAH: Big dreams?
CHRISTINE: Big data!
REBEKAH: I'm Rebekah Nix;
CHRISTINE: I'm Christine Maxwell;
BOTH: … and together we are vTapestry.
INTRODUCTION
CHRISTINE: Rebekah and I would like to welcome Dr
Ellen Wagner as both a dear friend and a very esteemed colleague. To introduce
you, I think it would be helpful if I read a very short bio from your
introduction at Online Educa Berlin where I believe you’ve been very recently.
(Source: http://www.online-educa.com/profile-bio-82163)
Ellen D Wagner is a Chief Research and Strategy Officer
for the PAR (Predictive Analytics Reporting) Framework. She served as Executive
Director of the WICHE (Western Interstate Commission for Higher Education)
Cooperative for Educational Technologies (WCET) from 2009 to 2013. She
continues to serve as Senior Analyst with Sage Road Solutions, LLC. She was
formerly Senior Director of Worldwide eLearning Solutions, Adobe Systems, Inc
where she helped set the strategic direction for e-learning solutions for specialized
markets, including education. Ellen previously served as Senior Director of
Global Education Solutions at Macromedia Inc. Dr Wagner's prior career as a
tenured university professor and administrator featured positions such as Chair
of the Educational Technology Program at the University of Northern Colorado,
Academic Affairs Coordinator of Instructional and Research Technologies and
Director of the Western Institute for Distance Education. She was also Visiting
Scholar and Project Director at the Western Cooperative for Educational
Telecommunication, Western Interstate Commission on Higher Education. She holds
a PhD in Educational Psychology from the University of Colorado Boulder.
Welcome Ellen.
ELLEN: Well, thank you Christine and thank
you, Rebekah. I’m really pleased to be here talking with you tonight.
Data-as-a-Meme
REBEKAH: Thank you for speaking with us Ellen.
It's hard to believe how much has changed since you and I met at the first
International Forum for Women in eLearning conference way back in 2005. I'm
eager to view the archive of that more recent opening session of the 2014
Online Educa Berlin conference. My understanding is that you and Inge deWaard
went up against Drs Mayer-Schonberger and Siemens to debate whether or not data
is corrupting education. After a long flight home and plenty of commentary on
that panel, what do you suggest leaders - in any field - need to think about in
terms of 'data-as-a-meme' as you phrased it?
ELLEN: You know I had to come up with some
way that I could argue in favor of a house motion that said that data was
corrupting education because when I first heard that I was quite non-plussed
because I really couldn’t imagine how someone like me – who in fact is just
really finding myself driven by wanting to know more about how to make better
use of data – could really argue for a position like that… and so I stepped
back and started reading so many of the comments of why data could be a problem
for us. And what I realized is that data really aren’t the problem. The problem
is that we use a term, like ‘data’ or terms like ‘big data’ as if we all know
what we mean and we nod and we wink and we all sort of think about what that
is, but none of us really know. What we know might look like a loss of privacy.
What we know might look like people knowing more about what we’re doing online
than we know about ourselves.
We fear that with that
type of knowledge of what one can do with data that... profiling: whether used
for good or for maybe less good means, depending on where you fall on being
profiled… there just seem as if there are a number of places where it’s really
not the data that we worry about, but what I do think we worry about – and
where there might be potential for corruption, if you will – is when data are in
the hands of people who don’t really know how to use it very well or who will
use it in ways where the methodologies don’t support its use or to use it to
bludgeon people into a particular position as opposed to informing them about
things that they could actually do better.
So, you know, it’s a long
answer to your question, but it really reminded me that (for those of us who
are working with data) we need to make sure that we are focused on what we
mean, that we are able to point to real examples of where we can make a
difference – measureable differences – and that we open this conversation up so
that everybody can see where it is that we are going so that the data can truly
be realized, can really realize their full potential.
CHRISTINE: Right. The thing that that had me
think of is that the challenge is that the Internet and technology is moving so
fast that, you know, machine learning and the semantic web and the synaptic web
are already upon us – and yet, hardly any of us – if we’ve heard of it, we certainly
don’t know what that really means for us. I don’t want to get into the details
of that just yet, but it does… it’s a big challenge to anyone who is involved
in online research today.
ELLEN: I’m struck, Christine, by the idea
that, as you speak of things like machine learning or when we start looking at
the true big unstructured web as a source of information that for so many of
us, particularly those of us working in education... we deal with data that
comes at us in columns and rows. (CHRISTINE: Yep.) And our ability to work
with columns-and-rows data seems to be really quite immature as I look across
the ecosystem. In some places it’s getting quite a bit better, but typically in
areas where we’re dealing with more concrete operational kinds of things, like
finance or fundraising or places where we can deal with tangible measures of
business operations.
When you start looking at
using big data to make decisions about academic performance, achievement,
goal-setting, outcomes, things like that, the lack of consistent and reliably
shared measures of what we mean when we talk about these things is going to
continue to hamper us. So, I share your enthusiasm for where we think we can go
with this and sometimes I do wonder what it is that it’s going to take for all
of us to really step up to that challenge when I know that for many of us,
we’re still not ready to deal with the data that are right in front of our
faces sometimes.
CHRISTINE: I just want to make one point about
what you said which is actually very scary what you said. Because, if we’re
still immature in dealing with data that’s structured, the world has been
focused on structured information for a very long time, until relatively
recently. And so, in some ways that’s rather a scary thought. In other ways,
unstructured data which, of course, is by far and away the format of the wealth
of data that’s now drowning us all... the question of how we learn to work with
that is where machine learning steps up to the plate to help us. At the same time,
unless one is aware of what the process is and what the potential dangers are
of bias, we may be led very easily down paths that we do not expect.
ELLEN: You know, being asked about the
question or the notion of data corrupting education and really looking at the
notion of corruption, which of course is such a loaded negative word, but if
you think of corruption as a deviation from an ideal for personal or some type
of self-gain, it’s actually kind of eye-opening to think about that because
deviation from an ideal – even if it’s for someone’s gain – it might not be
wrong. But it really raises the question of our own need at this point where we
are looking at the data systems surrounding us where looking at it through the
lens of our current times, our current technology capacities, is really going
to be essential.
I think another one of
the risks many of us are going to encounter will be the constraint of
legislation policy or what have you that has been written for a technology
world that we haven’t seen for 10 or 20 years.
CHRISTINE: Right.
REBEKAH: With my focus on integrating
technology into professional development and to make sure that the learning
environments that educators create are the most positive possible, I see this
new shift, the new capabilities that big data analytics and visualizations
bring to the table – to the everyday user almost, now – as really exciting. And
as an educator turned technologist (so to speak), you Ellen, bring a really
unique perspective to this realm of big data analytics.
Data as a Student 'Tracking' Decider
As a technologist turned
educator, it’s kind of funny that I was impressed with an exchange from the
movie Interstellar. It illustrated one of my concerns about how student data
might be impacting us already. It was set in the principal's office of a rural
school in the near future, and Matthew McConaughey, as the character Cooper,
said: "You're ruling out college for my son now? He's fifteen." Then
the Principal replied: "Tom's score simply isn't high enough." So
Cooper went on to suggest or guess that the principal's waist line was maybe a
32 with a 33 inseam to make the point that it took two numbers to figure out
the Principal’s seat size (ELLEN: uh
hum), but only one test score to measure his son's future... How likely is that
scenario to become true, based on what you see happening today?
ELLEN: I can see it as being a likely path
for most of us, but I do think that the idea of making judgments about how
student information is used is really illustrated very well in this example. We
don’t want to find ourselves in situations where we are making decisions like
that. My hope is that if you have information to help guide a student toward
places where they are going to be more successful, I hope that we would use those
data for things like, you know, saying that students like you who have found
themselves interested in these kinds of things, that met these types of
benchmarks, actually did pretty well in these areas. And, to not say that you
can’t do something, because who among us wants to be able to say something like
your example of Tom, the student in this movie, that would be denied a shot at
a future just because a number showed him to be a particular way?
We have developed
diagnostic tools in our PAR Framework that calculate a risk for students that
is constructed from a number of variables that seem to demonstrate risk at
significant levels for students like those people. Does it mean that those
individual people are going to in fact demonstrate that these things are true?
I mean, these are not causal kinds of things. So, I think that being smart
about what the numbers are actually telling you is going to be really key to
avoid these determinations that are made by some type of metrics that we don’t
even know about. But I will tell you that if I look ahead to a future where
every key stroke or every interaction or every exchange is going to be parsed
for some type of meaning, I do have occasional moments of pause and I know that
for myself I think that being mindful of some of the paths where this might
play out are absolutely things that we practitioners of the current day need to
be able to articulate with one another.
REBEKAH: Yeah, I think that what you mentioned
about the correlations, (ELLEN: uh
hum) that’s really exciting to me because it’s been hard, I mean it’s been
impossible for me as a researcher to prove causality in the things that I do.
There are so many variables that are measured but can’t be related or directly
linked that I know in my gut, or in my heart more often, make a difference. And
in terms of the learning environment and what people bring to that in each
different semester, with each different (ELLEN:
uh hum) collection of people... that’s what matters and I think that’s what
makes learning ‘stick’ and I’m really excited about the potential of these new
tools to bring some of those correlations, those underlying themes that I never
even imagined might matter, to light (ELLEN:
uh hum) so that I can do my job better and support those people that the
other data might identify as ‘at risk’. I think it’s really fantastic and like
you said, if used in context… content is really no longer king in my world, so
being able to figure out how to help people learn and do it better, faster, and
cheaper, is really exciting to me.
ELLEN: Well, you know Rebekah, one of the
things I’ve been finding in our current work is that when I focus on learning
it’s a harder question to answer because learning is a
scientifically-determined construct of course, but you know so many of us have
emotional reactions to it and so I know for us working with PAR, where we ended
up saying well let’s come up with our proxies for what we would mean for
learning but to do it around the questions that people say are current measures
for success right now, knowing that a graduation rate is a many-splendored
thing, but a completion score is still something where we can be very, very
actionable as we look at it, so being as actionable as we can with data really
I think can help us respond to the need both for individual students that
providing better pathways for people to determine success...
I don’t think that
there’s ever going to have to be a single pathway to greatness at this point
looking at the multiple ways in which type engines can be delivered and the
multiple ways in which programs can be provided or interventions can be wrapped
around specific types of students. But what I will say is that in the very,
very early days of doing this type of work, our opportunities to start becoming
far more systematic about our language, our constructs, our decision-making...
here’s where I’d make a big distinction for where I’m seeing the biggest
excitement about the use of data right now for student success…
With data in education,
we have typically done basic research where we were establishing accountability
as constructs that we would then include within the cannon of our particular
practices. And there’s going to continue to always be a need for people who can
in fact make sure that the theoretical constructs of our discipline are
examined with the type of rigor that we want them to be examined so that we
know that what we’re talking about makes sense. These are foundational ideas
that drive an entire practice; they have to be solid. For so much of where we
want to take this information and apply it in practice, the level of rigor
required to make effective decisions that maximize probabilities of success are
not necessarily constrained by those same types of theoretical models. If we can
build decision tables, decision tools, watch lists or dashboards, on top of
solid theoretical models, then we know that the information that we are
providing cannot necessarily suppose to be right or wrong answers, because to
your point, these are really rich nuanced environments where causation will be
difficult – and frankly, I would question the point from just a pure practical
perspective. What I want to be able to do is to maximize probabilities of
success. And if we’re able to use data to do that type of thing I think we will
be doing quite a service for our colleagues and our students.
REBEKAH: Absolutely.
CHRISTINE: How do you think – I mean it’s a
little difficult – but if you had to wave your magic wand and think you were
three years from now, what would you hope would have happened in your world
that would really be a very exciting milestone in the context of big data and
how it is helping students today?
ELLEN: I’ve been struck by my own realization
of a fundamental change in the technology, sort of, ecosystem around us. And
it’s the shift from a focus on infrastructure to exostructure. And what I mean
by that is that the world of big data is so big that I think for many of us we
don’t even consider just how truly pervasive our engagement with data
collecting and transmitting devices is currently and how much more it is going
to be when we start looking beyond our own use of data for decision making and
start getting involved in, say, the Internet of Things… where my house is
completely connected to my devices and completely connected to my services and
there’s an entire layer of my life which is being completely managed by virtue
of data collected about how I want things to operate. It is really exciting,
but when I think where we’re headed, that’s pretty amazing to me.
So, you know, my vision
for the next three years is personally to be looking at continuing work in an
area that for me is like a layer between all of the wonderful service providers
that are going to have products that they’ve brought to market that are really
going to help us do exactly the type of data sampling that I think many of us
imagine in our future – where tools have been developed so that we can pretty
much figure out how to make the best decisions about everything that we think
about. That’s pretty amazing.
But I want to be able to
have a place where those of us who really don’t know how to yet navigate
through this amazing array of opportunities will have a way to help make some
decisions. So, for me personally, I think we’re all going to have to get a lot
smarter about what the options are for our data choices and what we want to be
able to do with those because I think it’s really going to tax our imagination.
And I’m hoping to continue working in a place where being able to connect big
ideas with big technologies and facilitate conversations where people can
figure how to make better choices about what they need to achieve their dream.
(CHRISTINE: Right.) That’s pretty
exciting to me.
CHRISTINE: Yes. Maybe this is a good way to shift
into data as a ‘fear factor’ and Rebekah, you had some really interesting
comments about that so I’m going to put it over to you.
Data as a Fear Factor
REBEKAH: Yeah, in listening to you both, my
sights were sort of shifted to learning in the workplace, and the overwhelming
buzz – that’s really, really evident in education, but elsewhere also, and
probably will grow even more so – is about privacy and ownership rights and all
that stuff… it makes me wonder how powerful Jane Hart's, what she calls the,
'Learning Police' force may become sooner than later. As you know, Ellen, I'm
very excited about the potential of digital badges and alternative assessments that
new technologies have enabled just recently.
My hope is that we can
leverage and harness those 'big data' tools and techniques to liberate
education at all levels. I guess that my 'fear' is that self-managed learning
may face an unnecessarily long and uphill battle, like what we – as virtual
warriors – survived in the early days of distance education. In my experience,
fear is what causes the break downs in communication among stakeholders. Aung
San Suu Kyi noted that the "most insidious form of fear is that which
masquerades as common sense or even wisdom". What can 'normal' human
beings – everyday technology users – do to tip the scales in the direction that
you feel is right for us each on a global scale, Ellen?
ELLEN: I think we owe it to ourselves to look
at this notion of fear and to try to break it down into something which is a
little bit more concrete and operational. You know, the thing about fear is
that if you can’t keep your arms around pieces of it, there’s absolutely no way
to respond. There’s no faster way I know of to be able to help move from a
position of fear into a position of action is to giving people a path to get
there I guess. We’re talking a lot about pathways; pathways have been on my
mind. But in any event, I guess for me, being able to articulate the specific
aspect of what one fears would be a great place to begin, because as soon as it
can become operational it is easier to create a response. And with a response,
the likelihood of finding a solution to move past that point is just so much
more greatly enhanced.
CHRISTINE: I think also though that one of the
challenges is that the bottom line is that security – in the context of
technical security, which of course you know, the weakest link in the chain is
99% of the time human beings (ELLEN: uh
hum) – and with the Internet of Things beginning to peak its head above the
horizon and get people very excited, it is immensely exciting, but it is also
very daunting... daunting in the context of the desperate need to really beef
up security, because you know, if you’ve got 600 million fridges having a
denial of service attack all at the same time, it might be rather problematic.
(ELLEN: Oh yeah.) All of a sudden the
turkeys won’t come out of the fridge!
ELLEN: Well, that’s right Christine. So we
start looking at network security, data security and what that means in terms
of our own shifting views about what we will tolerate in terms of violation of
privacy in exchange for the security that we all claim that we want to have.
CHRISTINE: Ellen, this is where I think the role
of universities is very important because research, innovation, different views
of how to tackle those challenges… they’re immensely important and they also
have a tremendous effect going forward on changing the curriculums in our
schools. And that always takes quite a long time, but I think actually needing
to go faster these days than it’s ever done before, because change is so
prevalent.
ELLEN: I think we also should keep in mind
that the current credentialing systems have been in place for very specific
reasons of licensure, if you will, or of demonstration of completion of
experiences where people can make broad generalizations about everything from
goal-setting to basic math and English skills. I think for all of us, making
sure that we can separate the paths to education and completion and paths to
learning are diverse.
You know, Rebekah
mentioned interest in programs of badges as personalized learning or of ways to
be able to manage one’s own learning experience and her concern or her fear
that self-managed learning is going to take the long and winding road, and I
think that she’s smart to be worrying about that, because until there is some
opportunity to show either value or comparability, I think we are going to find
that from a pure credential standpoint – which is where the money matters –
it’s going to be very hard to break away from existing systems – until there
are better ways that are already in place that are shown to drive value. So, I
don’t know that I have an answer for that, but I do have a sense that it will
be very hard to change certain education structures without being able to
change a lot of the ways in which we take care of the entire education
ecosystem. And that seems all as daunting to me…
I’m looking for those
creative disruptors that are going to be able to leap-frog over all of those
incremental steps that so many of us find ourselves needing to take, frankly.
So if there’s anybody out there who’s coming up with those great big, sort of,
incremental leap ideas, I would love to hear about them.
REBEKAH: I think that’s the whole idea behind
the creative disturbance platform – and I’m thrilled that you’re on OUR
program, quite selfishly! I quote you quite often Dr Wagner. We have similar
backgrounds and you just have a way with the words and memorably characterize
both the challenges and the triumphs of technology advances. It really sticks
with me and continues to guide my work in many regards, so I look forward to
learning more from you.
Data as Carrots (v Sticks)
Personally, I have always
appreciated your standing up for the 'little' guys, like me as faculty. It
really helps to have a champion and to understand what’s going on in the world
around us when we’re in our own little silos doing what we’ve done all the
time, and just trying to keep up with all of the changes. Could you just
explain generally, for our listeners (which is a very broad audience), how you
hope data about faculty in particular might be used as 'carrots' as opposed to 'sticks',
as you suggested? I think that would be a really positive way to enter a
conversation in other areas and for people to find ideas or look for ways that
they can leverage big data in their own particular contexts.
ELLEN: I can’t think of another group that
would be more interested in figuring out strategies to do well by their
students than college and university faculty. It’s the natural place for us to
want to think about ways that we can get better in responding to what our
students want, what works for them. And frankly, because for so many of us in
faculty life, the opportunity to get a big picture view on how to facilitate
learning or how to do a better job as a teacher or professor or specialist. I
think what will help all of us is for us to reframe the opportunity where we
can find data that allow us to move ahead with the idea of being better
evidence-based decision makers. There will be opportunities for us in our
practice to look at the way we always done things and to think about how to extract
meaning from what we’ve been doing in some type of systematic, empirical way.
Does it have to be numbers? Well, not necessarily. Are there other ways of
capturing meaning? Well, sure. So, the idea of ‘carrots versus sticks’ is that
being able to think about ways that the data can motivate, what it is that one
can be doing better or differently, just seems like a natural for people who,
obviously in faculty life most of us have cared a lot about our grades. We all
want to continue to get ‘A’s. Well, why not use tools to help us do that?
What I think we also must
keep in mind is that it is going to be almost impossible for us to not have to
face the probability of data systems on faculty very similar to the kinds of
data systems that many of us are finding ourselves wanting to have on our
students. So this is just going to be something that those of us who are
working in online settings may find that some of our behaviors are going to be
tracked as well. And, that, you might want to keep that in mind. I just want to
share a quick little real-life story of how this type of two-way feedback
works. Many people listening may be familiar with a taxi alternative service
called Uber, which is an app-based taxi car service application which is
getting a lot of attention these days as being one of the more disruptive
companies and really just about to become one of the most highly valued
companies in quite a while. But what’s interesting about Uber is that, when you
leave your taxi ride, you’re supposed to rate your driver. Well what many of us
didn’t realize is that the drivers also rate us. So here we have a system where
people who ride in these taxis, may or may not find themselves being picked up
as quickly as they have been known by their taxi drivers to be very, very
difficult passengers.
I mention this just as an
example that I think we’re all going to have to be prepared for the fact that
data go two ways. And, so, we might want to be thinking about what we can do to
motivate ourselves before somebody might come along and use those same measures
to ask us some very pointed questions.
REBEKAH: Well, I didn’t know about that! I’ll
be nicer on my next Uber ride. (ELLEN: Isn’t
that crazy?!) These are certainly interesting times. And, you know, turn about
fair play. I welcome the feedback as a faculty member, but it scares me because
it’s hard for me to keep up. It’s hard for me to serve the students, but I see
this all as a great way to be able to do that with more knowledge, more
information, more understanding of what’s going on in those interactions.
CLOSING
So, I think it’s really
fantastic and I’m really thrilled that we have people, like Christine, on our
campus – at UT Dallas – and working outside of that with the technology
providers, the people who make these connections possible over the larger pipe,
ways to look at the new data that we’ve never even thought of as data before
that we’ve never been able to extract (the semantics and all that)… It’s just
amazingly fantastic and I’m glad I’m still in the game and have the input of
you and Christine to help guide me in that role. It’s really, really a
fantastically wonderful and very interesting time.
CHRISTINE: So I think that it’s a moment where,
you know, I can’t even think for pause. If you pause, you know there’s going to
be another deluge on you. So perhaps, this is a moment to say we’ll come
together again. We’ll take the dream in another direction. Thanks a lot Ellen.
ELLEN: You’re very welcome. Thank you both.
REBEKAH: Thanks for listening today.
CHRISTINE: You can find out more at vTapestry.com.
BOTH: Bye, for now!
(vTap beat and beeps)