Friday, December 12, 2014

Using Big Data to make Big - and Little - Decisions

recorded on December 12, 2014
Dr Ellen Wagner joins vTapestry to continue the conversation from Online Educa Berlin 2014 about whether or not big data is corrupting education.
Learn how big data can be used - in context - to drive evidence-based decisions at home, at play, and at work. You might be surprised at who is rating/grading you - and why!

 (vTap beat and beeps)
REBEKAH:    Big dreams?
CHRISTINE: Big data!
REBEKAH:    I'm Rebekah Nix;
CHRISTINE: I'm Christine Maxwell;
BOTH:           … and together we are vTapestry.


CHRISTINE: Rebekah and I would like to welcome Dr Ellen Wagner as both a dear friend and a very esteemed colleague. To introduce you, I think it would be helpful if I read a very short bio from your introduction at Online Educa Berlin where I believe you’ve been very recently. (Source:
Ellen D Wagner is a Chief Research and Strategy Officer for the PAR (Predictive Analytics Reporting) Framework. She served as Executive Director of the WICHE (Western Interstate Commission for Higher Education) Cooperative for Educational Technologies (WCET) from 2009 to 2013. She continues to serve as Senior Analyst with Sage Road Solutions, LLC. She was formerly Senior Director of Worldwide eLearning Solutions, Adobe Systems, Inc where she helped set the strategic direction for e-learning solutions for specialized markets, including education. Ellen previously served as Senior Director of Global Education Solutions at Macromedia Inc. Dr Wagner's prior career as a tenured university professor and administrator featured positions such as Chair of the Educational Technology Program at the University of Northern Colorado, Academic Affairs Coordinator of Instructional and Research Technologies and Director of the Western Institute for Distance Education. She was also Visiting Scholar and Project Director at the Western Cooperative for Educational Telecommunication, Western Interstate Commission on Higher Education. She holds a PhD in Educational Psychology from the University of Colorado Boulder.
Welcome Ellen.
ELLEN:           Well, thank you Christine and thank you, Rebekah. I’m really pleased to be here talking with you tonight.


REBEKAH:    Thank you for speaking with us Ellen. It's hard to believe how much has changed since you and I met at the first International Forum for Women in eLearning conference way back in 2005. I'm eager to view the archive of that more recent opening session of the 2014 Online Educa Berlin conference. My understanding is that you and Inge deWaard went up against Drs Mayer-Schonberger and Siemens to debate whether or not data is corrupting education. After a long flight home and plenty of commentary on that panel, what do you suggest leaders - in any field - need to think about in terms of 'data-as-a-meme' as you phrased it?
ELLEN:           You know I had to come up with some way that I could argue in favor of a house motion that said that data was corrupting education because when I first heard that I was quite non-plussed because I really couldn’t imagine how someone like me – who in fact is just really finding myself driven by wanting to know more about how to make better use of data – could really argue for a position like that… and so I stepped back and started reading so many of the comments of why data could be a problem for us. And what I realized is that data really aren’t the problem. The problem is that we use a term, like ‘data’ or terms like ‘big data’ as if we all know what we mean and we nod and we wink and we all sort of think about what that is, but none of us really know. What we know might look like a loss of privacy. What we know might look like people knowing more about what we’re doing online than we know about ourselves.
We fear that with that type of knowledge of what one can do with data that... profiling: whether used for good or for maybe less good means, depending on where you fall on being profiled… there just seem as if there are a number of places where it’s really not the data that we worry about, but what I do think we worry about – and where there might be potential for corruption, if you will – is when data are in the hands of people who don’t really know how to use it very well or who will use it in ways where the methodologies don’t support its use or to use it to bludgeon people into a particular position as opposed to informing them about things that they could actually do better.
So, you know, it’s a long answer to your question, but it really reminded me that (for those of us who are working with data) we need to make sure that we are focused on what we mean, that we are able to point to real examples of where we can make a difference – measureable differences – and that we open this conversation up so that everybody can see where it is that we are going so that the data can truly be realized, can really realize their full potential.
CHRISTINE: Right. The thing that that had me think of is that the challenge is that the Internet and technology is moving so fast that, you know, machine learning and the semantic web and the synaptic web are already upon us – and yet, hardly any of us – if we’ve heard of it, we certainly don’t know what that really means for us. I don’t want to get into the details of that just yet, but it does… it’s a big challenge to anyone who is involved in online research today.
ELLEN:           I’m struck, Christine, by the idea that, as you speak of things like machine learning or when we start looking at the true big unstructured web as a source of information that for so many of us, particularly those of us working in education... we deal with data that comes at us in columns and rows.  (CHRISTINE: Yep.) And our ability to work with columns-and-rows data seems to be really quite immature as I look across the ecosystem. In some places it’s getting quite a bit better, but typically in areas where we’re dealing with more concrete operational kinds of things, like finance or fundraising or places where we can deal with tangible measures of business operations.
When you start looking at using big data to make decisions about academic performance, achievement, goal-setting, outcomes, things like that, the lack of consistent and reliably shared measures of what we mean when we talk about these things is going to continue to hamper us. So, I share your enthusiasm for where we think we can go with this and sometimes I do wonder what it is that it’s going to take for all of us to really step up to that challenge when I know that for many of us, we’re still not ready to deal with the data that are right in front of our faces sometimes.
CHRISTINE: I just want to make one point about what you said which is actually very scary what you said. Because, if we’re still immature in dealing with data that’s structured, the world has been focused on structured information for a very long time, until relatively recently. And so, in some ways that’s rather a scary thought. In other ways, unstructured data which, of course, is by far and away the format of the wealth of data that’s now drowning us all... the question of how we learn to work with that is where machine learning steps up to the plate to help us. At the same time, unless one is aware of what the process is and what the potential dangers are of bias, we may be led very easily down paths that we do not expect.
ELLEN:           You know, being asked about the question or the notion of data corrupting education and really looking at the notion of corruption, which of course is such a loaded negative word, but if you think of corruption as a deviation from an ideal for personal or some type of self-gain, it’s actually kind of eye-opening to think about that because deviation from an ideal – even if it’s for someone’s gain – it might not be wrong. But it really raises the question of our own need at this point where we are looking at the data systems surrounding us where looking at it through the lens of our current times, our current technology capacities, is really going to be essential.
I think another one of the risks many of us are going to encounter will be the constraint of legislation policy or what have you that has been written for a technology world that we haven’t seen for 10 or 20 years.
REBEKAH:    With my focus on integrating technology into professional development and to make sure that the learning environments that educators create are the most positive possible, I see this new shift, the new capabilities that big data analytics and visualizations bring to the table – to the everyday user almost, now – as really exciting. And as an educator turned technologist (so to speak), you Ellen, bring a really unique perspective to this realm of big data analytics.

Data as a Student 'Tracking' Decider

As a technologist turned educator, it’s kind of funny that I was impressed with an exchange from the movie Interstellar. It illustrated one of my concerns about how student data might be impacting us already. It was set in the principal's office of a rural school in the near future, and Matthew McConaughey, as the character Cooper, said: "You're ruling out college for my son now? He's fifteen." Then the Principal replied: "Tom's score simply isn't high enough." So Cooper went on to suggest or guess that the principal's waist line was maybe a 32 with a 33 inseam to make the point that it took two numbers to figure out the Principal’s seat size (ELLEN: uh hum), but only one test score to measure his son's future... How likely is that scenario to become true, based on what you see happening today?
ELLEN:           I can see it as being a likely path for most of us, but I do think that the idea of making judgments about how student information is used is really illustrated very well in this example. We don’t want to find ourselves in situations where we are making decisions like that. My hope is that if you have information to help guide a student toward places where they are going to be more successful, I hope that we would use those data for things like, you know, saying that students like you who have found themselves interested in these kinds of things, that met these types of benchmarks, actually did pretty well in these areas. And, to not say that you can’t do something, because who among us wants to be able to say something like your example of Tom, the student in this movie, that would be denied a shot at a future just because a number showed him to be a particular way?
We have developed diagnostic tools in our PAR Framework that calculate a risk for students that is constructed from a number of variables that seem to demonstrate risk at significant levels for students like those people. Does it mean that those individual people are going to in fact demonstrate that these things are true? I mean, these are not causal kinds of things. So, I think that being smart about what the numbers are actually telling you is going to be really key to avoid these determinations that are made by some type of metrics that we don’t even know about. But I will tell you that if I look ahead to a future where every key stroke or every interaction or every exchange is going to be parsed for some type of meaning, I do have occasional moments of pause and I know that for myself I think that being mindful of some of the paths where this might play out are absolutely things that we practitioners of the current day need to be able to articulate with one another.
REBEKAH:    Yeah, I think that what you mentioned about the correlations, (ELLEN: uh hum) that’s really exciting to me because it’s been hard, I mean it’s been impossible for me as a researcher to prove causality in the things that I do. There are so many variables that are measured but can’t be related or directly linked that I know in my gut, or in my heart more often, make a difference. And in terms of the learning environment and what people bring to that in each different semester, with each different (ELLEN: uh hum) collection of people... that’s what matters and I think that’s what makes learning ‘stick’ and I’m really excited about the potential of these new tools to bring some of those correlations, those underlying themes that I never even imagined might matter, to light (ELLEN: uh hum) so that I can do my job better and support those people that the other data might identify as ‘at risk’. I think it’s really fantastic and like you said, if used in context… content is really no longer king in my world, so being able to figure out how to help people learn and do it better, faster, and cheaper, is really exciting to me.
ELLEN:           Well, you know Rebekah, one of the things I’ve been finding in our current work is that when I focus on learning it’s a harder question to answer because learning is a scientifically-determined construct of course, but you know so many of us have emotional reactions to it and so I know for us working with PAR, where we ended up saying well let’s come up with our proxies for what we would mean for learning but to do it around the questions that people say are current measures for success right now, knowing that a graduation rate is a many-splendored thing, but a completion score is still something where we can be very, very actionable as we look at it, so being as actionable as we can with data really I think can help us respond to the need both for individual students that providing better pathways for people to determine success...
I don’t think that there’s ever going to have to be a single pathway to greatness at this point looking at the multiple ways in which type engines can be delivered and the multiple ways in which programs can be provided or interventions can be wrapped around specific types of students. But what I will say is that in the very, very early days of doing this type of work, our opportunities to start becoming far more systematic about our language, our constructs, our decision-making... here’s where I’d make a big distinction for where I’m seeing the biggest excitement about the use of data right now for student success…
With data in education, we have typically done basic research where we were establishing accountability as constructs that we would then include within the cannon of our particular practices. And there’s going to continue to always be a need for people who can in fact make sure that the theoretical constructs of our discipline are examined with the type of rigor that we want them to be examined so that we know that what we’re talking about makes sense. These are foundational ideas that drive an entire practice; they have to be solid. For so much of where we want to take this information and apply it in practice, the level of rigor required to make effective decisions that maximize probabilities of success are not necessarily constrained by those same types of theoretical models. If we can build decision tables, decision tools, watch lists or dashboards, on top of solid theoretical models, then we know that the information that we are providing cannot necessarily suppose to be right or wrong answers, because to your point, these are really rich nuanced environments where causation will be difficult – and frankly, I would question the point from just a pure practical perspective. What I want to be able to do is to maximize probabilities of success. And if we’re able to use data to do that type of thing I think we will be doing quite a service for our colleagues and our students.
REBEKAH:    Absolutely.
CHRISTINE: How do you think – I mean it’s a little difficult – but if you had to wave your magic wand and think you were three years from now, what would you hope would have happened in your world that would really be a very exciting milestone in the context of big data and how it is helping students today?
ELLEN:           I’ve been struck by my own realization of a fundamental change in the technology, sort of, ecosystem around us. And it’s the shift from a focus on infrastructure to exostructure. And what I mean by that is that the world of big data is so big that I think for many of us we don’t even consider just how truly pervasive our engagement with data collecting and transmitting devices is currently and how much more it is going to be when we start looking beyond our own use of data for decision making and start getting involved in, say, the Internet of Things… where my house is completely connected to my devices and completely connected to my services and there’s an entire layer of my life which is being completely managed by virtue of data collected about how I want things to operate. It is really exciting, but when I think where we’re headed, that’s pretty amazing to me.
So, you know, my vision for the next three years is personally to be looking at continuing work in an area that for me is like a layer between all of the wonderful service providers that are going to have products that they’ve brought to market that are really going to help us do exactly the type of data sampling that I think many of us imagine in our future – where tools have been developed so that we can pretty much figure out how to make the best decisions about everything that we think about. That’s pretty amazing.
But I want to be able to have a place where those of us who really don’t know how to yet navigate through this amazing array of opportunities will have a way to help make some decisions. So, for me personally, I think we’re all going to have to get a lot smarter about what the options are for our data choices and what we want to be able to do with those because I think it’s really going to tax our imagination. And I’m hoping to continue working in a place where being able to connect big ideas with big technologies and facilitate conversations where people can figure how to make better choices about what they need to achieve their dream. (CHRISTINE: Right.) That’s pretty exciting to me.
CHRISTINE: Yes. Maybe this is a good way to shift into data as a ‘fear factor’ and Rebekah, you had some really interesting comments about that so I’m going to put it over to you.

Data as a Fear Factor

REBEKAH:    Yeah, in listening to you both, my sights were sort of shifted to learning in the workplace, and the overwhelming buzz – that’s really, really evident in education, but elsewhere also, and probably will grow even more so – is about privacy and ownership rights and all that stuff… it makes me wonder how powerful Jane Hart's, what she calls the, 'Learning Police' force may become sooner than later. As you know, Ellen, I'm very excited about the potential of digital badges and alternative assessments that new technologies have enabled just recently.
My hope is that we can leverage and harness those 'big data' tools and techniques to liberate education at all levels. I guess that my 'fear' is that self-managed learning may face an unnecessarily long and uphill battle, like what we – as virtual warriors – survived in the early days of distance education. In my experience, fear is what causes the break downs in communication among stakeholders. Aung San Suu Kyi noted that the "most insidious form of fear is that which masquerades as common sense or even wisdom". What can 'normal' human beings – everyday technology users – do to tip the scales in the direction that you feel is right for us each on a global scale, Ellen?
ELLEN:           I think we owe it to ourselves to look at this notion of fear and to try to break it down into something which is a little bit more concrete and operational. You know, the thing about fear is that if you can’t keep your arms around pieces of it, there’s absolutely no way to respond. There’s no faster way I know of to be able to help move from a position of fear into a position of action is to giving people a path to get there I guess. We’re talking a lot about pathways; pathways have been on my mind. But in any event, I guess for me, being able to articulate the specific aspect of what one fears would be a great place to begin, because as soon as it can become operational it is easier to create a response. And with a response, the likelihood of finding a solution to move past that point is just so much more greatly enhanced.
CHRISTINE: I think also though that one of the challenges is that the bottom line is that security – in the context of technical security, which of course you know, the weakest link in the chain is 99% of the time human beings (ELLEN: uh hum) – and with the Internet of Things beginning to peak its head above the horizon and get people very excited, it is immensely exciting, but it is also very daunting... daunting in the context of the desperate need to really beef up security, because you know, if you’ve got 600 million fridges having a denial of service attack all at the same time, it might be rather problematic. (ELLEN: Oh yeah.) All of a sudden the turkeys won’t come out of the fridge!
ELLEN:           Well, that’s right Christine. So we start looking at network security, data security and what that means in terms of our own shifting views about what we will tolerate in terms of violation of privacy in exchange for the security that we all claim that we want to have.
CHRISTINE: Ellen, this is where I think the role of universities is very important because research, innovation, different views of how to tackle those challenges… they’re immensely important and they also have a tremendous effect going forward on changing the curriculums in our schools. And that always takes quite a long time, but I think actually needing to go faster these days than it’s ever done before, because change is so prevalent.
ELLEN:           I think we also should keep in mind that the current credentialing systems have been in place for very specific reasons of licensure, if you will, or of demonstration of completion of experiences where people can make broad generalizations about everything from goal-setting to basic math and English skills. I think for all of us, making sure that we can separate the paths to education and completion and paths to learning are diverse.
You know, Rebekah mentioned interest in programs of badges as personalized learning or of ways to be able to manage one’s own learning experience and her concern or her fear that self-managed learning is going to take the long and winding road, and I think that she’s smart to be worrying about that, because until there is some opportunity to show either value or comparability, I think we are going to find that from a pure credential standpoint – which is where the money matters – it’s going to be very hard to break away from existing systems – until there are better ways that are already in place that are shown to drive value. So, I don’t know that I have an answer for that, but I do have a sense that it will be very hard to change certain education structures without being able to change a lot of the ways in which we take care of the entire education ecosystem. And that seems all as daunting to me…
I’m looking for those creative disruptors that are going to be able to leap-frog over all of those incremental steps that so many of us find ourselves needing to take, frankly. So if there’s anybody out there who’s coming up with those great big, sort of, incremental leap ideas, I would love to hear about them.
REBEKAH:    I think that’s the whole idea behind the creative disturbance platform – and I’m thrilled that you’re on OUR program, quite selfishly! I quote you quite often Dr Wagner. We have similar backgrounds and you just have a way with the words and memorably characterize both the challenges and the triumphs of technology advances. It really sticks with me and continues to guide my work in many regards, so I look forward to learning more from you.

Data as Carrots (v Sticks)

Personally, I have always appreciated your standing up for the 'little' guys, like me as faculty. It really helps to have a champion and to understand what’s going on in the world around us when we’re in our own little silos doing what we’ve done all the time, and just trying to keep up with all of the changes. Could you just explain generally, for our listeners (which is a very broad audience), how you hope data about faculty in particular might be used as 'carrots' as opposed to 'sticks', as you suggested? I think that would be a really positive way to enter a conversation in other areas and for people to find ideas or look for ways that they can leverage big data in their own particular contexts.
ELLEN:           I can’t think of another group that would be more interested in figuring out strategies to do well by their students than college and university faculty. It’s the natural place for us to want to think about ways that we can get better in responding to what our students want, what works for them. And frankly, because for so many of us in faculty life, the opportunity to get a big picture view on how to facilitate learning or how to do a better job as a teacher or professor or specialist. I think what will help all of us is for us to reframe the opportunity where we can find data that allow us to move ahead with the idea of being better evidence-based decision makers. There will be opportunities for us in our practice to look at the way we always done things and to think about how to extract meaning from what we’ve been doing in some type of systematic, empirical way. Does it have to be numbers? Well, not necessarily. Are there other ways of capturing meaning? Well, sure. So, the idea of ‘carrots versus sticks’ is that being able to think about ways that the data can motivate, what it is that one can be doing better or differently, just seems like a natural for people who, obviously in faculty life most of us have cared a lot about our grades. We all want to continue to get ‘A’s. Well, why not use tools to help us do that?
What I think we also must keep in mind is that it is going to be almost impossible for us to not have to face the probability of data systems on faculty very similar to the kinds of data systems that many of us are finding ourselves wanting to have on our students. So this is just going to be something that those of us who are working in online settings may find that some of our behaviors are going to be tracked as well. And, that, you might want to keep that in mind. I just want to share a quick little real-life story of how this type of two-way feedback works. Many people listening may be familiar with a taxi alternative service called Uber, which is an app-based taxi car service application which is getting a lot of attention these days as being one of the more disruptive companies and really just about to become one of the most highly valued companies in quite a while. But what’s interesting about Uber is that, when you leave your taxi ride, you’re supposed to rate your driver. Well what many of us didn’t realize is that the drivers also rate us. So here we have a system where people who ride in these taxis, may or may not find themselves being picked up as quickly as they have been known by their taxi drivers to be very, very difficult passengers.
I mention this just as an example that I think we’re all going to have to be prepared for the fact that data go two ways. And, so, we might want to be thinking about what we can do to motivate ourselves before somebody might come along and use those same measures to ask us some very pointed questions.
REBEKAH:    Well, I didn’t know about that! I’ll be nicer on my next Uber ride. (ELLEN: Isn’t that crazy?!) These are certainly interesting times. And, you know, turn about fair play. I welcome the feedback as a faculty member, but it scares me because it’s hard for me to keep up. It’s hard for me to serve the students, but I see this all as a great way to be able to do that with more knowledge, more information, more understanding of what’s going on in those interactions.


So, I think it’s really fantastic and I’m really thrilled that we have people, like Christine, on our campus – at UT Dallas – and working outside of that with the technology providers, the people who make these connections possible over the larger pipe, ways to look at the new data that we’ve never even thought of as data before that we’ve never been able to extract (the semantics and all that)… It’s just amazingly fantastic and I’m glad I’m still in the game and have the input of you and Christine to help guide me in that role. It’s really, really a fantastically wonderful and very interesting time.
CHRISTINE: So I think that it’s a moment where, you know, I can’t even think for pause. If you pause, you know there’s going to be another deluge on you. So perhaps, this is a moment to say we’ll come together again. We’ll take the dream in another direction. Thanks a lot Ellen.
ELLEN:           You’re very welcome. Thank you both.
REBEKAH:    Thanks for listening today.
CHRISTINE: You can find out more at
BOTH:           Bye, for now!
(vTap beat and beeps)