I have four questions for anyone engaged in (or interested in) science outreach activities — and you can define that term as broadly as you like. If you can answer any, or all, of these questions, I’d be very grateful. Just plug your answers into the comments section below. I’m hoping to use your answers to better organize my own thoughts on issues related to assessing the outcomes of outreach activities.
For Those Engaged in Outreach
- If you are part of a science outreach project, what are the goals of the project? Please be as specific as possible.
- What sort of questions are you asking (or could you ask) to help determine whether you are meeting those goals?
- What, if anything, are you doing to assess progress toward those goals?
For Everyone
- Which research disciplines have the requisite skillsets/interests to answer these questions?
Ultimately, I’d like to do two things. First, I’d like to identify existing bodies of research that address some of these issues and see how/if we can help incorporate relevant findings into the development of future outreach efforts. Second, I’d like to facilitate connections between researchers and those conducting outreach to develop and implement methodologies for collecting and assessing data on which outreach methods are most effective at accomplishing specific outreach goals — as I discussed in an earlier post.
I’m asking these questions because moving from the general idea of “quantifying outreach outcomes” to actual, specific actions requires more information. I’ve reached out to folks in several fields already, and they’ve all told me the same thing: “What is it specifically you want to know?” My questions were too broad. I’m hoping you can help me come up with some of those specific questions. (And, in some cases, I think the answers are already out there.)
Science Communication Breakdown 
1. Have been part of regional science outreach through Northwest Science Writers Association – planning events for public for more than 5 years as volunteer board member. Goals were to a. improve science literacy b. provide network for communicators where they could improve their own craft.
2. Questions we might ask: How has a specific event or series improved your “reach” as a communicator or made your materials better? (Big problem defining better)
3. Our membership has grown quickly – we assume increase in members is evidence of some success. Other metrics would be helpful.
LikeLike
Thanks, Sally. And this is actually a great example, because it raises a number of questions —
* What can be done to determine whether these events have expanded the “reach” of the attending communicators?
* How do you define reach?
* What discipline(s) would be interested in looking at something like that? Marketing? Sociology? Communication?
I don’t know. But I want to know.
LikeLike
1) My lab, the RJ Dunlap Marine Conservation Program at the University of Miami (SharkTagging.com) , takes students and community members into the field with us to participate in our long-term shark population surveys and other research. “The mission of RJD is to advance ocean conservation and scientific literacy by conducting cutting edge scientific research and providing innovative and meaningful outreach opportunities for students through exhilarating hands-on research and virtual learning experiences in marine biology”. We’re hoping to teach people about the value of sharks to the marine environment with an eventual goal of support for broader conservation policies.
2) We are interested in whether exposure to our presentations and field experiences alter people’s perspectives about sharks and marine conservation.
3) Stay tuned, more to report in about a year.
4) Our department (the Abess Center for Ecosystem Science and Policy) is interdisciplinary, so we have social science colleagues with offices right next to ours.
LikeLike
This is going to be long, but it’s because I wear a lot of outreach hats.
Wearing my citizen science project leader hat…
1. To extend and enhance informal science learning and democratize access to taxonomic expertise.
2. Our project ( http://nsf.gov/awardsearch/showAward?AWD_ID=1223210 ) will assess science learning outcomes in adult citizen science event participants in relation to two questions: 1) How do variations in the experiences of adult volunteer participants in DNA-assisted citizen science events impact informal science learning outcomes? 2) To what extent does DNA barcoding have the potential to help overcome the specimen identification bottleneck in citizen science projects?
3. Participants will be divided into three different tracks to enable comparative analysis of the impact on different experiences of on informal science learning outcomes. These tracks represent the different approaches to implementing citizen science projects involving species identification and we expect that the differences in learning outcomes between the tracks reflect the extent to which these different approaches affect learning outcomes. To assess participant learning outcomes in relation to these tracks, we will observe participants during the citizen science events, conduct semi-structured interviews, administer questionnaires and carry out follow-up communications using DEVISE measures to assess changes in behavior and attitude, which correspond to the broadly defined LSIE Strands 5 and 6, respectively. We will also assess more specific learning outcomes in relation to Strand 2 using DEVISE measures of awareness, knowledge, and understanding of the particular STEM content in this project. We expect that the results of these analysis will reveal which factors in a DNA-assisted citizen science project might extend and enhance – or detract from – different kinds of informal science learning, thereby improving our understanding of how DNA- assisted citizen science impacts informal science learning of adult volunteers as we begin to design and implement a full-scale BioTrails project. In relation to Design Question 2, we hope to better understand how incorporating DNA barcoding into citizen science events can lead to the practical benefits we predict through hands-on experience. Specifically, we hope to learn whether these benefits have the potential to remove or reduce barriers to participation in scientific research while producing data of adequate quality and volume to enable increases in research capacity.
Here is a download link to the project proposal: http://dl.dropbox.com/u/2500037/NSF%20ISE%20Proposal/James%20NSF%20abridged%20for%20sharing.pdf
Wearing my institutional outreach hat ( http://www.mdibl.org/ )…
1. To increase name recognition of our non-profit research institution on local, regional and national levels. To increase the public and professional profile of our scientists (the credibility of both institution and investigator are an important aspect of getting grants). To create and maximize fundraising opportunities.
2. Is name recognition improving at the various levels? By how much and because of which outreach efforts? Do the outreach activities of our scientists themselves have a positive effect on our fundraising goals and/or their ability to win grants? What is the ROI in our various outreach efforts in terms of annual donations?
3. We want to do something but aren’t sure what.
Wearing my tweets and (sometimes) blogs about science hat…
1.To put a human face on science, to share my enthusiasm for science and the natural world, to explain scientific concepts and results, to advocate for science (in our lives, in our societies)
2. Have my followers’/readers’ knowledge, opinions or intentions changed as a result of my tweets/blog posts? In what ways?
3. Nothing.
Wearing my co-founder and director of a charity ( http://www.hmsbeagleproject.org/ ) that does science outreach hat…
1. Through ship-board experiences: To provide a real-life platform for science, formal science education and informal science education, and an compelling online platform for internet outreach. Through social media, exhibitions and events: To inspire, inform and engage about climate change, biodiversity, exploration and history of science.
2. Same as above, have our audiences and participants knowledge, opinions or intentions changed as a result their experiences?
3. It’s still uncertain but we will probably use standard formative and summative evaluation instruments to assess learning outcomes.
LikeLike
(1) I’m Sama, founder and host of Carry the One Radio. Our show features short interviews with scientists. We talk about their most exciting projects and then close with a biographical “what got you into science”. We have three goals: (1) bridge the gap between scientists and the lay public by making science accessible and understandable to the public, (2) excite high school students about research, and (3) humanize scientists.
(2) We ask our listeners to chime in, to rate us, to review us, and to let us know how we can meet our goals better.
(3) We assess progress in a few ways. Our group has grown dramatically in a short time and we have gotten nothing but positive reviews from listeners and guests. This is a major morale boost, but does not speak directly to our goals. But because we are on facebook, twitter, and itunes – we have access to statistics of who is tapping into our radio show. We are also collaborating with high schools to build lesson plans around some of our episodes and the feedback from students has been fantastic.
(4) Our group is mostly PhD students at UCSF. We are all passionate about science and want to spread the news. I think having actual scientists work on this project is a major plus.
hope this helps!
Sama
@carrythe1radio
@ColumboAhmed
LikeLike
Hi, I’m a grad student at Arizona State University, and collaborator on an E/PO project funded through NASA, tied to an archival Hubble Space Telescope science project.
1. The goals of our outreach project are specifically to bring the general concepts of extra-galactic astronomy and the capabilities of the Hubble Space Telescope (and the future James Webb Space Telescope) to K-12 audience in under-served audiences in Arizona. Specifically we are targeting home-schooled students and Title-I school students in the Phoenix metro area, as well as First Nations school students throughout central and northern Arizona. We would like to see wider recognition of and familiarity with the kinds of science HST/JWST can do, which we believe/hope will lead to broader public support for these large-scale government-funded projects.
2.-3. These are sort of inter-related questions, so I’ll answer them together. We are working with a professional science education evaluator, who has helped us to design a variety of techniques for evaluating the effectiveness of our project in reaching the above goals. We need a variety of techniques because we are deploying our project in both informal and formal education settings. In informal education settings, such as museums and after-school community events, we are able to give questionnaires to parents. These questionnaires ask questions which assess attitudes towards space exploration and science funding. In formal education settings, such as classrooms, we can assess our impact on general knowledge and familiarity by asking teachers to give general exercises to kids in advance and afterwards— such as drawing a galaxy. These allow us to see what concepts are appearing after our visit that weren’t present beforehand (ie. galaxies are made of stars, galaxies are BIGGER than solar systems rather than the reverse, HST is in space, galaxies collide quite frequently, etc.) It’s much trickier to evaluate the effectiveness of the presentations in a informal setting… we can compare the output of the hands-on activities from groups of kids who haven’t yet seen the planetarium show with the output from kids who already have. But we’re still trying to figure out how collect this data consistently and ANONYMOUSLY, in a way that our evaluator can use. We are not collecting longitudinal data on the kids themselves, or producing publishable research. (We have to state this when interacting with the Internal Review Boards at both the university and at the schools and museums. If we were doing any of that, we would be doing “human subjects research”, and things would get a lot more involved.)
General question #1: Our science evaluator is a trained Education PhD. We didn’t connect with her through the Arizona State University department of Education… but that department does provide evaluation services for outreach and education projects, and most projects that are similar to ours do go through them. All of the proposals I’ve seen with E/PO components now have to have a concrete evaluation plan, and are expected to put aside at least 5-10% of their E/PO budget for evaluation.
LikeLike
Holy cow – you guys are engaged in some really awesome projects! I’ve got a few questions now, and will likely have more soon —
1). David: to the extent you can answer at this point (given pending results), what disciplines do the social scientists you’re working with represent? Which aspects of outreach assessment are the respective disciplines addressing? This is a big question for me, and reflects my often fuzzy understanding of the various social sciences, where they overlap, etc.
2)Sama – that sounds like an outstanding project. Are you focused on specific scientific subject areas/topics? Or are you taking a broad definition in re: making “science” accessible? If the broad route, does that make it harder to gauge progress? Very curious about this.
3). Jackie – the NASA program sounds, well, stellar. I understand the rigamarole that comes with “human subjects” testing, but it’s a shame the findings won’t be publishable. Will you be doing a public, qualitative assessment of the project? I think others may be able to learn from what works/doesn’t work with your efforts.
4). Karen – when do you sleep? I’m really excited about your NSF-funded project. There’s so much there that I’m curious about. Also – let’s correspond via email re: your institutional outreach hat. I have some questions/ideas about that (and you’re off to a very good start).
LikeLike
Believe it or not I usually get 8hrs of sleep, but I almost never blog anymore. Priorities. 😉 Please do email with your ideas and topics.
LikeLike
Prelude: My research is on the ATLAS experiment at the LHC. I am on the faculty at Duke University. I have undertaken several outreach initiatives and am currently the US ATLAS outreach/education Coordinator. Here I give one example (of a few ongoing projects) that I believe is particularly useful.
1. I have given several public lectures on the LHC and the future of particle physics, which generate a lot of interest, but often leave me thinking what their real benefit is. I’m now putting a lot more energy into actually involving others in research rather than simply talking to them about it. The LHC experiments have a great programme called Masterclasses. In mid-March I am trying this out at Duke. It involves bringing a group of high-school students to Duke (from a local HS), giving them a morning of lectures, tutorials and discussions to give them some background, then in the afternoon leading them through an actual analysis of real ATLAS data in a format that is specially designed to be more accessible. They will “re-discover” the Z boson, and even find other interesting events, to get a feeling of that eureka moment of discovery. By involving students in research (even if somewhat canned) they will experience (at least for a day) more of what it really takes to understand the data from the LHC, and ten hopefully convey that to their friends, teachers, parents, etc.. If successful, I’ll branch out to more HS’s and perhaps even invite a few reporters to give it a go !! I should note that there are several US universities implementing these LHC masterclasses, as well as international institutions.
2. The goal is to better experience the process of analysis rather than just being told results. This is intended to truly inspire students to further exploration and to begin discussions with their friends, family and mentors. We do not yet have a plan to follow up with students to see what they end up doing post-HS, but this would be a great thing to do.
3. Although not planned yet, it would be interesting to have follow-up visits to the HS(s) involved and perhaps interview the students (and teachers). Making the science we do more accessible will hopefully garner more public enthusiasm and support for it. It will take some further thought on how successful this will be.
Best,
Mark
LikeLike
@Matt: Currently, most of your episodes are neuroscience based (at least half of CTOR’s members are neuroscience PhD students). However, we want to present as many aspects of science as possible – we even hope to interview people who are working in industry, or science journalism. I don’t think this makes it any more difficult to assess our success. At the end of the day, we will have listener stats and as long as we hit our target audience – that is a success.
@Karen: you got me inspired!
LikeLike
Yes, it does seem a shame, doesn’t it? I don’t know what NASA does with the assessment information that we provide, if they use it internally or just put it in a pile somewhere. But the funded project is only $20,000 total. Several of the people involved, including our evaluator, are quite interested in seeing how the techniques work. It’s not unlikely that if there is some sign of a positive outcome, we might pursue it by applying for a larger-scale grant to test the techniques specifically, with goal of producing a publishable result.
(Additionally, it’s worth noting that going through the IRBs to do evaluation in the First Nations schools is extraordinarily difficult. During the 20th century, American academics had a very bad habit of treating First Nations citizens as research subjects minus the human subjects sensitivity. This was often perceived after the fact as extremely disrespectful and/or exploitive, e.g. many of the academics benefited tremendously from these interactions, but little to no gain was perceived in the Native American communities being studied. As a result, the modern IRBs associated with these communities are VERY sensitive and discouraging towards any hint of human subjects research.
In practical terms, this means we aren’t able to do much in the way of evaluation in these schools. Which is a tremendous handicap, it’s true. Hopefully we will have plenty of evaluation data from our other events. But more importantly, we will have the relationships we built during this exercise. Maybe more options will open up later. But even if they don’t, we still believe it’s worth pursuing these projects with these communities.)
LikeLike
Our goals are to make science on epigenetics (and sometime systems biology-though much more difficult) approachable for the lay public. We also seek to inspire school age children, and get them interested in science.
We have a small team, so measuring those goals tends to be things like count of articles published, twitter followers, # of unique visitors & views.
LikeLike
These are important questions, and the answers here are pretty awesome.
Here are my thoughts. For context, I’m involved in running science outreach at a biomedical research-oriented uni (as you know).
1) Re goals: this largely depends on your audience. When I am doing activities with high school students or middle school students, my goal is to get them to ask questions and be objective. When we hold events for adults (general/science interested), we want to engage in discussion, learn what folks are thinking, and maybe even entertain. We also have events tailored for specific adult audiences. For scientists: we do a lot of workshops/host a bunch of talks on scicomm-related topics. The goal here is to educate and maybe even inspire. For K-12 science teachers: The goal is to develop a better dialogue between local scientists and teachers, demonstrate/educate cool experiments that could be brought back to the classroom for hands on learning.
2) I am keen on learning how to asses effectiveness – what are the metrics we should use? What exactly are we measuring? Also, how do you figure out HOW to make the biggest possible impact in the context of your available resources?
3) Thinking, trial and error, talking to you!
LikeLike
I am one of the team behind OxfordSparks [https://www.oxfordsparks.co.uk], which launched nearly a year ago as a way of facilitating and extending engagement with science in general, with a focus on activities related to work across the University of Oxford.
Our main goal was to try and reach out to a ‘science-curious’ audience through the hook of some short (2 minute) fun animations on You Tube, and to develop teaching resources that link to the science behind the animations.
We have launched four animations so far (on the large hadron collider, cold chemistry, the heart and volcanoes), and have been monitoring ‘uptake’ so far using both standard metrics (views/downloads etc), and more qualitative approaches (e.g. encouraging feedback via social media platforms). Our teaching resources are being road-tested with the help of a teachers’ panel, and our hope is that the reach of the project will grow organically as we add more materials.
LikeLike
I am one of the team at the Wales Autism Research Centre at Cardiff University that is working on a ESRC (UK Govt) and Welsh Government funded project to build a “Research Policy Practice for Autism” online hub to bring these three communities closer together. The aim is to knowledge exchange between the communities and eventually to increase evidence based policy and practice in the those who work with people with autism.
I’m not sure uf this is the sort of project you’re interested in. As it is an online hub it will also be visible to the general puplic
LikeLike
You are so awesome! I don’t believe I have read through a single
thing like that before. So nice to discover someone with
some unique thoughts on this topic. Really.. thanks for starting this up.
This site is something that is needed on the web, someone with a little originality!
LikeLike