Good journalism is about starting a conversation, but journalists are generally agnostic about what kind of conversation that is. It may be a partisan, vitriol-filled, blame fest but at least people are talking about something they weren’t talking about before. What if the bar were higher? What if journalists could catalyze a more civil, substantive, productive conversation? Wouldn’t we as a society be better off? Wouldn’t trust in the media, now at an all time low, increase?

At Solutions Journalism Network, we have long hypothesized that solutions journalism can change the quality and tone of public discourse by catalyzing a conversation about possible solutions, rather than simply a contest about who is to blame. We have seen this anecdotally over the course of our work and we hear it from many of our newsroom partners. So we set out to test our theory. We wanted to know how, if at all, solutions journalism influences public discourse and if it does so differently than traditional news. We partnered with researchers at The Center for Investigative Reporting and The Engaging News Project at the University of Texas to analyze online mentions of ideas and themes surfaced in two solutions-oriented projects: The Seattle Times’ coverage of school discipline in “Education Lab” and The Chattanooga Times Free Press’s coverage of poverty in “The Poverty Puzzle.”

Spoiler alert: The studies didn’t confirm our theory (they didn’t disprove it, either). In some ways, they raised more questions than answers. One reason for that is the lack of accurate tools to gauge sentiment, but more on that later. First, what we did learn.

 

The Seattle Times’ coverage of school discipline in “Education Lab”

Education Lab (aka, Ed Lab) is a project of The Seattle Times aimed at surfacing emerging solutions to problems in public education. One issue that has received sustained coverage in Ed Lab is school discipline. We wanted to see if Ed Lab’s coverage correlated with any changes in the discourse around this topic and, if so, among who. We were particularly interested in three (potentially overlapping) groups: education stakeholders (parents, teachers and students); policymakers; and demographic groups outside of the Times’ traditional audience.screen-shot-2016-10-04-at-10-57-00-am

The study found that, while Ed Lab did not introduce issues around disciplinary practices to the political arena or public debate in Washington state—people were talking about these issues already—, it may have intensified public conversation on alternative approaches, particularly in community and ethnic media in the Seattle metro area. The Seattle Medium and The South Seattle Emerald, both geared toward African-American audiences, significantly increased their coverage of school discipline in 2015 (the year the Ed Lab stories ran) relative to 2014. The Seattle teachers’ union was also found to have increased its discussion of school discipline after the Ed Lab stories ran. Among policymakers, the correlation was less clear. Legislation to address disparities in disciplinary practices was in the works before Ed Lab’s coverage of the issue. Still, elected officials reported that Ed Lab’s coverage generated an outcry of public support for reform, which may have created the political space for the legislation to pass in 2015.

 

screen-shot-2016-10-04-at-10-52-48-am

The Chattanooga Times’ Free Press: “The Poverty Puzzle”

The city of Chattanooga has been called a model for urban renewal. But, like many cities, it struggles with very high levels of poverty. The Chattanooga Times Free Press addressed the issue in “The Poverty Puzzle,” a solutions-oriented series examining the issue of poverty and efforts to address it. The response, according to staff at the Free Press, was unlike anything they had ever seen before. Among other things, they reported a far more constructive public discourse around poverty than they typically saw. Researchers at Engaging News ran a content analysis to see what more it could tell us about the series’ impact on public discourse in Chattanooga. As with Education lab, they centered their analysis on three groups of (again, potentially overlapping) stakeholders: opinion leaders, policymakers and church leaders in Chattanooga.

Using Twitter as the primary data source, the study found that the public in Chattanooga tweeted more about poverty in the days following the publication of the series. The uptick in conversation was not, however, sustained beyond several days. There was also increased discussion about poverty among policymakers. In Chattanooga City Council meeting minutes, mentions of poverty-related terms increased more than six fold in the 12 meetings after publication of the series as compared to the 12 meetings before. Among church and opinion leaders, however, there was no discernible increase. These groups did not use more poverty-related terms in their tweets following the series (church leaders had a very limited presence on Twitter generally, though).


What can we make of these findings? They provide some support for the ability of long-form solutions journalism to affect public discussion among key stakeholder groups. But important questions remain. Did the way in which people talked about these issues change? To what extent are the increases in mentions tied specifically to the solutions orientation of the content? Unfortunately, these are questions for which there are no easy answers—or measurement tools. Sentiment analysis software attempts to assign sentiment to language, such as tweets or reader comments, but they are notoriously unreliable. Researchers at Engaging News did conduct a sentiment analysis as part of the Chattanooga study, but no meaningful conclusions could be drawn. One way to address this gap is to pair content analysis with additional research methods, like audience surveys or focus groups.

As for whether solutions journalism has the power to change public discourse, we’re pretty confident after working with nearly a hundred newsrooms all over the country that the answer is “yes.” And we’re going to keep working to prove it, because we think the best journalism starts a conversation worthy of having.