As technologies advance and the accessibility of data becomes ubiquitous, data literacy skills will likely gain increasing importance. The School of Data training resources have already laid an important foundation for social change efforts to harness data and improve their impact. Going forward, School of Data local communities will have to take into account their role as stewards of the curriculum, and continue to develop and incorporate new learnings as access to data continues to increase.

From what we (Mariel Garcia, and myself) have learned by conducting this research, we make the following recommendations:

• Training the trainers: The School of Data curriculum is the foundation for much of the Data Literacy training that is happening both inside and outside the School of Data network, as reported by interviewees; it would make sense to focus efforts on preparing materials not just for learner consumption, but also in a curriculum format for trainers.

• More research on pedagogical methods: Additional research and establishment of effective pedagogical methods of data literacy training would be beneficial – many interviewees mentioned the importance of this topic, and yet had no resources to share about it. In this regard, Peer to Peer University is the one participant that has invested most resources into this understanding, and is a great ally going forward in this area.

• More knowledge-sharing within the network: In this regard, the School of Data network also functions as a ‘community of practice’ for trainers who are sharing advice and tips on providing data literacy training, but this could be strengthened by actively promoting conversations around the topics covered in this research.

• Measuring the impact: As with different initiatives, impact evaluation is an area in which data literacy work can still grow. Both the School of Data local communities and data literacy related organisations need much stronger articulations of their long-term goals and intended impact in the short term.  School of Data events might be a good space to have the necessary conversations to find frameworks of evaluation that work for different work formats and budgets. Some organizations outside of the School of Data network (IREX and Internews) have worked extensively on this, and could be good references going forward.

• Promoting long term engagements: It appeared during the research that only older and established organisations had started long term projects and engagements related to data literacy. Consequently, it might make sense for School of Data to help smaller and newer organisations within its community to start and sustain long term engagements, by helping them find the necessary resources. This could provide an important focal point for collaborations within the network as it will likely yield important learnings.

• Data literacy at the organisation level: Articulate how individual data literacy training can complement and support long term engagements that will lead to organisational data literacy. Building local fellowship programs that can engage social change organisations over the long-term and build their capacity to utilise data in their campaigns will likely lead to deeper alliances and joint funding opportunities.

• Better collaboration with outside partners: The project would stand to benefit from more linkages and collaborations with academia, open data-related civil society efforts. Additionally, more efforts can be made to improve the accessibility of the School of Data curriculum, methodologies and trainings. This will likely lead to more diverse and sustainable funding.

The goal of this research was to empower the School of Data Steering Committee to take strategic decisions about the programme going forward along with helping the School of Data network members build on the successes to date. We hope that in providing this research and recommendations in an accessible format, both School of data and the wider network of data literacy practicioners will benefit from it. Hopefully, these research results will complement and contribute to the School of Data’s goal of improving the impact of social change efforts through data literacy.

In our next and final blog post, we will present a list of resources and references we used during our research.

After researching the definitions of data literacy, along with the methodologies and impact of data literacy efforts, we looked into the question of business models: are there sustainable business models that can support data literacy efforts in the long term? Along with looking at how data literacy efforts can support themselves financially, we also looked for opportunities for linkages with other efforts.

No clear business model for sustainability

Many of the School of Data local communities and external organisations that provide data literacy training are using a mix of foundational funding and fee for service to sustain themselves. The organisations using this model are opportunistic in getting organisations and individuals to pay for trainings when they can, but a lack of understanding about data processes among clients is often a problem. At this point, there is not a clear ‘sustainable business model’ that would direct data literacy organisations towards longevity. To understand better the business models of established organisations working at the intersection of technology and social change, we looked at a two of them: Aspiration and Tactical Technology Collective. 

Aspiration

Aspiration is a US-based NGO that operates globally, providing a range of services to build capacity and community around technology for social change. Over the last decade, under Allen Gunn’s (Gunner) leadership, Aspiration has gained a strong reputation for delivering trainings and events that focus on strategic and tangible outcomes while strengthening communities of practice.  Aspiration has championed an approach known as participatory events, developing knowledge-sharing and leadership development methodologies that prioritize active dialog-based learning. The philosophy and design focus on maximizing interaction and peer learning while making spare use of one-to-many and several-to-many session formats such as presentations and panels. Aspiration has been able to scale their model across a range of meeting sizes and purposes, from smaller team strategy sessions and retreats to large-scale events, such as the Mozilla Festival, which brings together over 1,500 participants.

Aspiration’s services are in high demand, with clients ranging from both small civil society organisations to larger international NGO’s and foundations. They have seen a gradual reduction in reliance on grants, and now generate the majority of their funds through earned income from strategic consulting services, events, and trainings. In order to scale service delivery, program staff have all been trained in the unique skill sets required for delivering participatory events and providing strategic services within the Aspiration frame of analysis. The organization now has five full-time staff able to deliver both live events and strategic services.

Tactical Technology Collective

Fee for service work on utilising data in social change has had an increased market in the civil society sector over the last five to seven years. However, many social change organisations have been unaware of the amount of resources and effort it takes to analyse and produce outputs such as data visualisations and info-graphics. A more mature organisation with experience in utilising information and technologies in activism, Tactical Technology Collective, set up a social enterprise, Tactical Studios, to undertake data-driven projects for large-scale NGO’s. They attempted to better educate clients by engaging them in creation of design briefs and a more intentional process. Tactical Studios was marginally successful as most advocacy and activist connected organisations look for quick and low-cost solutions to their data visualisations. 

Using collaborations and linkages to improve the understanding of data needs

A hopeful example in developing the capacity of organisations to understand the amount of resources needed to utilise data is in the School of Data’s Embedded Fellowship with Global Witness. Through a six-month engagement, the fellow, Sam Leon, was able to provide data trainings at all levels of the organisation – from senior management to the front line staff. This has helped the organisation, rather than just the individuals, to improve its data literacy.  What this points to is a need to differentiate between individual data literacy and organisational data literacy. While the School of Data curriculum addresses individual data literacy, efforts like the fellowship programme, that have long term engagements succeed in building organisational capacities. Being more intentional in articulating both the difference and how they complement each other will likely lead to a greater ability to raise funds and develop deeper relationships with allies.

Other potential areas for linkages and collaboration on the School of Data Curriculum that could lead to greater sustainablity for data literacy organisations:

• Schools and universities who are interested in expanding their course offerings to better address data literacy amongst students. An opportunity for chapters is to work with local academia in adapting the School of Data curriculum to address the needs of students who will potentially be using open-data in their careers. Teachers are also in need of training on the pedagogy in regards to understanding data and it’s contexts, as opposed to understanding how to use tools. Academic grants and funding could support this adaptation.
• Civil society efforts that are working towards the release of data, particularly by governments, for use in the public domain. One area that has a strong need for greater data literacy is the open governments, transparency and accountability movements, whose area of expertise is in pressuring governments through advocacy campaigns to release data. Many do not have capacity to provide training to those who might actually use the data. In this regard, a conclusion of the International Open Data Conference in 2015 (as stated in its final report) poses the need for work to identify and embed core competencies for working with open data within existing organizational training, formal education, and informal learning programs.
• Development initiatives, particularly those that are focused on supporting an emerging private sector that will be inspired by data for use in innovation.  Access and use-ability of open data could be exploited by the private sector in ways that could expand data literacy in emerging economies.  Current development initiatives could greatly benefit from engaging with School of Data chapters and engaging with the curriculum.

In order to sustain a long term data literacy initiative, it is likely that funding will need to come from a mix of foundational funding and fee for service work through expanding the diversity of clients, collaborations and linkages. As open-data usability and access continues to improve, it will be critical that Data Literacy organisations stay on top of future trends and continue to shape their curriculum to meet the needs of the communities they aim to serve. Hopefully, funders and social change organisations will also continue to evolve in their understanding of the importance of data and the resources involved in making it useful to stakeholders. As a network, the School of Data local communities will need to share information about how they grow and evolve sustainable business models.

In our next blog post ‘Recommendations for Improving Data Literacy Efforts’ we will discuss the conclusions that we have made as a result of undertaking this research effort.

As there are a wide range of methodologies for achieving data literacy in social change efforts, there is also a range of approaches to determining effectiveness. The degree to which a data literacy practice has the capacity to measure effectiveness is largely based on that practice’s maturity level. Participants who work for older, established organizations reported devoting considerable resources to M&E, whereas most individuals and participants from smaller organizations recognized that they were very limited in their evaluation possibilities. Methodologies for evaluating efficacy of data literacy efforts are not standardised. To measure the impact of data literacy work in environments with limited resources, participants in our research focus on the following sources of information:

• Analysis of data outputs: some of the participants mentioned the relative ease to measure the impact of data work (as compared to other ICT-related initiatives) because there will be outputs that you can analyze qualitatively.

• Sentiment analysis: Data literacy trainers frequently mentioned the importance of measuring outcomes by trying to get a feel for the reactions of people before ending a workshop, particularly in processes where follow-up is unlikely.

• Having an eye for the manifestation of organizational (vs individual) change: For some trainers, the true impact of data literacy work can be seen in how data work becomes internalised in an organization’s programmes and staffing.

• Direct skills assessment: Perhaps difficult to evaluate without exams, some participants rely on self-reporting from their beneficiaries and try to compare pre and post surveys to see the impact of particular training processes.

Diversity in approach

More recently established data literacy efforts are using basic evaluation forms distributed at the end of their trainings. Data literacy trainers frequently mentioned the importance of measuring outcomes by trying to get a feel for the reactions of people in their workshops. This involves including questions about the setup and fulfilment of expectations in post-workshop surveys, but also looking for signs of independent work and pondered questions. A few of the participants consider these signs of engagement are crucial in processes where follow up isn’t likely.

Code for Africa has developed a robust set of success indicators that allow them to chart a path towards success such as data responsibilities being included in organisational job descriptions. For some practitioners, the true impact of data literacy work can be seen in organizational change. Will someone be hired to do data work in the organizations? Do senior executives value data work more and are they willing to allocate more funds for this type of work?

Some of the participants mentioned the relative ease to measure the impact of data work (as compared to other ICT-related initiatives) because there will be outputs that you can analyze qualitatively. A couple of organizations mentioned detailed analysis frameworks to measure the quality of stories in data journalism, for example – employing local data journalists who could evaluate stories from before and after the processes took place to compare the performance of beneficiaries.

Some participants rely on self-reporting from their beneficiaries (through surveys that ask questions on their level of comfort/knowledge on specific skills) – and try to compare pre and post surveys to see the impact of particular training processes.

Even though most participants had given thought to monitoring and evaluation in a way or another, few of them had developed frameworks to use before, during and after the implementation of a project or program. Many of these efforts need more opportunity to articulate what success would look like for their project, and then work backwards to understand what steps and endeavours they need to accomplish to attain that success.

Determining effective ways to measure impact

During a workshop on impact assessment provided to data literacy practitioners connected to the School of Data network in March of 2015, it was determined that the term ‘impact assessment’ may not be an appropriate term, as it implies more robust and resource intensive endeavours that is often applied when evaluating public policy. There was a strong desire for lightweight methodologies that will help them learn how to improve offerings that will deliver greater impact in the long term. They determined that the methodology should contain some basic elements, such as baselines, working with beneficiaries to establish indicators, having feedback loops, articulating clear and transparent goals, having consistency throughout their programs and taking the time to document.

While some exchange between data literacy practitioners has begun around methodologies for evaluation that leads to learning and improved projects, there needs to be continued dialogue in the School of Data network to determine effective ways of measuring impact.

In our next post, ‘Sustainable Business Models for Data Literacy Efforts’, we will explore viable models and opportunities for data literacy practitioners to fund and support their work.

After exploring how to define data literacy, we wondered about the reality of the work of data literacy advocates. Which methodology do they use and what does it look like in practice? Unsurprisingly, there is a wide range of methodologies in use across different groups. Each of them fits the available opportunities, time and resources.

Short term efforts

Workshops

A large part of the training done by the data literacy advocates surveyed take the form of workshops. Participants agreed that workshops (rather than talks) were a good way to promote practical learning, and also the short timeframe allowed for the participation of individuals and organizations with resource constraints. Some workshops can be delivered inside of conferences and events (ranging from two hours to a half-day), and their learning goals are largely limited to introducing the basic topics to participants. Some of those workshops can be multi-day, ranging from two to five days: they allow for a longer exposure to processes and provide enough of a foundation to start concrete data projects. Multi-day workshops may also be augmented with follow-up sessions designed to provide guidance and support during the life of a data project.

The following short term workshops were mentioned:

• 2 hour workshops: they are the ones that take place in conferences, and which only provide space for an introduction (but require few resources to make happen)
• Half day workshops: they are seen as good for introductions to topics, as well as spaces to do practical work. For example: Data Therapy’s workshops.
• 2 to 3-day workshops: they provide enough practice time to make it feasible to start specific projects
• 5-day workshops: two of the organizations surveyed mentioned them as great opportunities to go through entire processes (like the data pipeline) with workshop participants
• 10-day workshops: the longest workshop format that was mentioned in interviews; they provide enough space to go through entire processes as well as work on specific projects from scratch.

The Data Pipeline

In regard to the content of these workshops, they often start with what participants described as “data basics” (what is data, what is data journalism, etc). After this introduction, it is common for trainers to explain the process of working with data. Here, a recurring concept is the School of Data pipeline, as shown in the illustration. It is a pedagogical device created to show that “data wrangling takes place in several stages; in order to process data, it must be moved through a ‘pipeline’. […] Raw data must usually travel through every stage of the pipeline – sometimes going through one stage more than once” (Newman, 2012). While the data pipeline is heavily promoted and used by the School of Data network, participants outside of the network were found to use the pipeline model to describe the type of content they cover in workshops.

Beside the data pipeline-like methodologies, another specific type of exercice that came up during the research was the “reverse engineering” of data exercises: deconstructing existing examples to explain how they came to be and make them more relatable for the trainees.

Community events

Along with workshops, community events have developed as a way to have a more social component to events, while getting informal but practical help and advice.

• Data clinics: those events provide space where people can develop their data skills, ask questions related to their own data and get help with challenges they are facing in their data projects
• Data meetups: many organizations that do trainings also devote resources to hosting informal meetups where people can share learnings on their own data projects, along with getting insights into other data projects.

Datathons

Another intensive format is the Datathon, which is based on the concept of hackathons. They are often named “data expeditions”, “data dives”, “data quests” and are popular with data literacy practitioners, along with individuals who have more established data-literacy skills. Quoting a participant, “acquiring data skills requires short, intensive bursts of focus from a group, rather than the type of attention you would have during 6-8 weeks with sessions that last a couple of hours per week”.

Medium term efforts

Some of the formats do not require individuals to be removed from their workplace. The training is brought to them, either physically at their workplace, or online, allowing participation from their place of work. The online format is generally conducted over a period of several weeks or months, a few hours a week.

• 5-week newslab model: in contexts where journalists cannot leave their newsroom, trainings can be brought to the newsroom.
• 4-week training: in opposition to the hypothesis that led to the birth of data expeditions, this model relies on a relatively modest demand of time each week, and relies on the accumulation of practice over four weeks.
• 1-week workshop with follow up: when there is interest in supporting a long term process, but offline training can’t be sustained over a long continuous period, follow up sessions can extend the process.

Some interviewees mentioned paying special attention to the need of developing communities of practice with alumni, in order to provide spaces where they can continue to develop expertise and learn from each other’s experiences.

Long term efforts

On the longer end of the scale, the long term efforts correspond to immersive endeavours where an individual is placed within a data project lasting anywhere from five months to a year. This takes the form of either fellowship programs, allowing an individual to gain expertise by being placed within a data project, or a mentorship program, where an individual with data expertise is placed within a data project to help build the skills of staff while working side by side.

• Fellowship programs: tend to last from 5 months to a year. Some participants favor fellowship programs for data journalists in environments where such intensive involvement is not disruptive to the media industry. School of Data has experience in this regard, too, with its own group of School of Data fellows.
• Research processes: some participants sustained long term capacity building through a research process that they documented. For communities that must collect and analyse their own data in the face of other challenges, such as marginalisation and illiteracy, an approach of a multi-year engagement towards empowerment can be successful. An example of this can be found in the FabRiders’ blog post series What we learned from Sex Workers.
• Six-month projects: rather than doing it through workshops, data literacy training can take place in the form of involvement in specific projects with the guidance of a mentor.  

Online vs offline

Most of the aforementioned formats (except for the follow-up community) take place primarily offline regardless of the duration, but some online formats were brought up by participants – primarily by those whose native language isn’t English, and whose communities don’t have a wealth of data literacy resources in their language. Some mentioned MOOCs (one of the participants ran a five-week MOOC on data journalism – the first one ever in Portuguese, the language spoken in her country, Brazil); others, dedicated websites (one of the participants was prompted by the desire to introduce trends she admired in other contexts, while translating the resources that could aid in this adoption); webinars as an attempt to replicate brief offline trainings, and paying attention to social media content as a source of learning.

Choosing an effective format

Despite the wide range of formats that are used to help build data literacy, the selected format for an event often comes down to two factors: the availability of funding and the amount of time participants are willing to invest. In some contexts, journalists and/or activists can’t give up more than two days at a time; in others, they can give up to half a year. The least disruptive formats are chosen for each community. There is a noticeable difference in the types of actors and the engagements they will favor. Larger and older organizations favor intensive, long term processes with relatively few beneficiaries; smaller and younger organizations or individuals favor short-term trainings to reach larger audiences.

Interviewees focused on developing developing data literacy capacity in both individuals and organisations favor providing experiential, project-driven work. Often it’s about providing people with a dataset and getting them to develop a story from it; other times, it’s hands-on training on different parts of the data pipeline. Most interviewees so far have made an emphasis on the importance of providing opportunities for hands-on experience with data.  They also strive for having concrete outcomes for their trainings where participants can see immediate impact of their work with data.

Curriculum Sources

The research participants mentioned several sources of inspiration around data literacy, which can be found below:

• School of Data’s curriculum, especially as it relates to the data pipeline. It is worth noting that School of Data curriculum development was not a primary effort of the core team at the Open Knowledge Foundation; it was created organically, and with end users (rather than trainers) in mind.
• Data Advocacy Essentials, FabRiders’ consolidation of information and advice they provide in trainings and projects.
• The resources at the Global Editors Network
• Listservs for professors and teachers in digital journalism, such as the ones compiled by Melody Kramer at Poynter
• Online curricula from data journalism courses in US universities
• Knight Center’s materials
• Others: Sciences Po Media Lab’s projects on data literacy; Data Science for Social Good; the Royal Statistical Society’s introductory courses on statistics.

In our next post ‘Measuring the Impact of Data Literacy Efforts’ we will look at how data literacy practitioners measure and evaluate their methodologies and efforts.