Learning to Evaluate a Facebook Group: How Does Social Media Foster Communities of Practice?

GHKC's Guide to Monitoring and Evaluating Knowledge Management in Global Health Programs was developed to describe key components of KM activities and help measure outcomes in learning and action. This resource is part of a series of case examples developed by GHKC members highlighting ways the Guide has been used and suggestions for future editions of the Guide.

Contact: Jarret Cassaniti, Program Officer II, Knowledge for Health Project (jarret.cassaniti@jhu.edu)

AMLSN eLearning Facebook Group Header

Overview

In 2011, USAID’s Knowledge for Health (K4Health) Project started working with local partners in Nigeria to implement an e-learning and professional development policy for the country’s medical laboratory scientists. K4Health developed six eLearning courses and promoted them through several channels, including a Facebook Group that attracted 8,500 members in 18 months. By 2015, the Group had grown to 13,610 members and the Association of Medical Laboratory Scientists of Nigeria (AMLSN) assumed its full leadership and management with four volunteer administrators.

The K4Health team studied the evolution of the Group’s discussion topics and network patterns, and found membership was positively correlated to completion of the eLearning courses. Since the Group’s initial purpose was collective learning, K4Health decided to document its successful evolution using a community of practice (CoP) lens. Using the Guide’s consolidated list of indicators, they developed a set of custom indicators and 38 survey questions, which they sent to 300 of the Group’s most active members. The results of the survey provided insights on the purpose of the Group, level and type of member contributions, network strength, and linkages to changes in practice. The AMLSN anticipates using certain key results to determine the group’s relevance and future direction.

While CoPs are a focal strategy for a number of global health projects—particularly those focusing on KM— relatively little has been documented about their effectiveness and efficacy as learning and teaching tools.

Background

In 2011, USAID’s K4Health Project began work with local partners to implement an eLearning and professional development policy for medical laboratory scientists based on needs identified by USAID/Nigeria. Six eLearning courses were developed and promoted through several channels including social media. A Facebook Group was created to share information about accessing and navigating the courses and attracted 8,500 members in 18 months. As the Group grew, the topics discussed evolved to include trade union news, employment opportunities, and technical resources. By 2015, the Group had grown to 13,610 members and the AMLSN assumed its full leadership and management of the Group. Four volunteer administrators were recruited and took on responsibilities of managing the community.

Evaluating Knowledge Management Activities

During the project, monitoring was limited to checking and responding to posts for relevance, disseminating news, and encouraging member participation. As the K4Health team studied the evolution of the Group’s discussion topics and network patterns, they found membership was positively correlated to completion of the eLearning courses as documented in an article published in Knowledge Management & E-Learning. Since the Group’s initial purpose was collective learning, K4Health decided to document its successful evolution using a CoP lens.

Social media platforms are increasingly being used as online CoPs. Global health practitioners, constrained by limited training and continuing education resources and opportunities, rely on these free platforms to share knowledge and acquire new skills. While CoPs are a focal strategy for many global health projects—particularly those focusing on knowledge management (KM)— relatively little has been documented about their effectiveness and efficacy as learning and teaching tools. Though there are several frameworks and sets of indicators that are discussed in peer-reviewed literature, no global standards or approaches predominate.

Use of the Monitoring and Evaluation Guide

A year after K4Health’s involvement in the Facebook Group ended, they sought to understand how the discussion topics and level of member involvement evolved. They carried out a literature review focused on evaluating learning and exchange within online CoPs. Based on analysis of 31 articles that discuss efforts to solve problems within the healthcare arena, K4Health compiled a list of criteria that could be evaluated. These criteria were used in a variety of contexts, some within frameworks and some without. To give the findings meaning in a KM context, they aligned criteria to the Global Health Knowledge Collaborative (GHKC) KM for Global Health logic model components of inputs, processes, outputs/usefulness, and initial outcomes.

Using the Guide to monitoring and evaluation knowledge management in global health programs' (KM M&E Guide) consolidated list of indicators, they developed a set of custom indicators and 38 survey questions, which they sent to 300 of the Group’s most active members. The process component and corresponding evaluation criteria are shown here as a snapshot of the framework that was used to develop the indicators, questions, and survey.

Snapshot of the framework that was used to develop the indicators, questions, and survey.

Nigeria Facebook Survey Framework

Key findings from the survey show the majority of respondents are satisfied with usefulness of the Group and their work has benefited as a result of membership. Recommendations based on these findings include the promoting these members as CoP champions, capturing and sharing how they have benefited, and promoting greater involvement among less active members.

Nigeria Facebook Sample Interaction

Future Suggestions for the Guide

Future iterations of the KM M&E Guide should include a fuller list of the components adapted for this survey that are purposefully aligned to the GHKC KM for Global Health logic model. It should also include an entire set of custom indicators and survey questions. One indicator and corresponding question per component is listed here as illustrative examples.

Nigeria Facebook Survey Example 2

Acknowledgement

The survey development and related work was suported by Rupali J. Limaye, Sarah Fohl, and Saori Ohkubo from the Johns Hopkins Center for Communication Programs, Baltimore, MD, USA and Daniel Adedigba, Adeyeye Adetunji Tam, and Arinze Anyiam, from the Association of Medical Laboratory Scientists of Nigeria, Abuja, Nigeria.