Part 1: AI-Enabled Human Rights Threats to Civil Society

Social media platforms deploy content moderation algorithms at scale to detect and remove content that violates their policies.⁸⁹ Algorithmic techniques in content moderation include hash matching “the process of transforming a known example of a piece of content into a ‘hash’ – a string of data meant to uniquely identify the underlying content” and classification, which “assesses newly uploaded content that has no corresponding previous version in a database” to categorise it and predict whether such content vmay or may not violate a platform’s policies.⁹⁰

These algorithms can have consequences for human rights and civic space when they result in over-moderation of legitimate speech, infringing on freedom of expression and access to information as well as hindering civil society’s ability to mobilise and reach audiences. In other cases, these algorithms may fail to detect content that is abusive, harassing, hateful or inciting, putting the safety and wellbeing of civil society activists and journalists at risk, and in turn, preventing them from doing their work.

Throughout the years, researchers, journalists, and activists have documented several challenges with algorithmic content moderation. First, there is the influence of socio-political factors and the Global North—particularly the U.S.– standards in the design and implementation of these systems, resulting in biases that disproportionately impact users in the Global South.⁹¹ For instance, Meta’s dangerous organisations and individuals (DOI) policy, based on which the company instructs its algorithms to remove content by “terrorist” organisations or in support of them, is highly influenced by U.S. foreign policy interests, and those affected by DOI-related removals are disproportionately of Middle Eastern, South Asian, and Muslim backgrounds.⁹²

Second, while algorithms can categorise content, they still struggle with the context in which content is posted.⁹³ For example, algorithms can accurately detect when content is graphic or should not be shown to younger users with high accuracy; however, it may still fail to recognise when violent or graphic content is posted for awareness-raising and educational purposes.⁹⁴As R0051 noted: “AI-driven content moderation and ranking systems sometimes suppress nutrition advocacy posts that mention hunger or child malnutrition, misclassifying them as ‘graphic’ or ‘sensitive’.” In the aftermath of the 7 October 2023 attacks by Hamas on northern Israel and Israel’s ensuing war on Gaza, Meta’s algorithmic content moderation systems erroneously removed videos for depicting graphic violence.⁹⁵ In another example, in 2020, Facebook and Instagram’s systems accidentally flagged posts about a big protest movement in Nigeria denouncing the controversial Special Anti-Robbery Squad (SARS) as “fake news”,⁹⁶ associating the acronym SARS with misinformation about the COVID-19 virus.⁹⁷

Third, the lack of “human in the loop” in algorithmic content moderation decisions can present challenges in human oversight, transparency, and access to effective remedy.⁹⁸ This makes it difficult for people to understand why their content was removed and appeal such decisions at a time when social media platforms have come to rely on AI in content moderation.

Another challenge is the lack of high-quality training data to train content moderation systems in low-resourced languages, many of which are spoken in the Global South. In fact, English dominates the field of AI due to the availability of digitised text data in this language to train models.⁹⁹ To address this challenge, platforms train their systems on multilingual, large language models which “infer connections between languages, allowing them to uncover patterns in higher-resourced languages and apply them to lower-resourced languages”.¹⁰⁰ These systems, however, have shortcomings, as concluded in a 2023 study by the Centre for Democracy and Technology (CDT)which found that they often rely on machine-translated text that contains errors and does not reflect local languages. They do not work well in all languages and fail to consider and reflect the contexts of local language speakers; and when problems arise, they are hard to identify and fix.¹⁰¹

The biases, mistakes, and inaccuracies in content moderation are compounded by content recommendation systems, which activists and civil society actors say negatively affect the reach and visibility of their content and expose them to harmful content. “The right-wing movement in the United States is using AI and algorithms (primarily in social media) to their advantage to spread disinformation and radicalise people as they promote hate, bigotry, misogyny, xenophobia, etc”, said a U.S.-based survey respondent who requested anonymity. Another respondent, R0035, wrote that “social media algorithms often downrank grassroots voices from regions like South Sudan, limiting the visibility of our campaigns”.

Over the past decade, research and investigative reporting have shed light on the way the algorithmic systems used by social media platforms to rank, curate, and recommend content are exacerbating the spread of hate speech, disinformation, and other harmful content. These algorithms are deployed as a part of business models aimed at generating profits from targeted advertising to the platforms and the companies that own them, determining the content users see, including ads, organic posts, recommended pages and accounts to follow, etc.¹⁰² These recommendations are made based on the data platforms collect and infer about users, including their location, language(s) they speak, age, profession, previous content they engaged with, interests, likes and dislikes, etc. This means that users are more likely to be recommended content that they would engage with so that they stay longer on the platforms, which can thus generate even more information for platforms to tailor further recommendations and targeted ads.

By prioritising engagement over quality of content, these algorithms often end up making sensationalist content,¹⁰³ including sexist, racist, and hateful rhetoric go viral. This can lead to content that undermines civic space becoming more visible for users such as misogynistic influencers and the anti-rights movement, both of which spread disinformation and hate that impacts women, feminist groups, and LGBTQIA+ people, among others. In addition, those subjected to online attacks may retreat from civic space and public life, self-censor or adopt lower profiles.¹⁰⁴ In a 2023 report exploring the negative impacts of gendered disinformation, a form of Tech-Facilitated Gender-Based Violence that is particularly deployed to target and silence women and -non-binary activists, HRDs, protesters and journalists, the UN Special Rapporteur on the promotion and protection of the right to freedom of opinion and expression highlighted the role of algorithms in spreading this type of content: “Algorithmic recommender systems play a major role in exacerbating the problem. Adversarial narratives exploit the human tendency towards negative content and disproportionately drive engagement on platforms. Algorithmic newsfeeds craft automatically generated, highly personalized adversarial content streams that keep users engaged on the platform, and monetized, and that in the end corrupt the entire global information ecosystem”.¹⁰⁵

The weaponisation of Generative AI to spread disinformation and target and discredit civil society is another threat to civic space. A Taxonomy of Human Rights Risks Connected to Generative AI, by the UN B-Tech Project, identified a number of risks that are relevant to civic space, including its use to create and spread disinformation, “inciting targeted physical violence against specific individuals or groups and non-consensual sexualised content”, and “derogatory or otherwise harmful outputs pertaining to people with marginalised identities”.¹⁰⁶

The authors also noted: “Generative AI may be leveraged to create false but convincing content presented as authentic for the purpose of misleading people. Generative AI’s capacity to create this content at scale can make online abuse campaigns easier to carry out, by both private and public actors. Such campaigns may successfully harass targets—including journalists, activists, and political actors—into self-censorship. Female public figures are especially at risk of targeted online harassment”.

One form of AI-generated content that has been particularly harmful is the spread of deepfake sexual abuse, which disproportionately targets women. This is exemplified when the Grok AI chatbot, developed by Elon Musk’s xAI, was abused on a wide scale to generate fake and nonconsensual nudes of women and children and spread them on X, the social media platform he owns.¹⁰⁷ A form of gendered disinformation,¹⁰⁸ deepfake sexual abuse, particularly targets women who are visible online, such as journalists, HRDs, feminist activists and politicians, to discredit and silence them and exclude them from participating in political and civic life.

“For women human rights defenders in Tanzania, in Uganda, the women candidates, vying for elections in Uganda, image-based abuse is the tactic that is now draining their swamp in terms of political participation”, said Irene Mwendwa, a Nairobi-based consultant on technology, AI, and law. She noted that, while moderation of text-based content moderation in some African languages has improved, “non-consensual image sharing is still very, very high, and there is very limited knowledge or understanding about pulling some of this content down”. She added, “the implications are higher, so that is what is really happening, and women human rights defenders have really tried to hold training opportunities, but also to hold some of these tech companies accountable. But, from the Big Tech side, the content moderation and public policy and human rights departments were all slashed down”.