Introduction

Vaccination campaigns are quickly turning the table against COVID-19. Massive efforts were put in place by most countries to acquire and distribute millions of doses all over the world1. Ever since their announcement in November 2020, vaccines were largely covered, described and debated by news and social media, creating a deluge of information consumed by individuals2,3,4,5.

Whereas many studies focused on the structural and dynamical features of COVID-19 knowledge flows in social media2,6,7, less well explored is the other half of news media consumption, represented by constellations of highly credible/mainstream and lowly credible/alternative journal venues8,9. Newspapers not only convey succinct information about the happening of events but can often bolster awareness about specific aspects of events10,11, e.g., bolstering the fatal consequences of statistically rare side effects of vaccines, or promote specific emotional perceptions12, e.g., painting the announcement of vaccine shortages with concern or hopefulness for the future.

Ultimately both social media and newspapers represent key components of information consumption6,13: they promote knowledge and specific perceptions about real-world events like vaccines. Exploring the semantic and emotional profiles of knowledge disseminated by such venues becomes therefore essential to reconstruct key ideas read by and influencing massive audiences12. Specifically for COVID-19 vaccines, this reconstruction is urgently needed to rethink how different news outlets, over time, structured knowledge around vaccinations that reached massive audiences8. While monitoring the presence of emotions like anger or trust in massively re-shared knowledge is important14, the main challenge to understand how these emotions can affect individuals is finding out which are the main concepts eliciting such emotions in massively read content12,15. Identifying the specific emotional and conceptual associations promoted by news media remains a crucial achievement for fighting misinformation and social manipulation9.

This manuscript adopts an interpretable natural language processing (NLP) framework of narratives centered around COVID-19 vaccines, promoted by mainstream/alternative media outlets subsequently re-shared on social media like Twitter and Facebook. We select the Italian news system as a complex yet relatively unexplored case study (cf. Stella et al.3). The current investigation adopts the recent frameworks of cognitive network science16 and forma mentis networks12 to interpret language processing and unveil the structure of knowledge embedded in news articles as syntactic/semantic networks of conceptual associations. Adopting semantic frame theory from psycholinguistics17, we reconstruct the meaning attributed to vaccines and other terms in language by looking at their semantic and emotional associations. Finding these associations operationalises a model for meaning reconstruction from text, a cognitive task where networks of conceptual associations between concepts cast meaning and emotional context to each individual concept17,18. This approach has been commonly used as content mapping through human coding18, which is clearly impractical to analyse thousands of news papers. The methodology outlined in this manuscript describes how artificial intelligence methods can automatise meaning reconstruction through cognitive networks and thus parse large volumes of texts at once.

Reconstructing the semantic and emotional frames surrounding COVID-19 vaccines is fundamental for understanding which perceptions pressured individuals taking part in vaccination campaigns. A lack of trust towards something can crucially inhibit adherence to norms promoted by institutions, slowing down vaccination enrolment. Similarly, semantic associations linking “vaccine” with “hoax” or “conspiration” could bolster conspiracy theories, altering the risk-perception of individuals and ultimately exposing them to contagion. Furthermore, recent evidence indicates that conspiratorial, underground, alternative and fake news spread at least as fast as real news19,20, contributing to promoting distorted perceptions within massive, rather than limited or peripheral, audiences5. Despite the analysis of these patterns often faces the complexity of trends and behaviours present in social media data, that might be unaffected by flickering emotions or contrasting content21, the scientific community must consider richer semantic and emotional maps of knowledge exchange to better understand how social media impact real-world behaviours.

The manuscript is organised in the following way. We first review relevant works that motivate our cognitive network approach to reconstructing vaccine perceptions in alternative and mainstream news media. Our results interestingly unveil drastic differences in the ways alternative and mainstream outlets framed COVID-19 vaccines within their news titles. We also detect structural shifts in the semantic frames of AstraZeneca and Pfizer, two specific types of COVID-19 vaccines. Our cognitive networks also highlighted the emergence of strong debates in news about the side effects of specific types of vaccines: Not all COVID-19 vaccines received the same treatment/semantic framing from alternative and mainstream news. We discuss our results in light of related literature and outline our machine learning methods at the end of the manuscript.

Figure 1
figure 1

(A) Mechanisms of data creation and key differences highlighted by them in mainstream and alternative news. Data is grouped by weeks. (A) highlights how our approach gives structure to knowledge in news. (B) Each tile \(<i, j>\) is coloured after the number of urls that have been shared i times on Facebook and j times on Twitter. Both axes are logarithmically binned. (C) Z-scores of emotions in news that contain the word “vaccin” against a neutral sample, grouped by day and smoothed with a weekly rolling average. Mainstream news about the word “vaccin” show consistent high levels of trust and anticipation, conveying hope for the vaccination campaign, and significantly less disgust. This positive leaning is not visible in alternative news. (D) Distribution of z-scores of bodies (right) and titles (left) into the emotional frame of the word “astrazenec”, divided by mainstream and alternative outlets. There is a striking difference in how titles are written by mainstream and alternative news outlets, with the latter evoking more sadness but also significantly less disgust than the former. Such a difference is not visible in the articles bodies. White filled petals represent emotions that are not significantly over- or under-represented in the corpus, if compared with a neutral baseline.

Full size image

Related works

Humans can communicate their ideas through language22, which makes it key to use linguistic data for mapping perceptions and attitudes. Both in computer science and psycholinguistics, the problem of detecting positive or negative perceptions from language is known as stance detection23. Although historically psycholinguistic focused on human coding of texts24, in the last decade several approaches from artificial intelligence in computer science have brought to accurate predictions of stances in texts via machine learning techniques like deep neural networks25,26 or sentiment analysis27. A key limitation of these approaches is their black-box nature, where the experimenter cannot easily interpret the inner structure of the model fitted from the trained data and its relationship with the specific stance under scrutiny28. To overcome this limitation, a novel stream of stance detection approaches adopts complex networks as tools for representing and visualising key aspects of text-based discourse29,30,31,32,33. Complex networks have the distinctive advantage of highlighting topological patterns of connectivity between interconnected entities, be it online users engaging in replies/mentions/re-sharing30,34 or specific hashtags or words co-occurring together within the same text31,35,36. Complex networks represent a powerful tool for enriching machine learning approach and highlighting the inner structure of a given stance promoted by online users, i.e., for understanding how ideas were associated and framed by online audiences12.

Complex networks of conceptual associations can also give stances a measurable structure, highlighting how ideas /concepts /emotions were associated and framed by specific textual narratives37,38. Among many options, associations between concepts can be reconstructed mainly in three ways: through word co-occurrences38,39, e.g., detecting adjacent words in sentences; (ii) through semantic associations35,40, e.g., words being synonyms or reminding of each other; or (iii) through syntactic relationships, e.g., words specifying the meaning of each other. Since word co-occurrences can approximate syntactic relationships once words devoid of meaning are filtered out36,41, the above methods can transform unstructured texts into networks of interconnected words/concepts. These structures can then be enriched with emotional data, identifying the key emotions42,43 or sentiment patterns44 evoked by concepts. The resulting network structure is indicative of associative knowledge and emotional signatures embedded in texts45, partially reflecting the way stances were organised in the authors’ psychology46. In fact, both the frameworks of content mapping in communication science18 and frame semantics in cognitive psychology ? indicate that the way words are linked in semantic/syntactic networks provides key insights for reconstructing the meaning attributed to words in texts by authors.

This relationship between network structure and meaning in narratives has been used for a variety of automatic tasks. Amancio and colleagues used co-occurrence networks of words to perform author identification of novels36,39. Stella and colleagues31 used hashtag co-occurrence networks to highlight the negative, hatred-inspiring stances injected by automated accounts on social discourse about the Catalan referendum in 2018. Colladon47 combined multiple network metrics of word-word co-occurrences in social discourse to identify key emotional and semantic features of news relative to brand advertisement. Ferrara and colleagues48 found that in social media, textual posts richer in positive emotions could reach larger audiences whereas a faster spreading rate was found for stances richer in negative emotional content. Radicioni and colleagues30 combined hashtag co-occurrences and online social interactions to identify discursive communities engaging in different stances of immigration. The authors identified a strong polarisation in favour and against humanitarian interventions to immigration mainly correlated with political factions, in agreement with previous social network approaches34. Teixeira and colleagues49 used semantic/syntactic networks to reconstruct stances expressed in suicide notes and found negative perceptions of concepts like “love”, distorted by suicide ideation when compared to control data. Mokryn and colleagues found that the emotional words expressed in movie reviews were predictive of the emotions inspired by those movies, further strengthening a connection between textual data and cognitive/emotional content.

Although differing in their scope and methods, the above approaches have a key common element: They all perform quantitative measurements of the semantic, syntactic and emotional content of texts. This automation makes it possible to perform stance detection through volumes of data/texts that would be intractable with human coding24.

In this work, we build upon the above past approaches by reconciling interpretable machine learning and networks of conceptual associations within the framework of textual forma mentis networks (TFMN)45. These networks perform dependency parsing—powered by recurrent neural networks50—to identify how individual words are syntactically related in sentences. Syntactic connections are also enriched with synonym relationships—indicating which words can overlap in meaning according to WordNet 3.051—and emotional data—indicating how words were positively/negatively/emotionally perceived in validated psycholinguistic experiments42,44. The resulting multi-layer, feature-rich network structure contains insights about how text authors organised their knowledge and perceptions in texts. Through the lens of cognitive network science12,16,40, discourse content, centrality and frames can all be measured via interpretable network metrics and, importantly, visualised. These aspects provide experimenters direct access into the structure of stances expressed in texts45 and also in the psychology of text authors46,49. A key advantage of TFMNs is their ability to unveil semantic frames surrounding specific concepts in discourse as network neighbourhoods of concept associations. This is a crucial methodological feature for investigating specific aspects of phenomena as complex as vaccination campaigns.

Results

Results are organised along the following timeline. We first start by providing evidence that social engagement is not enough to identify differences between mainstream and alternative sources of information relative to COVID-19 vaccines. In other words, users tend to post about alternative and mainstream news at similar rates. However, the emotional content of these two sources of news differ drastically in the way they frame the idea of vaccines. Subsequently, we focus our attention on perceptions about specific vaccines like AstraZeneca. Lastly, we outline how news media reported negative concepts related to vaccines, like “mort” (stem for the Italian for death) and “trombos” (thrombosis).

Prevalence of the discourse about vaccines on social networks

Vaccines have been extensively debated both on social and on news media. Keeping track of the popularity of the articles reshared on social media can disclose insights about the ways mainstream and alternative news media described vaccination campaigns. Along with the number of shares on Facebook and Twitter, we retrieved the cumulative number of likes for the URLs in the dataset. The news articles in the dataset have been shared 5.3 million times, and they gained 3.4 million likes (0.64 per post); the same articles have been shared 192 thousands times on Twitter, and they gained 336 thousands likes (1.74 each). Mainstream and Alternative news have been liked similarly (Mainstream: 4.9 million times on Facebook and 108 thousand times on Twitter, Alternative: 2.9 million times on Facebook and 83 thousands times on Twitter), but Alternative news on Facebook gained 1.1 likes per post on average. The average number of likes per post, however, is not enough to tell news coming from Mainstream venues from news coming from Alternative venues. A Wilcoxon test could not reject the hypothesis that the distribution of average likes per post could be the same for Mainstream and Alternative both on Facebook (p value 0.42) and Twitter (p value 0.55).

Figure 1B displays a correlation heatmap of the popularity of posts on the same set of news on Facebook and Twitter (see Methods), namely articles mentioning “vaccin” in their titles. It can be observed a positive correlation among the two measures of popularity (Pearson’s coefficient 0.38, p value \(\ll 0.001\)). The correlation vanishes for extremely popular content, which is quite rare, and for URLs unpopular on Twitter, which can have diverse outcomes on Facebook. The densest mass in the plot is in the middle, where news are shared around 14 times on Twitter and 300 to 600 times on Facebook, confirming an order of magnitude of shift between the volume of the two social networks. Further details on social media users posting activity about vaccine-related news are provided in the Supplementary Information. Roughly the same positive correlation between Facebook and Twitter resharings persisted for both mainstream (Pearson’s coefficient 0.44, p value \(\ll 0.001\)) and alternative (Pearson’s coefficient 0.41, p value \(\ll 0.001\)) news outlets. This lack of differences indicates that both mainstream and alternative news were re-shared in similar ways across social media platforms, in agreement with previous findings19. This also means that differences between mainstream and alternative news is to be searched not only within their online spread but rather in their semantic and emotional content, as investigated in the following subsection.

Differences and similarities in the narrative about vaccine in mainstream and alternative news media articles

Unable to distinguish between alternative and mainstream news, it has to be underlined that user activity on social media provides also little to no information on the narrative in which vaccines are framed. Hence, we rather focus our attention on quantifying the semantic/emotional content of news. To this end, we extract the semantic frameworks of selected concepts following a working pipeline explained in Fig. 1A and in Sect. 5 and, for starting, we explore the semantic framework of the term “vaccin”.

Figure 1C reports that different kinds of media framed the concept of “vaccin” with wildly different emotions. In fact, mainstream Italian news media expressed significantly more trust and anticipation, as well as less disgust, than expected at random (see Methods). Such richly emotional framing of COVID-19 vaccines is completely absent in alternative sources. Whereas mainstream sources consistently framed “vaccin” with mostly positive/trustful jargon for more than 50% of our sampling time window, alternative news framed the same concept as emotionless. Our results provide strong evidence that it is not social engagement but rather emotional profiling that strongly characterises sources coming from alternative and mainstream outlets.

However, Fig. 1C considers both titles and news bodies. Would these differences persist in case we focused on either of these elements? More importantly, would these differences be shared by specific types of vaccines like AstraZeneca?

In Fig. 1D we focus on the AstraZeneca vaccine. Petals without colours in Fig. 1D represent emotions that are not significantly over- or under-expressed in the corpus, if compared with a neutral baseline (see Sect. 5). We analyse the time-aggregated emotions that are expressed on this noun over the whole time window, always referring to a neutral baseline. It is also interesting to differentiate not only between mainstream and alternative sources, but also among bodies and titles of the articles, since article headlines are meant to convey information and to catch the reader’s eye in a very limited number of words. Using Plutchik’s flower to represent the eight primary emotions of the model, we find interesting differences. Both mainstream and alternative media adopt for their narratives the same emotional structure in the main bodies of their articles. Different choices were made for the titles. Alternative sources expressed way more sadness compared to the baseline, while mainstream ones adopt a more positive attitude, expressing trust and anticipation.

Perception of vaccine-related dangers: before and after March 15, 2021

We can further investigate the emotional approaches to vaccines of mainstream and alternative news media by studying the narratives before and after 15 March 2021. On this day the administration of the VaxZevria (then AstraZeneca) vaccine was temporarily suspended due to a small number of suspect side-effect thrombosis52. An important media fuss was raised over this event, since the AstraZeneca vaccine was, at that time, one of the most used vaccine in Italy on several public worker categories53 (according to the National Vaccination Guidelines valid at that time). To do so, we conduct a two-fold analysis by analysing the semantic frameworks in news titles around the words “pfizer” and “astrazenec” through TFMNs, as well as around the words “mort” (death) and “thrombos” (thrombosis).

Figure 2
figure 2

Semantic frames of tightly linked concepts around “astrazenec” in journal news titles from mainstream sources (right) and alternative sources (left), before (top) and after (bottom) the temporary suspension of 15 March 2021. We highlighted with a dot those words discussed in detail in the main text, i.e. thrombosis, reactions, death and allergies. Syntactic links between positive (negative, neutral) words are in cyan (red, grey). Syntactic links between positive and negative words are in purple. Green links indicate synonyms.

Full size image
Figure 3
figure 3

Semantic frames of tightly linked concepts around “pfizer” in journal news titles from mainstream sources (right) and alternative sources (left), before (top) and after (bottom) the temporary suspension of 15 March 2021. We highlighted with a dot those words discussed in detail in the main text, i.e. thrombosis, reactions, death and allergies. Syntactic links between positive (negative, neutral) words are in cyan (red, grey). Syntactic links between positive and negative words are in purple. Green links indicate synonyms.

Full size image

Figures 2 and 3 display the communities of tightly connected concepts surrounding, respectively, “astrazenec” and “pfizer” in their semantic frames as reconstructed from the TFMN. To highlight community structure we used the Louvain algorithm54 as in previous works45. Clustering concepts together in communities can better highlight more tightly connected and thus more semantically related concepts as debated in texts45. These network visualisations report the semantic content associated with the above two entities in the corpus of news titles. Each network visualisation compared titles from mainstream sources (right) and alternative sources (left), before and after (top and bottom) the temporary suspension of VaxZevria on March 15th 2021.

Investigating the semantic content of these networks reveals interesting insights. Before the suspension, alternative journals framed “astrazenec” with way less negative associations than “pfizer”. These journal venues concentrated negative associations like allergies, reactions and deaths mostly when reporting about Pfizer’s vaccine (cf. 3 top left). These associations are absent in the titles surrounding “astrazenec” (cf. 2 top left). This indicates that alternative journal venues produced titles giving more semantic prominence to allergic/negative reactions to the vaccines mostly when mentioning Pfizer and more rarely when mentioning “astrazenec”. This pattern is flipped when considering titles from mainstream journal venues, which feature more negative, reaction-related jargon when mentioning “astrazenec” (cf. 2 top right) rather than when talking about “pfizer” (cf. 3 top right). On top on this disparity in reporting negative side effects of different brands of vaccines, titles from mainstream journals framed Pfizer with a positive cluster of concepts that is missing from AstraZeneca’s semantic frame and relative to trust spawning from Pfizer’s approval from experts and institutions.

After the temporary suspension of AstraZeneca, the semantic frames of “astrazenec” and “pfizer” underwent some drastic changes in the TFMNs obtained from news titles. Pfizer underwent a drastic drop of network degree (− 79%) in the titles from alternative journal venues, indicating a reduced semantic richness of language surrounding “pfizer” in those titles and, consequently, a reduced semantic prominence of Pfizer’s vaccine in such titles (cf. 3 bottom left). Always within titles coming from alternative news media, the semantic community of “astrazenec” underwent a densification of negative associations. This included links with thrombosis, threat and dangerous that were not present before. Mainstream journals produced semantic communities framing “astrazenec” in similar ways before and after the temporary suspension of the vaccine. Noticeably, mainstream venues featured associations with clusters of concepts related to bureaucracy, underlining how the vaccine was under further scrutinise.

Figure 4
figure 4

(A) Fraction of news that mention “mort” (death) and “trombos” (thrombosis) in their bodies and titles over the total of mainstream news (blue) and alternative news (red), smoothed by a moving average of the last 7 days. (B) Distribution of emotions in the semantic frame of “mort” (death) in the titles of mainstream and alternative articles.

Full size image

A further analysis driven by the events of March 15 can be conducted by tracking the usage of words associated to the negative events: death and thrombosis, the main alleged side effect that was put under the spotlight. An analysis of the prevalence of these two words, as shown in Fig. 4A, suggests us that both mainstream and alternative media covered these events in a similar way. The words “mort” (death) and “trombos” (thrombosis) have a long history due to its association with the COVID-19 outbreak, especially during the peak of casualties in Fall 2020, in both mainstream and alternative news outlets. The two plots show how many of the news produced by mainstream and alternative outlets everyday included the words “mort” (death) and “trombos” (thrombosis) respectively. Daily values were averaged with a moving average of 7 days, reducing the noise and increasing the readability of the figures. Indeed, they both show a peak of the said words around or after March 15, mentioning death in almost 35% of their articles and thrombosis in the 25% (mainstream) and 30% (alternative) of their articles.

Interestingly, “thrombosis” started appearing in article bodies and titles on the eve of March 15, with analogous trends in both mainstream and alternative sources. Whereas “mort/death” in article bodies displays a relative increase of \(+\) 95% around March 15—compared to the start and end of March—in both alternative and mainstream outlets, article titles highlight a different pattern: Around March 15, titles of alternative outlets feature “death” 4 times more frequently than titles from mainstream sources. Furthermore, in titles from alternative outlets the occurrence of “death” evidently peaks on March 15, a phenomenon absent in titles from mainstream sources. These differences indicate how different sources provided different attention levels to the concept of death and COVID-19 vaccines. Further emotional analysis based on TFMNs reveals how mainstream and alternative titles framed the concept of “death” in emotionally different ways, cf. Fig. 4B. Before March 15, “mort/death” was framed with anticipation and trust by mainstream titles, perhaps due to the hope of vaccine’s effectiveness; alternative titles did not show such hopeful characterisation. After the suspension, however, mainstream titles lost the anticipation trait and also the trust decreased; meanwhile, “mort” was framed by alternative titles on vaccines as fearful. These quantitative patterns highlight how alternative outlets focused on fearful vaccine narratives after March 15, whereas mainstream sources kept trustful language when mentioning deaths and COVID-19 vaccines.

Discussion

News media shape the public opinion over many topics, including general health-related issues55,56 as well as specific aspects of the fight against the pandemic, such as vaccine hesitancy57,58. Many scientific efforts have been put into understanding how the media ecosystem can be affected by the different emotional framings of news and content. Modern journalism, that relies largely on social media interaction in order to achieve diffusion and to gain attention, is characterised by a progressively stronger emotional fingerprint59. Also in light of the frequent, heavy presence of negative emotions in disinformation pieces60, it is extremely important to carry out quantitative, evidence-based analyses of affective content in news media59,61, that could serve as an health check on the status of the media ecosystem itself, evaluating the interplay between emotional framing of news and engagement with disinformation content. Needless to say, during the last two years COVID-19 has been one of the major talking points in all kind of news media. With this regard, many studies have been conducted on both the nature of COVID-related news and the effects such news have on the audience. While many have already pointed out the burst in COVID-19 online misinformation14,62, it has also been shown that users’ online activity related to COVID-19 is strongly driven by media coverage11. Therefore, general mass media play a fundamental role in steering the unfolding of the social consequences of the pandemic: being constantly exposed to COVID-related news might impact not only the pandemic itself, modifying the people’s behaviour, but also the mental health of the audience63. This is a crucial aspect that should not be neglected: fear and anxiety can affect the general audience as well as those who are first in line in the response to the pandemic64 and, particularly, using caution relatively to the COVID-19 media coverage is advisable64. This is especially true when considering that social networks, which as we saw are an important channel to share news articles, are a powerful tool for emotional contagion, that can also happen without direct interaction between people65.

Indeed we found that, within the time window we considered, vaccines have been consistently portrayed with significantly more trust and anticipation in mainstream news, with no significantly emotional language displayed in alternative news. This general feeling towards the vaccines in alternative news was not the same reserved to the AstraZeneca vaccine for which, overall, carries significantly more sadness. This propensity for negative emotions in alternative news is reinforced as soon as we focus on the analysis around the date of 15 March 2020. This day has become a milestone in the Italian—and European—vaccination program, since the administration one of the most popular vaccines (AstraZeneca) underwent a series of disruptions due to a small number of serious side effects possibly related to the vaccine. Here, the usage of threat-related words such as death or thrombosis exploded in both mainstream and alternative media outlets, as did the emotional load, but it was in the latter where negative emotions prevailed, with a significant amount of fear dominating over the others. The analysis of the semantic neighbourhood of key concepts before and after the 15 March 2021 confirms that, after that date, there was a shift in the media’s attention from the vaccine Pfizer to AstraZeneca, that was framed under an increasingly negative light, with new associations with negative words.

Overall, we can appreciate significant differences between mainstream and alternative media sources, particularly in terms of the latter amplifying the notion of COVID-19 and deaths through fearful language in article titles. It is important to note though, that both kind of outlets insisted in presenting some topics (especially the AstraZeneca vaccine) in a strongly emotional way, with an almost always significantly higher emotional load with respect to a neutral language baseline. This is particularly true for the articles’ titles: since they have to convey a short, effective message, they are—predictably—more loaded with emotional content than the articles’ bodies, where the differences tend to taper. The media coverage of vaccines, throughout the whole pandemic, was emotionally intense; in a moment of crisis, such was the 15 March 2021, the media responded with a further diversification of the emotions, always higher than the neutral baseline. This fosters the findings presented at the beginning of this Section, where the authors highlight the strong interplay between media coverage of events and the mental health and response of the audience. Particular attention should be put both by the audience, for a well-reasoned consumption of media content, and by the media outlets for a careful choice of the narrative under which to describe sensitive matters, as are the events related to the vaccination campaign during a global pandemic. Given the proven influence of media coverage on vaccination campaigns66,67,68, conveying positive emotions, such as trust and anticipation whose combination, according to Plutchik’s model69, can be seen as an expression of hope, could potentially have positive effects. Indeed, at the present date Italy shows an above-the-average adoption rate of vaccines against COVID-19, with 81% coverage of the total population, performing better than United Kingdom (76%), United States of America (74%) and the average of the European Union (73%)70. On the other hand, an excess in optimism can also yield negative drawbacks71, proving once again how delicate is the balance in crisis communication and how important is the role of mass media in its management.

Methods

Data collection

We collected 5745 news articles about vaccines that circulated in Italy in a time window that spans from October 2020 to May 2021, along with their number of shares on social networks as Twitter and Facebook. We divided the dataset in articles coming from mainstream media sources and alternative sources. Mainstream sources are three among the most visited Italian newspapers, while alternative sources include a list of blogs, underground information and pseudo-newspapers already blacklisted by the independent fact checkers of Bufale.net (Last accessed: 02/01/2022.) as persistent spreaders of mis- and disinformation. Bufale.net is an independent collective of annotators who work in synergy as to agree whether an Italian piece of information is trustworthy or not. These annotators also produced global listings of Italian news sources that consistently share disinformation items. It must be noted, however, that none of the individual news in the dataset was verified by fact checkers nor by us. Thus, the two source categories are not indicative of true versus false contents, but rather of newspapers that have previously earned trust from the public opinion as authoritative and credible, versus unreliable and possibly partisan forms of news outlets. The full list of news sources is reported in Table 1 together with their categorisation. The news selection was initially operated through the Twitter APIs, by retrieving tweets that contained a word related to vaccines and a url from the above list. We considered to be related to vaccines keywords such as the Italian word “vaccino” itself, plus all the names of vaccines available worldwide by the time we collected data, i.e., “pfizer”, “astrazeneca”, “vaxzevria”, “moderna”, “johnson”, “sputnik” and “sinovac”. We then scraped the articles, downloading the date, the title and the text content of the news. After discarding miscast articles, cancelled articles, articles protected by a paywall, and articles that were dated before the observation period (and merely re-tweeted in the observation period), we collected in total 3447 news from mainstream sources and 2298 from alternative sources. The number of shares on Twitter for a single article was inferred by the number of tweets we downloaded. Due to the limited amount of data retrieved and the current Twitter’s policies about APIs and rate limits, we can safely assume that our procedure downloaded most, if not all, the tweets that responded to the criteria introduced above. Last, we inferred for each url the number of shares on Facebook trough the tool for monitoring social media reactions Sharescore72. Fig. 1B shows a hint of the efficacy of the data retrieval procedure, displaying a good level of correlation between number Twitter and Facebook shares.

Table 1 List of the media outlets web domains analysed.
Full size table

Emotion detection and analysis of semantic frames

The main goal of this work is the analysis of the emotional fingerprint of news articles about vaccines and related concepts. Generally speaking, this task is performed by checking the texts against a lexicon of word-emotion associations (the NRC Lexicon42), but we included into our methodology two additional steps that increase the sensitivity and the significance of the analyses.

First, we pre-processed all texts and extracted their Textual FormaMentis Networks (TFMNs)45. TFMNs are built through automatic syntactic parsing as implemented via a neural network architecture in the Stanford Parser (one hidden layer with 200 nodes, cf.73). Through a greedy cross-entropy loss minimisation, the parser is trained to identify syntactic relationships between words in sentences, e.g. in “love is weakness”, the noun “love” is syntactically specified by its linked noun “weakness”. Parsing represents a sentence as a tree, where a root word is subsequently specified by syntactic dependencies.45 used such automatic syntactic parser to reconstruct the structure of associative knowledge embedded in texts as a TFMN. In these networks built out of texts, processing sentence by sentence, nodes represent words and are linked either syntactically (if at distance leqT on the syntactic dependency tree extracted by the AI73) or semantically (if synonyms according to WordNet51). Tuning T to lower values selects local syntactic relationships. In this work, we used \(T=4\). Using syntactic parsing rather than the often commonly used word co-occurrences36 has higher computational costs but also the advantage of identifying meaning specifications/syntactic links between words far apart in sentences45. For instance, even in a sentence as short as “Climate change is inevitable by I think it is also threatening”, word co-occurrences would not capture the relationship between “change” and “threatening” because they are not adjacent nor close in text. Differently from co-occurrence networks, TFMNs are also enriched with emotional data42 in the form of labels attributed to individual words, e.g. words that elicit “fear” in participants of a psychological mega-study. The two types of links and the node-level features all contained in TFMNs make the latter feature-rich multiplex network models of associative knowledge and emotional stances in texts (see also12).

TFMNs, as we can see in Fig. 1A, provide a method for determining meaningful relationships between words, allowing for a fine-grain analysis about a single concept: by extracting the neighbourhood of a word from the TFMN of a (collection of) article(s), we were able to identify the words specifically associated to our target in the texts, filtering out words that merely co-occur in the same text but that have not a direct semantic or syntactic link with the target. Newspapers’ texts can cover a wide spectrum of emotions, due to the length and the variety of subjects and facts within the same article. For instance, while talking positively about the impact of vaccines on daily casualties rates, an article could convey negative emotions about the deaths, or concerns about the future evolution of the outbreak. Overall, positive or negative framing of a concept may be diluted into the numerous traces of different emotions, thus being of primarily importance to explore semantic frameworks of words.

To do so, we extracted the emotion distribution of the words belonging to the TFMN neighbourhood of a concept, by checking them against the Italian translation of the NRCLex lexicon. Last, we compared the emotions in the semantic frame to a null model, i.e., a random selection of words and their associated emotions. We computed the z-scores of the distribution of emotions that we found in TFMNs against the emotion distribution of 300 random samples of the lexicon itself. This methodology yields a numeric score for each emotion, which is an indication of how much that emotion is under-represented or over-represented. Figures 1D and 4B have been generated using a slight modification of the visualisation library PyPlutchik74, which allows for a quantitative representation of the Plutchik’s wheel of emotions. Petals were sized after the z-score of how much an emotion has been detected in the TFMNs against a neutral baseline, and coloured only when they were at greater than 1.96 (or lower that − 1.96), making it simple to identify emotions significantly over/under-expressed. A grey shadowed ring in such plots represents the space within 1.96 standard deviations from the average, where z-scores are not statistically odd. Similarly, in Fig. 1C, we coloured only the lines representing those emotions that were below − 1.96 or above 1.96 standard deviations from the average at least 50% of the time, again emphasising odd patterns that were consistent over time.

Limitations and future work

The present work has a number of limitations that should be taken into account while interpreting the results. The main limitation lies in the number of third-party resources used throughout all the study. Particularly, the URLs of the news media articles analysed have been collected by tracking down a selection of web domains and their diffusion on Twitter. These domains were flagged by independent fact-checkers (Bufale.net, last accessed: 02/01/2022) as either mainstream sources or misinformation spreaders, as per Table 1; to collect the URLs, we resorted to the Twitter APIs. This procedure brings two inherent limitations, related to both the media outlet labeling per se, as well as how representative is the portion of tweets retrieved through the APIs. As for the former issue, we made sure that the references used to identify mainstream and alternative news media, commonly used by many other studies about Italian infodemics13,75,76, were as current as possible at the moment of the data collection. We rely on third-party annotators, professionals and experts of the (national) news media landscape that keep the lists constantly up-to-date. Although the quality of this annotation—with respect to its neutrality and its thoroughness—could vary over time, it is constantly validated by the scientific community and by the public opinion thanks to the wide diffusion and popularity of these fact-checking services9, thus mitigating the risk of introducing biases in the analysis.

The present analysis could be further extended by inspecting into greater detail the emotional fingerprint from different perspectives. An outlet-wise analysis could reveal specific patterns about the positions of the several media sources with respect to the emotional framing of the COVID-19 epidemic77 or in terms of vaccine hesitancy5. Even more interestingly, the same aspects could be assessed on the user-generated comments to the news9,12,60, in order to evaluate the extent to which the emotional content of news titles and bodies are able to influence and propagate among millions of readers.