Vuyo.dev

Heads of Government Wikipedia Page Sentiment Analysis Using R (Part 1)

Thu, 12 Feb 2026 00:00:00 +0000

Introduction

As a budding data scientist, I have been uncertain how I can develop my skills independently. While LLMs today largely allow for many researchers to perform high quality text classification and cleaning, I have struggled with thinking how I can apply these tools.

In an attempt to answer this question, below is the first of a three part series on sentiment analysis. With this series, I’ll explore the limitations of traditional methods, equip myself with the theoretical foundation of LLMs, and eventually integrate an LLM. For this, I have chosen the popular statistical computing language R.

Why choose R? Well, mostly because I wanted re-familiarise myself with the language. Secondly, I wanted to explore how the language fits into the new data analysis paradigm. I’ll further explore this in parts 2 and 3.

The subject of this sentiment analysis is one of my favourite projects, Wikipedia. I find the breadth of its volunteer network and the work they are able to accomplish to be a realisation of what the internet was imagined to be. Therefore, as I set out my journey into sentiment analysis, I thought I would start there.

This post forms the first part of a three-part series on sentiment analysis. This first part will be a naive analysis, where we will simply parse the raw text and use the NRC Emotion Lexicon to get an idea of the sentiment. I will discuss naive sentiment analysis, the data of which I made use, and finally a visualisation of my conclusions. This analysis has been developed in R, to tease out its capabilities in sentiment analysis.

The second part will be an investigation into utilising LLM with R, using the Wikipedia pages of heads of government as a case study. The final part will be a text classification of the Wipageskipedia, finding the highlights and main topics of the text.

What is Sentiment Analysis?

Sentiment analysis is the use of natural language processing and machine learning to systematically identify and classify the emotional quality (positive or negative) of a text. Here is a breakdown of this definition:

Sentiment: a view or opinion that is held or expressed
Natural language: language developed organically, for speaking, writing, and reading by humans.

There can be varied workflows for sentiment analysis, depending on what tools you leverage, but here I outline what I think is the essential workflow:

You collect text data
Once you’ve processed the data to your ideal atomic unit, attach a sentiment to each unit (the most naive approach is to separate it into individual words).
You derive analysis scores and benchmarks from the labelled data.

Sentiment Analysis in R

R’s strong statistical analysis capacity is largely supplemented by packages. During the process of implementing my project goals and copy-pasting between drafts, it was easy to lose sight of which packages were useful. Therefore, below I’ve included a list of the packages I’ve used for this sentiment analysis with some explanations on how they were useful to me.

library(readr) # for reading files
library(lubridate) # for working with dates
library(dplyr) # for data frame manipulation
library(stringr) # for working with strings
library(tibble) # having tibble (similar to dataframes) objects
library(tidytext) # this is where stop_words comes from
library(data.table) # helpful for combining data structures to data frames
library(httr2) # for web scraping

This was my first time doing my own sentiment analysis, so my only experience is with R, but I hope in later posts to make comparisons between it and other languages.

Data: Policies, Parsing, and Supplementing

For this sentiment analysis, I wanted to look at the sentiment of the Wikipedia pages of heads of government in the years 2002 and 2020. To achieve, I needed three kind of datasets: heads of government, Wikipedia text, and a word-emotion dictionary.

Two-thirds of the data I needed required some level of web scraping. Previously, I have done small scale web scraping, so I did not so much concern myself with terms of service policies. But, as I was doing this project with this post in mind, I thought let me dot my i’s and cross my t’s here.

Heads of Government

The first step in data mining was looking for heads of government; these are the objects of our analysis. Given the early development of this project, I chose a dataset that presented the most parse-able format. Using Brambor et al.’s The Heads of Government Dataset, I was able to get information on the heads of government in 33 countries in 2002 and 2020. The parsing script for this can be found in naive-parsing-hog.

I was very surprised to see that there was no unified, mostly exhaustive historical dataset of heads of governments. Most of the ones I was able to find were websites, which would require a script for scraping. Perhaps the closest I got to access an exhaustive list with a clean UI was the US’s CIA website, but they don’t allow web scraping as part of their terms of service.

Brambor et al.’s dataset was quite clean, so processing it was quite simple. However, I did spend about 30 minutes confounded by why dplyr::group_by() was not working as I expected it. My understanding of it was that it changed the order of the data. But after hopping through several articles, someone used arrange(). From there, I was able to backtrack my misunderstanding.

dplyr::group_by() groups a table into a sections, and data operations (e.g. mean for height of people, grouped by country) are performed on each group separately
Using arrange() in conjuction means that we order the rows using our previously defined group.

Wikipedia

Through Wikipedia’s RESTful API, I was able to get the HTML of the desired pages. From there, I just parsed for paragraph elements, <p>, since that was where the majority of relevant content lay.

The workflow for Wikipedia parsing was split into two functions: get_html_from_title for using a head of government’s name to get the HTML from their wikipedia page, and clean_hog_wiki for joining all the paragraphs and filtering out non-alphanumeric characters, paragraph breaks, and Wikipedia’s reference (e.g. [43]).

Following, I converted the text into a dataframe, with the distinct words of the text in one column (one per row) and the name of the head of government in the other.

With the parsing of the data, we were now ready for sentiment analysis.

Supplementing

While I did not do it at this stage, I do feel like it’s worth speaking about supplementing one’s main data source. The main goal behind sentiment analysis is to get an idea of the feelings of the public towards a particular person, event, or organisation. The reason I chose Wikipedia was because I was curious about what a naive sentiment analysis would yield.

I’ll speak more about it in the conclusions and visualisation section, but I’ll state briefly here that the most common feeling from Wikipedia articles about heads of government was “trust”. At this stage, we can hypothesise about the reasons for this. My main one is that the vocabulary of politics obviously common in pages about a political figure may, removed from context, have the trust sentiment assigned to them, since they speak of structured things.

This immediately shows the pitfalls of a naive analysis of any text: it is more the usage of the words than the words themselves that produce an emotional response. In the second part of this sentiment analysis series, we will explore how we may bridge that gap using R, traditional sentiment analysis and large language models.

Analysis

This section follows the script naive-analysis-hog, explaining the different data objects created and their intended use.

For this naive sentiment analysis, I used the NRC Emotion Lexicon for labelling words with sentiments. For ease, I used the already processed version by Language Technology and Data Analysis Laboratory, which had a shape of 13872x2, loading it with this comamnd:

nrc <- readRDS(url("https://slcladal.github.io/data/nrc.rda", "rb"))

It contains two columns, word and sentiment:

word	sentiment
abacus	trust
abandon	fear
abandon	negative
abandon	sadness
abandoned	anger
abandoned	fear

Table 1. nrc.

With hog_annotations, we now have a table that has a head of government, words found in their Wikipedia page (hereafter “text”), and the sentiment of each word. With this, we are now able to derive statistics about the sentiment. I have done it in the following ways:

The percentage of the prevalence of emotions within a text
The polarity of a text

The percentage was calculated by getting the frequency of a sentiment across text, including those that were not categorised. From this attribute, we can get a picture of what these pages are written about; we can also see the distribution. The result of this process is percentage_hog, with shape 650x5:

hog	sentiment	sentiment_freq	words	percentage
Abel_Pacheco_de_la_Espriella	anger	4	231	1.7
Abel_Pacheco_de_la_Espriella	anticipation	14	231	6.1
Abel_Pacheco_de_la_Espriella	fear	3	231	1.3
Abel_Pacheco_de_la_Espriella	joy	7	231	3
Abel_Pacheco_de_la_Espriella	negative	8	231	3.5
Abel_Pacheco_de_la_Espriella	positive	26	231	11.3

Table 2. percentage_hog.

For the polarity of a text, we used this formula for polarity:

$ \frac{# of positive mentions - # of negative mentions}{total # of mentions} $

This processed resulted in polarity_hog, with shape 66x2 (33 countries, 2 leaders each):

hog	polarity
Abel_Pacheco_de_la_Espriella	0.191011235955056
Alberto_Fernández	0.0986159169550173
Alejandro_Toledo	0.212958551691282
Anders_Fogh_Rasmussen	0.0787309048178613
Andrés_Manuel_López_Obrador	0.0672241878417498
Andrés_Pastrana_Arango	0.15

Table 3. polarity_hog.

Results

For each sentiment, we can see that “trust” was the most prevalent emotion in texts, at about 8%.

The range of polarity scores can be from $-1$ to $+1$, but there were no values above 0.4 and only two below 0 (Jair Bolsonaro at $-0.0663$ and Kostas Simitis at $-0.0163$), with the majority between $0$ and $0.25$, and a mean of $0.12$. From this naive analysis, one could suppose then that their pages are written in a neutral manner.

Moreover, to show that data analysis is a really a cycle instead of a straight line, I’ll discuss one parsing mistake that only became evident while I was checking the polarity.

Initially, I had a graph that look liked this:

Someone having a polarity of $1$ was extremely odd because it meant that they had only positive words in their Wikipedia page, which was impossible. Going back, I saw that “Carlos Alvarado” (the Costa Rican president in 2020) had only 13 words. By checking the Wikipedia link myself, I saw that the title “Carlos Alvarado” goes to a disambiguation page on Wikipedia. I had to manually modify his name to “Carlos Alvarado Quesada”, which yielded the more accurate:

Conclusion

In this blog post, I discussed a sentiment analysis of the Wikipedia pages of 66 heads of government from 33 countries for years 2002 and 2020. Being the first part of an investigation into sentiment analysis, this served as foray into the space.

It was seen that trust was the most prevalent emotion in the texts, but this can also be explained by the vocabulary of politics being associated with trust. We finally looked at the polarity of each text, and we saw that 95% of the texts were between 0 and 0.25, and the remaining 5% were less than 0.1 outside that boundary.

Despite these simple results, I found this analysis an extremely useful task. I was able to gain familiarity with R using manageable data. I have many more questions about the language now. In this analysis, I mainly used R as a sequential language. On the one hand, as I’ve experienced with other small data science project, defining a function can be more trouble than its worth. But, any project bigger than a semester of work needs a more structured approach. Thus, going forward, I may explore OOP principles within R, and see if it’s appropriate.

Part II is upcoming in March!

data-science

Heads of Government Wikipedia Page Sentiment Analysis Using R (Part 1)

Guides

Posts by Category : {{ page.title }}

{% for post in site.categories.guides %}

{% endfor %}

Get Started

## Getting Started - How to use “devlopr-jekyll” theme ## What's Jekyll ? If you aren’t familiar with Jekyll yet, you should know that it is a static site generator. It will transform your plain text into static websites and blogs. No more databases, slow loading websites, risk of being hacked…just your content. And not only that, with Jekyll you get free hosting with GitHub Pages! If you are a beginner we recommend you start with [Jekyll’s Docs](https://jekyllrb.com/docs/installation/). Now, if you know how to use Jekyll, let’s move on to using this theme in Jekyll: ## Watch Tutorial ### Steps to create your blog using devlopr-jekyll and Host using Github Pages : > **Step 1.** Fork the repo - [click here](https://github.com/sujaykundu777/devlopr-jekyll/fork) ![Devlopr Jekyll Repo](/assets/img/posts/fork1.PNG){:class="img-fluid"} > **Step 2.** Use **your-github-username.github.io** as the new repo ( Replace your-github-username with yours). Remember if you use the name other than your-github-username.github.io , your blog will be built using gh-pages branch. ![Devlopr Jekyll Repo](/assets/img/posts/fork2.PNG){:class="img-fluid"} ![Devlopr Jekyll Repo](/assets/img/posts/fork3.PNG){:class="img-fluid"} > **Step 3.** Clone the new repo locally to make changes : ![Devlopr Jekyll Repo](/assets/img/posts/fork31.PNG){:class="img-fluid"} ![Devlopr Jekyll Repo](/assets/img/posts/fork32.PNG){:class="img-fluid"} ![Devlopr Jekyll Repo](/assets/img/posts/fork33.PNG){:class="img-fluid"} ```bash $ git clone https://github.com/yourusername/yourusername.github.io $ cd yourusername.github.io $ code . ``` > **Step 4.** Open the files using VSCode and edit _config.yml and edit with your details: - _config.yml file - replace with your own details - _posts - Add your blog posts here - _includes - You can replace the contents of the files with your data. (contains widgets) - _assets/img - Add all your images here ![Devlopr Jekyll Repo](/assets/img/posts/fork34.PNG){:class="img-fluid"} > **Step 5** - Install the development requirements: ### Set up local development environment 1. [Git](https://git-scm.com/) 2. [Ruby](https://www.ruby-lang.org/) and [Bundler](https://bundler.io/) 3. [VSCode](https://code.visualstudio.com/download) We need ruby and bundler to build our site locally. After installation check if its working: For ruby : ```bash $ ruby -v ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-gnu] ``` For bundler : ```bash $ gem install bundler $ bundler -v Bundler version 2.2.29 ``` Add jekyll : ```bash $ bundle update $ bundle add jekyll ``` This command will add the Jekyll gem to our Gemfile and install it to the ./vendor/bundle/ folder. You can check the jekyll version ``` $ bundle exec jekyll -v jekyll 4.2.0 ``` > **Step 6.** Install the gem dependencies by running the following command ```bash $ bundle update $ bundle install ``` > **Step 7.** Serve the site locally by running the following command below: ```bash $ bundle exec jekyll serve --watch ``` or you can also serve using : ```bash $ jekyll serve ``` Visit [http://localhost:4000](http://localhost:4000) for development server ![Devlopr Jekyll Repo](/assets/img/posts/fork41.PNG){:class="img-fluid"} ### Adding Content Start populating your blog by adding your .md files in _posts. devlopr-jekyll already has a few examples. #### YAML Post Example: ```yml --- layout: post title: Sample Post author: Sujay Kundu date: '2019-05-21 14:35:23 +0530' category: - jekyll summary: This is the summary for the sample post thumbnail: sample.png --- Hi ! This is sample post. ``` #### YAML Page Example: ```yml --- layout: page title: Sample Page permalink: /sample-page/ --- Hi ! This is sample page. ``` #### Editing stylesheet You’ll only work with a single file to edit/add theme style: assets/css/main.scss. ### Deploy your Changes Once happy with your blog changes. Push your changes to master branch. > **Step 8.** Push Your Local Changes ```bash $ git add . $ git commit -m "my new blog using devlopr-jekyll" $ git push origin master ``` Visit your Github Repo settings ! Enable master branch as Github Pages Branch : ![Devlopr Jekyll Repo](/assets/img/posts/fork6.PNG){:class="img-fluid"} > **Step 9.** Deploy your Blog : ![Devlopr Jekyll Repo](/assets/img/posts/fork7.PNG){:class="img-fluid"} > Congrats ! On your new shining Blog ! You can visit the blog using [http://your-github-username.github.io](http://your-github-username.github.io).

Jekyll

Posts by Category : {{ page.title }}

{% for post in site.categories.jekyll %}

{% endfor %}

Guides

Posts by Category : {{ page.title }}

{% for post in site.categories.sample_category %}

{% endfor %}

Our Sponsors

Thanks to all the amazing contributors and our Backers for the support. - [Dirish Mohan](https://dirishmohan.com)

Styleguide

### devlopr - Styleguide

Lets try the different text styles Bold , Strong , Emphasis , Italic

Now, lets try different heading styles :

Hello in h1 !

Hello in h2 !

Hello in h3 !

Hello in h4 !

Hello in h5 !

Hello in h6 !

Unordered List

List Item 1
List Item 2
List Item 3
List Item 4
List Item 5

Ordered List

List Item 1
List Item 2
List Item 3
List Item 4
List Item 5

This is a Block Quote, It can Expand Multiple Lines

You can use the mark tag to highlight text.

~~This line of text is meant to be deleted text~~

This line of text will render as underlined

This line of text is meant to be treated as fine print.

This line rendered as bold text.

This line rendered as italicized text.

attr

HTML

#	Heading	Heading	Heading	Heading	Heading	Heading	Heading	Heading	Heading
1	Cell	Cell	Cell	Cell	Cell	Cell	Cell	Cell	Cell
2	Cell	Cell	Cell	Cell	Cell	Cell	Cell	Cell	Cell
3	Cell	Cell	Cell	Cell	Cell	Cell	Cell	Cell	Cell

Instagram Embed

View this post on Instagram

A post shared by Sujay (@sujaykundu777)

Twitter Embed

I just published “Deploying a blog using Jekyll and Github Pages with SSL certificate for Free” https://t.co/B3T3IQVU93
— Sujay Kundu (@SujayKundu777) June 29, 2018

YouTube Responsive Embed

Vimeo Responsive Embed

TED Responsive Embed

Twitch Responsive Embed

SoundCloud Embed

CodePen Embed

Syntax Highlighting

'use strict';
var markdown = require('markdown').markdown;
function Editor(input, preview) {
  this.update = function() {
    preview.innerHTML = markdown.toHTML(input.value);
  };
  input.editor = this;
  this.update();
}

You can add inline code just like this, E.g. .code { color: #fff; }

pre {
  background-color: #f4f4f4;
  max-width: 100%;
  overflow: auto;
}

Vuyo.dev

Heads of Government Wikipedia Page Sentiment Analysis Using R (Part 1)

Introduction

What is Sentiment Analysis?

Sentiment Analysis in R

Data: Policies, Parsing, and Supplementing

Heads of Government

Wikipedia

Supplementing

Analysis

Results

Conclusion

data-science

Heads of Government Wikipedia Page Sentiment Analysis Using R (Part 1)

Guides

Posts by Category : {{ page.title }}

Get Started

Jekyll

Posts by Category : {{ page.title }}

Guides

Posts by Category : {{ page.title }}

Our Sponsors

Styleguide

Hello in h1 !

Hello in h2 !

Hello in h3 !

Hello in h4 !

Hello in h5 !

Hello in h6 !

Instagram Embed

Twitter Embed

YouTube Responsive Embed

Vimeo Responsive Embed

TED Responsive Embed

Twitch Responsive Embed

SoundCloud Embed

CodePen Embed

Syntax Highlighting

GitHub gist Embed

Input Style