Querying Twitter API Data

System Investigation Project

Introduction

Overview: The following project aims to programmatically retrieve and analyze all of Elon Musk’s tweets from the Twitter API. We compile over 10,000 of Musk’s tweets into a comprehensive dataset and perform sentiment analysis for each tweet object.

Elon Musk @elonmusk

Mars & Cars, Chips & Dips

Joined June 2009

18.92 K	118	102.8 M
Tweets	Following	Followers

Elon Musk, who transformed the electric car industry and accelerated the world’s space exploration efforts, provides the public an unfiltered look into his eccentric mind through his online Twitter presence. With more than 77 million followers, Musk is one of the most followed Twitter accounts, with each of his tweets getting thousands of shares, likes, and comments. Musk tweets frequently every day, covering a wide range of topics from technical ideas to lighthearted memes. Some of his tweets have a significant impact, making headlines, instigating controversy, and sometimes pushing the needle on everything from Tesla’s stock price to cryptocurrency markets. The companies that Musk runs are also hugely influential and disruptive.

To better understand Musk’s Twitter profile, we programmatically retrieve and analyze all of Elon Musk’s tweets using the rtweet package, which provides a convenient interface between the Twitter API and R code. We compiled over 10,000 of his tweets into a comprehensive dataset. Once we scraped a decade worth of Elon Musk tweets, we analyzed the data and sorted the information to answer the following questions.

1. (tweet types) What is the ratio of mentions, replies, retweets, quotes, and organic tweets?
2. (user engagement) How does Elon Musk engage with other twitter users?
3. (twitter activity) Are there any trends of when Musk tweets?
4. (topics) What are Elon Musk’s most tweeted topics?

---

Data Description

Using the Twitter API, we compile over 10,000 of Musk’s tweets into a comprehensive dataset. The dataset contains all of Elon Musk’s tweets from 2015 to 2022, stored in RDS format, where each tweet is in its separate row object. All Tweets are collected, parsed, and plotted using rtweet in R.

Data Set Variables
1	status_id	14	hashtags	27	quoted_followers_count	40	retweet_location
2	created_at	15	symbols	28	quoted_location	41	retweet_description
3	user_id	16	media_expanded_url	29	quoted_description	42	retweet_verified
4	screen_name	17	media_type	30	quoted_verified	43	name
5	text	18	mentions_screen_name	31	retweet_status_id	44	location
6	source	19	quoted_status_id	32	retweet_text	45	description
7	reply_to_screen_name	20	quoted_text	33	retweet_created_at	46	followers_count
8	is_quote	21	quoted_created_at	34	retweet_source	47	friends_count
9	is_retweet	22	quoted_source	35	retweet_favorite_count	48	statuses_count
10	favorite_count	23	quoted_favorite_count	36	retweet_retweet_count	49	account_created_at
11	retweet_count	24	quoted_retweet_count	37	retweet_user_id	50	verified
12	quote_count	25	quoted_user_id	38	retweet_screen_name
13	reply_count	26	quoted_screen_name	39	retweet_followers_count

Twitter API

To get the data for the Twitter account, we must first register a developer account and create an application with the necessary credentials to access the Twitter API. After setting up the application, we generate an authentication key and use the following commands to create a Twitter token that allows access to Twitter data.

library(rtweet) # load rtweet package
twitter_token <- create_token(app = "twitter_app", consumer_key = "api_key", consumer_secret = "api_secret", access_token = "access_token", access_secret = "access_secret")
use_oauth_token(twitter_token) # authenticate via web browser

Searching Twitter’s full archive API with the rtweet package, we run the search_fullarchive() function to access and extract the complete historical information of all the tweets associated with a particular user. We can then save these excerpted tweets to an RDS file. The sample code below captures Elon Musk’s tweets from January 01, 2010, to May 28, 2022.

df <- search_fullarchive(q = "from:elonmusk", n = 10000, env_name = enviroment_name, fromDate = "201001010000", toDate = "202205280000")

Data Set Preview:
created_at	screen_name	text	favorite_count	retweet_count	quote_count	reply_count	is_quote	is_retweet
2022-08-04 19:23:21	elonmusk	Livestream the 2022 Annual Tesla Shareholder Meeting today at 4:30pm CT https://t.co/KttPYLPcxi https://t.co/eF9kRVfFOr	0	0	0	0	FALSE	TRUE
2022-08-04 18:04:12	elonmusk	(garyblack00?) I had more kids in Q2 than they made cars!	58736	3593	1115	3173	FALSE	FALSE
2022-08-04 16:24:08	elonmusk	(dogeofficialceo?) (chicago_glenn?) (mySA?) (happydad?) Stone of Destiny	2935	178	30	436	FALSE	FALSE

Storing Data in Databricks

Optionally, we can load the Twitter API data into a data management system, such as Azure Databricks, and write queries to run a SQL job and retrieve the data. From the Azure portal, we create and launch a Databricks workspace, establish a Spark cluster, and configure a notebook on the cluster. In the notebook, we use SparkR to read the dataset into a Spark DataFrame and run a SQL job to query the data.

Azure Databricks is an analytics platform based on Microsoft Azure cloud services, enabling the latest versions of Apache Spark: an open-source engine providing large-scale APIs in general-purpose programming languages such as Scala, Python, and R. Specifically, Databricks provides a cloud-based interactive workspace with fully managed Spark clusters, allowing users to quickly execute Spark code in a easy-to-use environment.

---

Querying Twitter Data

When working with Twitter data, one of the first steps is distinguishing organic, user-written tweets from other tweet categories: retweets, replies, mentions, and quotes. By interpreting the tweet type ratio, we can glimpse into the general overview of a user’s Twitter feed and account type.

Query 1. Tweet Types

• What is the ratio of tweet types: mentions, replies, retweets, quotes, and organic tweets?

The different types of tweets that exist are general tweets, mentions, replies, retweets, and quotes. General tweets are original Twitter posts that include text, photos, GIFs, and videos but do not include mentions, replies, retweets, or quotes. Both mentions and replies are tweets containing other account usernames, though replies are specific tweets sent directly to another user’s tweets. Lastly, retweets and quotes are re-postings of another person’s tweets, although quotes allow users to post another tweet with their own added comment.

As a first step, we distinguish between organic tweets, retweets, and replies. With the data collected through the Twitter API, we can use specific columns such as is_retweet and reply_to_status_id to determine the tweet type. The following command removes retweets and replies from the data to keep only organic/generic tweets.

# Remove retweets and replies
dfGeneral <- df[df$is_retweet == FALSE,] %>% subset(is.na(reply_to_status_id))

Similarly, we want to create a different dataset for each data type.

dfMention <- subset(df, !is.na(df$mentions_user_id))
dfReply <- subset(df, !is.na(df$reply_to_status_id))
dfRtweet <- df[df$is_retweet == TRUE,]
dfQuote <- df[df$is_quote == TRUE,]

In the above, we subset the tweets into five datasets containing only general tweets, mentions, replies, retweets, or quotes. We then count the number of observations for each dataset using the nrow() function and store the information in a separate data frame containing the tweet types and their corresponding counts.

Now, for example, we can show information for each of Musk’s retweets and query the data to obtain his most frequently retweeted users. To identify the most frequently retweeted users, we use tidyr tools to unnest, count, and sort each user from Musk’s retweets.

---

Query 2. Mentions

• How does Elon Musk engage with other twitter users?

To track Elon Musk’s engagement with people on Twitter, we want to look into tweets containing conversations with and directed to other users. We begin by unpacking information for each of Musk’s tweets that mention another person’s username. Specifically, mentions are a type of tweet containing other account usernames, preceded by the “@” symbol.

Mentions
created_at	mentions_user_id	mentions_screen_name	text
2022-07-27 01:40:54	x924966648922157056 x593977467 x42555311 x16831353	chicago_glenn michaelsiconolf KirstenGrind EmilyGlazer	(chicago_glenn?) (michaelsiconolf?) (KirstenGrind?) (EmilyGlazer?) 99% of journalism is reading someone else’s story on the Internet, changing it up a little & pressing send
2022-08-01 17:28:24	x1492510878323081216	spideycyp_155	(spideycyp_155?) So much water under the bridge since then
2022-08-04 16:19:18	x924966648922157056 x9830752 x38672574	chicago_glenn mySA happydad	(chicago_glenn?) (mySA?) (happydad?) Yeah, pretty good. I like the name.

As shown above, tweets can mention more than one user in the body of the text. Therefore, we have to process the data so that each user mentioned in the tweet is recorded in a separate row so that we can count the total number of times Musk mentioned a unique user.

dfMentions %>% tibble(user = str_extract_all(text, "@\\w+")) %>%
  tidyr::unnest_longer(user) %>% dplyr::count(user, sort = TRUE)

The above command uses the str_extract_all() function to extract the mentioned users for each tweet and unnest_longer() to transform the nested lists into tidy rows so that each row contains only one user. Lastly, we count the total number of observations for each unique user.

screen_name	n	mention_time	mention_time2
spacex	1323
tesla	1050
erdayastronaut	705
ppathole	474
flcnhvy	470
teslaownerssv	459
wholemarsblog	384
teslarati	351
nasa	264
cleantechnica	217

Linking conversations together, a reply is a type of tweet sent directly to another user’s tweet. Analogous to mentions, replies allow users to direct tweets toward other Twitter users and interact in conversations. Continuing the same general procedure above, we get the following results.

---

Query 3. Twitter Activity

• Are there any trends of when Elon Musk tweets?

Here, we provide an overview of Musk’s Twitter activity by analyzing the frequency of tweets by timeframes such as year, month, weekday, hour, and time of day. Parsing the information from the created_at column, we extract timestamps to display the year, month, day, and hour of each tweet’s posting date.

df$created_at <- to_timestamp(df$created_at)
df$year <- year(df$created_at)
df$month <- date_format(to_date(df$created_at), "MMMM")
df$weekday <- date_format(to_date(df$created_at), "EEEE")

created_at	year	month	weekday	time
2021-11-10 17:06:52	2021	November	Wednesday	evening
2021-07-29 02:37:39	2021	July	Thursday	night
2020-11-09 01:55:55	2020	November	Monday	night
2020-08-04 07:41:11	2020	August	Tuesday	morning
2020-01-19 15:25:07	2020	January	Sunday	afternoon
2019-11-24 21:13:25	2019	November	Sunday	night

The above data allows us to explore Musk’s Twitter usage, ranging from the frequency of tweets over years to which days of the week or hours of the day have more or less activity. As a result, we see that Musk is most active on Thursdays and Fridays, mostly tweeting at night.

We can go even deeper and look at the exact time and day of the week in which Musk has the most activity posting tweets with a detailed histogram. Consistent with our previous results, we see the most activity on Thursdays at 6 pm.

---

Query 4. Tweet Topics

• What are Elon Musk’s most tweeted topics?

Continuing to analyze Musk’s habits on Twitter, we want to know which topics dominate Musk’s Twitter feed. To get such topics and themes of the user content, we examine the most frequently used hashtags and words for all of Musk’s tweets. Extracting unique hashtags and words from a tweet requires text mining tasks.

Hashtags:

We first extract hashtags, all words preceded with a # character, from the content of Musk’s tweets. The following command cleanses and unpacks the text of each tweet to get a string array containing only the hashtags. Then, we calculate the frequency of each unique hashtag.

hashtag <- df$text %>% str_extract_all("#[A-Za-z0-9_]+")
hashtag_word <- unlist(hashtag)
hashtag_word <- tolower(hashtag_word)
hashtag_word <- gsub("[[:punct:]ー]", "", hashtag_word)

Topic Words:

Next, let’s look at the words Musk mentioned the most in his tweets. We first clean up the text in the data by removing punctuation, URL links, and symbols, and use tools to convert the text into a tidy format. Then, we split the text into individual words and remove stop words.

# Regex for parsing tweets
replace_reg <- "https?://[^\\s]+|&amp;|&lt;|&gt;|\bRT\\b|^<"

# Clean text
words <- dfGeneral %>% 
  dplyr::mutate(
    text = str_remove_all(text, replace_reg),
    text = str_remove_all(text, "[[:punct:]]"),
    text = str_remove_all(text, "[[:digit:]]")) %>%
  # Split into words
  unnest_tokens(word, text, token = "tweets") %>% 
  # Remove stop words
  anti_join(stop_words, by = "word")

In the above command, the pattern matching function str_remove_all() removes unwanted text, and the unnest_tokens() function splits the text of each tweet into tokens, using a one-word-per-row format. We then use the str_detect() function to filter out words by removing stop words, unicode characters, and whitespace.

Above, we used the unnest_tokens function to tokenize by word; however, we can also use these functions to tokenize into consecutive sequences of words, called n-grams. We do this by adding the option token = "ngrams" and setting \(n\) to the number of words. Setting n to \(2\) allows us to examine pairs of two consecutive words, often called bigrams.

bigrams <- dfGeneral %>% 
  dplyr::mutate(text = str_replace_all(text, replace_reg, "")) %>%
  
  # split into word pairs
  unnest_tokens(bigram, text, token = "ngrams", n = 2) %>%
  separate(bigram, into = c("first","second"), sep = " ", remove = FALSE) %>%
  
  # remove stop words
  anti_join(stop_words, by = c("first" = "word")) %>%
  anti_join(stop_words, by = c("second" = "word")) %>%
  filter(str_detect(first, "[a-z]") & str_detect(second, "[a-z]"))

name	freq	time_posted
tesla model	35
boring company	28
falcon heavy	26
space station	25
cape canaveral	22
climate change	19
tesla team	19
tesla owners	18
static fire	16
giga berlin	15

---

Sentiment Analysis

Here we use the syuzhet R package to iterate over a vector of strings consisting of the text from all of Elon Musk’s tweets in our dataset. To obtain the vector of tweet text, the plain_tweets() function from the rtweet package is used to clean up the tweets character vector to cleaned up, plain text. We then pass this vector to the get_sentiment() function, which consequently returns the sentiment values based on the custom sentiment dictionary developed from a collection of human coded sentences.

round_time <- function(x, secs)
  as.POSIXct(hms::round_hms(x, secs))
sent_scores <- function(x)
  syuzhet::get_sentiment(plain_tweets(x)) - .5

df.sentiment <- gfg_data %>%
  dplyr::mutate(days = round_time(created_at, 60 * 60 * 24),
                sentiment = sent_scores(text)) %>%
  dplyr::group_by(days) %>%
  dplyr::summarise(sentiment = sum(sentiment, na.rm = TRUE))

Extending the above sentiment analysis, the next step is to understand the opinion or emotion in the text. First, we must clean the text from our dataset so that it’s in a tidy format. We accomplish this using the R function gsub() to replace unwanted text and the get_nrc_sentiment() function to get the emotions and valences from the NRC sentiment dictionary for each word from all of Musk’s tweet.

R Code:

txt <- c("rt|RT", "http\\w+", "<.*?>", "@\\w+", "[[:punct:]]", "\r?\n|\r", "[[:digit:]]", "[ |\t]{2,}", "^ ", " $")

cleanTweet <- as.vector(df$text)
cleanTweet <- grep::gsub(txt, "", cleanTweet)

textSentiment <- syuzhet::get_nrc_sentiment(cleanTweet)
nrc_sentiment <- cbind(df, textSentiment) %>% 
  dplyr::select(created_at, anger, anticipation, disgust, fear, 
                joy, sadness, surprise, trust, negative, positive)

In the above command, the gsub function replaces all occurrences of the given patterns and the get_nrc_sentiment function calculates the presence of eight different emotions and their corresponding valence. The resulting columns include the eight emotions disgust, fear, joy, sadness, surprise, trust and their respective positive or negative valence.

---

Conclusion

This project showed how to extract and analyze twitter data using various essential packages in R. The above Twitter analytics report for Elon Musk’s tweets covered insights regarding Musk’s tweeting behavior, insights on the content of Musk’s tweets, and a sentiment analysis capturing the tone of the tweets.

The first question examines Musk’s tweet distribution, which we answer by sorting each tweet into categories (based on tweet types) and ranking each category based on the volume of tweets. Analyzing Musk’s nature of engagement, we answer the second question by unpacking tweets containing conversations with and directed to other users. To answer the third question, we investigate Musk’s Twitter activity by sorting his feed into different timeframes based on the publish date. Dissecting the topics that predominate his profile feed, we answer the fourth question by examining Musk’s most frequently used hashtags and words.

 

References

Brown, Leif, Jason Howell, and Mary McCready. 2022. “Quickstart - Run a Spark Job on Azure Databricks Workspace Using Azure Portal.” Microsoft Technical Documentation. https://docs.microsoft.com/en-us/azure/databricks/scenarios/quickstart-create-databricks-workspace-portal.

Kearney, Michael W. 2019. “Rtweet-Workshop.” Data Science and Analytics Presentation. University of Missouri School of Journalism: Informatics Institute. https://rtweet-workshop.mikewk.com/.

Kearney, Michael W., Francois Briatte, Andrew Heiss. 2019. “Rtweet: Collecting and Analyzing Twitter Data.” Journal of Open Source Software 4 (42): 1829. https://doi.org/10.21105/joss.01829.