How to Analyze Articles About Data Science Using Data Science

In a previous post, we demonstrated how to use the Python3 library Newspaper to painlessly extract data from news articles. Using Newspaper, I was able to extract text from over a 1000 articles about topics including, but limited to Data Science, Artificial Intelligence, and Big Data. In this follow up post, we’ll use unsupervised machine learning tools to investigate this corpus of articles, specifically, we’ll be using clustering and topic modeling to derive the various kinds of data science articles that are published.

First up, let’s show a wordcloud of their corpus to get a feel for its content.

We see there is a lot of familiar terms that we’d expect from this collection of documents. What makes this wordcloud is that it uses so-called “keywords” derived from each article using the newspaper library. This library has a special function that parses the main topics or keywords from each article, which is a great way to prevent stopwords and other meaningless terms from polluting the word cloud.

Topic Modeling.

I decided to use Non-Negative Matrix Factorization to determine the topic modeling. In the following code, I used a TFIDF vectorizer to extract features from my corpus, which I then passed into an NMF object to derive the topics of my corpus.

These seven topics show the nature of the content in my corpus. Topics 1, 2, 3, and 6 are the ones most related to data science or data science topics. The other topics (4, 5, 7)  are about articles related to business and/or non-data science subject such as blockchain.

After determining the topics I wanted to cluster the articles and visualize those clusters. The following image display five color-encoded clusters (AI, business, analytics, tutorials, machine learning) on a two-dimension graph.

©ODSC2017