Data from the Web to R

Posted on May 12, 2014

Much of the behind-the-scenes of my research work and side projects have to do with the so-called Web Technologies. Consequently, questions like how to read/import data sets?, or how to preprocess data? become more relevant with data coming from the Web.

How do I parse XML / HTML documents in R? How do I scrape those pieces of valuable information? How do I use R to interact with my browser and make the right requests? These are some of the frequently asked questions I’ve been constantly coming across with when trying to get data from the Web.

Web Technologies for R

It’s no secret or news that the Web is a vast source of data and information. The amount of content and its exponential growth surpasses our capacity to handle it analytically. We hardly know what to do with it, not to mention how to make sense of it. Most of us —statisticians, analysts, data scientists, data miners— are in the same quest for our holy grails: gaining insight and learning from data. That’s where most of us are focusing our energies. However, in order to start crunching numbers, building models, and training machines to learn, we must first have access to the data itself.

For those of us who primarily work with R, there are various packages that can help us get access to our precious data sources. In addition, there are various resources that can guide us on how to use the tools and functions in those packages.

Deborah Nolan and Duncan Temple Lang have written an awesome book recently published at the beginning of this year.

XML and Web Technologies for Data Sciences with R is one of those books that I’ve been dreaming about for so long. It is THE compendium of reference for anyone interested in how to use R for interacting with Web Technologies.

(Parenthesis: I tip my hat to Duncan for all his amazing work, and his titanic effort leading the omegahat project. Bravo!)

Before Deb and Duncan’s book, the closest thing was Paul Murrell’s excellent book Introduction to Data Technologies (ItDT).

You are probably more familiar with Paul’s work around plots and data visualization with his famous book R Graphics, but his ItDT ebook is one of those rare jewels that most of my closest colleagues have highly ignored. If you haven’t taken a look at it before, stop reading this post and come back after you’ve checked Paul’s ItDT.

Also recently this year, a new Task View for Web Technologies has been added to CRAN. This only reflects the trending topic status that those Web tools have in the R community.

Getting Data from the Web with R

I’m way behind the experience and expertise of Deb, Duncan, Paul and many other R leading figures. But still I want to contribute with my own take on how to use R for getting data from the Web. After some delays due to my design hesitations, I’ve finished a set of slides focused on: How to get data from the Web with R.

In a nutshell, you’ll find the following content:

1. Introduction
2. Reading files from the Web
3. Basics of XML
4. Parsing XML and HTML Documents
5. Handling JSON data
6. Basics of HTTP and RCurl
7. Getting Data via Web Forms
8. Getting Data via Web APIs

PS

Inspired by some friends’ inquiry, I thought this could also be a nice opportunity for me to talk about some of the creative process I went through to design and prepare the material of these slides.