R Web Scraping and APIs
r
web-scraping
rvest
selenium
api
httr
Tutorials and packages for scraping websites, automating browsers, and calling web APIs from R.
This guide collects tutorials and packages for getting data from the web with R, whether by scraping pages, driving a browser, or calling an API. Begin with the three picks under Start here, then use the other sections as your project needs them. Check a site’s terms and robots.txt rules before scraping it. Sources of ready-made datasets are on the Data page.
Start here
- R Web Scraping Tutorial with rvest — DataCamp tutorial on web scraping with rvest, built around a real project.
- CRAN Task View: Web Technologies and Services — CRAN’s curated list of R packages for web technologies and services.
- Accessing APIs from R (and a little R programming) — R-Bloggers post on accessing web APIs from R.
Scraping pages
- BeautifulSoup vs. Rvest — Post comparing Python’s BeautifulSoup with R’s rvest for web scraping.
- Web Scraping with rvest: Exploring Sports Industry Jobs — R-Bloggers example of scraping sports-industry job listings with rvest.
- SelectorGadget — rvest vignette on using SelectorGadget to find the selectors to scrape (archived copy).
- R and the web (for beginners), Part III: Scraping MPs’ expenses in detail — Third part of a 2012 series on R and the web, on scraping MPs’ expenses.
- Web Scraping JS- and progressively-generated content — RPubs example of scraping JavaScript-generated content from Bandsintown.
- polite — Package for scraping websites politely.
Browser automation with Selenium
- RSelenium — R client for Selenium Remote WebDriver.
- RSelenium: Introduction — Introduction to RSelenium, linked at slide 18 of the presentation.
- Using RSelenium and Docker To Webscrape In R — Tutorial on scraping the WHO snake database with RSelenium and Docker (archived copy).
- RSelenium Tutorial — Tutorial on web scraping in R with RSelenium (archived copy).
- RSelenium For Mac: Update — Update on running RSelenium on a Mac, using Docker to run Selenium 3.
- seleniumPipes — R client that implements the W3C WebDriver specification.
- seleniumPipes: Basic operation — RPubs demonstration of basic operations with seleniumPipes.
APIs and JSON
- Using R to download and parse JSON — Zev Ross post on downloading and parsing JSON from an open data portal with R.
- Working with The New York Times API in R — Storybench tutorial on using the New York Times API from R.
- New York Times API and httr — Demo of using the New York Times API with the httr package.
- curl — Package that gives R a modern and flexible web client.
- plumber — Package that turns R code into a web API.
- How to hide a password in R with the keyring package — Post on keeping passwords out of your R scripts with keyring, useful when calling APIs.
- wayback — Tools for working with the Internet Archive’s Wayback Machine APIs from R.
- scholar — CRAN page for scholar, a package that extracts citation data from Google Scholar.