Anaconda Web Scraping
Jan 21, 2020 Installing Jupyter Notebook using Anaconda: Anaconda is an open-source software that contains Jupyter, spyder, etc that are used for large data processing, data analytics, heavy scientific computing. Anaconda works for R and python programming language. Spyder(sub-application of Anaconda) is used for python. Opencv for python will work in spyder. Introducing Anaconda and Conda. Since 2011, Python has included pip, a package management system used to install and manage software packages written in Python.However, for numerical computations, there are several dependencies that are not written in Python, so the initial releases of pip could not solve the problem by themselves.
- In this Python for Data Science tutorial, you will learn about Web scraping and Parsing in Python using Beautiful Soup (bs4) in Anaconda using Jupyter Noteb.
- First, learn the essentials of web scraping, explore the framework of a website and get your local environment ready to take on scraping challenges with Scrapy, BeautifulSoup, and Selenium. Next, set up a Scrapy crawler and cover the core details that can be applied to building datasets or mining.
- Jan 22, 2020 Anaconda is an open-source software that contains Jupyter, spyder, etc that are used for large data processing, data analytics, heavy scientific computing. Anaconda works for R and python programming language. Spyder(sub-application of Anaconda) is used for python. Opencv for python will work in spyder.
In this tutorial we will use a technique called web scraping to extract data from a website.
We’ll be using Python 3.7 through a Jupyter Notebook on Anaconda and the Python libraries urllib, BeautifulSoup and Pandas.
(If you don’t have Anaconda or Jupyter Notebook installed on your Windows machine, check out our tutorial How Do I Install Anaconda On Windows? before getting started. If you’re on Linux or Mac OS X you’ll have to Google it. Bill Gates fanboy in the house.…)
What is Web Scraping?
Web scraping (also known as screen scraping, data scraping, web harvesting, web data extraction and a multitude of other aliases) is a method for extracting data from web pages.
I’ve done a quick primer on WTF Is…Web Scraping to get you up to speed on what it is and why we might use it. Have a quick read and re-join the tour group as soon as possible.
Step By Step Tutorial
Anaconda Web Scraping Tool
OK. Now we know what web scraping is and why we might have to use it to get data out of a website.
How exactly do we get started scraping and harvesting all of that delicious data for our future perusal and use?
Standing on the shoulders of giants.
When I first tried screen scraping with Python I used an earlier version of it and worked through Sunil Ray’s Beginner’s Guide on the Analytics Vidhya blog.
Working with Python 3.7 now I had to change some libraries plus do a few further corrective steps for the data I’m looking to get hence not just pointing you straight to that article.
Which Python libraries will we be using for web scraping?
Urllib.request
As we are using Python 3.7, we will use urllib.request to fetch the HTML from the URL we specify that we want to scrape.
BeautifulSoup
Once urllib.request has pulled in the content from the URL, we use the power of BeautifulSoup to extract and work with the data within it. BeautifulSoup4 has a multitude of functions at it’s disposal to make this incredibly easy for us.
Learn more about Beautiful Soup.
Anything else we need to know before we kick this off?
Are you familiar with HTML?
HTML (Hypertext Markup Language) is the standard markup langauge for creating web pages. It consists of a collection of tags which represent HTML elements. These elements combined tell your web browser what the structure of the web page looks like. In this tutorial we will mostly be concerned with the HTML table tags as our data is contained in a table. For more reading on HTML, check out W3Schools Introduction to HTML.
Right, let’s get into it.
1. Open a new Jupyter notebook.
You do have it installed, don’t you? You didn’t just skip the advice at the top, did you? If so, go back and get that done then come back to this point.
2. Choosing our target Wikipedia page.
Like our friend Sunil, we are going to scrape some data from a Wikipedia page. While he was interested in state capitals in India, I’ve decided to pick a season at random from the English Premier League, namely the 1999/2000 season.
When I went and looked at the page I instantly regretted picking this season (fellow Tottenham Hotspur fans will understand why when they see the manager and captain at the end but I’ll stick with it as some kind of sado-masochism if nothing else).
3. Import urllib library.
Firstly, we need to import the library we will be using to connect to the Wikipedia page and fetch the contents of that page:
Next we specify the URL of the Wikipedia page we are looking to scrape:
Using the urllib.request library, we want to query the page and put the HTML data into a variable (which we have called ‘url’):
4. Import BeautifulSoup library.
Next we want to import the functions from Beautiful Soup which will let us parse and work with the HTML we fetched from our Wiki page:
Then we use Beautiful Soup to parse the HTML data we stored in our 'url' variable and store it in a new variable called ‘soup’ in the Beautiful Soup format. Jupyter Notebook prefers we specify a parser format so we use the “lxml” library option:
5. Take a look at our underlying HTML code.
To get an idea of the structure of the underlying HTML in our web page, we can view the code in two ways: a) right click on the web page itself and click View Source or b) use Beautiful Soup’s prettify function and check it out right there in our Jupyter Notebook.
Let’s see what prettify() gives us:
6. Find the table we want.
By looking at our Wikipedia page for the 1999/2000 Premier League season, we can see there is a LOT of information in there. From a written synopsis of the season to specific managerial changes, we have a veritable treasure trove of data to mine.
What we are going to go for though is the table which shows the personnel and kits for each Premier League club. It’s already set up in nice rows and columns which should make our job a little easier as beginner web scrapers.
Let’s have a look for it in our prettifyed HTML:
And there it is. (NB. since I first wrote this tutorial, Wiki has added another table with satadium name, capacity etc. that also has this class identifier. We'll allow for that further down in the code.)
Starting with an HTML <table> tag with a class identifier of 'wikitable sortable'. We’ll make a note of that for further use later.
Scroll down a little to see how the table is made up and you’ll see the rows start and end with <tr> and </tr> tags.
The top row of headers has <th> tags while the data rows beneath for each club has <td> tags. It’s in these <td> tags that we will tell Python to extract our data from.
7. Some fun with BeautifulSoup functions.
Before we get to that, let’s try out a few Beautiful Soup functions to illustrate how it captures and is able to return data to us from the HTML web page.
If we use the title function, Beautiful Soup will return the HTML tags for the title and the content between them. Specify the string element of 'title' and it gives us just the content string between the tags:
8. Bring back ALL of the tables.
We can use this knowledge to start planning our attack on the HTML and homing in only on the table of personnel and kit information that we want to work with on the page.
We know the data resides within an HTML table so firstly we send Beautiful Soup off to retrieve all instances of the <table> tag within the page and add them to an array called all_tables:
Looking through the output of 'all_tables' we can again see that the class id of our chosen table is 'wikitable sortable'. We can use this to get BS to only bring back the table data for this particular table and keep that in a variable called 'right_table'. As I said above, there is now another table with this classname in the HTML, we're going to use find_all to bring back an array and then look for the second element in the array which we know is the table we want:
9. Ignore the headers, find the rows.
Now it starts to get a little more technical. We know that the table is set up in rows (starting with <tr> tags) with the data sitting within <td> tags in each row. We aren't too worried about the header row with the <th> elements as we know what each of the columns represent by looking at the table.
To step things up a notch we could have set BeautifulSoup to find the <th> tags and assigned the contents of each to a variable for future use.
We’ve got enough to get getting on with getting the actual data though so let’s crack on.
10. Loop through the rows.
We know we have to start looping through the rows to get the data for every club in the table. The table is well structured with each club having it's own defined row. This makes things somewhat easier.
There are five columns in our table that we want to scrape the data from so we will set up five empty lists (A, B, C, D and E) to store our data in.
To start with, we want to use the Beautiful Soup 'find_all' function again and set it to look for the string 'tr'. We will then set up a FOR loop for each row within that array and set Python to loop through the rows, one by one.
Within the loop we are going to use find_all again to search each row for <td> tags with the 'td' string. We will add all of these to a variable called 'cells' and then check to make sure that there are 5 items in our 'cells' array (i.e. one for each column).
If there are then we use the find(text=True)) option to extract the content string from within each <td> element in that row and add them to the A-E lists we created at the start of this step. Let’s have a look at the code:
Still with me? Good. This all should work perfectly, shouldn’t it?
We're looping through each row, picking out the <td> tags and plucking the contents from each into a list.
Bingo. This is an absolute gift. Makes you wonder why people make such a fuss about it, doesn’t it?
11. Introducing pandas and dataframes.
To see what our loop through the Personnel and Kits table has brought us back, we need to bring in another big hitter of the Python library family – Pandas. Pandas lets us convert lists into dataframes which are 2 dimensional data structures with rows and columns, very much like spreadsheets or SQL tables.
We’ll import pandas and create a dataframe with it, assigning each of the lists A-E into a column with the name of our source table columns i.e. Team, Manager, Captain, Kit_Manufacturer and Shirt_Sponsor.
Let’s run the Pandas code and see what our table looks like:
Hmmm. That’s not what we wanted. Where's the Manager and Captain data?
Clearly something went wrong in those cells so we need to go back to our HTML to see what the problem is.
12. Searching for the problem.
Looking at our HTML, there does indeed seem to be something a little different about the Manager and Captain data within the <td> tags. Wikipedia has (very helpfully/unhelpfully) added a little flag within <span> tags to help display the nationality of the Managers and Captains in question.
It sure looks nice on the Wiki page but it's messing up my screen-scraping tutorial so I'm somewhat less than happy to have it in there.
Using the knowledge we've gained above, is there a simple way to workaround this problem and just lift out the Manager and Captain names as we planned?
This is how I decided to do it.
Looking at the HTML code, I can see that there are two sets of <a> tags i.e. hyperlinks within each cell for both the Manager and Captain data. The first is a link over the flag’s <img> tag and the second is a link on the Manager/Captain’s name.
If we can get the content string between the <a> and </a> tags on the SECOND of those, we have got the data we need.
I did a 'find_all' within the individual cells to look for the <a> tags and assign that to a variable (mlnk for Managers, clnk for Captains). I knew it was the second <a> tag's content string that I needed to get the name of the Manager and the Captain so I appended the content of the second element in the mlnk/clnk array I had created to the specific list (list B for Managers, list C for Captains).
As so:
Now run that and re-run our pandas code from before and 'hopefully' we'll fill in those blanks from the previous output:
Hurrah!
We now have 20 rows for the 20 clubs with columns for Team Name, Manager, Captain, Kit Manufacturer and Shirt Sponsor. Just like we always wanted.
(I’ll ignore the names in the Manager and Captain columns for Tottenham, must research my examples better before getting started…)
Want an even easier way to do this using just pandas?
Eternal thanks to reader Lynn Leifker who has sent me an even quicker and easier way to scrape HTML tables using only pandas. You'll still have to do some HTML investigation to find which table in the overall page code you are looking for but can get to the outcome quicker using this code:
Hurrah once again and a big thanks to Lynn for the top tip!
Wrapping Up
That successful note brings us to the end of our Getting Started Web Scraping with Python tutorial. Hopefully it gives you enough to get working on to try some scraping out for yourself. We've introduced urllib.request to fetch the URL and HTML data, Beautiful Soup to parse the HTML and Pandas to transform the data into a dataframe for presentation.
We also saw that things don't always work out just as easily as we hope for when working with web pages but it’s best to roll with the punches and come up with a plan to workaround it as simply as possible,
If you have any questions, please send me a mail (alan AT alanhylands DOT com). Happy scraping but if you get caught…we never met!
Someone on the NICAR-L listserv asked for advice on the best Python libraries for web scraping. My advice below includes what I did for last spring’s Computational Journalism class, specifically, the Search-Script-Scrape project, which involved 101-web-scraping exercises in Python.
Anaconda Web Scraping Software
See the repo here: https://github.com/compjour/search-script-scrape
Best Python libraries for web scraping
For the remainder of this post, I assume you’re using Python 3.x, though the code examples will be virtually the same for 2.x. For my class last year, I had everyone install the Anaconda Python distribution, which comes with all the libraries needed to complete the Search-Script-Scrape exercises, including the ones mentioned specifically below:
The best package for general web requests, such as downloading a file or submitting a POST request to a form, is the simply-named requests library(“HTTP for Humans”).
Here’s an overly verbose example:
The requests library even does JSON parsing if you use it to fetch JSON files. Here’s an example with the Google Geocoding API:
For the parsing of HTML and XML, Beautiful Soup 4 seems to be the most frequently recommended. I never got around to using it because it was malfunctioning on my particular installation of Anaconda on OS X.
But I’ve found lxml to be perfectly fine. I believe both lxml and bs4 have similar capabilities – you can even specify lxml to be the parser for bs4. I think bs4 might have a friendlier syntax, but again, I don’t know, as I’ve gotten by with lxml just fine:
The standard urllib package also has a lot of useful utilities – I frequently use the methods from urllib.parse. Python 2 also has urllib but the methods are arranged differently.
Here’s an example of using the urljoin method to resolve the relative links on the California state data for high school test scores. The use of os.path.basename is simply for saving the each spreadsheet to your local hard drive:
And that’s about all you need for the majority of web-scraping work – at least the part that involves reading HTML and downloading files.
Examples of sites to scrape
The 101 scraping exercises didn’t go so great, as I didn’t give enough specifics about what the exact answers should be (e.g. round the numbers? Use complete sentences?) or even where the data files actually were – as it so happens, not everyone Googles things the same way I do. And I should’ve made them do it on a weekly basis, rather than waiting till the end of the quarter to try to cram them in before finals week.
The Github repo lists each exercise with the solution code, the relevant URL, and the number of lines in the solution code.
The exercises run the gamut of simple parsing of static HTML, to inspecting AJAX-heavy sites in which knowledge of the network panel is required to discover the JSON files to grab. In many of these exercises, the HTML-parsing is the trivial part – just a few lines to parse the HTML to dynamically find the URL for the zip or Excel file to download (via requests)…and then 40 to 50 lines of unzipping/reading/filtering to get the answer. That part is beyond what typically considered “web-scraping” and falls more into “data wrangling”.
I didn’t sort the exercises on the list by difficulty, and many of the solutions are not particulary great code. Sometimes I wrote the solution as if I were teaching it to a beginner. But other times I solved the problem using the style in the most randomly bizarre way relative to how I would normally solve it – hey, writing 100+ scrapers gets boring.
But here are a few representative exercises with some explanation:
1. Number of datasets currently listed on data.gov
I think data.gov actually has an API, but this script relies on finding the easiest tag to grab from the front page and extracting the text, i.e. the 186,569 from the text string, '186,569 datasets found'. This is obviously not a very robust script, as it will break when data.gov is redesigned. But it serves as a quick and easy HTML-parsing example.
29. Number of days until Texas’s next scheduled execution
Texas’s death penalty site is probably one of the best places to practice web scraping, as the HTML is pretty straightforward on the main landing pages (there are several, for scheduled and past executions, and current inmate roster), which have enough interesting tabular data to collect. But you can make it more complex by traversing the links to collect inmate data, mugshots, and final words. This script just finds the first person on the scheduled list and does some math to print the number of days until the execution (I probably made the datetime handling more convoluted than it needs to be in the provided solution)
Anaconda Web Scraping Tutorial
3. The number of people who visited a U.S. government website using Internet Explorer 6.0 in the last 90 days
The analytics.usa.gov site is a great place to practice AJAX-data scraping. It’s a very simple and robust site, but either you are aware of AJAX and know how to use the network panel (and in this case, locate ie.json, or you will have no clue how to scrape even a single number on this webpage. I think the difference between static HTML and AJAX sites is one of the tougher things to teach novices. But they pretty much have to learn the difference given how many of today’s websites use both static and dynamically-rendered pages.
6. From 2010 to 2013, the change in median cost of health, dental, and vision coverage for California city employees
There’s actually no HTML parsing if you assume the URLs for the data files can be hard coded. So besides the nominal use of the requests library, this ends up being a data-wrangling exercise: download two specific zip files, unzip them, read the CSV files, filter the dictionaries, then do some math.
90. The currently serving U.S. congressmember with the most Twitter followers
Another example with no HTML parsing, but probably the most complicated example. You have to download and parse Sunlight Foundation’s CSV of Congressmember data to get all the Twitter usernames. Then authenticate with Twitter’s API, then perform mulitple batch lookups to get the data for all 500+ of the Congressional Twitter usernames. Then join the sorted result with the actual Congressmember identity. I probably shouldn’t have assigned this one.
HTML is not necessary
I included no-HTML exercises because there are plenty of data programming exercises that don’t have to deal with the specific nitty-gritty of the Web, such as understanding HTTP and/or HTML. It’s not just that a lot of public data has moved to JSON (e.g. the FEC API) – but that much of the best public data is found in bulk CSV and database files. These files can be programmatically fetched with simple usage of the requests library.
It’s not that parsing HTML isn’t a whole boatload of fun – and being able to do so is a useful skill if you want to build websites. But I believe novices have more than enough to learn from in sorting/filtering dictionaries and lists without worrying about learning how a website works.
Besides analytics.usa.gov, the data.usajobs.gov API, which lists federal job openings, is a great one to explore, because its data structure is simple and the site is robust. Here’s a Python exercise with the USAJobs API; and here’s one in Bash.
There’s also the Google Maps geocoding API, which can be hit up for a bit before you run into rate limits, and you get the bonus of teaching geocoding concepts. The NYTimes API requires creating an account, but you not only get good APIs for some political data, but for content data (i.e. articles, bestselling books) that is interesting fodder for journalism-related analysis.
But if you want to scrape HTML, then the Texas death penalty pages are the way to go, because of the simplicity of the HTML and the numerous ways you can traverse the pages and collect interesting data points. Besides the previously mentioned Texas Python scraping exercise, here’s one for Florida’s list of executions. And here’s a Bash exercise that scrapes data from Texas, Florida, and California and does a simple demographic analysis.
If you want more interesting public datasets – most of which require only a minimal of HTML-parsing to fetch – check out the list I talked about in last week’s info session on Stanford’s Computational Journalism Lab.