If you are here I assume you are interested in doing fun with web scraping. Great! In general, Web Scraping is a way of extracting information from websites automatically.

How this post can help you? I’ve found two reasons (there may be many more) for you to go through this post.
1) You are very much excited about scrapping
2) You may get help from my tutorial for your project

Right now I am working on an application (similar to Reddit and TopBuzz), its one of the core tasks is scrapping articles, videos from listed websites. Last day, I developed a small module. The module is only responsible for scrapping particular video information from a URL (eg. url). To share how I did this, in this post, we will walk through an example of extracting youtube video information with the help of Python.

Note: Web Scraping code depends on the structure of the web page. If the structure changes, you have to update your code too! 😒

Why Python?

In my opinion, in case of web scraping, there are no programming languages that are better than Python 🔥. And Python provides necessary packages specially for web scrapping (eg. beautifulsoup).

Setup

Before starting the tutorial, you have to setup Python environment. Install virtualenv which is going to be responsible for managing a particular project’s packages. It intends to make an isolated Python environment.

pip install virtualenv

Create a folder named utube(or what you want), inside this folder run

virtualenv venv

Activate the virtual environment,

source venv/bin/activate

Add requirements.txt file, which lists necessary packages,

bs4==0.0.1requests==2.22.0

Get the HTML Content and Meta Data

Now fire up your favorite text editor and create a file called main.py, import beautifulsoup, and requests

The target youtube video link is assigned to the link variable. As I am a big fan of Eminem, I put a video link one of his songs 😊. Below two lines will turn the HTML content of the given link into a structured tree.

print(soup.prettify())

soup variable shows metadata (related information of the video, eg. title, description, thumbnail, etc). Also, you will see the full raw HTML content of the given page.

Inspecting and Extracting Video Information

One of the important parts of scrapping is inspecting the webpage. Let’s examine and extract.

Find your necessary information and Extract

➼ Title of the Video