Web Scraping in Python

Implementing Web Scraping in Python with BeautifulSoup:

Use the API of the website. example: Facebook has the Facebook Graph API which  

- allows retrieval of data posted on Facebook.

- HTML of the webpage information/data will access.

Installing the required third-party libraries : 

pip install requests 
pip install html5lib

pip install bs4

Accessing the HTML content from webpage : 

import requests 
URL = "https://mindstick.com/forum/"

r = requests.get(URL)

print(r.content)

Parsing the HTML content :

#This will not run on online IDE 
import requests

from bs4 import BeautifulSoup

URL = "http://www.values.com/inspirational-quotes"

r = requests.get(URL)

soup = BeautifulSoup(r.content, 'html5lib')

print(soup.prettify())

soup = BeautifulSoup(r.content, 'html5lib') 

table = soup.find('div', attrs = {'id':'container'}) 

quote['theme'] = row.h5.text 

quote['url'] = row.a['href']