Web Scraping in Python

Posted by  Franco Falkov
 554  View(s)
Rate this:

How to Implement web Scraping in Python with BeautifulSoup ?

  1. Re: Web Scraping in Python

    Implementing Web Scraping in Python with BeautifulSoup:

    Use the API of the website. example: Facebook has the Facebook Graph API which  

    - allows retrieval of data posted on Facebook.

    - HTML of the webpage information/data will access.

    Installing the required third-party libraries : 

    pip install requests 
    pip install html5lib
    pip install bs4

    Accessing the HTML content from webpage : 

    import requests 
    URL = "https://mindstick.com/forum/"
    r = requests.get(URL)

    Parsing the HTML content :

    #This will not run on online IDE 
    import requests
    from bs4 import BeautifulSoup
    URL = "http://www.values.com/inspirational-quotes"
    r = requests.get(URL)
    soup = BeautifulSoup(r.content, 'html5lib')

    soup = BeautifulSoup(r.content, 'html5lib') 
    table = soup.find('div', attrs = {'id':'container'}) 
    quote['theme'] = row.h5.text 
    quote['url'] = row.a['href']