Web Scraping in Python

Total Post:113

Points:791
 183  View(s)
Ratings:
Rate this:

How to Implement web Scraping in Python with BeautifulSoup ?

  1. Post:140

    Points:984
    Re: Web Scraping in Python

    Implementing Web Scraping in Python with BeautifulSoup:

    Use the API of the website. example: Facebook has the Facebook Graph API which  

    - allows retrieval of data posted on Facebook.

    - HTML of the webpage information/data will access.

    Installing the required third-party libraries : 

    pip install requests 
    pip install html5lib
    pip install bs4

    Accessing the HTML content from webpage : 

    import requests 
    URL = "https://mindstick.com/forum/"
    r = requests.get(URL)
    print(r.content)


    Parsing the HTML content :

    #This will not run on online IDE 
    import requests
    from bs4 import BeautifulSoup
    URL = "http://www.values.com/inspirational-quotes"
    r = requests.get(URL)
    soup = BeautifulSoup(r.content, 'html5lib')
    print(soup.prettify())

    soup = BeautifulSoup(r.content, 'html5lib') 
    table = soup.find('div', attrs = {'id':'container'}) 
    quote['theme'] = row.h5.text 
    quote['url'] = row.a['href'] 

Answer

NEWSLETTER

Enter your email address here always to be updated. We promise not to spam!