Artificial intelligence and machine learning are transforming software development and modern applications.
""" ``` ## Step 5 — Remove HTML Tags Use BeautifulSoup to extract plain text. ```python # Parse HTML content soup = BeautifulSoup(html_content, 'html.parser') # Extract plain text text = soup.get_text() # Print extracted text print(text) ``` ## Step 6 — Clean the Text We remove: - Special characters - Numbers Extra spaces ```python # Convert text to lowercase text = text.lower() # Remove special characters and numbers text = re.sub(r'[^a-zA-Z\s]', '', text) # Remove extra spaces text = re.sub(r'\s+', ' ', text) # Print cleaned text print(text) ``` ## Step 7 — Tokenize the Text Tokenization splits text into individual words. ```python # Split text into words words = word_tokenize(text) # Print tokenized words print(words) ``` ## Step 8 — Remove Stop Words Stop words are common words like: - the - is - and - are These words usually don’t help identify keywords. ```python # Load English stop words stop_words = set(stopwords.words('english')) # Remove stop words filtered_words = [ word for word in words if word not in stop_words ] # Print filtered words print(filtered_words) ``` ## Step 9 — Count Keyword Frequency Now count how often each keyword appears. ```python # Count word frequency word_counts = Counter(filtered_words) # Print top 10 keywords print(word_counts.most_common(10)) ``` ## Full Working Code ```python # Import libraries from bs4 import BeautifulSoup import re from nltk.corpus import stopwords from nltk.tokenize import word_tokenize from collections import Counter import nltk # Download nltk datasets nltk.download('punkt') nltk.download('stopwords') # Sample HTML content html_content = """Artificial intelligence and machine learning are transforming software development and modern applications.
""" # Parse HTML soup = BeautifulSoup(html_content, 'html.parser') # Extract text text = soup.get_text() # Convert to lowercase text = text.lower() # Remove special characters text = re.sub(r'[^a-zA-Z\s]', '', text) # Remove extra spaces text = re.sub(r'\s+', ' ', text) # Tokenize words words = word_tokenize(text) # Load stop words stop_words = set(stopwords.words('english')) # Remove stop words filtered_words = [ word for word in words if word not in stop_words ] # Count frequency word_counts = Counter(filtered_words) # Print top keywords print("Top Keywords:") # Display keywords for word, count in word_counts.most_common(10): print(word, ":", count) ``` ## Example Output ```python Top Keywords: artificial : 2 intelligence : 2 machine : 1 learning : 1 transforming : 1 software : 1 development : 1 modern : 1 applications : 1 ``` ## How It Works Internally The workflow looks like this: ```plaintext HTML Content ↓ Remove HTML Tags ↓ Clean Text ↓ Tokenize Words ↓ Remove Stop Words ↓ Count Frequencies ↓ Top Keywords ``` ## Improvements You Can Add The basic keyword extractor works well, but real-world systems can be improved further. ## 1. Use TF-IDF TF-IDF helps identify important words more intelligently. Instead of only counting frequency, it considers: - Word frequency in document - Word rarity across documents Useful library: ```plaintext scikit-learn ``` ## 2. Extract Key Phrases Instead of single words, extract phrases like: - machine learning - artificial intelligence Libraries: - RAKE - spaCy - KeyBERT ## 3. Ignore Navigation Content Real HTML pages contain: - Menus - Headers - Footers - Ads You can remove unnecessary tags: ```python for tag in soup(['script', 'style', 'nav', 'footer']): tag.decompose() ``` ## 4. Use Stemming or Lemmatization Convert related words into base forms. Example: - running → run - applications → application ## 5. Process Live URLs You can fetch real websites using requests. ```python import requests response = requests.get("https://example.com") html_content = response.text ``` ## Real-World Use Cases Keyword extraction is widely used in: | Industry | Usage | | --- | --- | | SEO | [Content optimization](https://www.mindstick.com/forum/33900/please-tell-me-what-are-the-technique-to-content-optimization) | | Search Engines | Indexing pages | | E-commerce | Product tagging | | AI Chatbots | Context understanding | | News Platforms | Article categorization | | Recommendation Systems | Topic matching | ## Challenges in Keyword Extraction ### Noisy HTML Web pages often contain irrelevant content. ### Duplicate Words Repeated keywords may not always indicate importance. ### Context Understanding Simple frequency counting cannot fully understand meaning. ### Multilingual Content Different languages require different NLP models. ### Advanced Alternatives Modern AI systems use embedding-based methods and [transformer models](https://answers.mindstick.com/qa/116776/why-are-transformer-models-preferred-for-llms) for smarter keyword extraction. Popular tools include: - spaCy - KeyBERT - NLTK - Transformers These methods understand semantics rather than only word frequency. ## Final Thoughts Building a keyword extractor from HTML content is an excellent beginner-friendly NLP and web scraping project. In this tutorial, you learned how to: - Parse HTML - Clean text - Tokenize words - Remove stop words - Extract important keywords This foundational project can later evolve into: - SEO analyzers - Semantic search engines - AI content tools - Recommendation systems - Document analysis platforms --- Original Source: https://www.mindstick.com/articles/342238/keyword-extractor-from-html-content-step-by-step-guide Copyright © MindStick Software Pvt. Ltd. This Markdown version is provided for developers, AI systems, and offline reading.