---
title: "What is lexical analysis in a compiler, and how is it done?"  
description: "What is lexical analysis in a compiler, and how is it done?"  
author: "Utpal Vishwas"  
published: 2023-04-07  
updated: 2023-04-10  
canonical: https://www.mindstick.com/forum/157731/what-is-lexical-analysis-in-a-compiler-and-how-is-it-done  
category: "compiler design"  
tags: ["computer hardware", "computer system"]  
reading_time: 2 minutes  

---

# What is lexical analysis in a compiler, and how is it done?

What is lexical [analysis](https://www.mindstick.com/articles/323533/prospective-analysis) in a [compiler](https://www.mindstick.com/articles/27/just-in-time-compiler), and how is it done?

## Replies

### Reply by Krishnapriya Rajeev

Lexical analysis, also known as tokenization, is the process of *breaking down the source code* of a programming language into its fundamental building blocks called *tokens*. A token is a sequence of characters that represents a unit of meaning in the programming language, such as keywords, identifiers, literals, operators, and punctuations.

The *lexical analyzer* is responsible for performing lexical analysis. The lexical analyzer reads the input source code character by character and generates a stream of tokens that the parser can use to build the abstract syntax tree of the program.

The process of lexical analysis can be summarized in the following steps:

- **Input buffering:** The lexical analyzer reads the input source code in chunks, called buffers, to reduce the I/O overhead.
- **Scanning:** The lexical analyzer scans the input buffer character by character and identifies the tokens by matching them with the regular expressions defined in the lexer specification.
- **Tokenization:** The lexical analyzer generates a stream of tokens, where each token is represented by a type and an optional attribute value. For example, the token while might be represented as akeywordtoken type, and its attribute might be the string "while".
- **Error handling:** The lexical analyzer reports lexical errors, such as invalid tokens or unrecognized characters, to the parser.

The output of the lexical analyzer is a *stream of tokens*, which is passed to the parser for syntactic analysis. The parser uses the tokens to build a parse tree, which represents the syntactic structure of the program.


---

Original Source: https://www.mindstick.com/forum/157731/what-is-lexical-analysis-in-a-compiler-and-how-is-it-done

Copyright © MindStick Software Pvt. Ltd. This Markdown version is provided for developers, AI systems, and offline reading.