Services
Technology Web App Development Mobile App Development Windows App Development Database Development UI/UX Development Cloud Development SEO Digital Marketing Guest Posting Content Writing Artificial Intelligence Data Science ERP Development CRM Development CMS Development LMS Development Business Listing Blockchain Technology Startup Promotion AI Tools Integration
Products
Data Converter Cleaner TUC - Unit Converter Import Export Survey Manager Tweet Controller
Developer
Article Blog Forum Interview Beginner News Startup Quiz
Users
Career
Apply Job Internship
Pricing
Login Sign Up Get a Quote

forum

home / developersection / forums / what is lexical analysis in a compiler, and how is it done?
Ask Question
Utpal Vishwas
Utpal Vishwas

Student

I am Utpal Vishwas from Uttar Pradesh. Have completed my B. Tech. course from MNNIT campus Prayagraj in 2022. I have good knowledge of computer networking.

Message

1 Answers

Krishnapriya Rajeev
Krishnapriya Rajeev 10 Apr 2023
Report Abuse

Lexical analysis, also known as tokenization, is the process of breaking down the source code of a programming language into its fundamental building blocks called tokens. A token is a sequence of characters that represents a unit of meaning in the programming language, such as keywords, identifiers, literals, operators, and punctuations.

The lexical analyzer is responsible for performing lexical analysis. The lexical analyzer reads the input source code character by character and generates a stream of tokens that the parser can use to build the abstract syntax tree of the program.

The process of lexical analysis can be summarized in the following steps:

  • Input buffering: The lexical analyzer reads the input source code in chunks, called buffers, to reduce the I/O overhead.
  • Scanning: The lexical analyzer scans the input buffer character by character and identifies the tokens by matching them with the regular expressions defined in the lexer specification.
  • Tokenization: The lexical analyzer generates a stream of tokens, where each token is represented by a type and an optional attribute value. For example, the token while might be represented as a keyword token type, and its attribute might be the string "while".
  • Error handling: The lexical analyzer reports lexical errors, such as invalid tokens or unrecognized characters, to the parser.

The output of the lexical analyzer is a stream of tokens, which is passed to the parser for syntactic analysis. The parser uses the tokens to build a parse tree, which represents the syntactic structure of the program.

MindStick Training