4. Lexer and Parser Definition
The lexical analyzer defines how the contents of a file are broken into tokens, which is the basis for supporting custom language features. The easiest way to create a lexer is to use JFlex.
Define a Lexer
Define a Simple.flex
file with rules for the Simple Language lexer in package org.intellij.sdk.language
.
Generate a Lexer Class
Now generate a lexer class via from the context menu on Simple.flex file.
The Grammar-Kit plugin uses the JFlex lexer generation. When running for the first time, JFlex prompts for a destination folder to download the JFlex library and skeleton. Choose the project root directory, for example code_samples/simple_language_plugin.
After that, the IDE generates the lexer under the gen directory, for example in simple_language_plugin/src/main/gen/org/intellij/sdk/language/SimpleLexer.
Define a Lexer Adapter
The JFlex lexer needs to be adapted to the IntelliJ Platform Lexer API. Implement SimpleLexerAdapter
by subclassing FlexAdapter
.
Define a Root File
The SimpleFile
implementation is the top-level node of the tree of PsiElements
for a Simple Language file.
Define SimpleTokenSets
Define all sets of related token types from SimpleTypes
in SimpleTokenSets
.
Define a Parser
The Simple Language parser is defined in SimpleParserDefinition
by subclassing ParserDefinition
. To avoid unnecessary classloading when initializing the extension point implementation, all TokenSet
return values should use constants from dedicated $Language$TokenSets
class.
Register the Parser Definition
Registering the parser definition in the plugin.xml file makes it available to the IntelliJ Platform. Use the com.intellij.lang.parserDefinition
extension point for registration. For example, see simple_language_plugin/src/main/resources/META-INF/plugin.xml.
Run the Project
Run the plugin by using the Gradle runIde
task.
Create a test.simple file with the following content:
Now open the PsiViewer tool window and check how the lexer breaks the content of the file into tokens, and the parser transforms the tokens into PSI elements.