Indexing and PSI Stubs
Indexes
The indexing framework provides a quick way to locate specific elements, e.g., files containing a certain word or methods with a particular name, in large codebases. Plugin developers can use the existing indexes built by the IDE itself and build and use their own indexes.
It supports two main types of indexes:
File-based indexes are built directly over the content of files. Stub indexes are built over serialized stub trees. A stub tree for a source file is a subset of its PSI tree, which contains only externally visible declarations and is serialized in a compact binary format.
Querying a file-based index gets you the set of files matching a specific condition. Querying a stub index gets you the set of matching PSI elements. Therefore, custom language plugin developers typically use stub indexes in their plugin implementations.
Dumb Mode
Indexing is a potentially lengthy process. It's performed in the background, and during this time, IDE features are restricted to the ones that don't require index: basic text editing, version control, etc. This restriction is managed by DumbService. Violations are reported via IndexNotReadyException, please see its javadoc on how to adapt callers.
DumbService provides API to query whether the IDE is currently in "dumb" mode (where index access is not allowed) or "smart" mode (with all index built and ready to use). It also provides ways of delaying code execution until indexes are ready. Please see its JavaDoc for more details.
Gists
Sometimes, the following conditions hold:
The aggregation functionality of file-based indexes is not needed. One just needs to calculate some data based on a particular file's contents and cache it on disk.
Eagerly calculating the data for the entire project during indexing isn't needed (e.g., it slows down the indexing, and/or this data probably will ever be required for a minor subset of all project files).
The data can be recalculated lazily on request without significant performance penalties.
A file-based index can be used in such cases, but file gists provide a way to perform data calculation lazily, caching on disk, and a more lightweight API. Please see VirtualFileGist and PsiFileGist documentation.
Example:
VirtualFileGist:ImageInfoIndexcalculating image dimensions/bit depth needed to be displayed in specific parts of UI.PsiFileGist:JavaSimplePropertyGistproviding simple properties in Java
Improving Indexing Performance
Performance Metrics
Indexing performance metrics in JSON format are generated in logs directory (see sandbox directory for development instance) in 2020.2 and later. These are additionally available in HTML format starting with 2021.1.
Avoid Using AST
Use lexer information instead of parsed trees if possible.
If impossible, use light AST which doesn't create memory-hungry AST nodes inside, so traversing it might be faster. Obtain LighterAST by casting FileContent input parameter to PsiDependentFileContent and calling getLighterAST(). Make sure to traverse only the nodes you need to. See also LighterASTNodeVisitor and LightTreeUtil for useful utility methods.
For stub index, implement LightStubBuilder.
If a custom language contains lazy-parseable elements that never or rarely contain any stubs, consider implementing StubBuilder.skipChildProcessingWhenBuildingStubs() (preferably using Lexer/node text).
For indexing XML, also consider using NanoXmlUtil.
Consider Prebuilt Stubs
If your language has a massive standard library, which is mostly the same for all users, you can avoid stub-indexing it in each installation by providing prebuilt stubs with your distribution. See PrebuiltStubsProvider extension.