Multilingual text analysis and vocabulary tokenization for PostgreSQL 15–19.
Documentation · Releases · Issues
- Configure character filters, pre-tokenizers, and token filters through TOML.
- Analyze European languages with Snowball stemmers, Chinese with Jieba, and Japanese or Korean with Lindera.
- Use built-in vocabularies, Hugging Face models, or custom vocabularies trained on table data.
- Return text tokens, integer token IDs, or positioned
tsvectorvalues for vchord_bm25.
Install from the Pigsty package repository:
pig repo add pgsql -u
pig ext add pg_tokenizerPackages and source archives are also available from GitHub Releases. See the installation guide for platform and dictionary options.
Add pg_tokenizer to shared_preload_libraries, restart PostgreSQL, then enable the extension:
CREATE EXTENSION pg_tokenizer;Create a text analyzer and convert text into tokens or a tsvector:
SELECT tokenizer_catalog.create_text_analyzer('words', $$
pre_tokenizer = "unicode_segmentation"
[[character_filters]]
to_lowercase = {}
$$);
SELECT tokenizer_catalog.apply_text_analyzer('PostgreSQL Full Text Search', 'words');
SELECT tokenizer_catalog.analyze_to_tsvector('PostgreSQL Full Text Search', 'words');Use a vocabulary model to produce integer token IDs:
SELECT tokenizer_catalog.create_tokenizer('bert', $$
model = "bert_base_uncased"
$$);
SELECT tokenizer_catalog.tokenize('PostgreSQL Full Text Search', 'bert');
SELECT tokenizer_catalog.tokenize_to_tsvector('PostgreSQL Full Text Search', 'bert');Store analyzed vectors in a column and build a BM25 index with vchord_bm25. Use the same analyzer or tokenizer for documents and queries. See examples, pipeline configuration, and the SQL reference.
Requires Rust 1.99.1+, PostgreSQL development headers, and matching Clang, libclang, and LLVM tools.
cargo install cargo-pgrx --version 0.20.0 --locked
git clone https://github.com/pgsty/pg_tokenizer.git
cd pg_tokenizer
PG_CONFIG=/path/to/pg_config make build
sudo make PG_CONFIG=/path/to/pg_config installThe Makefile enables lindera-ipadic. Other dictionaries are optional Cargo features; external dictionaries must be compiled with Lindera 6. Set CLANG_PATH and LIBCLANG_PATH if multiple LLVM versions are installed, and CLANG_NO_DEFAULT_CONFIG=1 for Clang installations that load a default configuration file.
Install the new library and SQL files, restart PostgreSQL, then run in each database:
ALTER EXTENSION pg_tokenizer UPDATE;See the upgrade guide for earlier catalog and tokenizer changes that may require data migration or regenerating stored tokens.
cargo test --locked --features 'pg19 lindera-ipadic'
cargo clippy --locked --features 'pg19 lindera-ipadic'
sqllogictest --db "$USER" --user "$USER" './tests/**/*.slt'SQL tests require a running PostgreSQL instance with the extension installed. See development for the integration, preload, and standby test commands.
Copyright (c) 2025 TensorChord Inc. and 2026 PGSTY PTE. LTD. Maintained by Ruohang Feng at PGSTY.