Skip to content

About

PostgreSQL tokenizer extension for full-text search, fork by PGSTY

Topics

Resources

Stars

2 stars

Watchers

1 watching

Forks

Repository files navigation

pg_tokenizer

Website: vchord.pgsty.com Docs: tokenizer Catalog: tokenizer Version: v1.2.0 License: Apache-2.0 GitHub Stars PostgreSQL: 15-19

Multilingual text analysis and vocabulary tokenization for PostgreSQL 15–19.

Documentation · Releases · Issues

Features

  • Configure character filters, pre-tokenizers, and token filters through TOML.
  • Analyze European languages with Snowball stemmers, Chinese with Jieba, and Japanese or Korean with Lindera.
  • Use built-in vocabularies, Hugging Face models, or custom vocabularies trained on table data.
  • Return text tokens, integer token IDs, or positioned tsvector values for vchord_bm25.

Installation

Install from the Pigsty package repository:

pig repo add pgsql -u
pig ext add pg_tokenizer

Packages and source archives are also available from GitHub Releases. See the installation guide for platform and dictionary options.

Add pg_tokenizer to shared_preload_libraries, restart PostgreSQL, then enable the extension:

CREATE EXTENSION pg_tokenizer;

Usage

Create a text analyzer and convert text into tokens or a tsvector:

SELECT tokenizer_catalog.create_text_analyzer('words', $$
pre_tokenizer = "unicode_segmentation"
[[character_filters]]
to_lowercase = {}
$$);

SELECT tokenizer_catalog.apply_text_analyzer('PostgreSQL Full Text Search', 'words');
SELECT tokenizer_catalog.analyze_to_tsvector('PostgreSQL Full Text Search', 'words');

Use a vocabulary model to produce integer token IDs:

SELECT tokenizer_catalog.create_tokenizer('bert', $$
model = "bert_base_uncased"
$$);

SELECT tokenizer_catalog.tokenize('PostgreSQL Full Text Search', 'bert');
SELECT tokenizer_catalog.tokenize_to_tsvector('PostgreSQL Full Text Search', 'bert');

Store analyzed vectors in a column and build a BM25 index with vchord_bm25. Use the same analyzer or tokenizer for documents and queries. See examples, pipeline configuration, and the SQL reference.

Build from Source

Requires Rust 1.99.1+, PostgreSQL development headers, and matching Clang, libclang, and LLVM tools.

cargo install cargo-pgrx --version 0.20.0 --locked
git clone https://github.com/pgsty/pg_tokenizer.git
cd pg_tokenizer
PG_CONFIG=/path/to/pg_config make build
sudo make PG_CONFIG=/path/to/pg_config install

The Makefile enables lindera-ipadic. Other dictionaries are optional Cargo features; external dictionaries must be compiled with Lindera 6. Set CLANG_PATH and LIBCLANG_PATH if multiple LLVM versions are installed, and CLANG_NO_DEFAULT_CONFIG=1 for Clang installations that load a default configuration file.

Upgrading

Install the new library and SQL files, restart PostgreSQL, then run in each database:

ALTER EXTENSION pg_tokenizer UPDATE;

See the upgrade guide for earlier catalog and tokenizer changes that may require data migration or regenerating stored tokens.

Testing

cargo test --locked --features 'pg19 lindera-ipadic'
cargo clippy --locked --features 'pg19 lindera-ipadic'
sqllogictest --db "$USER" --user "$USER" './tests/**/*.slt'

SQL tests require a running PostgreSQL instance with the extension installed. See development for the integration, preload, and standby test commands.

License

Apache License 2.0.

Copyright (c) 2025 TensorChord Inc. and 2026 PGSTY PTE. LTD. Maintained by Ruohang Feng at PGSTY.

About

PostgreSQL tokenizer extension for full-text search, fork by PGSTY

Topics

Resources

Stars

2 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages