feat: Infix filtering support #2685 by mdhaduk · Pull Request #2813 · typesense/typesense

mdhaduk · 2026-03-04T05:10:39Z

feat: infix filtering support (field:value) #2685

Typesense supports exact (field:value) and prefix (field:value*) filtering on string fields, but had no way to filter by substring , users couldn't express "give me all docs where this field contains this value."

This PR adds field:value syntax to filter_by, enabling infix/contains filtering on string and string array fields that have infix: true in their schema.

Changes:

Detection (filter_result_iterator.cpp): value is now detected as an infix pattern before the existing prefix (value*) check, avoiding misidentification.
Validation: Returns a 400 error if the field does not have infix: true... the infix index is only built when explicitly enabled.
Lookup: Reuses the existing Index::search_infix() function (already used for query-time infix search). For multi-token filter values (e.g. Chris P), results
are intersected (AND). Multiple filter values in a list (e.g. [foo, bar]) are OR'd together.
Result assembly: Results are stored as a flat filter_result_t (sorted uint32_t[]) -> the same pattern used by numeric and geo filters. Mixed infix + exact/prefix in a single OR list is handled by flattening posting lists and OR'ing with infix results before returning.

Tesing (CollectionFilteringTest.InfixFilterOnTextFields)

Basic single-token infix on string[] field (ris matches "Chris Evans", "Chris Parnell", "Chris Pine")
Infix on string field
No matches case
OR of multiple infix values ([ris, art])
Error on field without infix: true (400 response)
Mixed infix + exact in OR ([ris, Martin])
Mixed infix + prefix in OR ([ris, Ma*])
Infix combined with AND on another field (cast: ris && points: >60)

…aduk/typesense/issues/2685)

tharropoulos · 2026-03-10T13:42:57Z

CC: @kishorenc

tharropoulos · 2026-03-10T13:43:25Z

Is this ready for review?

mdhaduk · 2026-03-10T14:09:30Z

New contributor, so I just wanted a review of this draft before submitting as PR in case of any obvious pitfalls

Let me know if I should just submit as PR, I think repo does not run clean build on issue of draft PR

tharropoulos · 2026-03-19T14:15:39Z

@happy-san Could you check this out?

mdhaduk · 2026-04-28T15:40:39Z

Went ahead and just opened it since some time has passed!

happy-san · 2026-04-30T08:39:27Z

Hey @mdhaduk, The PR looks good! It is missing implementation for handling lazy evaluation of filter_by though. Let me know if you want to take a stab at that. What needs to be done:

Instead of collecting the seq_ids https://github.com/typesense/typesense/blob/v31/src/index.cpp#L3420-L3422 only need to collect the posting lists.
Using the existing advance_string_filter_token_iterators and get_string_filter_next_match should work otherwise the logic to be implemented will be similar.
Need to implement test cases similar to https://github.com/typesense/typesense/blob/v31/test/filter_test.cpp to verify everything.

mdhaduk · 2026-04-30T18:58:28Z

Hey @happy-san, thanks for taking a look! I’m happy to finish out the missing implementation for lazy evaluation of filter_by

…upport Changes: - include/index.h, src/index.cpp: add search_infix_leaves(); refactor search_infix() to call it - src/filter_result_iterator.cpp: lazy infix path via search_infix_leaves() + posting_list_iterators; 400 error for multi-token; remove has_infix_results block; remove #include <algorithm> - test/filter_test.cpp: add FilterTest.InfixLazyEvaluation (5 cases including multi-token 400 error) Assumptions: - Infix filter values must be single tokens. A value that tokenises to N > 1 tokens returns 400. This is safe: every existing test uses single-token patterns (*ris*, *pta*, etc.); no documented behaviour depended on multi-token AND semantics.

mdhaduk · 2026-04-30T22:04:30Z

Hi @happy-san, implemented the lazy evaluation as requested. Here's a summary of what changed:

Lazy evaluation path (filter_result_iterator.cpp)

Replaced the eager search_infix() call (which materialised all doc IDs into a flat array) with search_infix_leaves(), which returns art_leaf* pointers without merging doc IDs.
Each matching vocab token becomes one outer entry in posting_list_iterators, ORed lazily by the existing get_string_filter_next_match() via the CONTAINS comparator branch — no position verification is needed, which is correct for infix.
Removed the has_infix_results finalization block; the normal string_filter_ids_threshold check now applies uniformly to infix.

search_infix_leaves() (index.h, index.cpp)

Extracted the scan logic from search_infix() into a new function that stops before merging doc IDs. search_infix() now delegates to it (no behaviour change for query-time infix search).

Tests (test/filter_test.cpp)

Added FilterTest.InfixLazyEvaluation with 5 cases: basic lazy iteration, no-match, OR of two infix values, infix AND numeric, and a multi-token error case.

assumptions made:

multi-token infix (cast:foo bar) was removed and replaced with a 400 error. The infix index stores individual word tokens only, so substring search across word boundaries is architecturally impossible... the old path was producing semantically wrong AND results with no adjacency guarantee and had no test coverage. Users can write cast:foo && cast:bar as separate conditions instead.

happy-san · 2026-05-01T05:26:49Z

+    auto res_op = coll->search("*", {}, "name_no_infix: *foo*", {}, {}, {0}, 10, 1, FREQUENCY, {false});
+    ASSERT_FALSE(res_op.ok());
+    ASSERT_EQ(400, res_op.code());
+    ASSERT_NE(std::string::npos, res_op.error().find("infix"));


We should match the error message verbatim here.

happy-san · 2026-05-01T05:27:05Z

+                                                     filter_tree_root, enable_lazy_evaluation);
+    ASSERT_FALSE(iter_multi_token.init_status().ok());
+    ASSERT_EQ(400, iter_multi_token.init_status().code());
+    ASSERT_NE(std::string::npos, iter_multi_token.init_status().error().find("single token"));


Same here, we should match the error message verbatim.

happy-san · 2026-05-01T05:30:54Z

+    // Test 5: Multi-token infix must return 400 — *ris pin* tokenizes to ["ris", "pin"].
+    // The infix index stores individual word tokens only; substring search across word
+    // boundaries is structurally impossible. Users should write cast:*ris* && cast:*pin* instead.
+    filter_op = filter::parse_filter_query("cast: *ris pin*", coll->get_schema(), store,


Let's add a test case having cast: *ris* && cast: *in* which should match doc id: 5.

… case - Replace substring find() checks with verbatim ASSERT_EQ on full error messages in FilterTest.InfixLazyEvaluation - Add FilterTest.InfixLazyEvaluation Test 5: cast:*ris* && cast:*in*

mdhaduk · 2026-05-01T07:07:46Z

Just pushed the small updates regarding error messages + cast: *ris* && cast: *in* test case!

happy-san · 2026-05-01T08:36:01Z

+        R"({"title": "Good Will Hunting", "cast": ["Matt Damon", "Ben Affleck"], "name_no_infix": "qux", "points": 83})"_json,
+        R"({"title": "Percy Jackson", "cast": ["Logan Lerman", "Alexandra Daddario"], "name_no_infix": "quux", "points": 59})"_json,
+        R"({"title": "Quantum Quest", "cast": ["Chris Pine"], "name_no_infix": "corge", "points": 52})"_json,
+    };


Let's add these final tests as well

{ "q": "will", "query_by": "title", "filter_by": "cast: [*an*, *in*]", "enable_lazy_filter": true }

which should match document id: 2.

{ "q": "will", "query_by": "title", "filter_by": "cast: ! [*an*, *in*]", "enable_lazy_filter": true }

which should match document id: 3.

You can call search like this as well: https://github.com/typesense/typesense/blob/v31/test/collection_filtering_test.cpp#L181-L198

We also haven't tested what happens when a filter like cast:= *in* is sent. := is the exact match filter operator.

That's a good point, will add this as well!

You can call search like this as well: https://github.com/typesense/typesense/blob/v31/test/collection_filtering_test.cpp#L181-L198

Ah I see because coll->search doesn't expose enable_lazy_filter...

It does, but you would've ended up initializing way too much stuff to get to it https://github.com/typesense/typesense/blob/v31/include/collection.h#L1084

It does, but you would've ended up initializing way too much stuff to get to it https://github.com/typesense/typesense/blob/v31/include/collection.h#L1084

I see I missed it, that makes sense

…ests Note: Tests 10 and 11 use collectionManager.do_search() with req_params to exercise the enable_lazy_filter HTTP parameter, which is not exposed through the coll->search()

mdhaduk · 2026-05-01T11:36:07Z

Updated collection_filtering_test with those three additional tests as outlined

happy-san · 2026-05-01T12:13:12Z

@kishorenc PR looks good to me. Ready for your review.

mdhaduk · 2026-05-04T18:09:43Z

Any update on this?

feat: Infix filtering support [typesense#2685](https://github.com/mdh…

a344811

…aduk/typesense/issues/2685)

mdhaduk mentioned this pull request Mar 4, 2026

[Feature Request] Infix filtering support #2685

Open

mdhaduk marked this pull request as ready for review April 28, 2026 15:40

mdhaduk added 2 commits April 30, 2026 16:42

Merge branch 'typesense:v31' into feat/2685-infix-filtering-support

a08ef89

happy-san reviewed May 1, 2026

View reviewed changes

test(infix-filter): verbatim error assertions + AND-of-two-infix test…

5614ac4

… case - Replace substring find() checks with verbatim ASSERT_EQ on full error messages in FilterTest.InfixLazyEvaluation - Add FilterTest.InfixLazyEvaluation Test 5: cast:*ris* && cast:*in*

happy-san reviewed May 1, 2026

View reviewed changes

test(infix-filter): add enable_lazy_filter and exact-match operator t…

bb83604

…ests Note: Tests 10 and 11 use collectionManager.do_search() with req_params to exercise the enable_lazy_filter HTTP parameter, which is not exposed through the coll->search()

Merge branch 'v31' into feat/2685-infix-filtering-support

fc9f590

Uh oh!

Conversation

mdhaduk commented Mar 4, 2026

Uh oh!

tharropoulos commented Mar 10, 2026

Uh oh!

tharropoulos commented Mar 10, 2026

Uh oh!

mdhaduk commented Mar 10, 2026

Uh oh!

tharropoulos commented Mar 19, 2026

Uh oh!

mdhaduk commented Apr 28, 2026

Uh oh!

happy-san commented Apr 30, 2026

Uh oh!

mdhaduk commented Apr 30, 2026

Uh oh!

mdhaduk commented Apr 30, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mdhaduk commented May 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

happy-san May 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mdhaduk commented May 1, 2026

Uh oh!

happy-san commented May 1, 2026

Uh oh!

mdhaduk commented May 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

happy-san May 1, 2026 •

edited

Loading