Grammatically Informed Sentence Segmentation for Semantic Search

Full report

On this page

Executive summary

For English semantic search, the most effective strategy is usually not to replace whole-sentence indexing with a single linguistic segmentation method, but to build a multi-view index: keep the original sentence, add clause-level segments derived from dependency or constituency parses, add base noun phrases and selected verb phrases, and optionally add semantic-role views such as predicate-argument tuples. In practice, that means using a fast syntactic backbone such as spaCy or Stanza, adding constituency parsing when phrase boundaries matter, and using SRL selectively for high-value corpora where “who did what to whom” improves retrieval. This recommendation follows from how the major open-source tools are designed: spaCy emphasizes production speed and offers dependency parsing, NER, noun chunks, and configurable pipelines; Stanza offers a broader linguistically oriented stack including dependency parsing, constituency parsing, NER, and coreference; AllenNLP still provides convenient SRL and coreference predictors, but its ecosystem is archived and in maintenance mode; Hugging Face provides flexible transformer pipelines for token classification and other tasks, but not a single unified English parsing stack; benepar remains one of the strongest open constituency parsers; and UDPipe is a strong lightweight UD-oriented parser/tagger with broad language coverage and simple CLI workflows.

The core design principle is to split only where the grammar gives you evidence that meaning can stand on its own. In English, the most useful boundaries are typically: coordinated clauses (conj/cc), clausal complements (ccomp, xcomp), adverbial clauses (advcl), noun-modifying clauses (acl, acl:relcl), and paratactic side clauses (parataxis). Quoted speech needs special handling because UD treats complete reported content as ccomp, but interrupted quotations as parataxis. Base noun phrases are valuable because they are short, stable retrieval units, while SRL gives a robust proposition view that can normalize active/passive alternations and improve matchability for event-centric search.

For a practical production recommendation, a strong baseline is: spaCy en_core_web_lg or en_core_web_trf for tokenization, sentence splitting, POS, dependencies, and NER; benepar or Stanza for constituency spans when you need reliable NP/VP/clause trees; optional Stanza coreference for document-level pronoun resolution; and AllenNLP SRL only in a pinned legacy environment, or a task-specific Hugging Face model if you prefer current transformer infrastructure. Keep both original and masked/normalized segment variants in the index, because removing proper nouns can improve semantic generalization, but over-masking can destroy discriminative lexical evidence. Use BM25 or another lexical index as a first-stage baseline, because BEIR shows BM25 remains robust across domains, and use embeddings and reranking on top for higher recall and semantic matching.

Linguistic foundations

What the relevant analyses contribute

Constituency parsing gives nested phrase-structure trees such as S, NP, and VP, which are especially useful when you want clean phrase spans for indexing, for example extracting a full noun phrase with modifiers or separating a subordinate clause from a matrix clause. Stanza’s constituency processor explicitly adds a phrase-structure parse tree to each sentence, and benepar exposes constituent labels and child spans through integrations with spaCy and NLTK. Constituency is the clearest route when you want phrase spans that look like traditional grammar.

Dependency parsing gives head-dependent relations over tokens, with one root and labeled relations such as subject, object, complement, modifier, and conjunction. This representation is especially useful for semantic search segmentation because it directly encodes predicate-argument structure and coordination. Universal Dependencies defines the syntax layer as a typed dependency tree and lists the standard relation inventory used across many English pipelines. spaCy’s parser is transition-based and jointly learns sentence segmentation and labeled dependency parsing; Stanza’s dependency parser follows the UD formalism as well.

POS tagging and morphological tagging are the lightest-weight signals and often the first useful filter. They are essential for detecting proper nouns (PROPN), finite verbs, auxiliaries, coordinating conjunctions, and punctuation classes. Stanza’s POS processor emits UPOS, XPOS, and morphological features; spaCy exposes Token.pos_ and related token attributes; UD defines PROPN as the class for names of specific individuals, places, or objects.

Chunking or shallow parsing sits between POS tagging and full parsing. The CoNLL-2000 shared task defines chunking as dividing text into syntactically related, non-overlapping groups of words. In modern practice, spaCy’s Doc.noun_chunks is a widely used approximation for base noun phrases, but it is intentionally limited: it excludes NP-level coordination, prepositional phrases, and relative clauses. That makes noun chunks excellent index units, but not a replacement for full NP extraction from constituency parses.

Clause splitting is usually not a separate pretrained task in these libraries; it is an algorithm you build on top of parses. The key relations are ccomp for clausal complements, xcomp for controlled open complements, advcl for adverbial clauses, acl and acl:relcl for noun-modifying clauses, and parataxis for side-by-side clauses or parentheticals. These provide a principled inventory of segments that are closer to propositions than raw fixed-length windows.

Coordination resolution matters because English packs a lot of searchable meaning into and, or, and but. UD treats coordination asymmetrically for technical reasons: the first conjunct is the technical head, later conjuncts attach with conj, and the conjunction token attaches with cc. That means naive string splitting can be wrong; you should split on parse relations, not on surface conjunctions alone.

Semantic role labeling adds another layer: predicate-argument semantics. PropBank describes predicate-argument annotation over Penn Treebank syntax, and AllenNLP’s SRL predictor returns per-verb structures containing the sentence tokens and BIO-style role tags. This is often the cleanest representation for event retrieval or relation-style search because it normalizes syntactic variation into roles such as agent, patient, temporal modifier, and location.

Named entity recognition detects named spans such as person, organization, geopolitical entity, date, or money. spaCy’s NER identifies contiguous labeled spans, and Hugging Face exposes token-classification pipelines with optional aggregation strategies for turning token predictions into entity spans. For semantic search segmentation, NER is the best first-line tool for masking or canonicalizing proper names before indexing.

Coreference resolution links mentions that refer to the same entity. That is crucial when you extract short segments, because many useful fragments otherwise become underspecified: “it,” “they,” “the company,” and “the team” are often only interpretable in document context. Stanza added a coreference model in version 1.7.0; spaCy offers experimental token-level coreference plus a span resolver in spacy-experimental; AllenNLP exposes a coreference predictor and helper methods for producing a resolved document.

What each representation is best for

The most important practical distinction is this: dependency representations are usually better for proposition extraction, while constituency representations are usually better for span extraction. If your search index needs “main clause,” “because-clause,” “subject + verb + object,” or “predicate with its obliques,” dependency parsing is the most direct substrate. If your index needs robust NP, VP, and SBAR-like spans, or if you want to avoid ad hoc span assembly from dependency subtrees, constituency parsing is the better fit. A high-quality production system often uses both, but with dependency parsing as the backbone and constituency parsing only for span cleanup and ambiguity checks.

Tooling landscape

Comparative view of the main open-source options

Recommended configurations

For most English semantic-search systems, the best practical choices are:

The main architectural decision is whether you want a single-tool pipeline or a hybrid pipeline. A single-tool pipeline is simpler to operate, but a hybrid usually wins on quality because it lets each component do the task it is best at: spaCy for fast production parsing and NER, benepar or Stanza for constituency spans, and SRL as an optional proposition layer. For semantic search, the hybrid is usually worth it when your corpus is high-value and query quality matters more than absolute throughput.

Segmentation algorithms

A recommended extraction strategy

The extraction logic should be hierarchical rather than flat:

1. Keep the original sentence as a parent segment.
2. Extract the matrix clause around the sentence root.
3. Extract subordinate clauses identified by UD relations.
4. Split coordinated clauses or coordinated phrases only when the parse shows a real coordination structure.
5. Extract base NPs and optionally VP-like spans.
6. Add SRL propositions as a parallel view.
7. Generate masked and optionally coref-resolved variants.
8. Rank, deduplicate, and index all variants with parent-child links.

graph LR
A[Raw document] --> B[Sentence segmentation]
B --> C[POS + dependency parse + NER]
C --> D[Clause extraction]
C --> E[Coordination splitting]
C --> F[Noun chunk extraction]
B --> G[Constituency parse]
B --> H[Optional SRL]
A --> I[Optional coreference]
D --> J[Candidate segments]
E --> J
F --> J
G --> J
H --> J
I --> K[Coref-resolved variants]
J --> L[Entity masking and normalization]
K --> L
L --> M[Ranking and deduplication]
M --> N[Lexical index]
M --> O[Embedding index]
M --> P[Metadata / graph links]

This pipeline is directly motivated by the official capabilities and relation inventories of the major tools: dependency parsing for clause and coordination structure, constituency parsing for phrase spans, SRL for predicate-argument views, and coreference for referential recovery.

Rules for masking proper nouns and named entities

A useful masking policy is:

• Mask explicit NER spans first, because learned NER is more reliable than POS alone for multi-token names such as “Bank of England” or “New York Stock Exchange.”
• Fall back to PROPN runs only for uncaught names.
• Replace spans with typed placeholders such as [PERSON], [ORG], [GPE], [DATE].
• Preserve count and local identity when needed for retrieval, e.g. [ORG_1] acquired [ORG_2].
• Keep a dual representation: the original text and the masked text.

Recommended refinement rules:

• Preserve head common nouns around masked names when they carry semantics: “CEO Tim Cook” should become CEO [PERSON], not just [PERSON].
• Treat appositive and title constructions conservatively; in search, titles and roles are often more useful than the name.
• Do not mask quantities, dates, or money by default unless your use case explicitly wants abstraction over them; these are often strong search anchors. spaCy’s NER label set includes types such as MONEY, DATE, and GPE, so you can choose selectively.

A strong operating pattern is to store three text views per segment:

• surface_text: original text
• masked_text: named entities and PROPN normalized
• lemma_text: lowercased, lemmatized lexical form

This supports exact lexical search, generalized search, and embedding generation from a normalized variant. The value of this three-view design is an inference from the available token, lemma, POS, dependency, and entity annotations exposed by spaCy and Stanza.

Rules for coordination splitting

Use the dependency parse as the source of truth. In UD, a later conjunct attaches to the first conjunct with conj, and the conjunction token attaches with cc. That means the correct algorithm is:

• Find heads with one or more conj children.
• Gather the full conjunct set: head + all conj siblings.
• Classify the coordination as clausal, VP-like, NP-like, or other from the POS and local dependents of each conjunct.
• Split only if the conjuncts are plausible standalone search units.
• Always retain the unsplit parent segment too.

Practical disambiguation rules:

• If a conjunct is a verb/adjective/noun predicate with its own subject, object, complement, or clause marker, treat it as clausal and emit one clause per conjunct.
• If the conjuncts are nouns, adjectives, or adverbs within a shared predicate, emit phrase segments rather than inventing full clauses.
• If the structure is a comma-separated list with a final and/or, emit both individual items and the aggregate list.
• If the parser confidence is uncertain or the phrase is classically ambiguous, such as “old men and women,” do not atomize aggressively; use constituency parsing as a second opinion and keep the unsplit span.

Rules for main clauses, subordinate clauses, verb phrases, noun phrases, and semantic roles

For main clauses, start at the sentence root and collect the minimal proposition around it: subject (nsubj / nsubj:pass), auxiliaries/copula, negation, objects (obj, iobj where appropriate), and selected obliques (obl) that are semantically integral, such as location or time. Exclude subordinate clausal dependents at first; index them separately.

For subordinate clauses, take heads whose dependency relation is one of:

• ccomp: a clausal complement that functions like a core argument
• xcomp: an open clausal complement with inherited subject
• advcl: a non-core adverbial clause
• acl / acl:relcl: noun-modifying clauses
• parataxis: side-by-side clauses, parentheticals, some interrupted quotations

Use mark dependents such as that, because, if, while, and infinitival to to label the clause type or connective. This is very useful for ranking because causal, conditional, concessive, and purpose clauses often have different retrieval value.

For noun phrases, use noun_chunks when you only need base NPs, and constituency NP nodes when you want larger spans that include internal PP or clause structure. Remember that spaCy’s noun chunks deliberately exclude NP coordination, PP attachment, and relative clauses, so they are excellent compact keys but not full semantic mentions.

For verb phrases, constituency parsers give the cleanest VP spans. If you only have dependencies, build VP-like spans from the verbal head plus auxiliaries, particles, negation, selected adverbs, and optionally objects/obliques depending on whether you want a predicate-only or predicate-plus-arguments representation. Dependency parses do not natively define VP spans, so this is a derived representation.

For semantic roles, run SRL over each sentence and emit one record per predicate. A useful search-oriented projection is:

• predicate_lemma
• ARG0
• ARG1
• ARG2+
• ARGM-TMP, ARGM-LOC, ARGM-MNR, ARGM-CAU, and so on

This gives you proposition-index keys that remain useful even when the surface syntax changes between active and passive or between direct and nominalized expression.

Handling punctuation, lists, and quoted text

Punctuation should mostly be treated as evidence about structure, not as the structure itself. Commas inside coordination often accompany the conj chain; semicolons and colons often correlate with parataxis or side-by-side propositions; and parentheses and dashes often indicate parenthetical material that should be indexed both separately and as part of the parent sentence. UDPipe’s documentation is also a useful reminder that preserving spaces and token ranges can matter when reconstructing segments faithfully.

Quoted text deserves special treatment. UD states that reported speech content usually attaches as ccomp, even for direct quotation, but if the speech verb interrupts the quoted content, parataxis is used instead. That means the safest policy is:

• Preserve the full quoted span as a segment.
• Preserve attribution separately when interrupted.
• Link quote content to speaker/verb metadata rather than flattening it away.

flowchart TD
A[Sentence] --> B{Quoted / list / parenthetical?}
B -- Quoted --> C[Preserve quote span]
C --> D{Interrupted attribution?}
D -- Yes --> E[Use parataxis branch]
D -- No --> F[Use ccomp branch]
B -- List --> G[Extract list items and aggregate list]
B -- Plain --> H[Find root predicate]
H --> I{Coordination present?}
I -- Clause-level --> J[Split conjunct clauses + keep parent]
I -- Phrase-level --> K[Split conjunct phrases + keep parent]
J --> L[Extract subordinate clauses]
K --> L
L --> M[Extract NP / VP / SRL views]
M --> N[Mask NER / PROPN]
N --> O[Score and index]

A worked example

Take this sentence:

“Acme Corp bought Beta Systems in London, but it kept the research team in Boston because the product was profitable.”

A good extractor would ideally produce at least these variants:

• Acme Corp bought Beta Systems in London
• it kept the research team in Boston
• the product was profitable
• the research team
• buy(ARG0=[ORG_1], ARG1=[ORG_2], ARGM-LOC=[GPE_1])
• keep(ARG0=[ORG_1], ARG1=the research team, ARGM-LOC=[GPE_2])
• masked variants such as [ORG_1] bought [ORG_2] in [GPE_1] and [ORG_1] kept the research team in [GPE_2]

If you also run coreference, you can create a resolved variant where it becomes [ORG_1]. The exact mechanism is supported by Stanza and AllenNLP coreference APIs, and spaCy’s experimental coref can also support token-level cluster creation and span resolution.

Ranking and index design

The best ranking strategy is to treat extraction as candidate generation, then use a learned or heuristic segment score before indexing. A simple and effective heuristic score can combine:

• segment type weight: main clause > subordinate clause > SRL proposition > VP > NP > list item
• syntactic completeness: has subject and predicate, or a clear predicate plus object/complement
• semantic density: more content words, fewer function words
• masking balance: neither entirely lexicalized with brittle names nor over-masked into generic placeholders
• position features: title, heading, topic sentence, or lead paragraph boosts when available
• redundancy penalty: downweight near-duplicates of the same sentence or proposition
• query-time compatibility: entity-type match, lemma overlap, embedding similarity, role match.

A useful deterministic baseline score is:

\[ \text{score} = 3.0 \cdot \text{main\_clause}

\]

• 2.0 \cdot \text{srl\_view}
• 1.5 \cdot \text{has\_subject}
• 1.5 \cdot \text{has\_object\_or\_complement}
• 1.0 \cdot \text{content\_density}
• 0.5 \cdot \text{entity\_balance}
• 1.0 \cdot \text{duplicate\_penalty}

Then normalize into a fixed range and keep the top k segments per sentence or paragraph. This formula is a recommendation, not a published standard, but it aligns with the structural distinctions encoded in UD and SRL outputs.

For semantic index keys, do not store only free text. Store structured fields too. A strong schema includes:

• predicate lemma
• masked text
• unmasked text
• noun chunk heads
• dependency head triples such as (buy, obj, systems)
• SRL fields such as ARG0, ARG1, ARGM-LOC
• entity bag with labels
• connective type (because, if, although)
• quote/list/parenthetical flags
• parent sentence and document IDs.

For retrieval architecture, use a dual index:

• a lexical index over surface_text, masked_text, lemmas, entities, and structural keys
• an embedding index over one or more normalized text views

This is the safest design because BEIR shows that BM25 remains a robust baseline across diverse tasks, while stronger neural and reranking methods often improve quality at higher compute cost. For segment retrieval, that means first-stage BM25 over structured fields, then embedding retrieval or reranking over a small candidate set.

Evaluation methodology

Intrinsic evaluation

Intrinsic evaluation should test whether the system extracted the right boundaries and right labels. Use:

• Exact-span precision / recall / F1 for extracted clauses, phrases, or entities.
• Boundary F1 if exact span equality is too strict.
• Overlap-based metrics such as span IoU or token-F1 for approximate matches.
• For component tasks, use the standard metrics of the underlying task:
• POS: tag accuracy
• dependency parsing: UAS and LAS
• constituency parsing: bracket precision / recall / F1
• chunking: precision / recall / F1
• NER: micro-F1
• SRL: CoNLL-style role F1
• coreference: either LEA or the CoNLL-style average over MUC, B-CUBED, and CEAFe.

The official UD evaluation page defines LAS as the percentage of words assigned both the correct head and dependency label. The CoNLL-2012 shared task discusses MUC, B-CUBED, and CEAF as the major coreference metrics and uses the unweighted mean of MUC, B-CUBED, and entity-based CEAFe as the official score. spaCy’s experimental scorer also exposes LEA for coreference clusters.

Human evaluation

Human judgment is indispensable because grammatically valid segmentation is not the same as search-useful segmentation. Ask annotators to rate:

• whether the segment is semantically complete enough to stand alone,
• whether it preserves the intended relation,
• whether masking removed too much meaning,
• whether the segment would help them find the right document faster.

Use at least pairwise comparisons against baselines such as full sentences, fixed token windows, and parser-only clause splits. This is a methodological recommendation, but it follows from the mismatch between component metrics and downstream retrieval goals emphasized by heterogeneous IR evaluation frameworks like BEIR.

Downstream retrieval evaluation

The most important test is whether segmentation helps retrieval. Compare at least these indexing strategies:

• full sentence only
• fixed-size sliding windows
• dependency clause segments
• constituency phrase + clause segments
• SRL proposition segments
• hybrid multi-view index

On a BEIR-style setup, report nDCG@10, Recall@100, and preferably MRR or MAP depending on the task. BEIR explicitly uses nDCG@10 and also reports Recall@100. In addition, track index size, average segments per sentence, candidate recall before reranking, and latency per query.

Suggested English datasets and corpora

A practical point is licensing: several classic English datasets, especially Penn Treebank and OntoNotes, are distributed through LDC and are not frictionless for open benchmarking. If you need an all-open setup, prioritize UD English EWT plus QA-SRL and retrieval corpora from BEIR.

Implementation blueprint

A practical pipeline plan

A good implementation plan has three phases.

Phase one builds the syntactic backbone with sentence segmentation, POS, dependency parsing, and NER. This alone is enough to create a strong clause-plus-NP index. Phase two adds constituency parsing to clean up phrase boundaries and coordination scope. Phase three adds optional SRL and coreference if the corpus contains many event queries or referentially sparse fragments. This sequencing matches the capabilities published by spaCy, Stanza, benepar, AllenNLP, and UDPipe, and it avoids paying SRL/coreference costs before your simpler pipeline is working well.

Example installation and API choices

The following choices are directly supported by the official docs or READMEs:

• spaCy models: en_core_web_sm/md/lg/trf and spacy.load(...) usage are documented in the models pages.
• benepar setup uses pip install benepar, benepar.download("benepar_en3"), and nlp.add_pipe("benepar", config={"model": "benepar_en3"}).
• Stanza uses stanza.Pipeline(lang="en", processors="...") for dependency, constituency, or coreference processors.
• AllenNLP predictor use is documented through pretrained.load_predictor(...) and predictor methods such as predict(...) and coref_resolved(...).
• Hugging Face token classification and NER can be loaded through pipeline("ner", ...) with aggregation strategies and device placement.
• UDPipe is well suited for batch CLI parsing with udpipe --tokenize --tag --parse ....

# Core dependencies for the hybrid Python pipeline
# Choose a pinned environment if you add AllenNLP.

# pip install spacy benepar stanza transformers torch
# python -m spacy download en_core_web_lg
# python -c "import benepar; benepar.download('benepar_en3')"
# python -c "import stanza; stanza.download('en')"

from __future__ import annotations

from dataclasses import dataclass, asdict
from typing import Any, Dict, Iterable, List, Optional

import spacy
import benepar
import stanza

# Optional: legacy SRL / coref
try:
    from allennlp_models import pretrained as allennlp_pretrained
except Exception:
    allennlp_pretrained = None


@dataclass
class Segment:
    doc_id: str
    sent_id: int
    seg_id: str
    seg_type: str                 # sentence | main_clause | sub_clause | np | vp | srl
    text: str
    masked_text: str
    head_lemma: Optional[str]
    connective: Optional[str]
    parent_seg_id: Optional[str]
    metadata: Dict[str, Any]


MASK_LABELS = {
    "PERSON", "ORG", "GPE", "LOC", "FAC", "NORP", "PRODUCT",
    "EVENT", "WORK_OF_ART", "LAW"
}


def build_spacy_pipeline() -> spacy.language.Language:
    nlp = spacy.load("en_core_web_lg")
    # benepar expects sentence boundaries to exist
    if "benepar" not in nlp.pipe_names:
        nlp.add_pipe("benepar", config={"model": "benepar_en3"})
    return nlp


def build_stanza_coref_pipeline() -> stanza.Pipeline:
    return stanza.Pipeline("en", processors="tokenize,coref", use_gpu=False)


def build_srl_predictor():
    if allennlp_pretrained is None:
        return None
    try:
        return allennlp_pretrained.load_predictor("structured-prediction-srl-bert")
    except Exception:
        return None


def mask_named_content(sent: spacy.tokens.Span) -> str:
    """Mask NER spans first; then fallback to uncaught PROPN runs."""
    replacement_by_token = {}
    entity_counter = {}

    for ent in sent.ents:
        if ent.label_ in MASK_LABELS:
            entity_counter[ent.label_] = entity_counter.get(ent.label_, 0) + 1
            repl = f"[{ent.label_}_{entity_counter[ent.label_]}]"
            replacement_by_token[ent.start] = (ent.end, repl)

    out = []
    i = sent.start
    while i < sent.end:
        if i in replacement_by_token:
            end, repl = replacement_by_token[i]
            out.append(repl)
            i = end
            continue

        tok = sent.doc[i]
        if tok.pos_ == "PROPN" and tok.ent_type_ == "":
            j = i + 1
            while j < sent.end and sent.doc[j].pos_ == "PROPN" and sent.doc[j].ent_type_ == "":
                j += 1
            out.append("[PROPN]")
            i = j
            continue

        out.append(tok.text_with_ws)
        i += 1

    return "".join(out).strip()


def span_text(span: spacy.tokens.Span) -> str:
    return span.text.strip()


def make_segment(doc_id: str, sent_id: int, seg_id: str, seg_type: str,
                 span: spacy.tokens.Span, parent_seg_id: Optional[str],
                 head_lemma: Optional[str] = None,
                 connective: Optional[str] = None,
                 metadata: Optional[Dict[str, Any]] = None) -> Segment:
    return Segment(
        doc_id=doc_id,
        sent_id=sent_id,
        seg_id=seg_id,
        seg_type=seg_type,
        text=span_text(span),
        masked_text=mask_named_content(span),
        head_lemma=head_lemma,
        connective=connective,
        parent_seg_id=parent_seg_id,
        metadata=metadata or {},
    )


def subtree_span(token: spacy.tokens.Token) -> spacy.tokens.Span:
    left = min(t.i for t in token.subtree)
    right = max(t.i for t in token.subtree) + 1
    return token.doc[left:right]


CLAUSAL_DEPS = {"ccomp", "xcomp", "advcl", "acl", "relcl", "acl:relcl", "parataxis"}


def extract_segments(doc_id: str, text: str, nlp, srl_predictor=None) -> List[Segment]:
    doc = nlp(text)
    segments: List[Segment] = []

    for sent_id, sent in enumerate(doc.sents):
        sent_seg_id = f"s{sent_id}"
        segments.append(make_segment(doc_id, sent_id, sent_seg_id, "sentence", sent, None))

        root = [t for t in sent if t.head == t][0]
        segments.append(
            make_segment(
                doc_id, sent_id, f"{sent_seg_id}_main", "main_clause",
                subtree_span(root), sent_seg_id, head_lemma=root.lemma_
            )
        )

        # Subordinate clauses
        for tok in sent:
            if tok.dep_ in CLAUSAL_DEPS:
                marker = next((c.text.lower() for c in tok.children if c.dep_ == "mark"), None)
                segments.append(
                    make_segment(
                        doc_id, sent_id, f"{sent_seg_id}_{tok.i}", "sub_clause",
                        subtree_span(tok), sent_seg_id,
                        head_lemma=tok.lemma_, connective=marker,
                        metadata={"dep": tok.dep_}
                    )
                )

        # Coordinated predicates / phrases
        for tok in sent:
            if tok.dep_ == "conj":
                seg_type = "coord_clause" if tok.pos_ in {"VERB", "AUX", "ADJ", "NOUN"} else "coord_phrase"
                segments.append(
                    make_segment(
                        doc_id, sent_id, f"{sent_seg_id}_conj_{tok.i}", seg_type,
                        subtree_span(tok), sent_seg_id,
                        head_lemma=tok.lemma_,
                        metadata={"cc": next((c.text for c in tok.children if c.dep_ == "cc"), None)}
                    )
                )

        # Base noun phrases
        for k, np in enumerate(sent.noun_chunks):
            segments.append(
                make_segment(
                    doc_id, sent_id, f"{sent_seg_id}_np_{k}", "np",
                    np, sent_seg_id, head_lemma=np.root.lemma_
                )
            )

        # SRL propositions
        if srl_predictor is not None:
            # AllenNLP SRL expects tokenized sentence for best alignment.
            words = [t.text for t in sent]
            try:
                srl = srl_predictor.predict_tokenized(words)
                for k, pred in enumerate(srl.get("verbs", [])):
                    segments.append(
                        Segment(
                            doc_id=doc_id,
                            sent_id=sent_id,
                            seg_id=f"{sent_seg_id}_srl_{k}",
                            seg_type="srl",
                            text=pred["description"],
                            masked_text=pred["description"],
                            head_lemma=pred["verb"],
                            connective=None,
                            parent_seg_id=sent_seg_id,
                            metadata={"tags": pred["tags"]},
                        )
                    )
            except Exception as exc:
                segments.append(
                    Segment(
                        doc_id=doc_id,
                        sent_id=sent_id,
                        seg_id=f"{sent_seg_id}_srl_error",
                        seg_type="debug",
                        text="",
                        masked_text="",
                        head_lemma=None,
                        connective=None,
                        parent_seg_id=sent_seg_id,
                        metadata={"error": str(exc)},
                    )
                )

    return segments


if __name__ == "__main__":
    nlp = build_spacy_pipeline()
    srl_predictor = build_srl_predictor()

    text = (
        "Acme Corp bought Beta Systems in London, but it kept the research team in Boston "
        "because the product was profitable."
    )

    segments = extract_segments("doc-1", text, nlp, srl_predictor=srl_predictor)
    for seg in segments:
        print(asdict(seg))

That code shows the shape of a usable hybrid pipeline: parse once, derive candidate spans from syntax, add noun chunks, optionally add SRL propositions, and produce original plus masked text for indexing. The exact segment inventory should be tuned empirically rather than hard-coded forever.

Example outputs to expect

For the example sentence above, a practical JSONL record set might look like this:

{"doc_id":"doc-1","sent_id":0,"seg_id":"s0","seg_type":"sentence","text":"Acme Corp bought Beta Systems in London, but it kept the research team in Boston because the product was profitable.","masked_text":"[ORG_1] bought [ORG_2] in [GPE_1], but it kept the research team in [GPE_2] because the product was profitable.","head_lemma":null}
{"doc_id":"doc-1","sent_id":0,"seg_id":"s0_main","seg_type":"main_clause","text":"Acme Corp bought Beta Systems in London","masked_text":"[ORG_1] bought [ORG_2] in [GPE_1]","head_lemma":"buy"}
{"doc_id":"doc-1","sent_id":0,"seg_id":"s0_conj_8","seg_type":"coord_clause","text":"it kept the research team in Boston because the product was profitable","masked_text":"it kept the research team in [GPE_1] because the product was profitable","head_lemma":"keep"}
{"doc_id":"doc-1","sent_id":0,"seg_id":"s0_14","seg_type":"sub_clause","text":"because the product was profitable","masked_text":"because the product was profitable","head_lemma":"profitable","connective":"because","metadata":{"dep":"advcl"}}
{"doc_id":"doc-1","sent_id":0,"seg_id":"s0_np_0","seg_type":"np","text":"Acme Corp","masked_text":"[ORG_1]","head_lemma":"Corp"}
{"doc_id":"doc-1","sent_id":0,"seg_id":"s0_np_1","seg_type":"np","text":"Beta Systems","masked_text":"[ORG_1]","head_lemma":"Systems"}
{"doc_id":"doc-1","sent_id":0,"seg_id":"s0_np_2","seg_type":"np","text":"the research team","masked_text":"the research team","head_lemma":"team"}

The important point is not the exact output text, which will vary by parser, but the coexistence of multiple segment granularities linked back to the same sentence and document. That structure is what makes retrieval robust.

Latency and complexity estimates

The official spaCy benchmark compares complete raw-text processing speed across libraries on 10,000 Reddit comments and reports approximately:

• spaCy en_core_web_lg: 10,014 WPS CPU
• spaCy en_core_web_trf: 684 WPS CPU
• Stanza en_ewt: 878 WPS CPU
• UDPipe english-ewt-ud-2.5: 1,101 WPS CPU

For a rough English sentence length of 30 tokens, that translates to approximate parser-layer latencies of:

These are back-of-the-envelope estimates derived from the published whole-pipeline WPS figures, so they are useful for capacity planning, not for SLA guarantees.

The expensive parts begin when you add constituency parsing, SRL, and especially coreference. benepar and Stanza constituency parsing are usually still practical at sentence scale, but coreference is document-level and can become memory-heavy on long documents. A later comparison paper reports that the popular AllenNLP coreference model required about 27GB of GPU memory and 12 minutes to process 2.8K OntoNotes documents on a V100 GPU, illustrating why coreference should be optional and carefully budgeted.

Pitfalls and open questions

The first major pitfall is over-splitting. Coordination, apposition, parentheticals, and relative clauses can all generate many plausible fragments, but not all fragments are useful search keys. The safest mitigation is to retain the parent sentence, emit only high-confidence child segments, and deduplicate aggressively. UD’s treatment of coordination and parataxis makes this feasible, but it still requires engineering judgment.

The second pitfall is over-masking named content. Proper nouns often create brittleness in embeddings and reduce recall across near paraphrases, but they are also frequently the very thing the user is searching for. The mitigation is to keep both surface and masked views and to let the retriever decide which one matters at query time. If you store only masked text, you will hurt exact-name search. If you store only surface text, you will hurt abstraction and generalization.

The third pitfall is parser disagreement. Dependency and constituency parsers will not always agree, and coordination scope is a classic failure case. In search systems, disagreement is not necessarily bad if you treat it as recall expansion rather than a single truth source: keep both interpretations when confidence is low, but rank the conservative one higher. benepar or Stanza constituency output is particularly helpful as a cross-check when dependency-only phrase segmentation looks brittle.

The fourth pitfall is coreference error propagation. Resolving pronouns can dramatically improve short segment search, but a bad coreference decision can corrupt many derived keys. The best mitigation is to keep coreference as an additional view, not a destructive rewrite of the only indexable text, and to apply it only where document-level anaphora is frequent enough to justify the cost. That tradeoff is especially important because coreference remains comparatively expensive and less turnkey than parsing or NER.

The fifth pitfall is evaluation mismatch. Better LAS, F1, or coref score does not guarantee better retrieval. The final arbiter should be downstream retrieval performance on a query set that reflects your actual application, ideally with BEIR-style metrics and manual judgments of segment usefulness.

The main open question is how much SRL and coreference help for your corpus specifically. The sources support their linguistic value and provide usable tooling, but there is no single official benchmark that answers the precise downstream question “Which segment inventory is optimal for semantic search over my English corpus?” The only reliable answer is an ablation study over your own retrieval tasks: sentence-only vs clause-only vs NP/VP-only vs SRL vs hybrid multi-view. That is the experiment I would treat as decisive.