Extraction API

Every type and function that pulls structure out of an analysed document: the declarative spec types and their extractors, the heuristic field and checkbox detectors, the repeating-row extractor, master-detail grouping and multi-page stitching.

This is a lookup page. For how a capability is used — the locator model, .schema extraction, grid sections — see Fields, forms and checkboxes and Tables and repeating structures.

Two extraction stacks

The names below belong to the spec stack: Go spec structs in, schema.Instance out. The .schema (JSON Schema x-*) extractors that a rule reaches through bbox are a separate stack with its own configuration surface and its own result shapes. They share vocabulary, not code.

Stack	Configured by	Entry points	Returns
Spec	Go structs (schema.*Spec)	form.Extract, masterdetail.Extract, spread.Stitch	schema.Instance
Heuristic	Detector config structs	field.Detector, checkbox.Detector, field.RepeatingFieldExtractor	field.Field, checkbox.Checkbox, field.Record
.schema	JSON Schema x-* keywords	Document.ExtractForm, Document.CheckedOption*	map[string]any, query.FieldResult

The heuristic detectors run automatically during document analysis and fill Document.Fields, Document.Checkboxes and Document.CheckboxGroups. The spec-stack extractors run only when you call them.

Both the kis CLI and rules.svc reach the same analysed document, so every result described here is available on either surface. See Running rules over documents.

Form extraction

func Extract(doc *document.Document, spec schema.FormSpec) schema.Instance

For each FieldSpec, anchors are tried in order — the first that matches locates the label, the rest are fallbacks. The locator hint moves from the label to the value. A named pattern optionally validates and normalises it. Fields that do not resolve and are marked Required land in Instance.Missing.

schema.FormSpec

Field	Type	Meaning
Name	string	Form identifier; copied to Instance.Name
Region	*RegionSpec	Search scope; nil means the whole document
Fields	[]FieldSpec	Field definitions

schema.FieldSpec

Field	Type	Meaning
Name	string	Key under which the value lands in Instance.Values
Anchors	[]AnchorSpec	Ordered label candidates; first match wins, the rest are fallbacks
Locator	LocatorHint	How to move from label to value
Pattern	*PatternSpec	Named pattern the value must satisfy
Required	bool	When unresolved, adds Name to Instance.Missing and costs 0.1 confidence

schema.AnchorSpec

Field	Type	Meaning
Text	string	Anchor string, or the regex when MatchType is AnchorRegex
MatchType	AnchorMatch	Comparison style
Fuzzy	bool	Allow approximate matching up to MaxDistance
MaxDistance	int64	Edit-distance budget; a value of 0 disables fuzzy matching entirely

AnchorMatch constants, and what each does in form.Extract:

Constant	String()	Comparison
AnchorExact (zero value)	exact	Case-insensitive equality after trimming
AnchorContains	contains	Case-insensitive substring
AnchorStartsWith	starts_with	Case-insensitive prefix after trimming
AnchorEndsWith	ends_with	Case-insensitive suffix after trimming
AnchorRegex	regex	Text compiled as a Go regex; a compile error yields no match

Caution

Fuzzy is honoured only for AnchorExact and AnchorContains. AnchorStartsWith, AnchorEndsWith and AnchorRegex never fall back to fuzzy matching, even with Fuzzy: true and a positive MaxDistance. Fuzzy matching uses OCR-aware distance and picks the single closest block under MaxDistance.

schema.PatternSpec

Field	Type	Meaning
Name	string	Pattern name resolved against the document’s pattern registry
Required	bool	When the pattern does not match, skip this anchor and try the next

A matched pattern sets Value.PatternName, Value.Normalized and Value.Confidence from the pattern match. A non-required pattern that fails leaves the raw text and falls back to the anchor-derived confidence.

schema.LocatorHint

Constant	String()	Resolution order in form.Extract
LocUnknown (zero value)	unknown	Right of the label, then below
LocInline	inline	Split the label block on the first :; if there is no right-hand part, fall back to right-of
LocAfter	after	Right of the label, then below
LocBelow	below	Below the label, then right-of
LocInRegion	in_region	Right of the label only
LocInSection	in_section	Right of the label only

Right-of takes the nearest block on the same page whose X0 is at or past the label’s X1 and which is horizontally aligned within half a line. Below takes the nearest block on the same page whose Y0 is past the label’s Y1 and which overlaps the label horizontally.

schema.RegionSpec

Field	Type	Meaning
StartAnchor	[]AnchorSpec	Region begins below the first matching anchor on each page
EndAnchor	[]AnchorSpec	Region ends above the first matching anchor on that page
PageRange	*PageRange	Page window, applied before anchor bounding

PageRange is {First, Last int64}. First is inclusive; Last == 0 means “to the end of the document”.

Caution

An anchor-bounded region is a per-page vertical band, not a rectangle — X is never constrained. A page on which no start anchor matches is dropped entirely, so a form whose section header is mis-OCR’d on page 2 yields nothing from page 2.

Results

schema.Instance:

Field	Type	Meaning
Name	string	The FormSpec.Name that produced it
Values	map[string]Value	Extracted values keyed by FieldSpec.Name
Missing	[]string	Required field names that did not resolve
Region	BoxSpan	Per-page union of every value’s box
Confidence	float64	Aggregate score
Provenance	Provenance	Instance-level location; form.Extract leaves this zero

Accessors: Get(name) (Value, bool), Text(name) string, Has(name) bool — Has reports false for a present-but-empty value.

schema.Value:

Field	Type	Meaning
Text	string	Raw extracted string, whitespace-trimmed
Normalized	any	Type-specific normalised form; nil when no pattern matched
PatternName	string	Name of the pattern that matched
Confidence	float64	Per-value score
Provenance	Provenance	Where the value came from

schema.Provenance — the fields form.Extract populates:

Field	Type	Meaning
SourceText	string	The block text that produced the value
MatchedAnchor	string	The anchor Text that located the label
FuzzyDistance	int64	Edit distance of the anchor match; 0 for exact
Page	int64	Page the value was read from
Box	Box	Bounding box of the value block

schema.Box is {X0, Y0, X1, Y1 float64} with Width(), Height() and Empty(). schema.BoxSpan holds Pages []PageBox ({Page int64, Box Box}) with FirstPage() and LastPage(), both returning 0 for an empty span.

Confidence

An anchor match with no pattern scores 0.9, scaled down linearly by fuzzy distance: 0.9 * (1 - dist/(MaxDistance+1)). A matched pattern uses the pattern’s own confidence instead. Instance.Confidence is the mean of the per-field scores minus 0.1 * len(Missing), floored at 0 — and is 0 outright when no field resolved.

Example

spec := schema.FormSpec{
  Name: "terms",
  Fields: []schema.FieldSpec{{
    Name: "rate",
    Anchors: []schema.AnchorSpec{
      {Text: "Interest Rate", MatchType: schema.AnchorExact},
      {Text: "Annual Interest Rate", MatchType: schema.AnchorExact},
      {Text: "Note Rate", MatchType: schema.AnchorExact},
    },
    Locator:  schema.LocAfter,
    Required: true,
  }},
}

in := form.Extract(doc, spec)
rate := in.Text("rate")

Heuristic field detection

func NewDetector() *Detector
func (d *Detector) DetectFields(blocks []text.Block, lines []text.Line) []Field
func (d *Detector) ExtractByDefinition(def FieldDef, blocks []text.Block, lines []text.Line) *Field

DetectFields needs no spec. It runs three strategies and deduplicates the result:

Strategy	Relation produced	Rule
Inline	RelInline	Block text matching ^([^:]+):\s*(.+)$, label at most 50 characters
Right-of	RelRightOf	Adjacent blocks in one line where the left looks like a label and the right does not, gap at most 3× the label width
Below	RelBelow	Consecutive non-wrapped lines, left-aligned within 0.2, first aligned value only

Deduplication keeps the highest-confidence detection among fields that share a label (case-insensitive) or whose value boxes intersect.

field.Field

Field	Type	Meaning
Label	string	Label text, trailing : stripped
Value	string	Value text
LabelBox	geometry.Box	Label bounds; equals ValueBox for RelInline
ValueBox	geometry.Box	Value bounds
Relation	RelationType	Which spatial relationship produced it
Confidence	float64	Mirror of OverallScore.Value
LabelConfidence	confidence.Score	Factor breakdown for the label
ValueConfidence	confidence.Score	Factor breakdown for the value
OverallScore	confidence.Score	Combined score
PatternType	pattern.PatternType	Set when the value matched a registered pattern
Normalized	any	Normalised value from that pattern match

func (f *Field) ComputeOverallConfidence()

Recomputes OverallScore and Confidence as a weighted average of LabelConfidence (0.4) and ValueConfidence (0.6). Call it after mutating either component score. DetectFields itself uses the geometric mean, so calling this changes the number.

field.RelationType

Constant	Meaning
RelInline (zero value)	Label: Value inside one block
RelRightOf	Value to the right of the label on the same line
RelBelow	Value on the line under the label
RelTableCell	Declared only. No detector produces it, and ExtractByDefinition treats it as the default hint — right of the label, then below

field.FieldDef

The caller-supplied contract for targeted lookup through ExtractByDefinition.

Field	Type	Meaning
Name	string	Becomes Field.Label on the result
LabelPatterns	[]string	Regexes tried first, each compiled case-insensitively; an uncompilable pattern is skipped
LabelAnchors	[]*anchor.Ngram	N-gram anchors tried after every label pattern fails
ValueType	pattern.PatternType	When set, the value is run through this pattern type
LocationHint	RelationType	Direction to search from the label
Required	bool	Never read
Validator	func(string) bool	Rejects a candidate value; the search continues past a rejection

Search extent: RelRightOf searches 5× the label width, RelBelow searches 3× the label height, any other hint tries right first and then below at the anchor defaults. ExtractByDefinition returns nil when nothing resolves.

Confidence is fixed rather than computed: 0.85 for a label-pattern hit, raised to 0.95 when ValueType also matched, and anchorConfidence * 0.9 for an n-gram hit. The 0.95 raise applies only on the label-pattern path — an n-gram hit whose value matches ValueType sets PatternType and Normalized but keeps anchorConfidence * 0.9.

Caution

FieldDef.Required is declared but never read. ExtractByDefinition returns nil for an unresolved definition whether or not Required is set, and nothing else in the package inspects the field. Enforce required-ness in your own caller, or use schema.FieldSpec.Required with form.Extract, which does populate Instance.Missing.

Grouping

func GroupFields(fields []Field, maxGap float64) []FieldGroup

Clusters fields whose label-value union boxes are within maxGap of each other. Returns FieldGroup{Name string, Fields []Field, Box geometry.Box}. Returns nil for an empty input.

Note

Name is always the literal string Field Group. The library is domain-neutral and does not classify groups; assign your own names from the returned Fields.

Tuning constants

Exported and read-only — DetectFields has no config struct.

Constant	Value	Applies to
DefaultMaxLabelLength	50	Inline detection: longer labels are rejected
DefaultShortLabelLength	20	Threshold for the short-label boost
DefaultLabelConfidenceNormalizationFactor	100.0	Divisor for the inline label-length factor
DefaultGapToWidthMultiplier	3.0	Max label-to-value gap, as a multiple of label width or height
DefaultAlignmentToleranceRatio	0.5	Vertical alignment tolerance, as a fraction of label height
DefaultBaselineTolerance	0.3	Horizontal alignment tolerance, as a fraction of label width
DefaultBaseLabelScore	0.5	Starting score for the label-pattern heuristic
DefaultColonBoost	0.3	Added when the label ends with :
DefaultShortLabelBoost	0.1	Added when the label is under DefaultShortLabelLength
DefaultCapitalBoost	0.1	Added when the label starts with A–Z
DefaultDigitPenalty	0.2	Subtracted when the label contains a digit
DefaultPatternMatchConfidence	0.95	Value confidence on a pattern match

Field packs

Domain vocabulary — field names, label anchors, expected value types — enters through packs. It is never built into the detector.

type Pack struct {
  Name   string
  Fields []FieldDef
}

func RegisterPack(p Pack)
func RegisteredPacks() []Pack
func RegisteredFieldDefs() []FieldDef
func RegisteredFieldDef(name string) *FieldDef
func ResetPacks()

Function	Returns	Behaviour
RegisterPack	—	Appends to the registry; safe to call concurrently
RegisteredPacks	[]Pack	Independent snapshot in registration order
RegisteredFieldDefs	[]FieldDef	Every definition across all packs, in registration order
RegisteredFieldDef	*FieldDef	First case-insensitive name match, as a copy; nil when absent
ResetPacks	—	Clears the registry. Tests only

Caution

Duplicate pack names are accepted and not merged, and duplicate field names are not deduplicated. When two packs define the same field name, RegisteredFieldDef returns the one registered first and RegisteredFieldDefs returns both. Apply override semantics yourself.

Document.ExtractRegisteredFields() runs every registered definition against a document and returns map[string]*field.Field keyed by field name, skipping definitions that resolve to nil.

Repeating rows

func NewRepeatingFieldExtractor(labels ...string) *RepeatingFieldExtractor
func (e *RepeatingFieldExtractor) Extract(lines []text.Line) []Record

A record starts wherever the first label appears in a line. Each subsequent label’s value runs from the end of the label to the next label or the nearest end marker, whichever comes first. The record is emitted only when the fraction of labels that produced a non-empty value reaches MinMatchRatio.

Fields and setters

The setters return the receiver, so they chain.

Field	Type	Default	Setter
Labels	[]string	—	constructor argument
EndMarkers	[]string	none	SetEndMarkers(markers ...string)
VerticalFields	[]string	none	SetVerticalFields(labels ...string)
MaxLines	int	10	SetMaxLines(n int)
MaxWords	int	3	SetMaxWords(n int)
FieldMaxWords	map[string]int	nil	SetFieldMaxWords(label string, n int)
MinMatchRatio	float64	0.5	set the field directly — there is no setter

MaxLines caps how many lines are folded into one record before it is flushed. MaxWords caps words per value; FieldMaxWords overrides it per label. A label listed in VerticalFields takes the nearest block below the label that overlaps it horizontally, searched across every line rather than just the record’s lines.

field.Record

Field	Type	Meaning
Fields	map[string]string	Label to extracted value; labels that produced nothing are absent
Line	text.Line	The record’s first line

Reachable from a rule as ExtractRepeatingFields(labels ...string) []field.Record and RepeatingExtractingFields(...) []map[string]any, which parses comma-separated label, end-marker, label:n word-cap and vertical-field strings into the same extractor. See Tables and repeating structures.

Checkbox and mark detection

func NewDetector() *Detector
func NewDetectorWithConfig(config DetectorConfig) *Detector
func DefaultConfig() DetectorConfig

func (d *Detector) DetectCheckboxes(blocks []text.Block, lines []text.Line) []Checkbox
func (d *Detector) DetectCheckboxGroups(blocks []text.Block, lines []text.Line) []CheckboxGroup

DetectCheckboxes runs three methods, deduplicates the results, then assigns labels. DetectCheckboxGroups runs DetectCheckboxes and then groups.

Method	Finds	Confidence
Unicode symbol scan	Checked and unchecked glyphs anywhere in block text	0.95
Regex patterns	Bracket, paren and underscore forms	0.85
Geometry, single mark	A near-square block holding one of X x ✓ ✔ •	0.7
Geometry, empty	A near-square block with no text	0.6

These four constants are the whole accuracy model. There is no OCR-confidence blending in this package. Deduplication keeps the highest-confidence detection among boxes that intersect on the same page — identical coordinates on different pages are different checkboxes.

The geometric method accepts a block only when width and height are both between 5 and 30 units and the aspect ratio is between 0.7 and 1.4.

Caution

Those are absolute pixel thresholds. At unusual scan resolutions the geometric method silently finds nothing; the symbol and regex methods still run.

checkbox.DetectorConfig

Field	Type	Default	Meaning
BracketPatterns	[]string	\[\s*[xX✓✔]\s*\], \[\s*\]	Regexes for [X] / [ ] forms
ParenPatterns	[]string	\(\s*[xX✓✔•]\s*\), \(\s*\)	Regexes for (X) / ( ) forms
BoxPatterns	[]string	☑ ☒ ■ ▣, ☐ □ ▢	Never read
CirclePatterns	[]string	● ◉ ⊙ ◆, ○ ◯ ◇	Never read
UnderscorePatterns	[]string	_[xX]_, _{2,5}	Regexes for _X_ / ___ forms
MaxLabelDistance	float64	200	Furthest a block can sit and still be taken as the label
LabelPosition	LabelPosition	LabelRight	Where to look for the label
GroupByProximity	bool	true	When false, every checkbox becomes its own single-member group
GroupMaxDistance	float64	100	Manhattan distance within which two checkboxes can group
CommonGroupLabels	[]string	Yes, No, N/A, True, False, plus marital, housing, occupancy and loan-purpose terms	Only checkboxes whose label is in this list are grouped with each other

Caution

BoxPatterns and CirclePatterns are populated by DefaultConfig and then never read. The Unicode symbol scan matches against a rune set hard-coded in the detector, so adding a glyph to either list does not make it detectable and removing one does not suppress it. The hard-coded set is ☑ ☒ ■ ▣ ● ◉ ⊙ ◆ ✓ ✔ ✗ ✘ for checked and ☐ □ ▢ ○ ◯ ◇ for unchecked. Only BracketPatterns, ParenPatterns and UnderscorePatterns change what is matched.

Caution

Within each of the three pattern lists that are read, the entry at index 0 is always classified as the checked form regardless of its content; the rest are classified by whether they contain x, X, ✓ or •. A custom config that lists the unchecked pattern first reports every match as checked.

Label assignment

Constant	Direction searched	Alignment required
LabelRight (zero value)	Blocks starting past the checkbox’s right edge	Mid-Y within half the taller box
LabelLeft	Blocks ending before the checkbox’s left edge	Mid-Y within half the taller box
LabelBoth	Either side	Mid-Y within half the taller box
LabelAny	Any direction, by Manhattan distance	None

Blocks that intersect the checkbox, sit on another page, or themselves look like a checkbox pattern are skipped. A checkbox with no qualifying block keeps an empty Label.

Results

checkbox.Checkbox:

Field	Type	Meaning
Box	geometry.Box	Checkbox bounds, estimated from character position within the block for pattern and symbol hits
Checked	bool	Whether the mark reads as set
Label	string	Assigned label text
LabelBox	geometry.Box	Bounds of the label block
GroupName	string	Always empty — the detector never sets it. Group names live on CheckboxGroup.Name
Confidence	float64	One of the four fixed method constants
Type	CheckboxType	Which form was matched
Page	int	Page the checkbox was found on

checkbox.CheckboxGroup:

Field	Type	Meaning
Name	string	Inferred group name, or the single member’s label
Checkboxes	[]Checkbox	Members
Selected	[]string	Labels of the members whose Checked is true
Type	GroupType	GroupMultiple only when more than one member is checked

CheckboxType: TypeCheckbox (zero value), TypeRadio, TypeBracket, TypeParen, TypeUnderscore. The symbol scan assigns TypeRadio for ○ ● ◯ ◉ and TypeCheckbox otherwise; the geometric method always assigns TypeCheckbox.

GroupType: GroupSingle (zero value), GroupMultiple.

Reachable from a rule as GetCheckboxByLabel(label), GetCheckboxGroup(name) and GetCheckedValue(groupName) []string, all matching case-insensitively on trimmed text.

Note

Group naming carries built-in vocabulary: a group whose joined labels contain Yes/No is named Yes/No, and there are four more such mappings. Anything unrecognised is named by joining the member labels with /.

Master-detail grouping

func Extract(doc *document.Document, spec schema.MasterDetailSpec) []schema.MasterDetailGroup

Pure composition over form and repeater extraction. It adds no detection of its own — it runs both extractors and decides which detail belongs to which master.

schema.MasterDetailSpec

Field	Type	Meaning
MasterForm	*FormSpec	Single master; used only when it extracted at least one value
MasterRepeater	*RepeaterSpec	Repeating masters; takes precedence over MasterForm
Detail	RepeaterSpec	The detail records

With neither master field set, Extract returns nil. It also returns nil when the master extractor produced no instances.

schema.RepeaterSpec — the detail configuration:

Field	Type	Meaning
Name	string	Result name
Row	FormSpec	The per-record form; every anchor, locator and pattern rule above applies
Region	*RegionSpec	Search scope
Separator	RowSeparator	How record boundaries are found
EndAnchors	[]AnchorSpec	Stop extracting at the first match
MaxRecords	int64	Cap on records emitted
MinMatchRatio	float64	Fraction of row fields that must resolve for a record to count

RowSeparator constants are SepAnchorRestart (zero value), SepBlankRow, SepRuledLine, SepStride, SepTableRow and SepIndentChange, with String() returning anchor_restart, blank_row, ruled_line, stride, table_row and indent_change. See Tables and repeating structures.

Grouping rule

Masters and details are each sorted by first page, then by the smallest Y0 across their page boxes. Each detail attaches to the latest master that is on a page at or before the detail’s page and, on the same page, above it.

Situation	Outcome
Detail before every master	Dropped
Master with no details	Kept, with an empty Details slice
Detail on a page with no master	Attaches to the nearest master on an earlier page

schema.MasterDetailGroup is {Master Instance, Details []Instance}.

Multi-page stitching

func Stitch(instances []schema.Instance, spec schema.StitchSpec) []schema.Instance

A post-pass over already-extracted instances, independent of which extractor produced them. It walks the input in order and merges each instance onto the previous output entry when every condition holds; otherwise it starts a new entry. Input order is preserved and an empty input returns nil.

schema.StitchSpec

Field	Type	Meaning
SpecName	string	Restricts stitching to instances with this Name; empty means stitch each Name group independently
ContinuationAnchors	[]AnchorSpec	When set, at least one must appear in the later instance before it merges
EndAnchors	[]AnchorSpec	When any appears in the earlier instance, it is terminal and never merges

Merge conditions

Condition	Rule
Name	Both instances share the same Name, and match SpecName when it is set
Adjacency	cur.Region.FirstPage() equals tail.Region.LastPage() or that plus 1
End anchors	No EndAnchors text present in the earlier instance
Continuation	When ContinuationAnchors are set, at least one is present in the later instance

Merge semantics

Result field	Rule
Values	Earlier value wins on key conflict; later-only keys are added
Missing	Union of both, minus any name now present in Values
Region	Per-page union of both BoxSpans
Confidence	Field-count-weighted mean: (tailConf*len(tailValues) + curConf*len(curValues)) / total
Provenance	The earlier instance’s

Caution

Two things surprise callers here. Earlier values win, so a later page cannot correct an earlier page’s reading. And the anchor test is a case-insensitive substring search over the instances’ extracted values and their Provenance.SourceText — not over page content — so AnchorSpec.MatchType and Fuzzy are ignored during stitching. A continuation marker that is not captured as a field value is invisible to Stitch.

Continue with

• Fields, forms and checkboxes — the locator model and .schema extraction
• Tables and repeating structures — tables, grid sections and repeating rows
• Matching and confidence — fuzzy distance and the confidence factor model
• Running rules over documents — how a rule reaches these results
• Operating rules.svc — the HTTP surface