Text acquisition API

Reference for the packages that turn OCR engine output into the spatial document model the Rules block queries: ocr (parsing and adapters), ocrclean (text normalisation), packs (domain vocabulary and classifiers), and parallel (page-level concurrency). These are the signatures behind kis hocr2bbox and kis ocr and the document capabilities described under Documents and Operating the service.

Caution

No OCR engine ships here. Nothing in these packages executes a recogniser, loads a model, or handles an image — there is no process execution, no engine dependency, and no language or model data of any kind. You run your own recogniser and hand in its output: hOCR (HTML) or a JSON word-box document. There is no engine-selection flag, key, or registry; the format of the file you supply, or the adapter you call, is the selection.

The document model

ocr.Document is what every downstream query, extraction, and rule sees. It is also the on-disk shape of a .bbox file.

type Document struct {
    Pages []Page `json:"pages"`
}

type Page struct {
    Number     int         `json:"number"`
    Width      float64     `json:"width"`
    Height     float64     `json:"height"`
    Words      []Word      `json:"words"`
    Lines      []Line      `json:"lines,omitempty"`
    Paragraphs []Paragraph `json:"paragraphs,omitempty"`
}

Word

Field	Type	JSON	Meaning
Text	string	text	Token text
Box	geometry.Box	box	Axis-aligned bounding box
Confidence	float64	confidence	0..1, normalised from the source
Page	int	page	Page number this word belongs to
LineID	int	line_id,omitempty	Line grouping, when the source provides one

geometry.Box carries X0, Y0 (top-left) and X1, Y1 (bottom-right), all float64, serialised as x0/y0/x1/y1.

Line

Field	Type	JSON	Meaning
ID	int	id	Line index on the page
Text	string	text	Member words joined by a single space
Box	geometry.Box	box	Union of the member word boxes
Confidence	float64	confidence,omitempty	Mean of member word confidences
WordStart	int	word_start	First index into Page.Words
WordEnd	int	word_end	One past the last index (exclusive)

Paragraph

Field	Type	JSON	Meaning
ID	int	id	Paragraph index on the page
Text	string	text	Member lines joined by newline
Box	geometry.Box	box	Union of the member line boxes
Confidence	float64	confidence,omitempty	Mean of member line confidences
LineStart	int	line_start	First index into Page.Lines
LineEnd	int	line_end	One past the last index (exclusive)

Caution

Line and Paragraph hold index ranges into Page.Words and Page.Lines, not copies. Reordering or filtering those slices invalidates every range on the page.

Document and Page methods

func (d *Document) GetPage(pageNum int) []Word
func (d *Document) AllWords() []Word
func (d *Document) PageCount() int
func (d *Document) UnescapeText()
func (d *Document) CleanDocument()
func (d *Document) CleanDocumentWithOptions(opts ocrclean.CleanOptions)
func (p *Page) CleanPage()
func (p *Page) CleanPageWithOptions(opts ocrclean.CleanOptions)
func (w Word) EstimateCharWidth() float64
func NewWord(text string, x0, y0, x1, y1, confidence float64, page int) Word

Symbol	Returns	Notes
GetPage	[]Word	Matches on Page.Number; nil when absent
AllWords	[]Word	Concatenates every page in slice order
PageCount	int	len(d.Pages)
UnescapeText	—	Applies HTML entity unescaping to every word text, in place
CleanDocument	—	CleanDocumentWithOptions(ocrclean.DefaultOptions())
CleanPageWithOptions	—	Rewrites Word.Text, then Line.Text, then Paragraph.Text
EstimateCharWidth	float64	Box.Width() / len(Text); 0 for empty text
NewWord	Word	Normalises the box so X0 <= X1, Y0 <= Y1

Caution

UnescapeText is a second decode pass. The hOCR parser already decodes HTML entities, so calling it turns & into &, and a word whose true text is literally & becomes &. kis hocr2bbox applies it; the parser functions below do not, so the same file yields different word text through the two routes.

Loading and parsing

func Load(path string) (*Document, error)
func LoadJSON(path string) (*Document, error)
func LoadHOCR(path string) (*Document, error)
func LoadHOCRWithOptions(path string, opts ParseOptions) (*ParseResult, error)
func ParseHOCR(r io.Reader) (*Document, error)
func ParseHOCRWithOptions(r io.Reader, opts ParseOptions) (*ParseResult, error)
func LoadWithCleanup(path string) (*Document, error)
func LoadWithCleanupOptions(path string, opts ocrclean.CleanOptions) (*Document, error)
func LoadHOCRWithCleanup(path string) (*Document, error)
func LoadHOCRWithCleanupOptions(path string, opts ocrclean.CleanOptions) (*Document, error)

Function	Input	Returns
Load	Any supported path, format sniffed	*Document
LoadJSON	JSON word-box file (.bbox, .json)	*Document
LoadHOCR	hOCR file	*Document, warnings discarded
LoadHOCRWithOptions	hOCR file + ParseOptions	*ParseResult
ParseHOCR	io.Reader	*Document, warnings discarded
ParseHOCRWithOptions	io.Reader + ParseOptions	*ParseResult
LoadWithCleanup	Any supported path	*Document, DefaultOptions() cleanup applied
LoadWithCleanupOptions	Any supported path + CleanOptions	*Document, cleaned
LoadHOCRWithCleanup	hOCR file	*Document, DefaultOptions() cleanup applied
LoadHOCRWithCleanupOptions	hOCR file + CleanOptions	*Document, cleaned

Format detection in Load

Extension	Behaviour
.json	JSON loader
.hocr	hOCR parser
.html, .htm	Read and sniffed for ocr_page, ocrx_word or ocr_word; hOCR if found, otherwise JSON
anything else (including .bbox)	JSON first; hOCR on failure

The extension is a hint, not a contract — .bbox is resolved by content, and an hOCR file is accepted anywhere a bbox file is.

Recognised hOCR classes

Level	Classes
Page	ocr_page
Content area (recursed into)	ocr_carea, ocr_photo, ocr_graphic, ocr_separator, ocr_noise
Block	ocr_par, ocr_textfloat, ocr_caption, ocr_header, ocr_footer
Line	ocr_line, ocrx_line, ocr_textline, ocr_header_line
Word	ocrx_word, ocr_word
Character	ocrx_cinfo, ocr_cinfo

That is the complete accepted vocabulary. Per page the parser runs a paragraph pass, a bare-line pass, and a bare-word pass, each skipping subtrees the earlier passes consumed; an orphan-text pass runs only if the page still has zero words.

Title-property and confidence parsing

An hOCR title is a ;-separated property list, for example image "/tmp/page_1.png"; bbox 0 0 2550 3300; ppageno 0. Splitting honours double-quoted values, and the property name must match as a whole token, so bboxes and a filename containing bbox are both rejected. A non-numeric bbox yields no box rather than scavenging digits from elsewhere in the title.

Property	Read as	Result
x_wconf N	integer 0–100	N / 100
x_conf N	float	divided by 100 when > 1
neither present	—	confidence 1.0
word synthesised from fallback text	—	confidence 0.0

Caution

Confidence is matched by plain substring search, not the whole-token rule used for bbox — another property or a filename containing x_conf can supply the value. A confidence of exactly 1.0 also means “the source reported nothing”, not “perfect”.

Caution

A word element with no bbox inherits the nearest enclosing box (line, paragraph, content area, or page) and gets confidence 0.0. With no ancestor box either it gets a zero-size box at the origin, so (0,0,0,0) boxes and clusters of words sharing one oversized box are both real outputs.

Caution

Page Width and Height of zero are silently fatal: percentage-based spatial queries scale by them, so the region collapses and matches nothing with no error raised. hOCR without an ocr_page bbox, a malformed page title, and the Docling adapter all produce this. The parallel package mitigates it by deriving dimensions from the word extent; other paths do not.

Parse options, results, and warnings

type ParseOptions struct {
    Verbose bool
}

type ParseResult struct {
    Document *Document
    Warnings []ParseWarning
}

type ParseWarning struct {
    Type    string
    Element string
    Text    string
    Context string
}

func (w ParseWarning) String() string

Field	Meaning
ParseOptions.Verbose	Accumulate warnings. The only option.
ParseWarning.Type	Warning kind, from the table below
ParseWarning.Element	Element tag or class that triggered it
ParseWarning.Text	Associated text, truncated to 50 characters with a ... suffix
ParseWarning.Context	Where in the tree it happened

String() formats as [type] element: "text" (context: ctx), dropping the quoted text when it is empty.

Warning types

Type	Emitted when
missing_bbox	An element carries no parseable bbox
no_bbox_fallback	A word took an ancestor’s box
no_structure	No ocr_page found and fallback extraction found no words
empty_paragraph	A paragraph element yielded no lines
empty_line	A line element yielded no words
empty_word	A word element yielded no text
orphan_text	Text rescued outside any recognised structure
unrecognized_class	A class starting with ocr that is not in the vocabulary
missed_content	Post-hoc diff of source text against extracted word text

missed_content compares punctuation-stripped, lowercased text, ignores single characters, and reports at most the first 10 missed words.

Caution

Warnings are accumulated only when Verbose is true. LoadHOCR, ParseHOCR, and every cleanup variant pass a zero ParseOptions, so their warning list is always empty. Missing content is invisible by default. The ParseWarning.Type doc comment names an empty_element type that is never emitted.

Caution

The HTML parser essentially never errors on arbitrary input. Feeding a plain text file or unrelated HTML produces a successful parse and a document with zero pages, or one synthetic page of zero-size orphan words. Check the word count, not the error.

Engine adapters

Three adapters build a Document from already-parsed engine output. They are in-process conversions over the structs below.

func FromPaddleOCR(result PaddleOCRResult, pageWidth, pageHeight float64) *Document
func FromTesseract(words []TesseractWord, pageWidth, pageHeight float64) *Document
func FromDocling(words []DoclingWord) *Document

Adapter	Completeness	Behaviour
FromPaddleOCR	Complete	Reduces each 4-corner polygon to an axis-aligned box; items with fewer than 4 corners are skipped. Pages come out in order, numbered from 1.
FromTesseract	Partial	Keeps only Level == 5 (word-level) rows with non-empty text; divides Confidence by 100. Page order is nondeterministic.
FromDocling	Partial	Reads Bbox as [x0,y0,x1,y1] (needs at least 4 values) and Score as confidence. Page order is nondeterministic, and page dimensions stay zero.

Caution

FromTesseract and FromDocling group words into a map and then range over it, so doc.Pages is not in page order and Pages[i].Number != i+1. Sort by Number before using the result. FromDocling additionally leaves Width and Height at zero — seed them before any percentage-based query.

Adapter input types

type PaddleOCRResult struct {
    Results [][]PaddleOCRItem `json:"results"`
}

type PaddleOCRItem struct {
    Boxes      [][]float64 `json:"boxes"`
    Text       string      `json:"text"`
    Confidence float64     `json:"confidence"`
}

Results is one slice per page; Boxes is [[x0,y0],[x1,y1],[x2,y2],[x3,y3]].

type TesseractWord struct {
    Level      int     `json:"level"`
    PageNum    int     `json:"page_num"`
    BlockNum   int     `json:"block_num"`
    ParNum     int     `json:"par_num"`
    LineNum    int     `json:"line_num"`
    WordNum    int     `json:"word_num"`
    Left       float64 `json:"left"`
    Top        float64 `json:"top"`
    Width      float64 `json:"width"`
    Height     float64 `json:"height"`
    Confidence float64 `json:"conf"`
    Text       string  `json:"text"`
}

Left/Top/Width/Height become the box; LineNum becomes Word.LineID.

type DoclingWord struct {
    Text   string    `json:"text"`
    Bbox   []float64 `json:"bbox"`
    Score  float64   `json:"score"`
    PageNo int       `json:"page_no"`
}

type HOCRWord struct {
    XMLName xml.Name `xml:"span"`
    Class   string   `xml:"class,attr"`
    Title   string   `xml:"title,attr"`
    Text    string   `xml:",chardata"`
}

HOCRWord is an XML-tag alternative to the HTML parse path. No function in the package consumes it.

Text cleanup and normalisation

The ocrclean package is pure text in, text out. It is character-level only — no dictionary, no language model, no language parameter.

Options

type CleanOptions struct {
    NormalizeUnicode        bool
    FixQuotes               bool
    FixDashes               bool
    FixLigatures            bool
    RemoveGarbage           bool
    FixMisreads             bool
    CleanWhitespace         bool
    RemoveControlChars      bool
    RemoveStrayPunct        bool
    PreserveLineBreaks      bool
    AggressiveClean         bool
    PreserveFormIndicators  bool
    NormalizeFormIndicators bool
}

Field	DefaultOptions()	FormAwareOptions()	Applies
NormalizeUnicode	true	true	Unicode NFC
FixQuotes	true	true	25 quote, prime and guillemet variants to ASCII ' or "
FixDashes	true	true	11 dash variants to -
FixLigatures	true	true	fi/fl/ff/ffi/ffl/st, AE/OE/IJ, and symbol expansions (No., (P), SM, TM, (R), (C))
RemoveGarbage	true	true	Deletes ~55 bullet, arrow, box-drawing and geometric glyphs
FixMisreads	true	true	Fullwidth ASCII, Greek/Cyrillic homoglyphs, Roman numerals, space variants, zero-width removal
CleanWhitespace	true	true	Collapses runs, trims lines, caps blank lines at two
RemoveControlChars	true	true	Drops control runes, keeping \n, \t, \r
RemoveStrayPunct	false	false	16 X patterns to a space
PreserveLineBreaks	true	true	Nothing — see below
AggressiveClean	false	false	Everything outside printable ASCII becomes a space
PreserveFormIndicators	true	true	Exempts checkbox and radio glyphs from garbage removal
NormalizeFormIndicators	false	true	Rewrites those glyphs to [ ], [x], ( ), (x)

FormAwareOptions() is DefaultOptions() with NormalizeFormIndicators set — that single field is the only difference.

Caution

PreserveLineBreaks is dead. It is declared and set to true, but CleanWithOptions never reads it. It is the only field of CleanOptions that is never consulted.

Constructors and the pipeline

func DefaultOptions() CleanOptions
func FormAwareOptions() CleanOptions
func CleanWithOptions(text string, opts CleanOptions) string
func CleanOCR(text string) string

CleanWithOptions applies steps in a fixed order: NormalizeUnicode, FixQuotes, FixDashes, FixLigatures, RemoveGarbageCharsWithOptions, NormalizeFormIndicators, FixCommonMisreads, RemoveStrayPunctuation, CleanWhitespace, RemoveControlChars, AggressiveCleanup.

CleanOCR is a fixed pipeline that ignores options entirely: NormalizeUnicode, FixQuotes, FixDashes, FixLigatures, RemoveGarbageChars, FixCommonMisreads, CleanWhitespace, RemoveControlChars.

Individual transforms

func NormalizeUnicode(text string) string
func FixQuotes(text string) string
func FixDashes(text string) string
func FixLigatures(text string) string
func FixCommonMisreads(text string) string
func RemoveGarbageChars(text string) string
func RemoveGarbageCharsWithOptions(text string, preserveFormIndicators bool) string
func NormalizeFormIndicators(text string) string
func CleanWhitespace(text string) string
func RemoveControlChars(text string) string
func RemoveStrayPunctuation(text string) string
func AggressiveCleanup(text string) string
func RemoveRepeatedChars(text string) string
func FixBrokenWords(text string) string
func StripPunctuation(s string) string

Function	Does	In a profile
NormalizeUnicode	NFC normalisation	yes
FixQuotes	Quote, prime and guillemet folding	yes
FixDashes	Dash folding to -	yes
FixLigatures	Ligature and digraph expansion	yes
FixCommonMisreads	Fullwidth and homoglyph folding	yes
RemoveGarbageChars	Garbage glyph removal, form indicators preserved	yes, via the options variant
RemoveGarbageCharsWithOptions	Same, with explicit preservation flag	yes
NormalizeFormIndicators	Checkbox and radio glyphs to ASCII	form-aware profile only
CleanWhitespace	Whitespace normalisation and trimming	yes
RemoveControlChars	Control rune removal	yes
RemoveStrayPunctuation	Isolated punctuation to a space	declared, off by default
AggressiveCleanup	Non-printable-ASCII to space, then CleanWhitespace	opt-in
RemoveRepeatedChars	Runs of 4+ identical runes collapsed to 2	no
FixBrokenWords	Rejoins word-\nword	no
StripPunctuation	Trims leading and trailing punctuation and quotes	no

FormIndicators is the exported map driving the last three form functions:

var FormIndicators = map[string]string{ /* glyph -> "[ ]" | "[x]" | "( )" | "(x)" */ }

Circles map to ( ) / (x); squares, ballot boxes, check marks and ballot crosses map to [ ] / [x].

Caution

RemoveRepeatedChars, FixBrokenWords and StripPunctuation are wired into nothing — not CleanOCR, not CleanWithOptions, not either profile. Call them explicitly or they never run. FixBrokenWords also has no effect on per-word cleanup, because a single word contains no newline.

Caution

Cleanup runs per word text and never adjusts boxes. A word made entirely of garbage glyphs becomes an empty string while keeping its box, so zero-text words are a normal output. For the same reason RemoveStrayPunctuation cannot fire in the shipped pipeline: all 16 of its patterns need a space on both sides, and a word has none.

Caution

The default profile is lossy in ways that matter for financial and legal text. FixDashes collapses en dash, em dash and the minus sign to -, making a leading minus indistinguishable from a dash. FixQuotes maps the prime character to an apostrophe, losing feet and inches marks. FixCommonMisreads maps the multiplication sign to x and folds Greek and Cyrillic homoglyphs to Latin. RemoveGarbageChars strips the section sign. All are on in DefaultOptions().

Domain packs

A pack registers domain vocabulary into the shared registries at package init. Packs are domain packs — there is no language pack, no per-language data, and no download step.

Registration entry points

func field.RegisterPack(p field.Pack)
func pattern.RegisterPack(p pattern.Pack)
func text.RegisterVocabulary(name string, v *text.Vocabulary)
func special.AddSignatureAnchors(anchors ...string)
func document.RegisterDoctype(spec document.DoctypeSpec)

Read them back with field.RegisteredFieldDefs(), text.RegisteredVocabularies() and text.MergedVocabulary().

The mortgage pack

func Register()
func Unregister()
func Doctypes() []document.DoctypeSpec
func CommonMortgageFields() []field.FieldDef
func Vocabulary() *text.Vocabulary
var SignatureAnchors []string

Symbol	Returns	Notes
Register	—	Installs fields, patterns, vocabulary, signature anchors and doctypes. Runs automatically at package init; guarded by sync.Once, so repeat calls are no-ops.
Unregister	—	Test-only teardown
Doctypes	[]document.DoctypeSpec	The 8 classifiers, for passing explicitly to document.Classify
CommonMortgageFields	[]field.FieldDef	The 13 field definitions, for driving an extraction loop directly
Vocabulary	*text.Vocabulary	A fresh vocabulary, seeded with terms and known corrections
SignatureAnchors	[]string	8 signature-label substrings, added to the domain-neutral defaults

Doctype classifiers registered: loan_estimate, closing_disclosure, urla_borrower, urla_additional_borrower, appraisal, title_commitment, promissory_note, deed_of_trust.

Field definitions registered under pack name mortgage: Borrower Name, Property Address, Loan Amount, Interest Rate, SSN, Date of Birth, Phone, Email, Monthly Income, Employer, NMLS ID, MERS MIN, APN.

Pattern types registered under pack name mortgage:

const (
    PatternNMLS            pattern.PatternType = "nmls"
    PatternAPN             pattern.PatternType = "apn"
    PatternMERS            pattern.PatternType = "mers"
    PatternCaseNumber      pattern.PatternType = "case_number"
    PatternLoanNumber      pattern.PatternType = "loan_number"
    PatternCreditScore     pattern.PatternType = "credit_score"
    PatternMortgageRatio   pattern.PatternType = "mortgage_ratio"
    PatternInterestRate    pattern.PatternType = "interest_rate"
    PatternBookPage        pattern.PatternType = "book_page"
    PatternRecordingNumber pattern.PatternType = "recording_number"
)

PatternMortgageRatio covers LTV, CLTV and DTI with one generic percentage regex — the distinction comes from the surrounding label, so use a field definition when you need a specific one.

The vocabulary carries mortgage terms across parties, property, loan, income, assets and closing, plus 8 hand-written corrections: 8orrower → Borrower, 1ncome → Income, Emp1oyer → Employer, Principa1 → Principal, Ba1ance → Balance, Tota1 → Total, Monthl y → Monthly, Annua1 → Annual.

Caution

Registration is process-global state. Unregister clears the entire registry for each subsystem it touches — fields, patterns, vocabularies, signature anchors and doctypes — because the registries have no per-pack removal. It is a test helper, not a way to swap packs at runtime. text.RegisterVocabulary also does not reject duplicate names.

Parallel page processing

The parallel package fans pages out over goroutines. Every workers argument follows the same rule: a value <= 0 means runtime.NumCPU().

Processor

type ProcessorConfig struct {
    Workers          int
    DetectTables     bool
    DetectFields     bool
    DetectCheckboxes bool
    DetectRegions    bool
    MergeWords       bool
    MergeConfig      text.MergeConfig
}

func DefaultProcessorConfig() ProcessorConfig
func NewProcessor() *Processor
func NewProcessorWithConfig(config ProcessorConfig) *Processor
func (p *Processor) ProcessDocument(doc *ocr.Document) (*DocumentResult, error)
func (p *Processor) ProcessDocumentContext(ctx context.Context, doc *ocr.Document) (*DocumentResult, error)

Field	Default	Effect
Workers	0 → NumCPU()	Goroutine count
DetectTables	true	Runs the table detector per page
DetectFields	true	Runs the field detector per page
DetectCheckboxes	true	Runs the checkbox detector per page
DetectRegions	true	Builds a region.PageLayout per page
MergeWords	true	Merges words into blocks; when false each word becomes its own block
MergeConfig	text.DefaultMergeConfig()	Gap multiplier 0.5, pattern merge on, dictionary merge off

NewProcessor() uses DefaultProcessorConfig(). ProcessDocument delegates to ProcessDocumentContext with context.Background(). Both return the first error encountered — a cancelled context or a failing page aborts the whole document and returns nil results.

Each page is stat-computed, then seeded with DocStats.SeedPageDims, which falls back to the words’ maximum extent when the page declares no dimensions and sets PageDimsDerived to report the reduced precision.

Results

type PageResult struct {
    PageNum    int
    Blocks     []text.Block
    Lines      []text.Line
    Tables     []table.Table
    Fields     []field.Field
    Checkboxes []checkbox.Checkbox
    Layout     *region.PageLayout
    Stats      text.DocStats
    Error      error
}

type DocumentResult struct {
    PageResults  []*PageResult
    Blocks       []text.Block
    Lines        []text.Line
    Tables       []table.Table
    Fields       []field.Field
    Checkboxes   []checkbox.Checkbox
    PageLayouts  []*region.PageLayout
    Stats        text.DocStats
    SpatialIndex *geometry.SpatialIndex[text.Block]
}

func (dr *DocumentResult) PageCount() int
func (dr *DocumentResult) GetPageResult(pageNum int) *PageResult

DocumentResult concatenates every per-page slice in page order and builds a spatial index over the combined blocks. GetPageResult matches on PageNum and returns nil when absent.

Caution

DocumentResult.Stats page dimensions are an arithmetic mean across pages with non-zero dimensions. For a document mixing portrait and landscape, or letter and legal, that number is physically meaningless — and it is what document-level percentage queries scale by. Use PageResult.Stats for anything dimension-sensitive.

Per-page functions

type PageFunc func(page ocr.Page, pageIdx int) (any, error)

type PageFuncResult struct {
    PageNum int
    PageIdx int
    Result  any
    Error   error
}

func ForEachPage(doc *ocr.Document, fn PageFunc, workers int) []PageFuncResult
func ForEachPageContext(ctx context.Context, doc *ocr.Document, fn PageFunc, workers int) []PageFuncResult
func ForEachPageOrdered(doc *ocr.Document, fn PageFunc, workers int) ([]any, []error)
func ForEachPageWithProgress(doc *ocr.Document, fn PageFunc, workers int, onProgress func(progress *Progress)) []PageFuncResult
func MapPages[T any](doc *ocr.Document, fn func(page ocr.Page) (T, error), workers int) ([]T, error)
func FilterPages(doc *ocr.Document, predicate func(page ocr.Page) bool, workers int) []ocr.Page
func ReducePages[T any](doc *ocr.Document, initial T, fn func(acc T, page ocr.Page) T) T
func ParallelReducePages[T any](doc *ocr.Document, initial T, mapFn func(page ocr.Page) T, combineFn func(a, b T) T, workers int) T

Function	Returns	Notes
ForEachPage	[]PageFuncResult	Results indexed by page position, so order is preserved even though execution is not
ForEachPageContext	[]PageFuncResult	A cancelled context fills remaining slots with ctx.Err() rather than aborting
ForEachPageOrdered	([]any, []error)	Splits ForEachPage output into parallel value and error slices
ForEachPageWithProgress	[]PageFuncResult	Wraps fn, incrementing a Progress and invoking onProgress after each page; onProgress may be nil
MapPages	([]T, error)	Returns the full result slice and the first non-nil error
FilterPages	[]ocr.Page	Predicate evaluated in parallel, output kept in document order
ReducePages	T	Sequential, despite the package. No workers argument.
ParallelReducePages	T	Parallel map phase, sequential combine phase in page order

onProgress in ForEachPageWithProgress is called from every worker goroutine. Anything it touches must be safe for concurrent use.

Batch and stream

type BatchProcessor struct {
    BatchSize int
    Workers   int
}

func NewBatchProcessor(batchSize, workers int) *BatchProcessor
func (bp *BatchProcessor) ProcessBatches(doc *ocr.Document, pageFn PageFunc, batchFn func(batchNum int, results []PageFuncResult) error) error

type StreamProcessor struct {
    Workers    int
    BufferSize int
}

func NewStreamProcessor(workers, bufferSize int) *StreamProcessor
func (sp *StreamProcessor) Process(ctx context.Context, doc *ocr.Document, fn PageFunc) <-chan PageFuncResult

Constructor	Argument	<= 0 becomes
NewBatchProcessor	batchSize	100
NewBatchProcessor	workers	runtime.NumCPU()
NewStreamProcessor	workers	runtime.NumCPU()
NewStreamProcessor	bufferSize	workers * 2

ProcessBatches slices the document into BatchSize-page sub-documents, processes each with ForEachPage, and calls batchFn after each batch with the absolute batch number and its results. A non-nil return from batchFn stops processing and is returned. pageFn receives the absolute page index, not the index within the batch.

Process returns a buffered channel and closes it when every worker finishes. Cancelling the context stops workers, and the channel closes with results already sent still readable.

Note

StreamProcessor.Process uses the struct’s Workers field verbatim. Construct it with NewStreamProcessor — a zero-valued StreamProcessor starts no workers and yields a channel that closes immediately.

Progress

type Progress struct {
    Total     int
    Completed int
    Failed    int
}

func NewProgress(total int) *Progress
func (p *Progress) Increment(success bool)
func (p *Progress) Percent() float64

Symbol	Behaviour
NewProgress	Sets Total; Completed and Failed start at zero
Increment	Mutex-guarded; always increments Completed, and Failed when success is false
Percent	Completed / Total * 100; returns 100 when Total is zero

Read Total, Completed and Failed through Percent or from inside onProgress. The fields are guarded by an unexported mutex, so reading them directly from another goroutine races.

Related pages

• Documents — the document model in use
• Text acquisition — ingesting pages end to end
• Extraction — fields, tables and checkboxes
• Authoring rules — writing rules over an ingested page
• Operating the service — running rules.svc