ScienceSoft/scnsoft-pii-encoder-doc-type-head

Release candidate for supportive monitoring. Not an enforcement tool. This is a document-type head, V3a, seed 7001. Its scores support a person looking at documents; nothing here claims that a document should be blocked, redacted or refused because of a score. The weights are published for evaluation and reproduction only (see Licence). The training data is not published.

Document-type head V3a (seed 7001)

This model adds a second output, doc_logits, to the same int8 graph as ScienceSoft/scnsoft-pii-encoder, so that a window of text is also scored for 17 document types (non-disclosure and master agreements, statements of work, other contracts, requests for proposals, proposals, price lists, source code, design and security documents, financial and personal-finance documents, personnel files, strategy documents, published documents and patient records). The PII output (logits) is the encoder's own.

What this release says, and does not say

• Four types have been evaluated, on one measure only. legal.nda, tech.design, tech.security and finance.personal met a false-record line on held-out benign documents on one machine (table below). That is the only evaluation of them. Their recall on documents of the type, their behaviour on other machines and their cost on customer hardware are not evaluated. Meeting the line does not make a type ready for any decision.
• legal.contract_other is not yet monitoring-grade. It missed the same line (8 false records on 18,075 benign documents). In doc_labels.json it is set to rules: the model adds nothing for it.
• Every other type with a calibration value is fused at that value as an unvalidated suggestion, never enforcement. That is eleven labels: legal.master_agreement, legal.sow, sales.rfp, sales.proposal, sales.pricing, tech.source_code, finance.report, finance.billing, hr.personnel, corporate.strategy and health.patient_record. They are not validated: no recall, no per-language result and no line was evaluated for them, and no accuracy is claimed. What was measured is false records on benign documents, in numbers: health.patient_record 54 of 18,075, corporate.strategy 12 of 18,075, legal.master_agreement 5 of 18,075 (all German), finance.report 5 of 18,075, sales.proposal 4 of 18,075, and 1 or 0 for the others (table below). A finding of any of them is a suggestion for a person to look at; the counts say how often a benign document reached the value, and nothing about how often a document of the type does. public.published has no calibration value and stays rules.
• One seed. The head is seed 7001, the highest of three seeds on the development split. No three-seed repeat was done.

Files

file	what it is
README.md	This model card.
onnx/model_quantized.onnx	Two outputs: logits (PII, unchanged) and doc_logits [batch, 17]. 279,041,365 bytes, sha256 0c9c1780673d719310afd1bce2538d8e078ea2adf0bc3a6e397c564ac6486b6e.
doc_labels.json	Version 2: the 17 labels in output order and, per label, a mode, t_add, t_confirm, delta and a status that says whether the type is evaluated or an unvalidated suggestion.
tokenizer.json	The encoder's tokenizer (vocabulary and rules), byte-identical to ScienceSoft/scnsoft-pii-encoder at the revision below. sha256 a514807cffabd8abaf028cfaffe7ff0c4f60b97ea2db80c41f14172ae6b018ca.
config.json	The encoder's model configuration, byte-identical to the encoder's. sha256 1e840bedc5fac622c796f19c2021c4040641326fcacb9d18ab87c5d2e050e0db.
tokenizer_config.json	The encoder's tokenizer settings, byte-identical to the encoder's. sha256 ccf223ba3d5b3cc7fa6c3bf451f3bb40557a5c92b0aa33f63d17802ff1a96fd9.
special_tokens_map.json	The encoder's special-token map (start, end, padding, unknown and mask tokens), byte-identical to the encoder's. sha256 06e405a36dfe4b9604f484f6a1e619af1a7f7d09e34a8555eb0b77b66318067f.
THIRD_PARTY_NOTICES.md	Attributions for the training sources that ask for them.
MANIFEST.sha256	The sha256 of every file above.

The encoder is the one at revision b1a36210f99ef293a4d42697ef6b2ba548dd9208 (blob sha256 c996fc5ad1118e0571263ee637f541b3d0981bec648e57211ad4676a3dc1ab47). The tokenizer and config.json are the encoder's own. No separate PII-only graph is included, so pii_only_graph in doc_labels.json is null.

The head

Two layers over the mean-pooled encoder states of a 450-character window: 768 to 256, exact GELU (written with Erf, as the released graph already does), 256 to 17. The encoder is frozen. The graph is the released graph with nodes added; none of its nodes is changed.

The PII output is unchanged. On this graph logits is byte-identical to the released encoder's on all 6,825 rows of the PII release check, at batch size 1:

host	onnxruntime	threads
aarch64 Linux (NVIDIA GB10 host)	1.24.2	1 and 4
Apple Mac, arm64	1.24.2	1 and 4
Apple Mac, arm64	1.19.2	1 and 4

Not checked: the fp32 graph, Windows, and any batch size above 1.

Modes and thresholds

mode	meaning
rules	A rule-based classifier decides. The head's score is information only.
add	The rule-based classifier's decision stands, unchanged. When the rules find nothing for the type and the document's score reaches t_add, the head's finding is reported as a model-added type: a suggestion, softer than a rule decision.

False records are counted at t_add on 18,075 benign documents (see "What was measured").

label	mode	t_add	false records	status
legal.nda	add	0.84	1 of 18,075	false-record line met; nothing else evaluated
legal.master_agreement	add	0.82	5 of 18,075	unvalidated suggestion
legal.sow	add	0.87	1 of 18,075	unvalidated suggestion
legal.contract_other	rules		8 of 18,075	not yet monitoring-grade: line missed
sales.rfp	add	0.70	1 of 18,075	unvalidated suggestion
sales.proposal	add	0.72	4 of 18,075	unvalidated suggestion
sales.pricing	add	0.90	1 of 18,075	unvalidated suggestion
tech.source_code	add	0.99	0 of 18,075	unvalidated suggestion
tech.design	add	0.97	0 of 18,075	false-record line met; nothing else evaluated
tech.security	add	0.97	2 of 18,075	false-record line met; nothing else evaluated
finance.report	add	0.98	5 of 18,075	unvalidated suggestion
finance.billing	add	0.69	1 of 18,075	unvalidated suggestion
finance.personal	add	0.81	0 of 18,075	false-record line met; nothing else evaluated
hr.personnel	add	0.93	0 of 18,075	unvalidated suggestion
corporate.strategy	add	0.66	12 of 18,075	unvalidated suggestion
public.published	rules		no reporting value	the rules decide
health.patient_record	add	0.95	54 of 18,075	unvalidated suggestion

How t_add was set. Each is the reporting value of the type: the smallest value from 0.05 to 0.99 at which a document's score (rule lead2, below) stays at or under a false-positive rate of 2.78 x 10^-4 in every one of seven languages, with the upper 95% Wilson bound over all languages under it too, read on 80,739 ordinary documents of a calibration set. Setting t_add equal to the reporting value is a choice made for this release; it is not a registered rule. The reporting value says nothing about recall.

How a document is scored

1. Windows. The text is cut into 450-character windows overlapping by 50 characters.
2. Scores. Each window is scored on its own, at batch size 1, with sigmoid(doc_logits).
3. Evidence rule, lead2. A document reaches a type when both of its first two windows reach the threshold (the only window, for a one-window text), or when at least max(3, ceil(n / 5)) of its n windows do.

The thresholds are valid at batch size 1 only. The features and thresholds were produced one window per call on one thread. On the same 200 windows, scores computed in padded batches of 64 differ from batch-1 scores by up to 0.53 (per type the largest difference is between 0.14 and 0.53; 197 of the 200 windows moved by more than 0.001 on some type). Thread count made no difference. Do not score in batches.

A file name or title is not used.

What was measured

False records (held-out benign documents). A false record is a benign document whose gold label lacks the type and whose score is at or above the type's calibration value. Population: 18,075 benign documents from two held-out sets (English 3,139; German 2,509; French 2,509; Italian 2,510; Spanish 2,509; Russian 2,450; Arabic 2,449), read once for this head. Machine: aarch64 NVIDIA GB10 host, onnxruntime 1.24.2, batch 1, 1 thread. The line, fixed before the read: met when the pooled count is at most 5 and no language has more than 2.

type	value	pooled	by language	line
legal.nda	0.84	1	es 1	met
legal.contract_other	0.86	8	en 3, de 3, es 1, ru 1	missed
tech.design	0.97	0	none	met
tech.security	0.97	2	ru 2	met
finance.personal	0.81	0	none	met

Reported only, with no line and no verdict (these eleven types are fused as unvalidated suggestions; the numbers describe the head's score on benign documents, not an evaluation of the type):

type	value	pooled	by language
health.patient_record	0.95	54	en 33, es 6, it 6, fr 4, de 3, ru 2
corporate.strategy	0.66	12	ru 3, en 2, de 2, it 2, es 2, fr 1
legal.master_agreement	0.82	5	de 5
finance.report	0.98	5	en 3, de 1, it 1
sales.proposal	0.72	4	de 3, ar 1
legal.sow	0.87	1	es 1
sales.rfp	0.70	1	de 1
sales.pricing	0.90	1	fr 1
finance.billing	0.69	1	en 1
tech.source_code	0.99	0	none
hr.personnel	0.93	0	none

public.published has no calibration value. Meeting the false-record line does not make a type ready: recall, agreement between machines and cost on customer hardware are further conditions, and none of them is met.

legal.contract_other against the earlier preview (doc-head-v2), recall. 484 held-out contract documents from 430 filers, each model at its own calibration value (V3a 0.86, v2 0.85), batch 1 for both. V3a found 448 (recall 0.9256, 95% interval 0.9017 to 0.9481), v2 found 442 (0.9132, 0.8866 to 0.9380). Paired difference +0.0124 (interval -0.0063 to +0.0327). The difference is inside the interval around zero: V3a is not shown to be better than v2 on this type. Absolute recall is optimistic, because the set keeps only documents on which the rules and a model agreed; only the difference is a comparison, and it is inside noise.

Cost. p50 time per 450-character window at batch size 1 (aarch64 NVIDIA GB10 host, onnxruntime 1.24.2, 200 windows, one pass, no repeat): 41.6 ms on 1 thread and 14.1 ms on 4 threads for this graph; 41.5 and 13.7 ms for the released encoder alone. The head's cost is not distinguishable from zero at this resolution (about plus or minus 0.5 ms); it is not claimed to be zero. Peak memory 502 MB, against 501 MB for the encoder alone. Cost on a Mac, on Windows and on customer machines was not evaluated, nor was engine time per file.

PII output. Byte-identical to the encoder's on 6,825 rows at batch 1 (table above).

Limitations

• Machines disagree. Two files of doc_logits for the same 6,825 rows, computed on the aarch64 Linux host and on an Apple Mac, differ in bytes on every row. Against the Mac with onnxruntime 1.24.2, 611 rows fall on opposite sides of a threshold for some evaluated type (legal.nda 466, legal.contract_other 167, tech.security 7); against the Mac with 1.19.2, 138 (103, 35 and 2). Those files were produced without the engine's graph-arithmetic setting, so they are not a measurement of an engine and not a test of agreement between machines; the engine-level cells on a Mac and on Windows are pending, and the agreement bar between machines is not met. A document near a threshold may reach it on one machine and not on another. Thresholds were read on the aarch64 Linux host only.
• Batch composition changes scores (above). Only batch size 1 is described here.
• Not evaluated. Recall of the four types on documents of the type; the cross-machine bar; cost on a Mac, on Windows and on customer machines; engine time per file; a three-seed repeat; the fp32 graph; the PII output at batch sizes above 1; hr.personnel; and, beyond the false records above, every type other than the four.
• Training data. Six types have no real training document, only generated text: corporate.strategy, finance.billing, health.patient_record, sales.pricing, sales.proposal and sales.rfp. For the ten other types that have real training documents, all the real documents are in English and their other-language training documents are machine translations and generated text; only public.published, the examples of no type, has real documents in other languages.

Training data

47,824 training documents and 6,185 development documents. The training documents are 20,561 real, 6,680 machine translations and 20,583 generated. The data is not published. Provenance, as it is:

• Real public documents, under open licences or in the public domain: agreements from CUAD, ContractNLI, the KL3M collection of EDGAR agreements, Common Paper and Bonterms (CC BY 4.0), SEC EDGAR filings, official legislation (EU, French and Italian texts), United Nations documents, European Union and UK government pages (the GOV.UK pages are used as examples of documents that are none of the types, under the Open Government Licence v3.0), software and security documentation and advisories, Python Enhancement Proposals, US drug labels, payslip examples, and Wikinews.
• Machine translations of registered training documents, made with OPUS-MT (CC BY 4.0).
• Generated text written by an open-weight language model released under a permissive open licence, from prompts written for each type, and documents based on templates written by ScienceSoft.

Attributions that the sources ask for are in THIRD_PARTY_NOTICES.md. Text was cut into 450-character windows, pooled by the frozen encoder and used only to fit the head; no source text is redistributed with the model.

Licence

Proprietary to ScienceSoft. The weights are published so that the results on this page can be evaluated and reproduced; no licence to use them in production, to modify them or to redistribute them is granted. For licensing, contact ScienceSoft (https://www.scnsoft.com).

Basis for these terms. The model is derived from FacebookAI/xlm-roberta-base (MIT licence). Its head was fitted on documents from the sources listed in THIRD_PARTY_NOTICES.md and described above. The licences those sources carry that this release has checked (CC BY 4.0, the Open Government Licence v3.0, the Licence Ouverte 2.0, the European Commission's reuse policy for EUR-Lex, the United Nations corpus terms, and public-domain and permissive software and documentation licences) permit a derivative work to be distributed under proprietary terms, provided notices are retained and attribution is given; none is share-alike and none restricts commercial use. The SEC EDGAR filings used for training come through a dataset whose card says only that EDGAR data is publicly available; this release has not confirmed a licence statement for them beyond that.