Ingestion and parsing

Ingestion is more than a file upload. SAG normalizes different inputs to Markdown, then runs chunking, embedding, and event-entity extraction while background jobs track the complete process.

Supported sources

The reference application currently includes file uploads and a web connector. Common parseable formats include Markdown, plain text, PDF, Word, PowerPoint, and Excel. Exact format support comes from MarkItDown and the optional MinerU integration.

The API can also ingest a text block or a batch of messages directly, which works well for continuous synchronization of conversations, notes, and application data.

Processing pipeline

source
  -> normalize to Markdown
  -> semantic chunks
  -> chunk embeddings
  -> one event per chunk
  -> entities per chunk
  -> relational + vector indexes
  -> ready

The original text of every chunk remains intact. Events and entities locate and expand candidates, but display, generation, and citations always return to the chunk.

Document parser

The default is SAG_DOCUMENT_PARSER=auto:

• PDF uses MinerU 2.5 first when MinerU is fully configured.
• If MinerU is not configured, times out, or fails, SAG falls back to local MarkItDown.
• Office and text formats use MarkItDown by default.

MinerU job state and results can resume from cache, preventing background retries or concurrent work from causing duplicate charges.

Chunk settings

The default chunk limit is 1,000 tokens in standard mode:

SAG_DOCUMENT_CHUNK_MAX_TOKENS=1000
SAG_DOCUMENT_CHUNK_MODE=standard

Smaller chunks usually improve local targeting but create more vectors, extractions, and relationship records. Larger chunks preserve more context but can reduce precision. After changing the strategy, reprocess existing documents; changing environment variables does not update old indexes.

Background jobs

Document ingestion returns a document record while a background job continues processing. The default maximum is three attempts:

SAG_JOB_MAX_ATTEMPTS=3
SAG_DOCUMENT_EXTRACT_CONCURRENCY=5

Jobs can be paused, resumed, and reprocessed. When a failure occurs, inspect the document state and API logs before repeatedly uploading the same file.

Update and delete

Reprocessing rebuilds the document index with the current parser and model settings. When a document is deleted, the application adapter also removes related chunk, event, and entity associations along with the uploaded file so that stale content cannot remain searchable.

For batch or automated ingestion, use POST /api/v1/sources/{source_id}/documents/ingest. See the REST API reference for the complete request structure.