--- title: "Ingest Documents" description: "How to ingest text, streams, files, metadata, and XION-signed content into a Lucent collection." published: 2026-05-14T12:11:30.963447+00:00 updated: 2026-05-14T12:11:30.963447+00:00 tags: ["guide", "ingestion", "lucent"] url: https://xiobjects.com/docs/xio/lucent/guides/ingest-documents source: XI Objects --- # Ingest Documents All ingestion goes through `IKnowledgeEngine.AddDocumentAsync`. The method is overloaded in a few useful ways. ## Prerequisites - `Xio.Lucent` installed and registered with `services.AddLucent()` - `engine.ProvisionAsync()` called at startup ## Ingest plain text Pass content as a string. The content type is detected automatically, or set a hint to skip detection. ```csharp var result = await engine.AddDocumentAsync("my-collection", new AddDocumentRequest { Content = "The quick brown fox...", DocumentId = "fox-article", ContentTypeHint = "text/plain" }); ``` `DocumentId` is optional. Omitting it generates a deterministic ID from a BLAKE3 hash of the content. The same content always produces the same ID, which makes idempotent ingestion straightforward. ## Ingest a file stream Pass a forward-only stream for binary formats. Lucent reads the stream once and doesn't seek. ```csharp await using var file = File.OpenRead("/docs/spec.pdf"); var result = await engine.AddDocumentAsync("my-collection", new AddDocumentRequest { ContentStream = file, Source = new SourceInfo { FileName = "spec.pdf", FilePath = "/docs/spec.pdf", LastModified = File.GetLastWriteTimeUtc("/docs/spec.pdf") } }); Console.WriteLine($"{result.ChunkCount} chunks from {result.ContentType}"); ``` The `Source` record is optional, but providing `FileName` improves content type detection and populates `ChunkMetadata.FilePath`. `ContentStream` and `Content` are mutually exclusive; setting both throws. ## Attach metadata Any key-value pairs in `AddDocumentRequest.Metadata` are attached to every chunk from that document. They're queryable via filter expressions at retrieval time. ```csharp var result = await engine.AddDocumentAsync("policies", new AddDocumentRequest { ContentStream = file, Source = new SourceInfo { FileName = "privacy-policy-v3.pdf" }, Metadata = new Dictionary { ["category"] = "legal", ["version"] = "3", ["status"] = "approved", ["owner"] = "legal-team" } }); ``` Later, filter to only approved legal docs: ```csharp using static Lucent.Filter; var result = await engine.QueryAsync("policies", new QueryRequest { Text = "data retention obligations", Filter = And(Eq("category", "legal"), Eq("status", "approved")) }); ``` ## Ingest XION-signed content Set `IsXionContent = true` to trigger XION verification before ingestion. The document must be a valid XION-signed artifact; verification failure rejects the document with an exception. ```csharp var xionMarkdown = File.ReadAllText("report.xion.md"); var result = await engine.AddDocumentAsync("verified-docs", new AddDocumentRequest { Content = xionMarkdown, DocumentId = "q1-report", IsXionContent = true }); ``` On success, every chunk carries a `XionAttribution` record on its `ChunkMetadata`. See [XRAG: Verified-Provenance RAG](/docs/xio/lucent/concepts/xrag) for the full details. XRAG requires `AddLucentXionVerification` in your DI registration. Attempting to ingest XION content without it throws at runtime. ## Update a document Calling `AddDocumentAsync` with an existing `DocumentId` is an upsert. Lucent compares the BLAKE3 hash of the incoming content against the stored hash: - Same hash, same embedder model: no-op. Returns immediately with the existing chunk count and zero durations. - Same hash, different model: re-embeds all chunks with the new model. - Different hash: runs the full pipeline, replaces all chunks. This means you can call `AddDocumentAsync` on every deploy without worrying about redundant work for unchanged files. ## Delete a document ```csharp await engine.DeleteDocumentAsync("my-collection", "fox-article"); ``` Removes the document registry entry and all associated chunks from both the vector store and full-text index. ## List documents ```csharp var docs = await engine.ListDocumentsAsync("my-collection"); foreach (var doc in docs) { Console.WriteLine($"{doc.DocumentId} — {doc.ChunkCount} chunks, ingested {doc.IngestedAt:u}"); } ``` `DocumentInfo` carries the document ID, content type, chunk count, model ID, source info, and ingest timestamp. ## Export and import Lucent can export a collection to a single binary stream and restore it elsewhere. The SQLite adapter uses `VACUUM INTO` for an atomic snapshot with no WAL or SHM sidecars. ```csharp // Export await using var backup = File.Create("collection-backup.lucent"); await engine.ExportAsync("my-collection", backup); // Import into a fresh engine await using var restored = File.OpenRead("collection-backup.lucent"); await engine.ImportAsync("my-collection", restored); ``` ## Next Steps - [Query a Collection](/docs/xio/lucent/guides/query-collection) — search, filters, score thresholds - [XRAG: Verified-Provenance RAG](/docs/xio/lucent/concepts/xrag) — provenance querying and signed source retrieval - [Chunking Strategies](/docs/xio/lucent/concepts/chunking) — choose the right chunker for your content