Platform

One governed loop, every step recorded.

Datris is the open-source data control plane for AI agents. An agent asks for data; the platform acquires it, validates it, lands it in the stores you already run, and returns it with a receipt.

Have an engineer try it Commercial licensing

Introduction

More walkthroughs →

The operating loop

One agent-driven loop

Acquire, validate, land, observe, explain, repair. Same loop every run, same audit trail every time — so the agent's job is reasoning about the data, not improvising infrastructure.

🎯

Acquire

AI-Generated Taps

Describe a source in plain English — an API, a vendor drop, a folder of PDFs. Datris generates a Python tap, wires up scheduling, and brokers credentials through Vault. Schema is auto-generated from a sample.

✓

Validate

Plain-English Rules

Data quality rules described in natural language, turned into a script you can read when you save, and executed on every row in an isolated container. Bad records quarantine at the door — not after they pollute your warehouse.

🚀

Land

Multi-Destination Pipelines

One pipeline lands the same records in Postgres, MongoDB, Snowflake, Databricks, S3, and vector stores in parallel. Idempotent by key — reruns don't double-write.

👁

Observe

Provenance & Job State

Every run gets a durable record: which script, which commit, which inputs, which rows landed. Every change to an AI script is kept as a numbered version you can diff and restore without a model call. Agents poll job status through MCP; humans see it in the UI. The transcript is not the system of record.

🩹

Explain & Repair

Incidents & Recovery Agent

When something fails, Datris opens an incident, explains the root cause in plain English, and proposes a fix. The fix is rehearsed against a scratch copy, shown as before-and-after evidence, and landed with a real run or held for your approval — within limits the model cannot override.

Self-healing data flows

A recovery agent that shows its work

When a scheduled tap fails, a pipeline errors, a source goes stale, volume swings, or columns change, the platform opens an incident, diagnoses it, and repairs it within limits you set. You see the before and after before anything lands.

observe diagnose propose rehearse gate execute verify record

Evidence before you approve

An incident shows what the resource feeds, columns added and removed, rows and data-quality pass rate before and after, the biggest null-rate changes, each threshold with a pass or fail, and a line-by-line diff of the script. Column values are never shown. The same numbers reach your webhook.

Rehearsed in a scratch lane first

A proposed fix runs against a scratch copy of the pipeline and is compared with the last good run. A fix that clearly breaks the data is stopped with nothing written. One that changes the data's shape waits for a person. Only a fix that passes lands with a real run, and a real run that fails is reverted.

Problems are named, not guessed

A tap whose columns changed opens a schema-drift incident that names the columns and what is downstream. A source that rejects its credentials is marked as a credential problem and names the secret to check, never retried or rewritten. A failure the doctor traces to the platform carries the doctor's fix, before any AI is used.

The model proposes, the platform decides

Limits, verification, and the revert are deterministic code the model cannot talk its way past. Every action passes through Agent Policy. On autopilot the per-action matrix still applies, and a single critical tap can be held on propose while the rest run on their own.

It repairs its own scripts

When a pipeline fails because its AI data-quality or transformation script fails, the agent can regenerate the script from the pipeline's instruction, rehearse it, prove it with a real run, and restore the previous version if either fails. The incident names both versions and opens their comparison.

It closes itself and learns

An incident waiting on a person closes as resolved when the tap or pipeline next runs healthy, and withdraws anything it had queued. A source quirk the agent noticed is saved as a suggested prompt fragment that the tap generator uses only after a person accepts it.

Three modes, chosen by an admin. It ships off and never changes on its own.

off Failures get an AI explanation in run history. Nothing else happens.

propose Incidents open and are diagnosed. Every proposed action waits for a person.

autopilot Actions follow your per-action policy: run, queue for approval, or refuse.

The Agent Policy page recommends settings for the install's environment label: autopilot with looser limits on dev, propose with strict limits on QA, production, and unlabelled installs. One click applies them.

Incidents and the recovery agent → Agent Policy →

Where Datris fits

Agents don't need a new data platform. They need a way into yours.

Datris sits beside the warehouse and the lake, never in front of them. It is the intake valve: the agent asks, Datris acquires and validates, and the rows land in the stores your teams already query. Nothing moves out of your stack and nothing new becomes the system of record.

Agents Don't Need a New Data Platform →
Load the Lake. Don't Live in It. →

The write path

SnowflakeAnalytics warehouse
DatabricksLakehouse
PostgreSQLOperational store
MongoDBDocuments
S3 / MinIOObject storage, Parquet
pgvector · Qdrant · Weaviate · Milvus · ChromaVector search

One pipeline lands the same validated records in several of these in parallel, idempotent by key. Keep the intake layer neutral and every future architecture decision stays yours.

Governance

Enforced by the platform, not by the prompt

The controls a risk committee asks about, in the open-source build, switched on for production. This is the part of Datris a head of data repeats to an auditor.

Agent Policy

Per-action allow, approve, or refuse, enforced by the platform rather than by a prompt. An agent that is not allowed to delete a pipeline cannot, no matter what it was told. A schedule can only point at a script that has passed a test run, whoever edited it. The recovery agent runs under the same matrix: off, propose, or autopilot, with limits recommended per environment and set by an admin.

The AI approval you already have →

Audit log

With the audit log on, every create, change, run, delete, login, and denied request, attributed to a login or API key. Export to CSV or mirror to your SIEM.

Stop giving agents master keys →

Provenance

With provenance stamping on, every landed row cites the pipeline, the run, the script commit, and the source. Agent-written scripts are versioned in git, and every change to a pipeline's AI script is kept as a numbered version you can compare line by line and restore, so what ran is what you can diff.

If an agent wrote it, git should remember it →

Lineage

Where each dataset came from and where it went, across every destination a pipeline writes to.

Validate once, deliver everywhere →

Field protection

Mark a PII or PHI column and it is pseudonymized, masked, redacted, dropped, or encrypted on your server before data quality, AI steps, previews, or any destination see it. A HIPAA Safe Harbor preset classifies columns by name. Encrypted values can be revealed only by a role granted that permission on purpose, and every reveal is audited.

Field protection docs →

Single sign-on

People sign in through your identity provider over OpenID Connect: Okta, Microsoft Entra ID, Google Workspace, Keycloak, Auth0, or any OIDC provider. MFA is whatever your provider enforces, unknown people are refused unless you choose a default role, and every SSO sign-in and refusal is in the audit log.

Single sign-on docs →

Commercial licensing Security overview

Dev, QA, and production

Promote a catalog, not a pile of settings

Run one install per environment and move work forward as a catalog archive, with code in one GitHub repository and a branch per environment. The same secret names everywhere, with different values.

dev Build taps and pipelines

qa Test against governed settings

production Run on a schedule

A catalog is one file

Download a catalog as a zip: every tap and pipeline definition, each tap's script and incremental state, and the validation schemas it uses. Secret values, destination data, and run history are never included, so the archive sits safely with your other configuration backups. From the UI, the CLI, or an agent.

Restores never surprise you

Import on the same install or another one. A restore never merges into an existing catalog: clashes are listed and nothing changes unless you choose to overwrite. Secrets travel by name only, and the result says which ones the target is missing. Restored taps start disabled, so nothing runs on its own.

Production runs what QA tested

Promoted pipelines run the exact AI data-quality and transformation scripts QA tested, with the model and time they were generated, and the restore makes no model call. Enable in order tests, enables, and runs a restored catalog's taps in dependency order, waiting for each source to land.

You always know which install you're on

Label each install dev, qa, or production and the header shows it, red for production. Connected agents are told the environment before they act, tools that change things name it, and on production agents confirm with you before a change.

Dev, QA and production guide → Data catalogs →

Live Read

Not every answer needs a table

Most of what an agent asks for is a one-off: what a source returns for these ids, whether today's file is clean, what a transformation would produce. Live Read runs the full governed pipeline and hands the rows back to the caller instead of landing them. If the rows turn out to be worth keeping, switch the destination. The tap, its schedule, and its sync bookmark do not move.

Land it

Data you will want again

Rows land beside your lake, in the stores your teams already query. Observed, catalogued, refreshed on a schedule, and with provenance stamping on every row cites its source.

Read it live

An answer you need once

The same run, but no loader. Validated rows go straight back to the agent, are held briefly, and expire. Nothing is written, nothing new becomes a system of record.

What stays the same in both lanes

The tap, generated or hand-written, with credentials brokered through Vault
Data quality rules and the transformation chain
Agent Policy, the audit log, and the run record
The agent never holds a key, in either lane

The built-in Assistant picks the lane for you: a one-off question goes to Live Read, not a new table. Live Read in the docs →

Model Context Protocol

Your AI agents are
first-class pipeline operators

Datris ships with a native MCP server. Claude, Cursor, OpenClaw, and any MCP-compatible AI agent can register pipelines, trigger jobs, and query your structured, document, and vector data in real time — all through natural conversation.

Transports: stdioSSE (Server-Sent Events)Streamable HTTP

Compatible agents:

MCP Capabilities

Register pipelines and generate schemas from sample data
Create, schedule, and run AI-generated taps
Ingest documents into vector databases (extract → chunk → embed)
Upload data for processing
Trigger and monitor pipeline jobs
Profile data and get AI insights
Semantic search across vector databases
Query PostgreSQL, MongoDB, Snowflake, and Databricks directly
Get an answer without landing a table — Live Read pipelines return validated rows straight to the agent
Run the built-in doctor to diagnose the platform before retrying a failed run
Manage credentials via Vault — without ever holding the key

Example prompt

"Generate a tap for our prime broker margin API, schedule it daily, validate that 'all account_ids must be present and balances must be non-negative', and load into Postgres."

Formats

Speaks every data language

Ingest structured data, unstructured documents, and archives. Output to vector stores, structured stores, or optimized columnar formats.

Format	Input	Default Destination
CSV		SQL DB
JSON / NDJSON		NoSQL DB
XML		NoSQL DB
Excel (.xlsx)		SQL DB
PDF		Vector DB
Word (.docx)		Vector DB
PowerPoint (.pptx)		Vector DB
HTML		Vector DB
Email (.eml)		Vector DB
EPUB		Vector DB
Archives (.zip, .tar, .gz)		Unpacked, routed
Plain Text		Vector DB

Destinations are fully configurable. Route any format to any target — SQL databases, NoSQL stores, vector databases, REST endpoints, Kafka topics, or ActiveMQ queues. Object-store destinations write Parquet, ORC, or Apache Iceberg tables with atomic commits, upserts by key, partitioning, and schema evolution. On Databricks, Iceberg tables are registered through Unity Catalog, and descriptions, tags, and lineage are written to the catalog by default. Or skip landing entirely: a Live Read pipeline runs the same validation and transformation chain and hands the rows straight back to the caller.

RAG Pipeline

Full RAG pipeline built in

Extract, chunk, embed, and upsert documents into any major vector database. Build retrieval-augmented generation workflows without leaving your pipeline.

🗄️

5 Vector Databases

Qdrant Weaviate Milvus Chroma pgvector

✂️

Chunking Strategies

Choose the right chunking strategy for your use case:

🧠

Embedding Providers

Generate embeddings with cloud or local models:

📄

Document Extraction

Extract text from virtually any document format:

📒

Document Ledger

Content-hashed ledger of every document seen. Re-runs skip unchanged files automatically — no re-embedding the same folder every night, no surprise OpenAI bill.

RAG Pipeline Flow

1 Document Extraction

→

2 Chunking

→

3 Embeddings

→

4 Vector Upsert