# Data



> Informational documentation, based on the paper Data Language Models: A New Foundation Model Class for Tabular Data (arXiv:2605.06290, https://arxiv.org/abs/2605.06290). Reference material, not a hosted service; hosts, keys, and ids are placeholders.

Data is one group covering connections and datasets, at `/v2/data` and on the platform's Data page. A **connection** is where data comes from. A **dataset** is an ingested or registered table. Each ingest or sync mints an immutable **snapshot** (`dsv_...`), which is what endpoints pin and held-out reports stamp.

## Sources

| Family | Sources | Ref shape |
|---|---|---|
| Upload | CSV, Excel, JSON | `upload` (platform), or inline `tables` in a request |
| Databases | PostgreSQL, MySQL, Supabase, MongoDB, Databricks, Snowflake, Pinecone, Chroma | `postgres://...`, `snowflake://...`, ... |
| Cloud storage | Google Drive, GCP Storage, AWS S3 | `gdrive://...`, `gs://...`, `s3://...` |
| API | REST, GraphQL | `rest+https://...`, `graphql+https://...` |

Small files upload directly on the platform or travel inline as `tables` in a request. Everything else is a connection: registered once, referenced by id or scheme ref.

## Register a connection

cURL:

```bash
curl -X POST https://api.example.com/v2/data/connect \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "source": "postgres://db.example.com:5432/crm/public/contacts",
        "name": "crm_contacts",
        "credentials": { "user": "reader", "password": "…" },
        "options": { "query": "select * from contacts where deleted_at is null" } }'
```
Python:

```python
import os, requests

API_KEY = os.environ["API_KEY"]
r = requests.post("https://api.example.com/v2/data/connect",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={"source": "s3://example-exports/crm/2026/", "name": "crm_exports",
          "credentials": {"access_key_id": "…", "secret_access_key": "…"}})
```
JavaScript:

```javascript
const API_KEY = process.env.API_KEY;
const res = await fetch('https://api.example.com/v2/data/connect', {
  method: 'POST',
  headers: { Authorization: `Bearer ${API_KEY}`, 'Content-Type': 'application/json' },
  body: JSON.stringify({ source: 'rest+https://api.example.com/v1/customers',
    name: 'customers_api', credentials: { bearer: '…' } }),
});
```

The response returns the connection (`conn_...`) and the dataset it produced (`ds_...`) with its first snapshot (`dsv_...`). A quick run uses it transiently and registers nothing new; endpoint creation pins it. `GET /v2/data/:id` returns the catalog profile: rows, columns, current snapshot, pin state, and which endpoints pin it. Full family: Data API.

## Snapshots and sync

Every ingest and every `POST /v2/data/:id/sync` mints a new immutable `dsv_...`. Endpoints and reports reference the exact snapshot they ran on; an endpoint keeps serving its current snapshot until you `refresh` it, even after a sync has minted a newer one. Schedule syncs from your orchestrator, then refresh dependent endpoints (see [Workflows and data stack](/model/integrations/data-stack)).

## Pinning

Pinning is a state on a registered dataset, set with `POST /v2/data/:id/pin` and `/unpin`.

An endpoint keeps serving its pinned snapshot until you refresh it.

Endpoint creation pins its datasets automatically. A dataset pinned by a live endpoint cannot be unpinned or deleted while that endpoint serves; the `409` names the dependent endpoint.

## Batch and large runs

Batch is a service tier, elected per job with `"options": { "processing": "batch" }`. It applies to inference and synthetic generation.

```http
POST /v2/run
{
  "data": ["snowflake://example/warehouse/events_2025"],
  "target": { "column": "converted" },
  "options": { "processing": "batch", "out": "warehouse://analytics.outputs" }
}
```

`202` with a job. The bundle is written to `out` (a file or warehouse table) on completion, or retrievable from the job when `out` is omitted and the result fits. A job that misses its window expires unbilled and you resubmit.

## Synthetic data

`POST /v2/data/generate { "sector": "hospital operations", "rows": 5000, "name": "hospital_ops_synth" }` produces a synthetic dataset for a sector with realistic per-column ranges, as an ordinary dataset you can run on, pin, or create an endpoint on. Useful when representative data is thin or as a safe test fixture. The response is the new dataset; large generations run as batch jobs. `sector` uses the same vocabulary the sector output returns; if unsure of the exact name, quick-run a small real table and use its `sector.top1.name`.

## Persistence follows the operation

| Operation | Stored |
|---|---|
| `run`, endpoint calls | No dataset is registered; submitted tables are used for the single pass and discarded. The run's job and run record remain visible in Jobs and Reports. Endpoint logs keep operation events, never row data. |
| Endpoint create, refresh, upgrade | The pinned snapshots, the held-out report with its split seed and partition, the configuration and system prompts. Inline `tables` at creation become an upload-type dataset under Data. |
| Data connect, sync | The dataset snapshot; connection credentials encrypted at rest, write-only. |

Everything stored is visible under Data, deletable immediately, and counted toward storage. Residency and deployment options: Trust Center.

## Deletion

| Resource | Policy |
|---|---|
| Datasets and connections | `DELETE /v2/data/:id` is immediate and irreversible. A dataset pinned by a live endpoint is refused with `409 conflict` naming the endpoint: delete the endpoint or refresh it onto other data first. |
| Endpoints | Immediate on `DELETE`. Reports remain in the audit trail; its datasets remain under Data and are unpinned. |
| Reports | Retained for the life of the org. |
| API keys | Revoked immediately with `DELETE /v2/keys/:id`; the key stops working and leaves the listing. |

**To remove data and its influence entirely**: remove the rows at source (or delete the dataset), `sync`, then `refresh` dependent endpoints. Purged rows leave the stored snapshot and the pinned context in one step. Data handling terms are in the DPA.

## Credentials and audit

Connection credentials are encrypted at rest and are write-only; rotate by re-entering. The audit log records connection, sync, key, and deletion events with actor and timestamp.

## Related

Trust Center · Privacy · DPA · Sub-processors · Security