duckle

Quick links

What is Duckle?

A visual data pipeline studio that runs on your laptop. Drag sources, transforms, validators, and sinks onto a canvas. Wire them together. Press Run. Duckle compiles the graph to SQL and executes it through a real columnar engine, with live previews, generated SQL on every node, and zero hidden state.

Three things make Duckle different from the heavyweights and the toy ETL tools:

1. An AI assistant that ships in the box. Describe the pipeline you want in English; Duckie writes the JSON and drops it onto the canvas. The model runs locally - no API key, no telemetry, no cloud round-trip.
2. 290+ connectors at install time. Files, lakehouses, SQL databases, warehouses, NoSQL, vector DBs, streaming brokers, SaaS REST/GraphQL APIs, even FTP and IMAP - working today, not coming-soon.
3. A self-contained binary you can audit. ~65 MB download. Engines install on first launch. Workspaces are plain files in a folder you choose. Diff them, branch them, ship them.

Meet Duckie - the local AI pipeline assistant

Describe what you need. Duckie writes the pipeline.

The sidebar on the right is Duckie AI Assistant - powered by Qwen 2.5 Coder 1.5B running through llama.cpp, downloaded once (~1.1 GB) and then run entirely on your CPU. Ask in plain English; Duckie streams back a valid Duckle pipeline definition. One click drops it onto the canvas, ready to inspect, tweak, and run.

Why Duckle is different

Status

Duckle is in public beta. The visual designer, the DuckDB execution engine, the scheduler, the cloud connectors, and the Duckie AI assistant all work today and are covered by 170+ integration tests across Linux, macOS, and Windows. The catalog is still growing and APIs may evolve before 1.0, but the day-to-day surface is stable enough for real work.

Scope, stated plainly: Duckle is a single-machine, embedded studio. If you outgrow one box, point Duckle's output at the system that scales (a warehouse, an object store, a lakehouse). It will not pretend to be a cluster.

The component palette ships 313 nodes so the roadmap is visible in the product itself:

• 292 available runs on the DuckDB engine today
• 5 preview is configurable in the designer (drag, wire, set properties); execution is being wired engine-by-engine
• 16 planned is reserved in the palette but not yet executable - see docs/roadmap.md

Screenshots

Real pipelines, built and run in Duckle - not mockups.

_{A 5M-row pipeline: a CSV, a Parquet file, a DuckDB table, and a SQLite table enriched through one visual Map (3-way join), no SQL.}

_{Left: the visual Map editor - main plus lookups, per-output expressions, an inline filter. Right: Parallelize fanning out aggregate, window, and top-N branches.}

_{One run, many branches: 16 nodes finish in a few seconds. Concurrency auto-detects from CPU cores; branches write to Parquet, CSV, DuckDB, and SQLite at once.}

_{Left: DuckLake CDC change-feed mirrored via upsert + delete propagation (100k rows). Right: watermark incremental load over 5M rows, advancing state only on a fully successful run.}

Capabilities

Duckle is not a CSV tool with extras. It reads a broad set of formats and sources, ships a deep transform library, and writes to files, databases, object storage, vector DBs, message buses, and email.

Sources (74 available)

For CSV / TSV sources, the Schema panel accepts an optional per-column Format (a strptime token string such as %d/%m/%Y) on Date and Timestamp columns. Several date columns can each parse a different layout in one read - the column is read as text and re-parsed with its own format, working around DuckDB's single global date format. A value that does not match its format becomes null rather than failing the run.

Transforms (126 available)

All 6 AI transforms ship today. Three need a model API (LLM, Classify, Embeddings) and ride the apiKey-in-props pattern; three are pure-local (Chunk, PII Redact, Dedupe).

Data quality (12 available)

Validators split their input: passing rows continue on the main port, failures route to a reject port you can sink, count, or inspect.

Custom code (7 available)

Sinks (58 available)

Control flow (19 available)

Advanced settings (per-node)

Every node has an Advanced tab with fields the engine honours at run time:

Orchestration and workspace

Clean data before it reaches your AI

Models inherit the quality of their inputs. RAG indexes, embedding stores, and training sets quietly accumulate duplicates, nulls, malformed rows, mixed encodings, and inconsistent schemas. Duckle is built to scrub that data before it lands in a vector store:

• Deduplicate with exact Distinct, Uniqueness, and Fuzzy Deduplicate (Jaro-Winkler / Levenshtein); use Record Match to find near-duplicate pairs with a similarity score
• Semantic dedupe with xf.ai.dedupe over a precomputed embedding column
• Profile + describe every column up front (Column Profile, Describe, Histogram) so issues surface before they reach a model
• Validate and filter malformed, empty, or out-of-range records and route failures to a reject port
• Normalize types, encodings, casing, and null handling across messy sources (Standardize, Cast, regex / string transforms)
• Redact PII (emails, phones, SSNs, credit cards) via xf.ai.pii before embedding
• Chunk + embed long text via xf.ai.chunk -> xf.ai.embed for RAG indexing
• Classify rows with an LLM (xf.ai.classify constrains the model to one of N user-supplied categories)
• Retrieve with both halves of hybrid search, locally, no model API required: Vector Similarity Search (cosine / L2 / inner product) and Full-Text Search (BM25)
• Land it in your store - pgvector ships, and Pinecone, Qdrant, Weaviate, Milvus all have working sinks that POST batches through each vendor's HTTP API

Engines

Duckle ships a thin shell and installs its engines on first launch.

First-launch extension pre-fetch

When the installer downloads the DuckDB CLI it also pre-fetches the extensions Duckle uses, with per-extension progress, so the first time you touch a Postgres source or an Iceberg table there is no surprise network hop mid-pipeline:

httpfs (S3 / GCS / HTTP), azure (Azure Blob native), sqlite, postgres, mysql, excel, iceberg, delta, ducklake, vss, fts.

spatial is lazy-loaded (~50 MB GDAL bundle) - it installs on first use of a geospatial source/sink to keep the initial download small.

Download / Install

Pick the binary for your OS from the latest release:

The single-file binary above is all you need for Build Pipeline too: the headless runner is embedded into the app at build time, and exporting a pipeline produces ONE self-contained executable (the engine, the DuckDB CLI, any needed extensions, and the resolved pipeline are all inside that one file). Copy that single file to your server and run or schedule it - no separate runner download required.

The binary is ~55-78 MB depending on platform (it embeds the headless runner and the bundled MCP server). On first launch you'll be guided through downloading two engines into your app-data directory:

App-data location:

• Windows: %APPDATA%\io.duckle.app\engines\
• macOS: ~/Library/Application Support/io.duckle.app/engines/
• Linux: ~/.config/io.duckle.app/engines/

Delete the engines/ folder if you ever want to force a fresh install.

Quickstart (60 seconds)

1. Download the binary for your OS (see Download / Install above) - or build from source.
2. Launch it. First run shows the setup modal:
   • Click Install on DuckDB (required, takes ~30 s).
   • Optionally click Install on Duckie AI Assistant (~1.1 GB, takes 5-10 min on average broadband).
3. Pick a workspace folder. Pipelines, connections, context variables, and routines live there as plain files.
4. Build a pipeline two ways:
   • Drag + wire: drag a CSV source in, point it at samples/orders.csv, hit Autodetect schema. Drag a Filter, wire it up. Drag a Parquet sink with an output path. Press Run, watch the nodes light up.
   • Ask Duckie: click the Sparkles icon (top-right of the toolbar), type "read orders.csv, filter where status = 'paid', write to paid.parquet". When Duckie streams back a pipeline, click Insert into canvas.
5. Inspect. Click any node to see its generated SQL in the Plan tab and a live row sample in the Preview tab.

That's a real, native ETL pipeline built and run in under a minute. CSV is just the easiest first node; swap in Parquet, JSON, S3, Snowflake, MongoDB, or Stripe the same way.

Run your first pipeline

A worked example using the bundled samples/orders.csv data.

1. Add a source

• Open the Components sidebar (left). Click Sources -> Files -> CSV.
• Drag it onto the canvas.
• In the right-side Properties panel:
  • Path: browse to samples/orders.csv
  • Click Autodetect schema - the Schema tab fills in column types from the file, the Preview tab shows the first 20 rows.

2. Add a transform

• Components -> Transforms -> Rows -> Filter. Drag onto canvas.
• Wire the CSV source's main output port to the Filter's main input.
• In Properties:
  • Predicate: status = 'paid' (you can write raw SQL or use the visual builder)
  • Filter has two output ports: pass (rows matching) and reject (rows that don't).

3. Add a sink

• Components -> Sinks -> Files -> Parquet.
• Wire Filter's pass port to the Parquet sink.
• Path: paid_orders.parquet. Write mode: overwrite. Compression: zstd.

4. Run it

• Press Run in the toolbar. Nodes light up in execution order; row counts appear under each.
• Open the Output tab (bottom panel) to see per-stage timing.
• Click any node to inspect generated SQL in Plan + sampled rows in Preview.

5. Iterate

• Add a Group By before the sink to aggregate. Re-run. Sub-second on small data.
• Cancel mid-run with the Stop button - the DuckDB process is killed cleanly.
• Save your work: Cmd/Ctrl-S writes a JSON pipeline file to your workspace folder.

How to use Duckle

A wider tour of the workflow.

Recipes and examples

Ready-to-adapt patterns. Each one is a few nodes you wire on the canvas (or ask Duckie to sketch).

CSV cleanup

"Read orders.csv, drop nulls, deduplicate by order_id, write to orders_clean.parquet"

src.csv -> qa.not_null -> qa.uniqueness -> snk.parquet

Set qa.not_null to the columns that must be present; set qa.uniqueness to order_id. Rejected rows go to a snk.csv on the reject port for inspection.

Postgres -> Snowflake nightly load

"Read all rows from Postgres events, upsert into Snowflake table analytics.events on event_id"

src.postgres -> snk.snowflake (mode=upsert, conflict=event_id)

Attach a ctl.schedule with cron 0 2 * * * to run nightly at 02:00.

S3 -> partitioned Parquet

"Read all .json.gz files in s3://logs/2026/*/*.json.gz, parse, write Hive-partitioned by event_date"

src.s3 (glob, autodetect json.gz)
  -> xf.derive (event_date = CAST(ts AS DATE))
  -> snk.parquet (path=out/, partitionBy=event_date, mode=overwrite_or_ignore)

RAG ingestion

"Chunk our docs, embed with OpenAI, dedupe near-identicals, store in pgvector"

src.s3 (markdown files)
  -> xf.ai.chunk (chunkSize=1500, overlap=150)
  -> xf.ai.pii (redact)
  -> xf.ai.embed (model=text-embedding-3-small, baseUrl=https://api.openai.com)
  -> xf.ai.dedupe (threshold=0.95)
  -> snk.pgvector (table=docs)

Slack channel digest

"Pull yesterday's Slack messages from #support, classify by sentiment, email a summary"

src.slack (channels.history with oldest=yesterday)
  -> xf.ai.classify (categories=positive,negative,neutral)
  -> xf.aggregate (group by sentiment, count)
  -> snk.email (to=oncall@..., subject=Daily Support Digest)

Webhook -> S3 archive

"Receive 100 webhooks, archive each one as JSON in S3"

src.webhook (port=8080, maxRequests=100, timeoutMs=300000)
  -> snk.s3 (path=s3://archive/events/, format=jsonl, partitionBy=event_date)

Git commit-log analytics

"Build a dashboard of who's been committing what in the last 30 days"

src.git (mode=log, maxRows=10000)
  -> xf.filter (date > current_date - INTERVAL '30 days')
  -> xf.aggregate (group by author_email, count)
  -> snk.csv (path=author-stats.csv)

More examples live in samples/ - drop the pipeline files into a workspace and open them.

Git integration (GitHub + GitLab)

Push, pull, branch, and watch CI from inside Duckle. No terminal required.

Click the Git icon in the topbar to open the workspace Git panel. Built-in integration with GitHub and GitLab, on the system git CLI (no FFI, no embedded git library):

Workflow. Workspaces are plain folders (see Workspace and Git flow) - any standard Git workflow works:

Create / clone -> open in Duckle -> edit pipelines -> commit + push -> 
PR / MR -> CI runs your pipeline tests -> merge -> pull

You can do the entire push / pull / merge loop without leaving Duckle. Heavy operations (interactive rebase, conflict resolution, log archaeology) still live in your terminal or external Git tool - the panel is designed for the everyday flow, not as a full Git replacement.

Provider detection. The remote URL host determines which CI API the badge polls:

The badge uses the same PAT you saved for pushes - no separate auth step.

Workspace and Git flow

A workspace is a folder you pick on first launch. Everything you build lives there as plain text:

my-workspace/
  pipelines/
    orders_etl.pipeline.json     # the node graph
    nightly_load.pipeline.json
  connections/
    prod-postgres.connection.json # saved DB credentials (encrypted)
    snowflake-analytics.connection.json
  contexts/
    dev.context.json              # variables for dev environment
    prod.context.json
  routines/
    cleanse-addresses.sql         # reusable SQL snippets
  documents/
    runbook.md                    # plain-Markdown docs
  schedules.json                  # all scheduled runs in this workspace
  run-history/
    orders_etl/                   # one folder per pipeline
      2026-05-25T14-30-00.json    # one file per run

Git-friendly by design. Every file is human-readable JSON or Markdown. Standard workflows work:

git init my-workspace && cd my-workspace
git add . && git commit -m "Initial pipelines"

# Pull a teammate's update
git pull --rebase

# Push your changes
git push

# Branch for a risky migration
git checkout -b feature/upsert-mode
# ...edit pipelines in Duckle...
git diff       # readable JSON diffs
git push -u origin feature/upsert-mode
# open PR / MR

Sensitive values in connections get encrypted with a workspace-local key (workspace/.duckle/keys/). Don't commit that file - add **/.duckle/keys/ to .gitignore. The connection JSON files themselves only hold the ciphertext, which is safe.

Schedules and triggers

Pipelines can run on cron, fixed interval, or file-watch triggers. Configure these in the Schedule panel (toolbar -> Schedule icon), not as graph nodes.

Schedules persist to workspace/schedules.json and execute via the in-process scheduler crate. They survive app restarts but require Duckle to be running.

For headless / always-on schedules that run when Duckle is closed, build the pipeline into a standalone file and let the operating system's own scheduler run it - see Server deployment below.

Server deployment (Build Pipeline)

The in-app scheduler runs only while Duckle is open. To run a pipeline on a server with no desktop app, Build Pipeline turns it into ONE self-contained executable - the equivalent of a standalone "Job".

Right-click a pipeline (in the project tree or on the canvas) and choose Build Pipeline. The output is a single file named after the pipeline (orders_etl.exe on Windows, orders_etl on macOS / Linux) that embeds everything it needs:

• the headless execution engine,
• the DuckDB CLI,
• only the DuckDB extensions that pipeline's components actually use,
• the resolved pipeline (context variables substituted, routines inlined),
• its secrets (see below).

On first run it self-extracts to a temp cache and uses its own embedded DuckDB, so the server needs nothing installed - no Duckle, no DuckDB. There is no folder to copy, no run.sh, and no separate runner download. A CSV-to-CSV pipeline builds to about 28 MB; only the extensions a pipeline uses are bundled, so the file stays lean.

./orders_etl            # or orders_etl.exe on Windows

The process exits 0 on success and non-zero on failure, and writes the same NDJSON run logs under logs/ (Splunk / Dynatrace friendly).

Build options

Schedule it with whatever the server already has - point the OS scheduler straight at the file:

# Linux cron - run every day at 02:00
0 2 * * * /opt/duckle/orders_etl >> /var/log/orders_etl.log 2>&1

On Windows use Task Scheduler; on macOS a launchd plist; on Linux a systemd timer. Full examples in docs/current/scheduler.md.

Run against an existing workspace - the same embedded headless runner can also execute a pipeline JSON directly, resolving context the way the app does:

duckle-runner --pipeline /path/to/pipeline.json [--workspace /path/to/workspace] [--duckdb /path/to/duckdb]

MCP server (connect Claude or any LLM to Duckle)

Duckle ships its own Model Context Protocolserver, so Claude (or any MCP client - Claude Desktop, Claude Code, Cursor, orany other LLM agent) can drive Duckle directly: browse the full component catalogand per-component property schemas, generate a pipeline straight into a workingdirectory you choose, validate it (compile without running), run it headlessly,read existing pipelines and their run logs, build a standalone artifact, andmanage saved connections.

Connect in one click (recommended)

The MCP server is bundled inside the app - there is nothing extra to install.In the designer, click Connect to Claude in the top bar to open the connectorpopup, then pick your client:

• Connect to Claude Code - registers the duckle server for you (runsclaude mcp add under the hood).
• Add to Claude Desktop / Add to Cursor - writes the duckle entry intothat client's config, with the resolved engine paths filled in (both theMicrosoft Store / MSIX and standalone Claude Desktop layouts are handled).
• Or copy the command / config for any other MCP client.

Restart the AI client, then try "Use duckle to list the available components"to confirm the connection.

Manual / headless

For a build-from-source or server setup, point any client at the duckle-mcpbinary directly. It speaks JSON-RPC over stdio and reuses the DuckDB enginein-process (no GUI, no Node runtime).

cargo build -p duckle-mcp --release      # target/release/duckle-mcp
claude mcp add duckle -- /path/to/duckle-mcp

For Claude Desktop and other clients, add it to mcpServers:

{
  "mcpServers": {
    "duckle": {
      "command": "/path/to/duckle-mcp",
      "env": {
        "DUCKLE_DUCKDB_BIN": "/path/to/duckdb",
        "DUCKLE_RUNNER_BIN": "/path/to/duckle-runner"
      }
    }
  }
}

Tools: list_components, get_component_schema, create_pipeline,validate_pipeline, run_pipeline, list_pipelines, read_pipeline,read_run_logs, build_pipeline, list_connections, create_connection.run_pipeline / build_pipeline need a DuckDB binary (DUCKLE_DUCKDB_BIN);build_pipeline also needs duckle-runner (DUCKLE_RUNNER_BIN). Full guide:docs/current/mcp.md.

Connection management

Saved connections become DuckDB secrets at runtime so credentials never leak into the pipeline JSON.

Connections live in workspace/connections/ as JSON. The token/password field is encrypted with the workspace key; the rest is plain text.

To use a connection in a pipeline, the Properties panel of any compatible source/sink shows a Connection dropdown - pick one and the fields auto-fill.

The Copy SQL / Export SQL output is display-only and never executed. Secret values (passwords, tokens, keys, connection strings) are replaced with named placeholders such as ${DUCKLE_PASSWORD}, so the exported script stays valid and is safe to share - substitute the real value at run time. To emit the real credentials instead (so the script runs unchanged), set the environment variable DUCKLE_EXPORT_INCLUDE_SECRETS=1; the output then contains live secrets and should be handled accordingly.

Context variables

Bind any field to a context variable that resolves at run time. Useful for dev vs prod, per-environment paths, secrets injected from CI, etc.

In a context file (workspace/contexts/prod.context.json):

{
  "name": "prod",
  "vars": {
    "DB_HOST": "db.internal.acme.com",
    "S3_BUCKET": "acme-prod-data",
    "BATCH_SIZE": "10000"
  }
}

In the Properties panel of any node, switch a field from Manual to Context and pick DB_HOST. Or inline-reference one with ${DB_HOST} in a string field.

Pick the active context from the topbar's Context dropdown. Switch contexts and re-run without editing the pipeline.

Build from source

Prerequisites

• Rust (stable)
• Node.js 18+ and npm
• cargo-tauri CLI: cargo install tauri-cli --version "^2"
• Platform webview dependencies per the Tauri prerequisites. WebView2 is preinstalled on Windows 10 and 11.

Clone and install

git clone https://github.com/SouravRoy-ETL/duckle
cd duckle
npm --prefix frontend install

Run in development (hot-reloading frontend plus the native shell):

cargo tauri dev

Build a release binary:

# The --features custom-protocol flag is required: without it, tauri-codegen
# embeds the dev URL instead of the bundled frontend.
cargo build --release --manifest-path apps/desktop/Cargo.toml --features custom-protocol

Outputs land in target/release/duckle (or duckle.exe). The engine is not statically linked: DuckDB downloads at first launch, which is why the build is fast and the binary is tiny.

Run the tests:

cargo test                                                          # workspace unit + plan tests
DUCKLE_DUCKDB_BIN=/path/to/duckdb cargo test -p duckle-duckdb-engine # full integration suite

Architecture

duckle/
  apps/desktop/         Tauri 2 shell: Tauri commands, engine installer, llama runtime, window
  frontend/             React 19 + Vite + TypeScript: the designer UI + chat panel
  crates/
    duckdb-engine/      Compiles the node graph to SQL and drives the DuckDB CLI
    slothdb-engine/     SlothDB adapter
    scheduler/          Cron / interval / file-watch triggers
    metadata/           Schema and type model
    plugin-sdk/         Connector / inspector traits
    connectors/         Source and sink connectors
    runtime, workflow-engine, transform-engine, stream-engine, execution-core

• The frontend (React with @xyflow/react) is the visual designer; it talks to the Rust core over Tauri commands.
• duckdb-engine topologically sorts the graph, lowers each node into SQL, and executes by shelling out to the downloaded DuckDB CLI. Non-sink nodes materialize as tables so later stages can reference them; sinks become COPY ... TO statements; cancel kills the process. No statically linked database, so the binary stays small.
• Duckie is a llama-server subprocess on 127.0.0.1 exposing an OpenAI-compatible chat-completions API. The chat panel streams from it via SSE. The model is sandboxed: no fs, no net, no tools - it can only emit text.
• Everything persists to the workspace folder you choose, as plain JSON and Markdown files.

Configuration

A few knobs you can set without touching code.

Performance tips

A few patterns that consistently produce sub-second runs at small / medium data scale, and tractable runs at warehouse scale.

FAQ

Yes, free + open source. Dual-licensed MIT OR Apache-2.0. You can use it commercially, fork it, sell what you build with it. No usage limits, no telemetry.

No. The app runs entirely on your machine. The engines (DuckDB, llama.cpp) are downloaded from official upstream releases on first launch and then run locally. The only network calls Duckle makes on your behalf are the ones your pipelines explicitly do (e.g. a src.s3 reading from your S3 bucket, or xf.ai.embed if you configure it to hit OpenAI).

Duckie AI Assistant runs fully offline once the model is downloaded.

DuckDB is excellent on data that fits on one machine - tens of GB on a laptop, hundreds on a workstation. Beyond that, point Duckle's output at a warehouse / lakehouse that scales horizontally. Duckle is honest about being single-machine.

No - Duckle downloads it for you on first launch. The download is ~30 MB and includes the most-used extensions (httpfs, postgres, mysql, iceberg, delta, vss, fts, etc.) so the first time you touch a Postgres source there's no mid-pipeline network pause.

About 55-78 MB depending on platform (macOS ~54-67, Windows ~59-68, Linux ~66-78); it embeds the headless runner and the MCP server. The engines aren't statically linked - DuckDB (~50 MB with extensions) and the Duckie LLM (~1.1 GB for the Qwen GGUF) both download on first launch with a guided installer into your app-data folder, so they update independently of the app.

Yes. The AI transforms (xf.ai.embed, xf.ai.llm, xf.ai.classify) accept a baseUrl prop. Point it at any OpenAI-compatible /v1/... endpoint and an apiKey and Duckle uses that instead. The local Duckie chat panel is hardwired to localhost; the pipeline AI transforms are configurable.

In the workspace folder you pick on first launch (see Workspace and Git flow). Pipelines are plain JSON files you can commit to Git, diff, branch, and review.

Via Git, yes - check the workspace into a repo and use standard branch/PR flows. Duckle does not have a real-time multiplayer mode (single-machine by design).

Yes. Build Pipeline (right-click a pipeline) produces a single self-contained executable that runs anywhere with nothing installed - drop it on a server or CI runner and execute it, or schedule it with cron / systemd / Task Scheduler. The embedded duckle-runner can also run a workspace pipeline JSON directly (duckle-runner --pipeline pipeline.json). See Server deployment. You can also import the engine crate (duckle-duckdb-engine) into your own Rust binary.

For 90% of common pipelines (read source -> simple transforms -> sink), yes - the Qwen 2.5 Coder model is tuned for structured-JSON generation. For long, complex pipelines you'll likely want to iterate: describe the first half, click insert, then ask for the next half. You can also swap the model: point xf.ai.llm's baseUrl at GPT-4 or Claude for more capable pipeline drafting.

No. Once llama-server and the Qwen GGUF are downloaded into your app-data directory, Duckie runs fully offline. Tested by killing wifi and asking it for a pipeline - works fine.

DuckDB's SQL surface is wide enough to express most ETL work, it's vectorized and fast on a laptop, it has first-class Iceberg/Delta/Parquet readers, and its extension model lets us add vector + full-text + Postgres ATTACH without code changes. Polars is great but doesn't ship the cloud/format/extension breadth we need; Spark is a great cluster but overkill for the local-first niche we're in.

See the Contributing section and crates/duckdb-engine/src/plan.rs (planner branch) + crates/duckdb-engine/src/lib.rs (executor). The shortest path: copy an existing connector with similar shape (e.g. src.rabbit for a streaming source, src.dynamodb for an HTTP+auth API), adapt, add a test, flip the palette tile.

Troubleshooting

If you see something not listed, please open an issue with steps to reproduce + the relevant log line.

CI / CD

Duckle's CI pipeline runs on both GitHub and GitLab - the project mirrors to both. Push / pull-request / merge-request / tag events all trigger builds.

What each pipeline does:

1. Frontend - npm ci + npm run build (type-check + bundle)
2. Rust test matrix - cargo test --workspace on Linux + macOS + Windows
3. Live-service integration tests - PostgreSQL + MySQL + MinIO services spun up via Docker, real connector code runs against them
4. Desktop release-build smoke check - cargo build --release --features custom-protocol then grep the binary for the embedded frontend JS chunk (catches the v0.0.7-class "binary loads devUrl" bug at PR time)
5. Format + clippy - informational (does not block merge)
6. On tag: build the Duckle binary on all three OSes, upload as release assets

See .github/workflows/ and .gitlab-ci.yml for the exact steps. The two pipelines are kept feature-equivalent so contributors can fork to either platform.

Releasing a new version

Nothing regenerates this README, the hero / flow SVGs, or the downloadlinks automatically - they are hand-maintained, so they drift unless eachrelease updates them. Treat the README as a release artifact: walk thischecklist every time before tagging.

# 0. Update the README in the SAME commit as the version bump:
#    - bump every vX.Y.Z reference (the Download / Install link, badges)
#    - refresh capability tables for any new sources/transforms/sinks
#    - add/replace screenshots in docs/assets for shipped features
#    - re-check the hero/flow SVG wording if positioning changed
# 1. Bump version in apps/desktop/tauri.conf.json
# 2. Commit (README + version together)
git commit -am "Release: bump to vX.Y.Z"
# 3. Tag + push
git tag vX.Y.Z
git push origin main vX.Y.Z
# Both GitHub Actions and GitLab CI pick up the tag and build the
# release artifacts automatically. Once green, the draft release on
# GitHub gets the binaries uploaded; un-draft + mark Latest with:
gh release edit vX.Y.Z --draft=false --latest

Roadmap

A complete planned-component breakdown lives in docs/roadmap.md. Highlights:

• Multi-shard Kinesis and Pulsar streaming (Pulsar blocked on protoc at build time)
• Apache ORC read / write (blocked on the Arrow version conflict between orc-rust and our workspace pin)
• SFTP source (shipped - russh + russh-sftp on the ring backend, password / key auth, host-fingerprint pin)
• OAuth-heavy SaaS (Google Sheets, Excel Online, full Salesforce OAuth, Gmail / O365 IMAP)
• Embedded Python / Rust code stages (current code.* family: SQL, Shell, JavaScript, WebAssembly all ship)
• Hosted documentation site
• Plugin marketplace via the connector SDK
• In-process Native engine - a Rust streaming / incremental executor as an alternative to shelling out to the DuckDB CLI

Contributing

Contributions, issues, and ideas are welcome. Duckle is young and there is a lot of green field. Open an issue to discuss a change before a large PR, match the existing code style, and keep changes focused. Run cargo test and npm --prefix frontend run build before submitting. See CONTRIBUTING.md.

License

Licensed under either of MIT or Apache-2.0 at your option.