HHostingByteSoftware
Ctrl K
ComunidadGuías…

CHANGELOG.md

Rama main · réplica de lectura del repositorio original.

Archivo original ↗Preguntar a KiBot
# Changelog

All notable changes to Ciaren will be documented in this file.

Ciaren follows the spirit of [Keep a Changelog](https://keepachangelog.com/)
and uses semantic versioning. Until the first stable release (`1.0.0`),
breaking changes may still happen between `0.x` releases.

## [Unreleased]

## [0.3.1] - 2026-10-12

This release improves Python export parity and the documentation for both
engines. Existing flows keep their configuration; corrected exports can change
results that previously differed from a Ciaren run.

### Added

- **Semi and anti joins** select matching or nonmatching left-hand rows without
  adding right-hand columns (#198).
- **Cumulative product** is available in Window Function (#208).
- **Runnable pandas and Polars references** cover input/output formats, SQL,
  row and column transformations, combining data, windows and ML workflows.
  The ML guides explain training, model persistence, prediction and evaluation.
- **Text-cleaning recipes and troubleshooting** show how to split irregular
  fields, choose fallback values and diagnose missing generated columns.

### Fixed

- **pandas Split Column exports** preserve missing split parts and requested
  output columns when rows have fewer delimiters (#231).
- **Polars exports** match runtime duplicate counts for Assert Unique (#235),
  preserve aggregation group order (#236), and handle optional/fractional
  Assert Range bounds without truncating them (#238).
- **ML exports** preserve model definitions and preprocessing (#239), explain
  the model-definition requirement in Cross Validation errors (#240), and
  honor prediction batch sizes, including nullable features (#241).
- **Text input exports** preserve UTF-8, blank lines and boundary whitespace.
- **Icon-only editor controls** have accessible labels (#230).
- **Documentation checks on Windows** invoke npm tooling correctly (#232).
- **Python dependency security floors** apply to pip installs as well as the
  development lockfile, excluding affected multidict, urllib3, Werkzeug and
  Pillow releases from the existing MLflow dependency stack.

### Changed

- **Release validation** runs the complete backend inventory in disposable
  module processes, combines coverage and blocks PyPI publication until the
  full OS/Python matrix and both package builds pass.
- **Frontend tooling** uses Vite 8 with verified vendor chunks; installed
  packages continue to bundle the editor. Tailwind 4 removes an unpatched
  dependency chain while retaining the purple theme and keyboard focus styles.
- **Documentation dependencies** use patched Vite and source-map packages.
  The docs audit now blocks high-severity advisories, matching the editor.
- **Documentation navigation and metadata** use specific descriptions,
  current capability notes and examples that distinguish runtime safeguards
  from ordinary exported Python. The roadmap describes alpha limitations.
- **Codespaces opens the editor directly.** The dev container builds from a
  slim image with the expired Yarn apt source removed (the old setup fell back
  to a recovery container without Ciaren), installs the released package from
  PyPI onto the `/workspaces` volume, starts Ciaren in the terminal when the
  editor attaches, and opens it in a new browser tab. The README puts the "Try
  it in your browser" button right under the tagline.

## [0.3.0] - 2026-09-25

Notebook export, run drift, CSV dialect detection for storage inputs, and a
way to try Ciaren in the browser with GitHub Codespaces. There are no breaking
changes. The polars `pivot` count fix changes results where the values column
has nulls, so the engines now agree.

### Added

- **Try Ciaren in GitHub Codespaces.** The repository ships a dev container,
  and the README's "Open in GitHub Codespaces" badge opens the editor with the
  Demo project in your browser, with nothing to install. The server trusts
  exactly the codespace's forwarded URL through `CIAREN_CORS_ORIGINS`, so the
  cross-site request guard stays on. The installation guide covers it.
- **`ciaren serve` asks for a GitHub star.** One static line after the app URL.
  It makes no network call and records nothing.
- **Jupyter notebook export.** The code export dialog can download each engine
  variant (pandas, polars, lazy polars) as a `.ipynb` notebook, and
  `POST /api/flows/{flow_id}/export/python?include_notebooks=true` returns them
  in the new `notebook`, `notebook_polars`, and `notebook_polars_lazy` fields
  (`null` unless requested). Cells split only between top-level statements, so
  every cell runs on its own. The notebook exporters are listed in
  `GET /api/catalog/exporters`, and the Python client's `export_flow_python`
  takes `include_notebooks`. Thanks to @tusharui.
- **Run drift.** The run detail page shows a "Since last run" panel: per-node
  row-count change and added or removed columns compared with the previous run
  of the same flow, plus nodes added or removed when the graph changed (#142).
  Thanks to @tusharui.
- **CSV dialect detection for storage inputs.** CSV files read from a local
  folder, S3, GCS, or Azure Blob connection now get the same delimiter,
  encoding, and decimal detection as dataset uploads. The node panel shows the
  detected values, and a delimiter, encoding, or decimal set in the node config
  still wins (#200).
- **Recently used nodes.** The node palette shows the last five node types you
  placed when the search box is empty (#191).
- **`lstrip` and `rstrip`** operations on the String transform node (#188).
  Thanks to @rashmeetchhabra12.
- **Validator example plugin** in `examples/plugins/validator-plugin/`, a
  data-quality node between the Hello and MLP Classifier examples (#120).
  Thanks to @tusharui.

### Fixed

- **`pivot` with `aggfunc="count"`** now counts non-null values on polars, so
  the pandas and polars engines and their exported code agree when the values
  column has nulls (#143). Thanks to @tusharui.
- Transformation validation messages follow one documented format (#119).
  Thanks to @tusharui.
- An S3 error without a response object returned HTTP 500 instead of the
  scrubbed connector error (#200).

### Changed

- Contributions no longer need a DCO `Signed-off-by` line. They are licensed
  under the repository licenses through the GitHub Terms of Service (#202).
- The PyPI project links and the app's footer and landing links point at
  `/docs/latest` instead of the redirecting `/docs` URL. The docs home drops
  its hero glow and card hover effects to match ciaren.com (#203).
- The test suite builds, signs, installs, and runs every example plugin from
  source (#190).
- Pull-request checks also run against release-preparation branches, and
  routine Dependabot updates for dependencies and GitHub Actions are folded in
  (#137).

### Security

- Patched vulnerable dependencies: aiohttp, anyio, cryptography, gitpython,
  mlflow, pyasn1, and sqlparse in the backend lock file, and `npm audit fix`
  for the frontend and docs (#189).

### Documentation

- Every docs page has a specific search description; README and client links
  point at the live `/docs/latest/` URLs; the docs build now fails on dead
  internal links (#193).
- One start path from installation to the first flow, one sidebar home per
  page, and clearer roles for the plugin tutorial, guide, and reference (#199).
- Source installs need Node.js 20 or newer (vitest 4).
- The README, PyPI description, keywords, and project URLs are clearer, and
  the docs have a social card for link previews (#177).
- The comparison page compares Ciaren with KNIME, Alteryx, and Flowfile by
  name (#204).
- The docs state what plugins can run today: nodes, connectors, and ML model
  types work end to end, while engine, exporter, and validator providers only
  register metadata so far. The ML classification example now imports pandas
  before using it.
- The docs site has a Changelog page with these release notes, checked in and
  kept identical to `CHANGELOG.md` by a test.

## [0.2.0] - 2026-07-20

A repo-wide correctness and hardening pass from an internal audit. Most of it is
straightforward bug fixing, but a few changes alter results or reject input that
previously passed — read **Breaking changes** before upgrading.

### Breaking changes

- **`groupByAggregate` rejects aggregations with no exact polars equivalent**
  (`sem`, `skew`, `kurt`, `size`, `mad`). These previously ran on the pandas
  backend and failed or silently differed elsewhere. *Migration:* switch to a
  supported aggregation, or compute the statistic downstream in a
  `pythonTransform` node.
- **polars `first` / `last` / `nunique` now skip nulls, matching pandas.**
  Groups containing nulls produce different (now correct and engine-consistent)
  values. *Migration:* re-run affected flows; polars results now agree with the
  pandas backend and with the exported code.
- **`concatRows` on polars unions mismatched columns** and null-fills, instead
  of raising. Flows that previously errored now succeed. *Migration:* none.
- **`GET /api/datasets` and `GET /api/flows` return at most 500 rows** by
  default. *Migration:* none for the UI; API consumers that relied on an
  unbounded list should expect truncation until `limit`/`offset` query
  parameters land.
- **A Train node whose model fails to save now fails the run** instead of
  reporting success with an unusable model reference. *Migration:* none — the
  run was already broken downstream; the failure is now visible.
- **Run creation rejects an unknown or cross-project `input_dataset_id`**
  (404 / 400). *Migration:* pass a dataset that exists in the flow's project,
  or omit the field.
- **"Default" is a reserved project name** — creating or renaming a project to
  it returns 400. *Migration:* pick another name.
- **Plugins are re-verified at load.** An installed plugin whose files were
  modified on disk after installation is refused (*migration:* reinstall it to
  re-pin the baseline); an install whose id case-collides with an existing
  plugin is rejected; a plugin whose metadata id differs from its manifest id
  is refused; and a node's declared `provider` is forced to the owning plugin's
  id.
- **Hardened connector mode** (`CIAREN_CONNECTOR_BLOCK_PRIVATE_HOSTS=true`) now
  rejects multi-host / URI-style hostnames (e.g. a comma-separated libpq seed
  list) and refuses `verify_tls: false`. *Migration:* use a single plain
  hostname per connection and keep TLS verification on, or leave the guard off.
- **`pythonTransform` strict mode** (`CIAREN_PYTHON_TRANSFORM_STRICT=true`)
  blocks additional capability modules (`io`, `tempfile`, `socket`, `pickle`,
  …). Imports that are allowed now genuinely work at runtime, where they
  previously failed with an opaque error.

### Security

- Connector SSRF guard now fails closed on multi-host / URI-style hostnames
  (which drivers could expand into a seed list or full URI) and blocks the
  NAT64 range that maps to the cloud metadata endpoint.
- The REST connector re-validates every HTTP redirect hop against the SSRF
  guard, rejects non-`http(s)` redirect targets, refuses to disable TLS
  verification in hardened mode, and no longer forwards `Authorization` /
  API-key headers when a redirect crosses to a different host.
- Marketplace artifact resolution is confined for untrusted (URL-shaped) index
  sources — `file://`, absolute, and `..` paths are rejected — ahead of any
  hosted plugin index.
- Plugins: code integrity is now verified at load (previously only the manifest
  was hashed), a planted or altered bytecode cache can no longer execute in
  place of trusted code, a node's `provider` is bound to its owning plugin, and
  an install whose id case-collides with an existing plugin is rejected.
- SQLite foreign-key enforcement is enabled; the `/api/settings/webhook` auth
  exemption is scoped to `GET`; `pythonTransform` strict mode blocks additional
  capability modules and frame/traceback traversal.
- CI: all GitHub Actions are pinned to commit SHAs (including the OIDC PyPI
  publish jobs), and `API_TOKEN` now documents the need for a long,
  high-entropy value.

### Fixed

- Projects: creating or renaming a project to the reserved name "Default", or
  deleting a project that holds a dataset whose name also exists in the default
  project, no longer returns a 500.
- A `PUT /api/flows` with an explicit `null` project id now moves the flow to
  the default project instead of erroring; an impossible-but-valid cron
  expression (e.g. Feb 30) now returns a validation error instead of a 500.
- Scheduler: daily schedules no longer fire twice during the DST fall-back
  hour; a concurrent user edit is no longer overwritten by a just-finished run;
  a manual "Run now" can no longer start a second concurrent run of a flow the
  scheduler is already running.
- Engine: `groupByAggregate` and `concatRows` now produce identical results on
  the pandas backend, the polars backend, and the exported code (previously a
  config valid on one could fail or differ on another).
- ML: a Train node whose model fails to save now fails clearly instead of
  reporting success with an unusable model; prediction aligns feature columns
  and warns before overwriting an existing column; training hyperparameters are
  bounded to prevent a runaway fit.
- Frontend: file downloads work when an `API_TOKEN` is configured; the flow
  editor warns before discarding unsaved changes; storage-restricted browser
  contexts no longer crash the editor; the "not authorized" message points to
  the real `?api_token=` mechanism.
- Database: networked Postgres/MySQL engines use connection health checks
  (`pool_pre_ping`) so a stale pooled connection no longer surfaces as a 500,
  and startup no longer runs best-effort column patching on Alembic-managed
  databases.
- Listing datasets no longer loads every version of every dataset into memory —
  the latest version and version count come from an aggregate query, so a
  workspace with a long run history stays responsive.

## [0.1.0] - 2026-07-14

First public release of Ciaren — a local-first visual builder for data and ML
workflows that exports clean, readable pandas/polars Python. This is alpha
software: APIs, data models, and generated code may still change before a
stable `1.0`.

### Visual builder

- 80 nodes across 9 categories — inputs, cleaning, columns, reshape,
  analytics, data quality, charts, machine learning, and outputs — including
  8 chart nodes (bar, line, area, scatter, pie, histogram, box plot,
  correlation heatmap) that render instantly from a compact run artifact and
  support one-click PNG export.
- File, SQL, and cloud-storage I/O (S3, GCS, Azure Blob, Snowflake) plus a
  built-in, read-only REST API connector (auth, custom headers, pagination,
  SSRF-guarded).
- Per-node data preview, undo/redo, duplicate flows, and copy/paste nodes
  with their connecting edges.
- Import options with auto-detection — CSV/TSV dialect, encoding, and
  multi-sheet Excel workbooks are detected on upload, with an explicit
  override always available.

### Execution and code export

- Dual-engine execution — **polars** (default) or **pandas**, selectable per
  run.
- Exported Python reads like hand-written code (keyword `assign()`, chainable
  `.loc[lambda _d: …]` filters, fused method chains, reused dataframe
  variables on straight-line flows) and is verified equivalent to the in-app
  run by an automated pandas/polars, eager/lazy equivalence harness.
- Cancel a running run from the UI or API; optional failure webhook
  notifications when a run fails or a schedule auto-disables.

### Data quality and Machine Learning

- Data contract nodes: assert not-null, unique, value range, row count, and
  arbitrary expressions.
- Built-in ML: split, feature engineering, train, predict, and evaluate
  nodes, with MLflow tracking and a Models page. `pip install ciaren` alone
  gives a working ML palette (scikit-learn, MLflow, and joblib are core
  dependencies); XGBoost and LightGBM are available via the `ml` extra.

### Scheduling and automation

- Built-in cron scheduler with retries, catch-up, and auto-disable after
  repeated failures.
- REST, CLI, and webhook-based ways to trigger flows outside the UI.

### Plugin platform

- A versioned, contract-first Plugin API/SDK (`backend/app/plugin_api/`,
  Apache-2.0) — plugins can contribute transformation nodes, ML model types,
  connectors, and schema-driven config forms without forking the core.
- Signed `.ciarenplugin` packaging, local and entry-point discovery, and
  permission-gated model loading.
- `ciaren-client` — a Python SDK for scripting against the server (flows,
  runs, datasets, plugins).
- Two documented example plugins (hello-world node, MLP classifier) covering
  both extension paths.

### Documentation and project

- Full documentation site: getting started, guides, transformation
  reference, recipes, plugin authoring, and API reference.
- Contributing guide, Developer Certificate of Origin (DCO) policy, issue and
  PR templates, and public GitHub Discussions.
- Dual licensing: Ciaren Core is AGPL-3.0-only; the public Plugin API/SDK is
  Apache-2.0 so plugin authors can choose their own license.
- Docker and Docker Compose support for single-container local/self-hosted
  use.

### Security

- pip-audit and npm audit run on every push to `main` and fail CI on new
  high/critical advisories in pinned dependencies.

### Known limitations

- Ciaren is alpha software; recommended for controlled internal workflows
  while the project matures, not yet for critical production jobs.
- Not a distributed or streaming data engine — batch-style local and
  self-hosted workflows are the product center.
- Has not yet completed a formal independent third-party security audit.