PendingReleaseNotes
Rama main · réplica de lectura del repositorio original.
>=22.0.0
* RBD: The ``schedule_time`` field in the output of
``rbd mirror snapshot schedule status`` and
``rbd trash purge schedule status`` has been renamed to
``next_scheduled_time``. Scripts or tools that parse this output
must be updated accordingly.
* libcephsqlite: concurrent readers is now supported. The VFS implements the
full sqlite3 locking protocol.
* RBD: The ``rbd mirror snapshot schedule add`` and
``rbd trash purge schedule add`` commands now accept a ``w`` suffix to
specify the schedule interval in weeks (e.g. ``2w``), in addition to the
existing ``d``, ``h``, and ``m`` suffixes.
* Cephadm: ``ceph orch upgrade start`` and ``ceph orch upgrade check`` now
verify that every host's CPU supports the x86-64 microarchitecture level
required by the target release (official builds of 21.x and later require
``x86-64-v3``). An upgrade that would deploy binaries a host's CPU cannot
execute is refused, or paused with a new ``UPGRADE_INCOMPATIBLE_HOST_CPU``
health alert. cephadm now reports the host's detected level as a new
``cpu_isa_level`` host fact (e.g. ``x86-64-v2``). The check can be disabled
with ``ceph config set mgr mgr/cephadm/upgrade_cpu_isa_check false``, e.g.
when running custom-built images targeting an older CPU generation.
* RBD: The Python ``Image.list_lockers()`` method now returns a dictionary
containing ``tag``, ``exclusive``, and ``lockers`` fields even when no clients
hold the lock, instead of an empty list. Callers should check the
``lockers`` field rather than the truth value of the returned object
to determine whether any clients hold the lock.
* RGW: The "Bytes Sent" field is always set to "-" when bucket-logging is enabled
in Standard mode.
* RGW: Changing ``rgw_dynamic_resharding`` used to turn queuing buckets
for reshard on or off while not affecting resharding itself until the
next realm reload. Now changes to ``rgw_dynamic_resharding`` require a
restart. Running RGWs will ignore the change.
* RGW: The Elasticsearch sync module is deprecated in Vampire and will
be removed in the X release. People wanting to continue using
Elasticsearch can populate it with bucket notifications. The
upcoming S3 Inventory feature also covers some of the use case.
* RGW: The default value of ``rgw_rest_conn_connect_to_resolved_ips`` changed
from ``false`` to ``true``.
* RGW: The default value of ``rgw_rest_conn_ip_fail_timeout_secs`` changed
from ``2`` to ``30`` seconds.
* RGW: RGW can now return free TCMalloc memory to the OS periodically. It is
off by default. Set ``rgw_heap_release_interval`` to a number of seconds to
enable it. ``rgw_heap_release_bytes`` (64 MiB by default) sets how much it
tries to return each time; a pass is skipped while less than that is free.
* RGW: ``radosgw-admin bilog trim`` without ``--end-marker`` now requires
``--yes-i-really-mean-it``, because such a trim has no upper bound and can
remove bilog entries that other zones have not synced yet.
* RGW: ``radosgw-admin bilog trim`` no longer accepts ``--start-marker``, since
FIFO-backed bilogs can only be trimmed from the front.
* RGW: S3 Select is deprecated. The default value of ``rgw_disable_s3select``
changed from ``false`` to ``true``.
* RGW: Fixed a bug in same-account evaluation of bucket policy statements that
allow the account principal. Users and roles in that account now require a
corresponding allow from identity policy to match AWS behavior. The evaluation
of bucket policy against user or role principals is unchanged and still does
not require identity policy.
* RGW: The STS ``AssumeRole`` and ``AssumeRoleWithWebIdentity`` actions now consult
identity policy according to the AWS same-account and cross-account policy
evaluation rules. They had previously only consulted the role's trust policy.
This means that account users will now require an Allow from identity policy
to assume roles in other accounts.
* RGW: Lifecycle transition to a cloud-s3 storage class now skips current
object versions that have an unexpired object lock retention period or a
legal hold, as it already did for non-current versions. Previously such a
version could be replaced by a delete marker or, with ``retain_head_object``
and ``retain_current_version`` enabled, by a cloud-tiered stub, leaving its
data only on the cloud endpoint.
* ``src/script/ceph-backport.sh`` now runs on macOS in addition to Linux: it
re-execs under GNU bash >= 4 when the system bash is too old, and honors a
``$GETOPT`` environment override so GNU getopt can be selected
(``brew install bash gnu-getopt``).
* CephFS: Added a multi-tiered scoring matrix to the MDS Monitor to dynamically
prioritize standby promotion during failover events. The ``mds_join_fs`` configuration
variable is now checked in combination with infrastructure topology rules to enforce
node anti-affinity constraints. An isolated, generic standby daemon (Score 5) will
now be chosen over a preferred standby daemon targeting that file system if the
preferred standby resides on the same physical host as the failed active rank (Score 3).
This safely mitigates localized hardware failure boundaries during rapid automatic
recovery cycles. This behavior can be disabled per file system using the command
``ceph fs set <fs_name> standby_enable_host_anti_affinity false`` (default:
``true``) to restore legacy strict ``mds_join_fs`` prioritization.
* CephFS: New MDS configuration options for non-blocking dirfrag prefetch have
been added. ``mds_dir_prefetch_backend``, when enabled with
``mds_dir_prefetch=false``, serves dentry requests via a fast targeted keyed
OSD fetch while lazily launching a full dirfrag prefetch in the background.
The background prefetch is triggered only after
``mds_dir_prefetch_backend_hit_threshold`` (default: ``16``) accesses to the same
dirfrag, avoiding wasted full-omap reads for directories that are only hit
a few times in random workloads. ``mds_dir_prefetch_backend_max`` (default: ``1``)
limits the number of concurrent background fetches. This reduces latency in
large directories while retaining cache warmth for hot directories.
* BlueStore: The experimental PMEM device backend has been removed, together
with the ``pmem`` value of the ``bdev_type`` option and the DML/DSA offload
path. The hardware it targeted, Intel Optane DC persistent memory, was
discontinued in 2022.
Existing OSDs backed by persistent memory in fsdax mode (a block device such
as ``/dev/pmem0``) keep working on the default ``aio`` backend over the same
data. Remove any explicit ``bdev_type = pmem`` from their configuration. OSDs
on a devdax namespace (a character device such as ``/dev/dax0.0``) will no
longer start, because no remaining backend can open one; reconfigure the
namespace to fsdax with ``ndctl`` or re-provision them before upgrading.
* RBD: The PMEM backend of the persistent write log cache has been removed,
together with the ``rwl`` value of the ``rbd_persistent_cache_mode`` option
and the ``WITH_RBD_RWL`` build option. It targeted the same discontinued
Intel Optane DC persistent memory as the BlueStore backend above, and with
both gone Ceph no longer builds against PMDK at all.
The ``ssd`` cache mode is unaffected, but ``rwl`` was enabled in the shipped
x86_64 packages, so drain any cache it left **before** upgrading, and in this
order. While still running the old version, close or unmap each image that
has one: a clean shutdown flushes the cache and removes it, after which
``rbd status`` no longer reports a dirty cache. If the client crashed and the
workload will not be restarted, use ``rbd persistent-cache flush`` instead.
Only then change ``rbd_persistent_cache_mode`` and upgrade.
Changing the mode first strands the cache instead. Either way the image then
fails to acquire its exclusive lock, so all of its I/O fails: with the cache
disabled, one that still records a dirty cache is refused outright, and with
the cache enabled, one whose metadata records a dirty ``rwl`` cache is
refused because the backend that could replay it is gone. Recovering means
setting the mode to ``ssd``, giving the client ``rbd_plugins = pwl_cache``,
and running ``rbd persistent-cache invalidate``, which discards the cache
along with whatever it had not flushed.
A stale ``rwl`` in ceph.conf or the Monitor config store is merely ignored,
and such an image simply runs without a cache. Note that ``ceph config set
... rwl`` now fails, which may break automation.
>=21.0.0
* CephFS: The ``fs subvolumegroup create`` command now supports the
``--namespace-isolated`` option. When a subvolume group is created with this flag,
the group directory is placed in a separate RADOS namespace, and all subvolumes
subsequently created within that group automatically inherit RADOS namespace
isolation without needing the ``--namespace-isolated`` flag on each
``fs subvolume create`` invocation. An optional ``--pool-namespace`` flag allows
specifying an explicit RADOS namespace to override the auto-generated name.
This provides per-tenant RADOS-level security isolation at the subvolume group
granularity. The option is opt-in: upgraded clusters are unaffected and no
existing subvolume group is migrated automatically. An existing group can be
converted by re-running ``fs subvolumegroup create`` with
``--namespace-isolated``, but only subvolumes created afterwards inherit the
isolation. See the "Subvolume groups" section of ``doc/cephfs/fs-volumes.rst``
for the restrictions that apply when converting an existing group.
* CephFS: The ``client_force_lazyio`` configuration option is now correctly marked
as not supporting runtime updates. Previously, the configuration schema indicated
this option could be changed at runtime, but changes had no effect on opened file
handlers in ceph-fuse or libcephfs clients because there is no logic to propagate
changes to open file handles.
* RADOS: PG autoscaler allows overlapping roots. Each root receives a PG target based
on its OSDs, with OSDs shared across multiple roots contributing proportionally
less to each root's allocation.
* CephFS: CephFS snapshots metadata is now mutable. It is now possible to add,
update and remove existing key-value pairs that are part of a snapshot
metadata via ``libcephfs`` API ``ceph_do_snap_md_op()``.
* CephFS: MDS dmclock support for subvolume QoS. With this feature, MDS assigns QoS
to throttle client metadata requests (e.g., create, mkdir, lookup, and so on) to
subvolumes, where each subvolume QoS is shared among multiple client sessions at
that time.
* Monitor: Each Monitor now reports four new perf counters covering its data
directory: ``data_disk_total_bytes``, ``data_disk_avail_bytes``,
``data_disk_avail_percent``, and ``db_total_bytes`` (estimated on-disk size
of the key/value store). They surface automatically as ``ceph_mon_*``
Prometheus metrics via the ``prometheus`` mgr module and ``ceph-exporter``,
enabling trending of Monitor DB growth and filesystem free space.
* librados/neorados: The C++ APIs for executing Ceph Class (CLS) methods
have undergone a breaking change to enforce compile-time type safety, replacing
legacy string-based parameters with strongly-typed ClsMethod structs. C++
developers must update their method calls to use these new structs, which ensure
read/write semantics are correctly applied.
* RGW: S3 ListObjects and ListObjectVersions now support the
``x-amz-optional-object-attributes: RestoreStatus`` request header to include
restore status in listing responses. Restore status is stored in the bucket
index, so only objects written or restored after this upgrade will populate
the field. Existing objects are unaffected.
* RGW: New S3 Control APIs to apply PublicAccessBlock configuration to User Accounts.
* RGW: PutBucketEncryption and GetBucketEncryption now support the
``BlockedEncryptionTypes`` element. Blocking ``SSE-C`` on a bucket rejects new
SSE-C uploads with ``403 AccessDenied`` while leaving existing SSE-C objects
readable. Newly created buckets are unchanged by default. Enable the new
``rgw_s3_block_sse_c_by_default`` option to block SSE-C on them instead,
matching the AWS behavior for new buckets.
* RGW: New config option ``rgw_keystone_accepted_project_reader_roles``.
A Keystone user whose only access-granting role is one of the configured project-reader
roles is capped to read-only access within its project, mirroring the
``project_reader_roles`` behavior of Swift's keystoneauth middleware. A bucket
policy naming the user can still grant additional access. The reader role
admits the user on its own; it need not also be in
``rgw_keystone_accepted_roles``.
* RGW: New config option ``rgw_keystone_implicit_deny_roles``.
A Keystone user whose only access-granting role is one of the configured
implicit-deny roles is authenticated but granted no implicit permissions.
Access must be granted explicitly through a bucket policy or IAM policy, and
applies only to the buckets and objects the policy names. The role admits the
user on its own.
* ceph-volume: Raw BlueStore OSD preparation now pre-formats NVMe devices and
skips the slower BlueStore discard phase, reducing mkfs time on
very large namespaces.
* RGW: Omap backing for the RGW Datalog is deprecated and support will be removed in a future version.
* The ``rgw default data log backing`` option is removed and it is no longer
possible to create clusters with an omap based datalog.
* ``radosgw-admin datalog type`` will only accept ``--log_type=fifo``.
* RGW: ``iam:RemoveClientIdFromOIDCProvider`` is now a recognized action for
policy, corrected from a typo of ``iam:RemoveCientIdFromOIDCProvider``
* Manager: The default values of ``mon_target_pg_per_osd`` and ``mon_max_pg_per_osd``
have been increased from 100 and 250 to 200 and 500, respectively. These values
enhance the PG autoscaler's ability to calculate reasonable ``pg_num`` values
for pools under its control. These in turn increase parallelism and performance,
and improve the ``balancer`` module's ability to achieve uniform OSD utilization.
`This concomitant PR <https://github.com/ceph/ceph/pull/67694>`__ ensures that
upgrades are not delayed or disrupted by PG splitting as a result of this change.
Clusters with effective ``bluestore_osd_memory_target`` values smaller than
the default 4GiB may wish to explicitly set the previous default values in
the central config store before upgrading. This will reduce the potential
for increased memory starvation. Clusters with SSDs split into more than two
OSDs may also wish to pin their current values in advance, or redeploy those
SSDs with at most two OSDs per.
* RGW: Bucket Logging supports creating log buckets in EC pools.
Implicit logging object commits are now performed asynchronously.
* RGW: ``radosgw-admin bucket list`` now supports pagination for versioned buckets by using
both ``--marker`` and ``--object-version`` options together (e.g., ``--marker=obj1 --object-version=abc123``).
This enables proper pagination through versioned bucket listings without duplicates or
infinite loops. For non-versioned buckets, use only ``--marker`` as before (backward compatible).
* RGW: ``radosgw-admin bucket suspend`` and ``bucket unsuspend`` can suspend an
individual bucket without suspending its owner. S3 requests against a suspended
bucket fail with ``403 BucketSuspended``. The ``suspended`` field is now included
in ``radosgw-admin bucket stats`` output.
* RGW: OpenSSL engine support is deprecated in favor of provider support.
* Removed the ``openssl_engine_opts`` configuration option. OpenSSL engine configuration in string format is no longer supported.
* Added the ``openssl_conf`` configuration option for loading specified providers as default providers.
Configuration file syntax follows the OpenSSL standard (see https://github.com/openssl/openssl/blob/master/doc/man5/config.pod).
If the default provider is required when also using custom providers,
it must be explicitly loaded in the configuration file or code (see https://github.com/openssl/openssl/blob/master/README-PROVIDERS.md).
* RGW: Fixed bucket notification events so the ``x_amz_request_id`` in NotificationEvent now matches the ``x_amz_request_id`` returned by the corresponding S3 operation.
* RGW: The ``rgw_gc_max_deferred`` and ``rgw_gc_max_deferred_entries_size`` options have been removed, as they did not do anything.
* Updated the default for ``rgw_gc_max_queue_size`` to account for the extra space from removing ``rgw_gc_max_deferred_entries_size``.
* RGW: The default value of ``rgw_thread_pool_size`` has been reduced from 512 to 128.
Benchmarks showed that the lower thread count improves throughput and reduces latency
on NVMe and HDD based environments. Users with specific workload requirements can
tune this value; see also ``rgw_max_concurrent_requests``.
* RGW: This release contains a fix for CVE-2026-54330. RGW now rejects
SigV4 requests with ``host`` or ``x-amz-*`` headers not included in the
signed subset. This applies to every SigV4 client, including multisite,
for both header-signed and presigned requests; affected clients receive
``403 AccessDenied``. See :ref:`CVE-2026-54330-upgrade-steps`.
If you are running multisite, you must set the
``rgw_sigv4_insecure`` option to true before you begin to
upgrade. After all clusters are upgraded, set the option to
``false`` again.
* RGW: New CephX profile ``profile rgw`` grants a Ceph Object Gateway the
permissions it needs. Create keys with
``mon 'profile rgw' osd 'profile rgw'``.
* Dashboard: A new Overview landing page provides an at-a-glance cluster summary
with health status, capacity, performance metrics, and quick-access links. All the
previous legacy landing pages are removed.
* Dashboard: RGW Service form updated to take input regarding QAT compression.
QAT compression is an optional field which can be set to 'Hardware' or
'Software' by selecting options from provided dropdown. If 'None' is selected,
compression is removed altogether.
* Dashboard: NVMe-oF management workflows have been revamped with DH-HMAC-CHAP
in-band authentication support for both unidirectional (host-to-controller) and
bidirectional (mutual) authentication, namespace host masking, and an integrated
subsystem creation wizard that combines NQN setup, host addition, and
authentication configuration in a single guided flow. Subsystem HA is now enabled
by default. Gateway group creation and deletion are now supported directly from
the dashboard, including per-gateway node management. NVMe-oF performance and
overview Grafana dashboards are embedded directly in the Block > NVMe/TCP tab.
* Dashboard: RGW storage classes now support GLACIER and LOCAL storage class
types with full create, list, edit, and delete operations.
* Dashboard: RGW S3 bucket notifications are now fully manageable from the dashboard.
Notification destinations (topics) support full CRUD operations (create, list, edit, delete)
for configuring webhook endpoints, Kafka brokers, and AMQP queues. Bucket notification
configurations can be created, listed, edited, and deleted to define which S3 events
trigger notifications and which destination to use.
* Dashboard: RGW user accounts can now be linked to individual RGW users,
root account user functionality has been added, and the bucket form
differentiates account users from standalone RGW users.
* Dashboard: RGW tiering now supports read-through configuration and bucket
tiering options in lifecycle policies.
* Dashboard: RGW multisite setup improvements: the RGW module is now
auto-enabled on primary and secondary clusters during multisite automation,
FQDN endpoints are supported (not only IPs), and users can re-use existing
realm/zonegroup/zone configurations when setting up replication.
* Dashboard: RGW archive zone configuration is now available from the dashboard.
* Dashboard: SMB shares now support QoS rate limiting, including per-share
bandwidth limits and cluster-wide rate limiting configuration.
* Dashboard: NFS export management has been enhanced with subvolume group and
subvolume selection in the edit export form, CephFS snapshot visibility
toggling per export, and IPv6 address support. NFS cluster and export listing
is now available.
* Dashboard: CephFS volume creation now supports selecting existing pools for
metadata and data instead of always creating new ones.
* Dashboard: Certificate management is now integrated into the dashboard.
Service detail views include a Certificate tab showing certificate status, and
the service create/edit modal supports certmgr-based certificate provisioning.
Certificate health alerts are surfaced in the dashboard and Prometheus.
* Dashboard: The onboarding (cluster creation) wizard has been redesigned with
an updated layout and improved visual flow.
* Dashboard: The dashboard UI has been modernized with the Carbon Design
System — including forms, modals, notifications, icons, performance charts,
and multi-cluster views.
* Dashboard: Logging infrastructure has been migrated from Promtail to Grafana
Alloy.
* Dashboard: Prometheus configuration now supports a remote write section for
forwarding metrics to external systems.
* Dashboard: ``ceph dashboard sso enable oauth2`` checks for the ``oauth2-proxy``
service for SSO enablement. If the ``oauth2-proxy`` service goes down, cephadm
will automatically disable Dashboard OAuth2 SSO.
* Monitoring: New Grafana dashboards have been added for Application Overview,
CephFS, CephFS Subvolume, NVMe-oF (performance and overview), SMB Overview,
and RGW Bucket Notification. The Grafana dashboards naming convention has been
standardized. Grafana has been upgraded to version 12.3.1.
* Monitoring: Added ``NVMeoFHostKeepAliveTimeout`` Prometheus alert that fires
when an NVMe-oF host keepalive times out.
* Monitoring: Added ``CephOSDFreeSpaceFragmentation`` Prometheus alert that fires
when the ``BLUESTORE_FREE_FRAGMENTATION`` health check is active (warning severity).
* Monitoring: Certificate management health checks now generate Prometheus
alerts and dashboard warnings for certificate expiration and errors.
* ``CephCertificateError``: Fires when a Ceph certificate has expired (critical severity).
* ``CephCertificateWarning``: Fires when a Ceph certificate is about to expire (warning severity).
* CephFS: The `peer_add` command is deprecated in favor of the `peer_bootstrap` command.
* RGW: For `radosgw-admin account rm`, the `--purge-data` option previously applied only
to buckets and objects, but now deletes account users, roles, groups, and oidc-providers
too instead of failing with ENOTEMPTY.
* RADOS: When objects are read during deep scrubs, the data is read in strides,
and the scrubbing process is delayed between each read in order to avoid monopolizing
the I/O capacity of the OSD.
The default stride size (``osd_deep_scrub_stride``) was 512 KiB, and is now 4 MiB.
* RADOS: When an OSD is overloaded with queued snap-trim operations, no
regular (non-urgent) scrubs will be scheduled on that OSD. This is
determined by comparing the total snap-trim queue lengths for all PGs
for which the OSD is a primary against ``osd_scrub_queued_snaptrims_limit``,
which defaults to 500.
This restriction does not apply to operator-initiated scrubs, nor to repair scrubs.
It can be disabled by setting ``osd_scrub_queued_snaptrims_limit`` to 0.
* RGW: Add SSE-KMS secrets cache
* RGW: New buckets now default to the FIFO bilog backend instead of the
older InIndex (omap-based) backend, controlled by the new
``rgw_default_bucket_bilog_type`` configuration option (default:
``fifo``). Existing InIndex buckets are unaffected until their next
reshard, at which point they are automatically migrated to FIFO.
>=21.0.0
* CephX: a new key type ``aes256k`` has been introduced to address
vulnerabilities in the existing encryption schemes. It is necessary to
perform an upgrade of all CephX keys used by daemons or clients. Please see
:ref:`cephx-upgrade` for more information. New clusters allow only
``aes256k``, which kernel clients support since Linux 7.0; see
:ref:`cephx-older-kernel-clients`.
* The ``auth_supported`` config has been removed.
* The Monitor map includes new settings relating to CephX:
``auth_service_cipher``, ``auth_allowed_ciphers``, and
``auth_preferred_cipher``. These are used to control which CephX key types
may be used in the cluster.
* A new ``--key-type`` argument has been introduced for all commands which
produce CephX keys. This includes Monitor commands as well as Ceph cluster
bootstrap commands like ``ceph-authtool`` and ``monmaptool``.
* The ``mon.`` credential is now authoritatively stored in the Monitor "auth"
database. The Monitor keyring file is only used as a fallback. The ``ceph-mon``
command now includes a ``--use-mon-keyring`` switch for recovery if the
``mon.`` key in the auth database is lost.
* RADOS: Stretch mode can now be entered even if the two dividing buckets differ
in weight by a small fraction (default 0.1). This is tunable via
``mon_stretch_max_bucket_weight_delta``.
* RADOS: Added a debug-level guard that protects against accidentally leaving
debug subsystems at high log levels in production. When ``high_debug_level_threshold``
is set (default: ``0`` = disabled), any subsystem whose log level sits at or above
the threshold for longer than ``high_debug_level_reset_timeout`` seconds (default:
``300``) is automatically set back to the level it had before it was raised, and a
``HIGH_DEBUG_LEVEL`` health warning surfaces in ``ceph health detail`` while the
level remains elevated. Setting ``high_debug_level_reset_timeout`` to ``0`` gives
warn-only mode (health warning without auto-revert).
Related Tracker: https://tracker.ceph.com/issues/75467
* CephFS: The offline CephFS tools (cephfs-data-scan, cephfs-journal-tool,
and cephfs-table-tool) now include progress tracking with ETA (Estimated Time of
Arrival) for long-running operations. Progress updates are displayed automatically
at regular intervals, showing completion percentage, processed items, and time
estimates. This feature is enabled by default for relevant commands including
scan_extents, scan_inodes, and other state-changing operations.
Related Tracker: https://tracker.ceph.com/issues/63191
* RBD: Fixed incorrect behavior of the "start-time" argument for mirror
snapshot and trash purge schedules, where it previously offset the schedule
anchor instead of defining it. The argument now requires an ISO 8601
date-time. The ``schedule ls`` output displays the start time in UTC, including
the date and time in the format "%Y-%m-%d %H:%M:00". The ``schedule status``
output now displays the next schedule time in UTC.
* CephFS Mirroring: Now utilizes a multi-threaded architecture to improve synchronization
performance. The workload is split into two distinct thread pools: a crawler thread pool, which
manages snapshot crawl and a data synchronization thread pool, which handles concurrent file
transfers. Users can fine-tune these operations using configuration parameters:
``cephfs_mirror_max_concurrent_directory_syncs`` (controlling the number of
concurrent snapshots being crawled) and ``cephfs_mirror_max_datasync_threads``
(controlling the total threads available for data sync).
For more information, see https://tracker.ceph.com/issues/73452
* CephFS Mirroring: Improved incremental synchronization behavior in CephFS mirroring. Previously,
block-level delta synchronization was used for all files regardless of size. With this change,
blockdiff is applied only to files larger than a configurable threshold, while smaller files are
synchronized using full copy, as blockdiff is not efficient for small files. The threshold is
controlled by the new configuration option
``cephfs_mirror_blockdiff_min_file_size`` (default: ``16_M``).
For more information, see https://tracker.ceph.com/issues/73452
* CephFS Mirroring: Improved mirror daemon status reporting. The command
``ceph fs snapshot mirror daemon status`` now shows the remote cluster's
Monitor addresses and cluster ID for each configured peer, making it easier
to verify peer connectivity and troubleshoot mirroring issues.
* CephFS Mirroring: The ``fs mirror peer status`` admin socket command reports
additional per-directory sync metrics (sync mode, throughput, crawl and
data-sync queue timing, bytes/files progress, and ETA). Output is grouped
under ``metrics/<mirrored-dir-path>/peer/<peer-uuid>``. For more information,
see https://tracker.ceph.com/issues/73453
* CephFS Mirroring: Per-directory snapshot sync progress is exposed as labeled perf
counters in the ``cephfs_mirror_directory`` group (``counter dump`` on the mirror
daemon admin socket, exportable via ``ceph-exporter``). Counters mirror
``fs mirror peer status`` fields (directory state, current sync, last sync, and
snap summary) and are labeled by peer UUID and mirrored directory path. See
https://tracker.ceph.com/issues/73457
* CephFS Mirroring: The mirroring module provides ``ceph fs snapshot mirror status``,
a Ceph CLI command similar to the ``fs mirror peer status`` admin socket interface
for viewing per-directory snapshot sync metrics. The output layout matches
``fs mirror peer status`` for core sync fields and additionally includes
``metrics_updated_at`` (time of the last omap write). Optional filters by
mirrored directory path and peer UUID are supported. For more information,
see https://tracker.ceph.com/issues/76686
* CephFS Mirroring: Introduced snapshot checkpoints feature that allows users to mark
specific snapshots as important milestones and track their replication status to remote
sites. Checkpoints automatically detect if a snapshot has already been synced and provide
visibility into disaster recovery readiness. The feature includes four new CLI commands:
``ceph fs snapshot mirror checkpoint add`` to mark a snapshot,
``ceph fs snapshot mirror checkpoint now`` to checkpoint the latest snapshot,
``ceph fs snapshot mirror checkpoint ls`` to list all checkpoints with their status
(created/complete/failed), and ``ceph fs snapshot mirror checkpoint remove`` to remove
a checkpoint. Checkpoint metadata is persistent across daemon restarts. The mirror daemon
automatically prunes old COMPLETE checkpoints per directory, retaining the newest ones;
retention is controlled by ``cephfs_mirror_checkpoint_keep_count`` (default: 10). The
count may briefly exceed that limit until the next initialize, and a checkpoint completed
in the current initialize pass is not pruned in that pass. CREATED and FAILED checkpoints
are not pruned. This feature is useful for compliance auditing,
application consistency verification, and SLA tracking.
For more information, see https://tracker.ceph.com/issues/73454
* RBD: Mirror snapshot creation and trash purge schedules are now automatically
staggered when no explicit "start-time" is specified. This reduces scheduling
spikes and distributes work more evenly over time.
* RBD: Introduced a new ``RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT`` policy for
``rbd_lock_acquire()``. This is a low-level interface intended to allow
a peer to grab exclusive lock manually for short periods of time with other
peers pausing their activity and waiting for the lock to be released rather
than instantly aborting I/O and returning an error. It's possible to switch
from ``RBD_LOCK_MODE_EXCLUSIVE`` to ``RBD_LOCK_MODE_EXCLUSIVE_TRANSIENT``
policy and vice versa even if the lock is already held.
* RGW: In multisite deployments, zone endpoints configured in the zonegroup
(e.g. ``https://zone-a.example.com``) can now be resolved to all IP addresses
returned by DNS, with inter-zone traffic distributed across them using
round-robin and per-IP health tracking. This enables DNS-based service discovery
for inter-zone traffic without an external load balancer. This feature is disabled
by default: to opt in, set ``rgw_rest_conn_connect_to_resolved_ips = true``. The
per-IP retry-after-failure timeout is controlled by ``rgw_rest_conn_ip_fail_timeout_secs``
(default: 2 seconds). Deployments that work around libcurl's single-address behavior
by repeating the same endpoint multiple times in zone configuration should review that
setup before enabling, as round-robin distribution makes such duplication unnecessary.
* OSD: A health warning is reported when BlueFS usage exceeds the
configured ratio of the main OSD data device size. This warning is
informational and can be muted with:
``ceph health mute BLUESTORE_BLUEFS_OVERSIZED``
* Manager: The Manager now automatically increases ``mgr_stats_period`` when its
message queue is congested, reducing daemon reporting frequency to prevent
overload. The period recovers automatically once the queue clears. This
behavior is controlled by the new ``mgr_stats_period_autotune`` (default:
``true``) and ``mgr_stats_period_autotune_queue_threshold`` (default: ``100``)
config options.
* Manager: The Manager cache system has been redesigned from a TTL-based (time-to-live)
approach to an event-driven invalidation strategy. The cache now automatically
invalidates entries when underlying cluster maps are updated. The cache is enabled
by default. The previous ``mgr_ttl_cache_expire_seconds`` configuration option has
been removed and replaced with ``mgr_map_cache_enabled`` (default: ``true``).
* The ``last_degraded`` timestamp is added to the ``pg_stat_t`` structure to
track the initial point of redundancy loss when a PG enters an undersized or
degraded state. This timestamp is latched until the PG returns to a clean
state and a subsequent redundancy loss occurs. Used in conjunction with
``last_clean``, the ``last_degraded`` timestamp enables the calculation of
data vulnerability and durability scores.
* Two new perf counters, ``pg_vulnerability_duration`` and
``pg_rebuild_duration`` (each with a companion ``_min`` gauge and a
``max_inc``), track how long PGs spend at reduced redundancy and how
much of that time was active data movement. See
:ref:`rados_operations_monitoring_pg_vulnerability_rebuild_duration`
for details and example PromQL queries.
* RBD: It's possible to specify source cluster's ``mon_host`` and ``key`` for
``native`` format migration via the migration spec now. This eliminates the
dependency on ``<cluster-name>.conf`` file in a known location which is
rather rigid and also challenging to disseminate in some environments. The
key can be embedded in the migration spec or just referenced from there while
stored in the Monitor config-key store.
* RBD: ``Group::list_images`` python API will now correctly raise an ``ObjectNotFound``
error when invoked on a non-existent group. The C APIs ``rbd_group_image_list``,
``rbd_snap_list``, ``rbd_group_snap_list`` and C++ API ``group_image_list`` will now
return ``ENOENT`` instead of masking the error and returning 0.
* BlueStore: fsck no longer aborts on an undecodable onode; it reports the object
and continues. Post-crash allocation recovery tolerates corruption only during
read-only fsck, gated by the new ``bluestore_accept_corrupted_onode_recovery``
option (``never`` or ``read-only``, default ``read-only``); mount, repair, and
``ceph-bluestore-tool trim`` still abort.
>=20.0.0
* Client: New config value osd_min_split_replica_read_size, which allows control
of the minimum size a read can be when a client breaks up the read into multiple
smaller reads across PG shards.
* RADOS: The lead Monitor and stretch mode status are now displayed by `ceph status`.
Related Tracker: https://tracker.ceph.com/issues/70406
* RGW: The User Account feature introduced in Squid provides first-class support for
IAM APIs and policy. Our preliminary STS support was based on tenants, and
exposed some IAM APIs to admins only. This tenant-level IAM functionality is now
deprecated in favor of accounts. While we'll continue to support the tenant feature
itself for namespace isolation, the following features will be removed no sooner
than the V release:
* Tenant-level IAM APIs including CreateRole, PutRolePolicy and PutUserPolicy,
* Use of tenant names instead of accounts in IAM policy documents,
* Interpretation of IAM policy without cross-account policy evaluation,
* S3 API support for cross-tenant names such as `Bucket='tenant:bucketname'`
* RGW: Lua scripts will not run against health checks.
* RGW: For compatibility with AWS S3, LastModified timestamps are now truncated
to the second. Note that during an upgrade, users may observe these timestamps
moving backwards as a result.
* RGW: IAM policy evaluation now supports the conditions ArnEquals and ArnLike, along
with their Not and IfExists variants.
* RGW: Adding missing quotes to ETag values in S3 CopyPart,
PostObject and CompleteMultipartUpload responses.
* RGW: Add support for S3 GetObjectAttributes.
* RGW: Add the Beast front end option 'so_reuseport', which facilitates running multiple
RGW instances on the same host that share a single TCP port.
* RBD: All Python APIs that produce timestamps now return "aware" `datetime`
objects instead of "naive" ones (i.e. those including time zone information
instead of those not including it). All timestamps remain to be in UTC but
including `timezone.utc` makes it explicit and avoids the potential of the
returned timestamp getting misinterpreted -- in Python 3, many `datetime`
methods treat "naive" `datetime` objects as local times.
* RBD: The `rbd group info` and `rbd group snap info` commands are introduced to
show information about a group and a group snapshot respectively.
* RBD: The output of `rbd group snap ls` now includes the group snapshot IDs. The
heading for group snapshot status in the unformatted CLI
output is changed from `STATUS` to `STATE`. A group snapshot
that was previously shown as `ok` is now shown as `complete`, which is more descriptive.
* CephFS: Directories may now be configured with case-insensitive or
normalized directory entry names. This is inheritable and
applies to an entire directory tree. For more information, see
https://docs.ceph.com/en/latest/cephfs/charmap/
* RGW: "STS Lite" and the sts:GetSessionToken operation are now deprecated.
* Based on tests performed at scale on an HDD based Ceph cluster, it was found
that scheduling with mClock was not optimal with multiple OSD shards. For
example, when multiple OSD nodes failed, client
throughput was found to be inconsistent across test runs and multiple
slow requests were reported. However, the same test with a single OSD shard and
with multiple worker threads yielded significantly better results in terms of
consistent client and recovery throughput across multiple test runs.
As an interim measure until the issue with multiple OSD shards
(or multiple mClock queues per OSD) is investigated and fixed, the following
changes to option value defaults have been made:
- osd_op_num_shards_hdd = 1 (was 5)
- osd_op_num_threads_per_shard_hdd = 5 (was 1)
For more details see https://tracker.ceph.com/issues/66289.
* RGW: Added support for S3 Object Ownership to disable object ACLs.
* MGR: The Ceph Manager's always-on modulues/plugins can now be force-disabled.
This can be necessary when we wish to prevent the Manager from being
flooded by module commands when Ceph services are down or degraded.
* CephFS: It is now possible to pause the threads that asynchronously purge
the deleted subvolumes by using the config option
"mgr/volumes/pause_purging".
* CephFS: It is now possible to pause the threads that asynchronously clone
the subvolume snapshots by using the config option
"mgr/volumes/pause_cloning".
* CephFS: Modifying the setting `max_mds` when a cluster is
unhealthy now requires users to pass the confirmation flag
(--yes-i-really-mean-it). This has been added as a precaution to inform
admins that modifying `max_mds` may not help with troubleshooting or recovery
efforts. Instead, it might further destabilize the cluster.
* RADOS: Added convenience function `librados::AioCompletion::cancel()` with
the same behavior as `librados::IoCtx::aio_cancel()`.
* mgr/restful, mgr/zabbix: both modules, already deprecated since 2020, have been
finally removed. They have not been actively maintained and
suffer from vulnerabilities in their dependency chain (e.g.:
CVE-2023-46136). As alternatives, for the `restful` module, the `dashboard` module
provides a richer and better maintained RESTful API. Regarding the `zabbix` module,
there are alternative monitoring solutions, notably the Prometheus Alertmanager,
which scales more readily and is widely adopted within the Ceph community.
* CephFS: EOPNOTSUPP (Operation not supported ) is now returned by the CephFS
fuse client for `fallocate` for the default case (i.e. mode == 0) since
CephFS does not support disk space reservation. The only flags supported are
`FALLOC_FL_KEEP_SIZE` and `FALLOC_FL_PUNCH_HOLE`.
* pybind/rados: Fixes WriteOp.zero() in the original reversed order of arguments
`offset` and `length`. When pybind calls WriteOp.zero(), the argument passed
does not match rados_write_op_zero, and offset and length are swapped, which
results in an unexpected response.
* The HeadBucket API now reports the `X-RGW-Bytes-Used` and `X-RGW-Object-Count`
headers only when the `read-stats` querystring is explicitly included in the
API request.
* RGW: PutObjectLockConfiguration can now be used to enable S3 Object Lock on an
existing versioning-enabled bucket that was not created with Object Lock enabled.
* RADOS: The `ceph df` command reports incorrect `MAX AVAIL` values for stretch mode pools when
CRUSH rules use multiple take steps for datacenters. PGMap::get_rule_avail
incorrectly calculates available space from only one datacenter.
As a workaround, define CRUSH rules with `take default` and `choose firstn 0 type
datacenter`. See https://tracker.ceph.com/issues/56650#note-6 for details.
Upgrading a cluster configured with a CRUSH rule that includes multiple `take` steps
can lead to data shuffling, as CRUSH changes may necessitate data
redistribution. In contrast, a stretch rule with a single `take` step
will not cause data movement during the upgrade process.
* RGW: The `x-amz-confirm-remove-self-bucket-access` header is now supported by
`PutBucketPolicy`. Additionally, the root user will always have access to modify
the bucket policy, even if the current policy explicitly denies access.
* CephFS: The ``ceph fs subvolume snapshot getpath`` command now allows users
to get the path of a snapshot of a subvolume. If the snapshot is not present
``ENOENT`` is returned.
* CephFS: The ``ceph fs volume create`` command now allows users to pass
metadata and data pool names to be used for creating the volume. If either
is not passed or if either is a non-empty pool, the command will abort.
* RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been added that allows
admins to clear all pg-upmap-primary mappings in the osdmap when desired.
Related trackers:
- https://tracker.ceph.com/issues/67179
- https://tracker.ceph.com/issues/66867
* RADOS: The default plugin for erasure coded pools has been changed
from Jerasure to ISA-L. Pools created on Tentacle or later releases will
use ISA-L as the default plugin. Pools within clusters upgraded
to Tentacle or Umbrella will continue to use their existing plugin.
The default values can be overriden by creating a new erasure code profile and
selecting it when creating a pool.
ISA-L is recommended for new pools because the Jerasure library is
no longer maintained.
* CephFS: The format of pool namespaces for CephFS volumes has been changed
from `fsvolumens__<subvol-name>` to `fsvolumens__<subvol-grp-name>_<subvol-name>`
to avoid namespace collision when two subvolumes located in different
subvolume groups have the same name. Even with namespace collision, there were
no security issues since the MDS auth cap is restricted to the subvolume path.
Now, with this change, namespaces are completely isolated.
* RGW: Add support for the `RestrictPublicBuckets` property of the S3 `PublicAccessBlock`
configuration.
* RBD: Moving an image that is a member of a group to trash is no longer
allowed. `rbd trash mv` command now behaves the same way as `rbd rm` in this
scenario.
* CephFS: If the subvolume name passed to the command `ceph fs subvolume info`
is a clone, the output now will also contain a "source" field that tells user
the name of source snapshot along with the name of volume, subvolume group and
subvolume in which the source snapshot is located. For clones created with
Tentacle or earlier release, the value of this field will be 'N/A'. Regular
subvolumes don't have a source subvolume and therefore the output for them
won't contain a "source" field regardless of the release.
* RGW: The default value of `rgw_lc_max_wp_worker` has been raised from 3 to 128
to accelerate lifecycle processing. These workers now use coroutines instead of
separate threads.
* RGW: Replication policies now validate permissions using `s3:ReplicateObject`,
`s3:ReplicateDelete`, and `s3:ReplicateTags` for destination buckets. For source
buckets, both `s3:GetObjectVersionForReplication` and `s3:GetObject(Version)`
are supported. Actions like `s3:GetObjectAcl`, `s3:GetObjectLegalHold`, and
`s3:GetObjectRetention` are also considered when fetching the source object.
Replication of tags is controlled by the `s3:GetObject(Version)Tagging` permission.
* RADOS: A new command, ``ceph osd pool availability-status``, has been added that allows
users to view the availability score for each pool in a cluster. A pool is considered
unavailable if any PG in the pool is not in active state or if there are unfound
objects. Otherwise the pool is considered available. The score is updated every
one second by default. This interval can be changed using the new config option
``pool_availability_update_interval.``. The feature is off by default. A new config option
``enable_availability_tracking`` can be used to turn on the feature if required.
Another command is added to clear the availability status for a specific pool,
``ceph osd pool clear-availability-status <pool-name>``.
This feature is in tech preview.
Related trackers:
- https://tracker.ceph.com/issues/67777
* RBD: Fetching the mirroring mode of an image is invalid if the image is
disabled for mirroring. The public APIs -- C++ `mirror_image_get_mode()`,
C `rbd_mirror_image_get_mode()`, and Python `Image.mirror_image_get_mode()`
-- will return EINVAL when mirroring is disabled.
* RBD: Promoting an image is invalid if the image is not enabled for mirroring.
The public APIs -- C++ `mirror_image_promote()`,
C `rbd_mirror_image_promote()`, and Python `Image.mirror_image_promote()` --
will return EINVAL instead of ENOENT when mirroring is not enabled.
* RBD: Requesting a resync on an image is invalid if the image is not enabled
for mirroring. The public APIs -- C++ `mirror_image_resync()`,
C `rbd_mirror_image_resync()`, and Python `Image.mirror_image_resync()` --
will return EINVAL instead of ENOENT when mirroring is not enabled.
* Dashboard: NVMe/TCP Gateway group support
* Dashboard: NVMe/TCP multiple namespaces support
* Dashboard: Multi-cluster support
* Dashboard: Management Gateway and oAuth2 integration with dashboard
* Dashboard: RGW multi-site automation & sync Policies
* Dashboard: RGW Tiering, Rate Limting and Account management
* Dashboard: RGW bucket Lifecycles and ACLs
* Dashboard: RGW Topics and Bucket Notification Management
* Dashboard: RGW granular bucket replication
* Dashboard: SMB monitoring and management
* Dashboard: `ceph dashboard sso enable oauth2` checks for oauth2-proxy service for SSO enablement. If the `oauth2-proxy` service goes down, cephadm will automatically disable Dashboard OAuth2 SSO.
* Monitoring: New monitoring Dashboards for Application, NVMe, CephFS, and SMB Overview
* RGW: Introduce the `rgw_usage_log_key_transition` configuration option to handle
the co-existence of old and new usage log keys. This option is enabled by default
to ensure compatibility during upgrades, but can be disabled once old usage logs
are no longer present to avoid performance overhead.
* NVMe-oF: A new Ceph Manager module, `nvmeof`, is now available. When enabled, it provisions the
dedicated `.nvmeof` RADOS Block Device (RBD) pool required for NVMe-oF integration,
simplifying initial setup and ensuring the pool is created consistently across deployments.
* MGR: A new command, `ceph osd ok-to-upgrade`, has been added that allows
users and orchestration tools to determine a safe set of OSDs within a CRUSH
bucket to upgrade simultaneously without impacting data availability. To help
converge to a safe set, a new config option
``mgr_osd_upgrade_check_convergence_factor`` is introduced. This option can be
modified (if necessary) to help converge to an optimal set. Higher values
maximize the set of OSDs to upgrade at the cost of longer command response
times. Conversely, a lower value improves the command response time but
results in a non-optimal or smaller set of OSDs which impacts the overall time
to upgrade all OSDs in the cluster. For more details see tracker:
https://tracker.ceph.com/issues/73031.
* NFS: cluster/export delete commands now throw a deprecation warning when
used and will be removed in V release.
>=19.2.1
* CephFS: The `fs subvolume create` command now allows tagging subvolumes through option
`--earmark` with a unique identifier needed for NFS or SMB services. The earmark
string for a subvolume is empty by default. To remove an already present earmark,
an empty string can be assigned to it. Additionally, commands
`ceph fs subvolume earmark set`, `ceph fs subvolume earmark get` and
`ceph fs subvolume earmark rm` have been added to set, get and remove earmark from a given subvolume.
* RADOS: Add ``messenger dump`` command to retrieve runtime information
on connections, sockets, and kernel TCP stats from the messenger.
* RGW: deleted buckets are automatically cleaned up as part of trimming process.
DeleteBucket will start returning 409 BucketNotEmpty errors until empty
on all zones when sync policy is enabled.
* RADOS: A performance botteneck in the balancer mgr module has been fixed.
Related Tracker: https://tracker.ceph.com/issues/68657
>=19.0.0
* cephx: key rotation is now possible using `ceph auth rotate`. Previously,
this was only possible by deleting and then recreating the key.
* Ceph: a new --daemon-output-file switch is available for `ceph tell` commands
to dump output to a file local to the daemon. For commands which produce
large amounts of output, this avoids a potential spike in memory usage on the
daemon, allows for faster streaming writes to a file local to the daemon, and
reduces time holding any locks required to execute the command. For analysis,
it is necessary to retrieve the file from the host running the daemon
manually. Currently, only --format=json|json-pretty are supported.
* RGW: GetObject and HeadObject requests now return an x-rgw-replicated-at
header for replicated objects. This timestamp can be compared against the
Last-Modified header to determine how long the object took to replicate.
* The cephfs-shell utility is now packaged for RHEL / CentOS / Rocky 9 as required
Python dependencies are now available in EPEL9.
* RGW: S3 multipart uploads using Server-Side Encryption now replicate correctly in
multi-site deployments Previously, replicas of such objects were corrupted on decryption.
A new tool, ``radosgw-admin bucket resync encrypted multipart``, can be used to
identify these original multipart uploads. The ``LastModified`` timestamp of any
identified object is incremented by one ns to cause peer zones to replicate it again.
For multi-site deployments that make use of Server-Side Encryption, we
recommended running this command against every bucket in every zone after all
zones have upgraded.
* Tracing: The blkin tracing feature (see https://docs.ceph.com/en/reef/dev/blkin/)
is now deprecated in favor of Opentracing (https://docs.ceph.com/en/reef/dev/developer_guide/jaegertracing/)
and will be removed in a later release.
* RGW: Introducing a new data layout for the Topic metadata associated with S3
Bucket Notifications, where each Topic is stored as a separate RADOS object
and the bucket notification configuration is stored in a bucket attribute.
This new representation supports multisite replication via metadata sync and
can scale to many topics. This is on by default for new deployments, but is
is not enabled by default on upgrade. Once all radosgws have upgraded (on all
zones in a multisite configuration), the ``notification_v2`` zone feature can
be enabled to migrate to the new format. See
https://docs.ceph.com/en/squid/radosgw/zone-features for details. The "v1"
format is now considered deprecated and may be removed after 2 major releases.
* CephFS: The MDS evicts clients which are not advancing their request tids, which causes
a large buildup of session metadata, which in turn results in the MDS going read-only
due to RADOS operations exceeding the size threshold. `mds_session_metadata_threshold`
config controls the maximum size to which (encoded) session metadata can grow.
* CephFS: A new "mds last-seen" command is available for querying the last time
an MDS was in the FSMap, subject to a pruning threshold.
* CephFS: For clusters with multiple CephFS file systems, all snap-schedule
commands now expect the '--fs' argument.
* CephFS: The period specifier ``m`` now implies minutes and the period specifier
``M`` now implies months. This is consistent with the rest of the system.
* RGW: New tools have been added to radosgw-admin for identifying and
correcting issues with versioned bucket indexes. Historical bugs with the
versioned bucket index transaction workflow made it possible for the index
to accumulate extraneous "book-keeping" olh entries and plain placeholder
entries. In some specific scenarios where clients made concurrent requests
referencing the same object key, it was likely that extra index
entries would accumulate. When a significant number of these entries are
present in a single bucket index shard, they can cause high bucket listing
latency and lifecycle processing failures. To check whether a versioned
bucket has unnecessary olh entries, users can now run ``radosgw-admin
bucket check olh``. If the ``--fix`` flag is used, the extra entries will
be safely removed. An additional issue is that some versioned buckets
may maintain extra unlinked objects that are not listable via the S3/Swift
APIs. These extra objects are typically a result of PUT requests that
exited abnormally in the middle of a bucket index transaction, and thus
the client would not have received a successful response. Bugs in prior
releases made these unlinked objects easy to reproduce with any PUT
request made on a bucket that was actively resharding. In certain
scenarios, a client of a bucket that was a victim of this bug may find
the object associated with the key to be in an inconsistent state. To check
whether a versioned bucket has unlinked entries, users can now run
``radosgw-admin bucket check unlinked``. If the ``--fix`` flag is used,
the unlinked objects will be safely removed. Finally, a third issue made
it possible for versioned bucket index stats to be accounted inaccurately.
The tooling for recalculating versioned bucket stats also had a bug, and
was not previously capable of fixing these inaccuracies. This release
resolves those issues and users can now expect that the existing
``radosgw-admin bucket check`` command will produce correct results.
We recommend that users with versioned buckets, especially those that
existed on prior releases, use these new tools to check whether their
buckets are affected and to clean them up accordingly.
* RGW: The "user accounts" feature unlocks several new AWS-compatible IAM APIs
for self-service management of users, keys, groups, roles, policy and
more. Existing users can be adopted into new accounts. This process is optional
but irreversible. See https://docs.ceph.com/en/squid/radosgw/account and
https://docs.ceph.com/en/squid/radosgw/iam for details.
* RGW: On startup, radosgw and radosgw-admin now validate the ``rgw_realm``
config option. Previously, they would ignore invalid or missing realms and
go on to load a zone/zonegroup in a different realm. If startup fails with
a "failed to load realm" error, fix or remove the ``rgw_realm`` option.
* RGW: The radosgw-admin commands ``realm create`` and ``realm pull`` no
longer set the default realm without ``--default``.
* CephFS: Running the command "ceph fs authorize" for an existing entity now
upgrades the entity's capabilities instead of printing an error. It can now
also change read/write permissions in a capability that the entity already
holds. If the capability passed by user is same as one of the capabilities
that the entity already holds, idempotency is maintained.
* CephFS: Two FS names can now be swapped, optionally along with their IDs,
using "ceph fs swap" command. The function of this API is to facilitate
file system swaps for disaster recovery. In particular, it avoids situations
where a named file system is temporarily missing which would prompt a higher
level storage operator (like Rook) to recreate the missing file system.
See https://docs.ceph.com/en/latest/cephfs/administration/#file-systems
docs for more information.
* CephFS: Before running the command "ceph fs rename", the filesystem to be
renamed must be offline and the config "refuse_client_session" must be set
for it. The config "refuse_client_session" can be removed/unset and
filesystem can be online after the rename operation is complete.
* RADOS: A POOL_APP_NOT_ENABLED health warning will now be reported if
the application is not enabled for the pool irrespective of whether
the pool is in use or not. Always tag a pool with an application
using ``ceph osd pool application enable`` command to avoid reporting
of POOL_APP_NOT_ENABLED health warning for that pool.
The user might temporarily mute this warning using
``ceph health mute POOL_APP_NOT_ENABLED``.
* The `mon_cluster_log_file_level` and `mon_cluster_log_to_syslog_level` options
have been removed. Henceforth, users should use the new generic option
`mon_cluster_log_level` to control the cluster log level verbosity for the cluster
log file as well as for all external entities.
CephFS: Disallow delegating preallocated inode ranges to clients. Config
`mds_client_delegate_inos_pct` defaults to 0 which disables async dirops
in the kclient.
* S3 Get/HeadObject now support query parameter `partNumber` to read a specific
part of a completed multipart upload.
* RGW: Fixed a S3 Object Lock bug with PutObjectRetention requests that specify
a RetainUntilDate after the year 2106. This date was truncated to 32 bits when
stored, so a much earlier date was used for object lock enforcement. This does
not effect PutBucketObjectLockConfiguration where a duration is given in Days.
The RetainUntilDate encoding is fixed for new PutObjectRetention requests, but
cannot repair the dates of existing object locks. Such objects can be identified
with a HeadObject request based on the x-amz-object-lock-retain-until-date
response header.
* RADOS: `get_pool_is_selfmanaged_snaps_mode` C++ API has been deprecated
due to being prone to false negative results. It's safer replacement is
`pool_is_in_selfmanaged_snaps_mode`.
* RADOS: For bug 62338 (https://tracker.ceph.com/issues/62338), in order to simplify
backporting, we choose to not
condition the fix on a server flag. As
a result, in rare cases it may be possible for a PG to flip between two acting
sets while an upgrade to a version with the fix is in progress. If you observe
this behavior, you should be able to work around it by completing the upgrade or
by disabling async recovery by setting osd_async_recovery_min_cost to a very
large value on all OSDs until the upgrade is complete:
``ceph config set osd osd_async_recovery_min_cost 1099511627776``
* RADOS: A detailed version of the `balancer status` CLI command in the balancer
module is now available. Users may run `ceph balancer status detail` to see more
details about which PGs were updated in the balancer's last optimization.
See https://docs.ceph.com/en/latest/rados/operations/balancer/ for more information.
* CephFS: Full support for subvolumes and subvolume groups is now available
for snap_schedule Manager module.
* RGW: The SNS CreateTopic API now enforces the same topic naming requirements as AWS:
Topic names must be made up of only uppercase and lowercase ASCII letters, numbers,
underscores, and hyphens, and must be between 1 and 256 characters long.
* RBD: When diffing against the beginning of time (`fromsnapname == NULL`) in
fast-diff mode (`whole_object == true` with `fast-diff` image feature enabled
and valid), diff-iterate is now guaranteed to execute locally if exclusive
lock is available. This brings a dramatic performance improvement for QEMU
live disk synchronization and backup use cases.
* RBD: The ``try-netlink`` mapping option for rbd-nbd has become the default
and is now deprecated. If the NBD netlink interface is not supported by the
kernel, then the mapping is retried using the legacy ioctl interface.
* RADOS: Read balancing may now be managed automatically via the balancer
manager module. Users may choose between two new modes: ``upmap-read``, which
offers upmap and read optimization simultaneously, or ``read``, which may be used
to only optimize reads. For more detailed information see https://docs.ceph.com/en/latest/rados/operations/read-balancer/#online-optimization.
* CephFS: MDS log trimming is now driven by a separate thread which tries to
trim the log every second (`mds_log_trim_upkeep_interval` config). Also,
a couple of configs govern how much time the MDS spends in trimming its
logs. These configs are `mds_log_trim_threshold` and `mds_log_trim_decay_rate`.
* RGW: Notification topics are now owned by the user that created them.
By default, only the owner can read/write their topics. Topic policy documents
are now supported to grant these permissions to other users. Preexisting topics
are treated as if they have no owner, and any user can read/write them using the SNS API.
If such a topic is recreated with CreateTopic, the issuing user becomes the new owner.
For backward compatibility, all users still have permission to publish bucket
notifications to topics owned by other users. A new configuration parameter:
``rgw_topic_require_publish_policy`` can be enabled to deny ``sns:Publish``
permissions unless explicitly granted by topic policy.
* RGW: Fix issue with persistent notifications where the changes to topic param that
were modified while persistent notifications were in the queue will be reflected in notifications.
So if user sets up topic with incorrect config (password/ssl) causing failure while delivering the
notifications to broker, can now modify the incorrect topic attribute and on retry attempt to delivery
the notifications, new configs will be used.
* RBD: The option ``--image-id`` has been added to `rbd children` CLI command,
so it can be run for images in the trash.
* PG dump: The default output of `ceph pg dump --format json` has changed. The
default json format produces a rather massive output in large clusters and
isn't scalable. So we have removed the 'network_ping_times' section from
the output. Details in the tracker: https://tracker.ceph.com/issues/57460
* mgr/REST: The REST manager module will trim requests based on the 'max_requests' option.
Without this feature, and in the absence of manual deletion of old requests,
the accumulation of requests in the array can lead to Out Of Memory (OOM) issues,
resulting in the Manager crashing.
* CephFS: The `subvolume snapshot clone` command now depends on the config option
`snapshot_clone_no_wait` which is used to reject the clone operation when
all the cloner threads are busy. This config option is enabled by default which means
that if no cloner threads are free, the clone request errors out with EAGAIN.
The value of the config option can be fetched by using:
`ceph config get mgr mgr/volumes/snapshot_clone_no_wait`
and it can be disabled by using:
`ceph config set mgr mgr/volumes/snapshot_clone_no_wait false`
* RBD: `RBD_IMAGE_OPTION_CLONE_FORMAT` option has been exposed in Python
bindings via `clone_format` optional parameter to `clone`, `deep_copy` and
`migration_prepare` methods.
* RBD: `RBD_IMAGE_OPTION_FLATTEN` option has been exposed in Python bindings via
`flatten` optional parameter to `deep_copy` and `migration_prepare` methods.
* CephFS: Command "ceph mds fail" and "ceph fs fail" now requires a
confirmation flag when some MDSs exhibit health warning MDS_TRIM or
MDS_CACHE_OVERSIZED. This is to prevent accidental MDS failover causing
further delays in recovery.
* CephFS: fixes to the implementation of the ``root_squash`` mechanism enabled
via cephx ``mds`` caps on a client credential require a new client feature
bit, ``client_mds_auth_caps``. Clients using credentials with ``root_squash``
without this feature will trigger the MDS to raise a HEALTH_ERR on the
cluster, MDS_CLIENTS_BROKEN_ROOTSQUASH. See the documentation on this warning
and the new feature bit for more information.
* CephFS: Expanded removexattr support for cephfs virtual extended attributes.
Previously one had to use setxattr to restore the default in order to "remove".
You may now properly use removexattr to remove. You can also now remove layout
on root inode, which then will restore layout to default layout.
* cls_cxx_gather is marked as deprecated.
* CephFS: cephfs-journal-tool is guarded against running on an online file system.
The 'cephfs-journal-tool --rank <fs_name>:<mds_rank> journal reset' and
'cephfs-journal-tool --rank <fs_name>:<mds_rank> journal reset --force'
commands require '--yes-i-really-really-mean-it'.
* Dashboard: Rearranged Navigation Layout: The navigation layout has been reorganized
for improved usability and easier access to key features.
* Dashboard: CephFS Improvments
* Support for managing CephFS snapshots and clones, as well as snapshot schedule
management
* Manage authorization capabilities for CephFS resources
* Helpers on mounting a CephFS volume
* Dashboard: RGW Improvements
* Support for managing bucket policies
* Add/Remove bucket tags
* ACL Management
* Several UI/UX Improvements to the bucket form
* Monitoring: Grafana dashboards are now loaded into the container at runtime rather than
building a grafana image with the grafana dashboards. Official Ceph grafana images
can be found in quay.io/ceph/grafana
* Monitoring: RGW S3 Analytics: A new Grafana dashboard is now available, enabling you to
visualize per bucket and user analytics data, including total GETs, PUTs, Deletes,
Copies, and list metrics.
* RBD: `Image::access_timestamp` and `Image::modify_timestamp` Python APIs now
return timestamps in UTC.
* RBD: Support for cloning from non-user type snapshots is added. This is
intended primarily as a building block for cloning new groups from group
snapshots created with `rbd group snap create` command, but has also been
exposed via the new `--snap-id` option for `rbd clone` command.
* RBD: The output of `rbd snap ls --all` command now includes the original
type for trashed snapshots.
* CephFS: "ceph fs clone status" command will now print statistics about clone
progress in terms of how much data has been cloned (in both percentage as
well as bytes) and how many files have been cloned.
* CephFS: "ceph status" command will now print a progress bar when cloning is
ongoing. If clone jobs are more than the cloner threads, it will print one
more progress bar that shows total amount of progress made by both ongoing
as well as pending clones. Both progress are accompanied by messages that
show number of clone jobs in the respective categories and the amount of
progress made by each of them.
* RGW: in bucket notifications, the `principalId` inside `ownerIdentity` now contains
complete user id, prefixed with tenant id
* NFS: The export create/apply of CephFS based exports will now have a additional parameter `cmount_path` under the FSAL block,
which specifies the path within the CephFS to mount this export on. If this and the other
`EXPORT { FSAL {} }` options are the same between multiple exports, those exports will share a single CephFS client. If not specified, the default is `/`.
* CephFS: MDS emits a warning with estimated replay completion time when replay
runs for more than 30 seconds.
>=18.0.0
* The RGW policy parser now rejects unknown principals by default. If you are
mirroring policies between RGW and AWS, you may wish to set
"rgw policy reject invalid principals" to "false". This affects only newly set
policies, not policies that are already in place.
* The CephFS automatic metadata load (sometimes called "default") balancer is
now disabled by default. The new file system flag `balance_automate`
can be used to toggle it on or off. It can be enabled or disabled via
`ceph fs set <fs_name> balance_automate <bool>`.
* RGW's default backend for `rgw_enable_ops_log` changed from RADOS to file.
The default value of `rgw_ops_log_rados` is now false, and `rgw_ops_log_file_path`
defaults to "/var/log/ceph/ops-log-$cluster-$name.log".
* The SPDK backend for BlueStore is now able to connect to an NVMeoF target.
Please note that this is not an officially supported feature.
* RGW's pubsub interface now returns boolean fields using bool. Before this change,
`/topics/<topic-name>` returns "stored_secret" and "persistent" using a string
of "true" or "false" with quotes around them. After this change, these fields
are returned without quotes so they can be decoded as boolean values in JSON.
The same applies to the `is_truncated` field returned by `/subscriptions/<sub-name>`.
* RGW's response of `Action=GetTopicAttributes&TopicArn=<topic-arn>` REST API now
returns `HasStoredSecret` and `Persistent` as boolean in the JSON string
encoded in `Attributes/EndPoint`.
* All boolean fields previously rendered as string by `rgw-admin` command when
the JSON format is used are now rendered as boolean. If your scripts/tools
relies on this behavior, please update them accordingly. The impacted field names
are:
* absolute
* add
* admin
* appendable
* bucket_key_enabled
* delete_marker
* exists
* has_bucket_info
* high_precision_time
* index
* is_master
* is_prefix
* is_truncated
* linked
* log_meta
* log_op
* pending_removal
* read_only
* retain_head_object
* rule_exist
* start_with_full_sync
* sync_from_all
* syncstopped
* system
* truncated
* user_stats_sync
* RGW: The beast frontend's HTTP access log line uses a new debug_rgw_access
configurable. This has the same defaults as debug_rgw, but can now be controlled
independently.
* RBD: The semantics of compare-and-write C++ API (`Image::compare_and_write`
and `Image::aio_compare_and_write` methods) now match those of C API. Both
compare and write steps operate only on `len` bytes even if the respective
buffers are larger. The previous behavior of comparing up to the size of
the compare buffer was prone to subtle breakage upon straddling a stripe
unit boundary.
* RBD: compare-and-write operation is no longer limited to 512-byte sectors.
Assuming proper alignment, it now allows operating on stripe units (4M by
default).
* RBD: New `rbd_aio_compare_and_writev` API method to support scatter/gather
on both compare and write buffers. This compliments existing `rbd_aio_readv`
and `rbd_aio_writev` methods.
* The 'AT_NO_ATTR_SYNC' macro is deprecated, please use the standard 'AT_STATX_DONT_SYNC'
macro. The 'AT_NO_ATTR_SYNC' macro will be removed in the future.
* Trimming of PGLog dups is now controlled by the size instead of the version.
This fixes the PGLog inflation issue that was happening when the on-line
(in OSD) trimming got jammed after a PG split operation. Also, a new off-line
mechanism has been added: `ceph-objectstore-tool` got `trim-pg-log-dups` op
that targets situations where OSD is unable to boot due to those inflated dups.
If that is the case, in OSD logs the "You can be hit by THE DUPS BUG" warning
will be visible.
Relevant tracker: https://tracker.ceph.com/issues/53729
* RBD: `rbd device unmap` command gained `--namespace` option. Support for
namespaces was added to RBD in Nautilus 14.2.0 and it has been possible to
map and unmap images in namespaces using the `image-spec` syntax since then
but the corresponding option available in most other commands was missing.
* RGW: Compression is now supported for objects uploaded with Server-Side Encryption.
When both are enabled, compression is applied before encryption. Earlier releases
of multisite do not replicate such objects correctly, so all zones must upgrade to
Reef before enabling the `compress-encrypted` zonegroup feature: see
https://docs.ceph.com/en/reef/radosgw/multisite/#zone-features and note the
security considerations.
* RGW: the "pubsub" functionality for storing bucket notifications inside Ceph
is removed. Together with it, the "pubsub" zone should not be used anymore.
The REST operations, as well as radosgw-admin commands for manipulating
subscriptions, as well as fetching and acking the notifications are removed
as well.
In case that the endpoint to which the notifications are sent maybe down or
disconnected, it is recommended to use persistent notifications to guarantee
the delivery of the notifications. In case the system that consumes the
notifications needs to pull them (instead of the notifications be pushed
to it), an external message bus (e.g. rabbitmq, Kafka) should be used for
that purpose.
* RGW: The serialized format of notification and topics has changed, so that
new/updated topics will be unreadable by old RGWs. We recommend completing
the RGW upgrades before creating or modifying any notification topics.
* RGW: New rate limiting options have been added to control the number of
bucket listing requests (`radosgw-admin ratelimit set ... --max-list-ops=`)
and delete operations (`radosgw-admin ratelimit set ... --max-delete-ops=`)
per accumulation interval, along with the ability to configure the
accumulation interval duration (`rgw_ratelimit_interval`).
Their default value is `0`, which keeps backward compatibility to the
previous behavior of READ and WRITE operations rate limiting.
* RBD: Trailing newline in passphrase files (`<passphrase-file>` argument in
`rbd encryption format` command and `--encryption-passphrase-file` option
in other commands) is no longer stripped.
* RBD: Support for layered client-side encryption is added. Cloned images
can now be encrypted each with its own encryption format and passphrase,
potentially different from that of the parent image. The efficient
copy-on-write semantics intrinsic to unformatted (regular) cloned images
are retained.
* CEPHFS: Rename the `mds_max_retries_on_remount_failure` option to
`client_max_retries_on_remount_failure` and move it from mds.yaml.in to
mds-client.yaml.in because this option was only used by MDS client from its
birth.
* The `perf dump` and `perf schema` commands are deprecated in favor of new
`counter dump` and `counter schema` commands. These new commands add support
for labeled perf counters and also emit existing unlabeled perf counters. Some
unlabeled perf counters became labeled in this release, with more to follow in
future releases; such converted perf counters are no longer emitted by the
`perf dump` and `perf schema` commands.
* `ceph mgr dump` command now outputs `last_failure_osd_epoch` and
`active_clients` fields at the top level. Previously, these fields were
output under `always_on_modules` field.
* `ceph mgr dump` command now displays the name of the mgr module that
registered a RADOS client in the `name` field added to elements of the
`active_clients` array. Previously, only the address of a module's RADOS
client was shown in the `active_clients` array.
* RBD: All rbd-mirror daemon perf counters became labeled and as such are now
emitted only by the new `counter dump` and `counter schema` commands. As part
of the conversion, many also got renamed to better disambiguate journal-based
and snapshot-based mirroring.
* RBD: list-watchers C++ API (`Image::list_watchers`) now clears the passed
`std::list` before potentially appending to it, aligning with the semantics
of the corresponding C API (`rbd_watchers_list`).
* The rados python binding is now able to process (opt-in) omap keys as bytes
objects. This enables interacting with RADOS omap keys that are not decodeable as
UTF-8 strings.
* Telemetry: Users who are opted-in to telemetry can also opt-in to
participating in a leaderboard in the telemetry public
dashboards (https://telemetry-public.ceph.com/). Users can now also add a
description of the cluster to publicly appear in the leaderboard.
For more details, see:
https://docs.ceph.com/en/latest/mgr/telemetry/#leaderboard
See a sample report with `ceph telemetry preview`.
Opt-in to telemetry with `ceph telemetry on`.
Opt-in to the leaderboard with
`ceph config set mgr mgr/telemetry/leaderboard true`.
Add leaderboard description with:
`ceph config set mgr mgr/telemetry/leaderboard_description ‘Cluster description’`.
* CEPHFS: After recovering a Ceph File System post following the disaster recovery
procedure, the recovered files under `lost+found` directory can now be deleted.
* core: cache-tiering is now deprecated.
* mClock Scheduler: The mClock scheduler (default scheduler in Quincy) has
undergone significant usability and design improvements to address the slow
backfill issue. Some important changes are:
* The 'balanced' profile is set as the default mClock profile because it
represents a compromise between prioritizing client I/O or recovery I/O. Users
can then choose either the 'high_client_ops' profile to prioritize client I/O
or the 'high_recovery_ops' profile to prioritize recovery I/O.
* QoS parameters like reservation and limit are now specified in terms of a
fraction (range: 0.0 to 1.0) of the OSD's IOPS capacity.
* The cost parameters (osd_mclock_cost_per_io_usec_* and
osd_mclock_cost_per_byte_usec_*) have been removed. The cost of an operation
is now determined using the random IOPS and maximum sequential bandwidth
capability of the OSD's underlying device.
* Degraded object recovery is given higher priority when compared to misplaced
object recovery because degraded objects present a data safety issue not
present with objects that are merely misplaced. Therefore, backfilling
operations with the 'balanced' and 'high_client_ops' mClock profiles may
progress slower than what was seen with the 'WeightedPriorityQueue' (WPQ)
scheduler.
* The QoS allocations in all the mClock profiles are optimized based on the above
fixes and enhancements.
* For more detailed information see:
https://docs.ceph.com/en/latest/rados/configuration/mclock-config-ref/
* mgr/snap_schedule: The snap-schedule mgr module now retains one less snapshot
than the number mentioned against the config tunable `mds_max_snaps_per_dir`
so that a new snapshot can be created and retained during the next schedule
run.
* `ceph config dump --format <json|xml>` output will display the localized
option names instead of its normalized version. For e.g.,
"mgr/prometheus/x/server_port" will be displayed instead of
"mgr/prometheus/server_port". This matches the output of the non pretty-print
formatted version of the command.
* CEPHFS: MDS config option name "mds_kill_skip_replaying_inotable" is a bit
confusing with "mds_inject_skip_replaying_inotable", therefore renaming it to
"mds_kill_after_journal_logs_flushed"
>=17.2.1
* The "BlueStore zero block detection" feature (first introduced to Quincy in
https://github.com/ceph/ceph/pull/43337) has been turned off by default with a
new global configuration called `bluestore_zero_block_detection`. This feature,
intended for large-scale synthetic testing, does not interact well with some RBD
and CephFS features. Any side effects experienced in previous Quincy versions
would no longer occur, provided that the configuration remains set to false.
Relevant tracker: https://tracker.ceph.com/issues/55521
* telemetry: Added new Rook metrics to the 'basic' channel to report Rook's
version, Kubernetes version, node metrics, etc.
See a sample report with `ceph telemetry preview`.
Opt-in with `ceph telemetry on`.
For more details, see:
https://docs.ceph.com/en/latest/mgr/telemetry/
* OSD: The issue of high CPU utilization during recovery/backfill operations
has been fixed. For more details, see: https://tracker.ceph.com/issues/56530.
>=15.2.17
* OSD: Octopus modified the SnapMapper key format from
<LEGACY_MAPPING_PREFIX><snapid>_<shardid>_<hobject_t::to_str()>
to
<MAPPING_PREFIX><pool>_<snapid>_<shardid>_<hobject_t::to_str()>
When this change was introduced, 94ebe0e also introduced a conversion
with a crucial bug which essentially destroyed legacy keys by mapping them
to
<MAPPING_PREFIX><poolid>_<snapid>_
without the object-unique suffix. The conversion is fixed in this release.
Relevant tracker: https://tracker.ceph.com/issues/56147
* Cephadm may now be configured to carry out CephFS MDS upgrades without
reducing ``max_mds`` to 1. Previously, Cephadm would reduce ``max_mds`` to 1 to
avoid having two active MDS modifying on-disk structures with new versions,
communicating cross-version-incompatible messages, or other potential
incompatibilities. This could be disruptive for large-scale CephFS deployments
because the cluster cannot easily reduce active MDS daemons to 1.
NOTE: Staggered upgrade of the mons/mgrs may be necessary to take advantage
of the feature, refer this link on how to perform it:
https://docs.ceph.com/en/quincy/cephadm/upgrade/#staggered-upgrade
Relevant tracker: https://tracker.ceph.com/issues/55715
* Introduced a new file system flag `refuse_client_session` that can be set using the
`fs set` command. This flag allows blocking any incoming session
request from client(s). This can be useful during some recovery situations
where it's desirable to bring MDS up but have no client workload.
Relevant tracker: https://tracker.ceph.com/issues/57090
* New MDSMap field `max_xattr_size` which can be set using the `fs set` command.
This MDSMap field allows to configure the maximum size allowed for the full
key/value set for a filesystem extended attributes. It effectively replaces
the old per-MDS `max_xattr_pairs_size` setting, which is now dropped.
Relevant tracker: https://tracker.ceph.com/issues/55725
* Introduced a new file system flag `refuse_standby_for_another_fs` that can be
set using the `fs set` command. This flag prevents using a standby for another
file system (join_fs = X) when standby for the current filesystem is not available.
Relevant tracker: https://tracker.ceph.com/issues/61599
* mon: add NVMe-oF gateway monitor and HA
This PR adds high availability support for the nvmeof Ceph service. High availability
means that even in the case that a certain GW is down, there will be another available
path for the initiator to be able to continue the I/O through another GW.
It is also adding 2 new mon commands, to notify monitor about the gateway creation/deletion:
- nvme-gw create
- nvme-gw delete
Relevant tracker: https://tracker.ceph.com/issues/64777
* MDS now uses host error numbers, as defined in errno.cc, for current platform.
`errorcode32_t` is converts, internally, error codes from host to Ceph when encoding, and vice versa,
when decoding, resulting in Linux codes on the wire, and host codes on the receiver.
All CEPHFS_E* defines have been removed across Ceph (including the Python binding).
Relevant tracker: https://tracker.ceph.com/issues/64611
* RGW: Persistent bucket notifications will use queues with multiple shards instead of one queue. Number of shards
can be configured using the `rgw` option `rgw_bucket_persistent_notif_num_shards`. Note that pre-existing topics will continue to function as is, i.e, they are mapped to only one RADOS object.
For more details, see:
https://docs.ceph.com/en/latest/radosgw/notifications/
Relevant tracker: https://tracker.ceph.com/issues/71677
* RGW: Multipart upload part heads are now tracked under a new
``rgw.multipart`` category in bucket stats, separate from completed
objects (``rgw.main``). This makes ``radosgw-admin bucket stats``
and the ``X-RGW-Object-Count`` header on HEAD Bucket more accurate:
in-progress or abandoned multipart parts no longer inflate the
object count. Existing in-progress uploads retain the old category
until they complete or abort.
Relevant tracker: https://tracker.ceph.com/issues/77509
* RGW: The new ``rgw_topic_ownership_update_period`` option sets how often a
radosgw refreshes the persistent topic queue list. The ownership (failover)
timeout of a persistent topic queue is three times this value, so lowering it
speeds up failover of persistent topics between gateways. Setting it too low
may cause false-positive failures. Changing it affects the ownership of an
existing persistent topic queue only on its next renewal. It defaults to 30
seconds, which matches the previously hardcoded behaviour.
Relevant tracker: https://tracker.ceph.com/issues/78727
* RADOS: A new ``BLUESTORE_NO_DB_SHARDING`` health warning reports OSDs whose
RocksDB database does not use column family sharding, i.e. OSDs created
prior to Pacific that have not been resharded yet with
``ceph-bluestore-tool reshard``. This warning can be disabled with the new
``bluestore_warn_on_no_db_sharding`` option.
* RADOS: A new ``BLUESTORE_LEGACY_MIN_ALLOC_SIZE`` health warning reports
OSDs that use an allocation unit (``min_alloc_size``) that differs from
the one a newly deployed OSD would use on the same device, e.g. HDD-backed
OSDs deployed prior to the Pacific release with a 64 KiB allocation unit.
It can be disabled with the new ``bluestore_warn_on_legacy_min_alloc_size``
option, globally or per OSD / device class using a configuration mask.