Integrating Kudu Prometheus Metrics with Grafana, A Step-by-Step Guide

2026-07-03T00:00:00+00:00

Introduction

With the completion of KUDU-3691, Apache Kudu now offers full Prometheus metrics support - making it straightforward to integrate Kudu monitoring into Grafana. This guide covers the basics, from installing Prometheus and Grafana to configuring them. It provides a practical approach to crafting Grafana dashboards for a better understanding of Apache Kudu.

Full Prometheus metrics support is available on the master branch and will ship with the upcoming Kudu 1.19 release.

Integration Benefits

The built-in Kudu web server exposes metrics as a point-in-time snapshot per daemon. Prometheus adds historical retention, cross-node aggregation, and alerting - Grafana provides the visualization layer on top. Together they allow correlating Kudu metrics with the rest of the infrastructure in a single place.

Prerequisites

Ensure that you have the following:

A running Apache Kudu cluster: At least one master and one tablet server. See the Kudu quickstart guide if you need to set one up.
Prometheus: Follow the official Prometheus installation guide to set up Prometheus on your system.
Grafana (optional): Install Grafana by following the official Grafana documentation if you want to visualize the collected metrics.

Prometheus Configuration for Kudu

Open the Prometheus configuration file, typically located at /etc/prometheus/prometheus.yml. If using a custom location, specify it accordingly.

Add the following snippet to the file to configure Prometheus to scrape Kudu metrics:

 global:
   scrape_interval: 15s
   evaluation_interval: 15s

 scrape_configs:
   - job_name: "kudu"
     metrics_path: "/metrics_prometheus"
     static_configs:
       - targets: ["master-1:8051", "master-2:8051", "master-3:8051"]
         group: "masters"
       - targets: ["tserver-1:8050", "tserver-2:8050", "tserver-3:8050"]
         group: "tservers"

Adjust the targets to match the Kudu instances in your environment. For more configuration options, refer to the Prometheus documentation.
Start Prometheus, if using a custom config location, use the config.file command line flag, for example:
```
 $ prometheus --config.file="<custom_path>/prometheus.yml"
```
Verify that Prometheus is running on localhost:9090 and check the /targets endpoint to ensure all Kudu services are up.

Grafana Setup

Start Grafana, which runs by default on port 3000.
Configure Prometheus as a data source in Grafana:
- Follow the steps outlined in the Grafana documentation to add Prometheus as a data source.
- Set the Prometheus URL to the Prometheus server (e.g., http://localhost:9090).
- Save and test the connection.
For a comprehensive list of available metrics in Kudu, please refer to the official metrics reference. Kudu Prometheus metrics all have the “kudu_” prefix.
Create a Kudu Dashboard:
- Navigate to the “+” menu on the left sidebar and select “Dashboard”.
- Add a new panel, select Prometheus as the data source, and use PromQL queries to create visualizations based on Kudu metrics.

Working with a large number of TServers

For clusters with many tablet servers, manually listing every target in prometheus.yml is impractical and breaks when nodes are added or removed. Kudu masters expose an HTTP service discovery endpoint at /prometheus-sd that Prometheus can poll directly, returning all registered masters and tablet servers automatically.

The endpoint:

Is served only by the leader master (non-leaders return an empty response with HTTP 200)
Returns one target per server with labels: group (masters/tservers), cluster_id, location, and __scheme__ (http/https)
Automatically excludes presumed-dead tablet servers
Supports authentication via the --webserver_prometheus_token_cmd flag

Example Prometheus config using http_sd_configs:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "kudu"
    metrics_path: "/metrics_prometheus"
    http_sd_configs:
      - url: "http://master-1:8051/prometheus-sd"
      - url: "http://master-2:8051/prometheus-sd"
      - url: "http://master-3:8051/prometheus-sd"

By pointing at all masters, Prometheus discovers all targets regardless of which master is currently the leader. Non-leader masters return an empty list, so there is no duplication.

A sample response from the endpoint:

[
    {
        "targets": [
            "master-1:8051"
        ],
        "labels": {
            "group": "masters",
            "cluster_id": "fb61f15b7f184f009731f52a1b973bfa",
            "location": "n/a",
            "__scheme__": "http"
        }
    },
    {
        "targets": [
            "tserver-1:8050"
        ],
        "labels": {
            "group": "tservers",
            "cluster_id": "fb61f15b7f184f009731f52a1b973bfa",
            "location": "n/a",
            "__scheme__": "http"
        }
    },
    ...
]

For more details on HTTP service discovery configuration options, refer to the Prometheus documentation.

Capability Notes

Tablet-level metrics are available at the /metrics_prometheus endpoint since Kudu 1.18. The --metrics_prometheus_use_entity_labels flag controls how they are represented in the output:

false (default): Entity IDs are embedded in metric names (e.g. kudu_tablet_<tablet_id>_on_disk_data_size). This preserves backward compatibility with existing dashboards and alerting rules but prevents label-based aggregation in PromQL.
true (recommended for new deployments): Entity attributes appear as Prometheus labels on a shared metric name, enabling queries like sum by (table_name)(kudu_on_disk_data_size).

Apache Kudu 1.18.1 Released

2026-01-09T00:00:00+00:00

The Apache Kudu team is happy to announce the release of Kudu 1.18.1!

This maintenance release fixes critical bugs and packaging issues discovered in Apache Kudu 1.18.0. Users of Kudu 1.18.0 are encouraged to upgrade to 1.18.1 as soon as possible.

This release includes the following incremental updates since Apache Kudu 1.18.0:

Improved Java and Spark ecosystem reliability through fixes to dependency publishing, Spark class packaging, and data source discovery.
Enhanced TLS support for the embedded webserver, including configurable minimum TLS versions up to TLS 1.3 and control over TLSv1.3 cipher suites.
Upgraded Apache Spark support to Spark 3.5.
Increased stability of the CLI tools and the C++ client library, fixing crash in the kudu CLI binary upon exiting and crash in the C++ client library with verbose logging enabled. The latter was a regression introduced in the 1.18.0 release.
Improved compatibility with newer system libraries, including OpenSSL 3.4+ and platforms with 64KB memory pages.

The above is just a list of the highlights, for a more complete list of new features, improvements and fixes please refer to the release notes.

The Apache Kudu project only publishes source code releases. To build Kudu 1.18.1, follow these steps:

Download the Kudu 1.18.1 source release
Follow the instructions in the documentation to build Kudu 1.18.1 from source

For your convenience, binary JAR files for the Kudu Java client library, Spark DataSource, and other Java integrations are published to the ASF Maven repository and are now available.

The Python client source is also available on PyPI.

Additionally, experimental Docker images are published to Docker Hub.

Introducing REST API for metadata management to Apache Kudu

2025-11-10T00:00:00+00:00

Apache Kudu has long provided client APIs in C++, Java, and Python for building client applications. Today, we’re excited to announce the introduction of a REST API for table DDL operations that makes Kudu even more accessible to developers and integration tools.

This feature is available in the master branch and will be released with Kudu 1.19.0.

Overview

The Apache Kudu REST API exposes table metadata and administrative operations over HTTP on top of Kudu’s system catalog. It lets users handle these tasks with lightweight REST calls instead of building a full client application that depends on the Kudu client library and uses KRPC to interact with Kudu clusters.

Key Features

DDL Operations: Create, read, update, and delete tables through HTTP endpoints
JSON-based: All requests and responses use JSON format for easy parsing
RESTful Design: Follows standard HTTP methods and status codes
Kerberos/SPNEGO Authentication: Integrates with Kudu’s existing authentication mechanisms

API Overview

The REST API is available on the master’s webserver at /api/v1/ and provides endpoints for table DDL operations. The API supports standard HTTP methods (GET, POST, PUT, DELETE) and returns JSON responses.

Available Endpoints

GET /api/v1/tables - List all tables
POST /api/v1/tables - Create a new table
GET /api/v1/tables/<table_id> - Get table details
PUT /api/v1/tables/<table_id> - Update table metadata
DELETE /api/v1/tables/<table_id> - Delete a table
GET /api/v1/leader - Discover the current leader master

Example Request

GET /api/v1/tables/<table_id>

curl http://master-host:8051/api/v1/tables/a2810622a25b4a3e8ce0be3ece103c50

For secure clusters with Kerberos/SPNEGO authentication enabled, include authentication flags in the curl command (e.g., --negotiate -u :).

Response:

{
  "name": "example_table",
  "id": "a2810622a25b4a3e8ce0be3ece103c50",
  "schema": {
    "columns": [
      {
        "id": 10,
        "name": "column_name",
        "type": "INT8",
        "is_key": true,
        "is_nullable": false,
        "encoding": "AUTO_ENCODING",
        "compression": "DEFAULT_COMPRESSION",
        "cfile_block_size": 0,
        "immutable": false
      }
    ]
  },
  "partition_schema": {
    "hash_schema": [
      {
        "columns": [
          {
            "id": 10
          }
        ],
        "num_buckets": 2,
        "seed": 0
      }
    ],
    "range_schema": {
      "columns": [
        {
          "id": 10
        }
      ]
    }
  },
  "owner": "default",
  "comment": "",
  "extra_config": {}
}

For complete API documentation, the OpenAPI specification is available in the Kudu repository. On a Kudu cluster running this feature, the API documentation is accessible at http://master-host:8051/api/docs:

Limitations

Multi-Master Setup

In multi-master Kudu clusters, only the leader master will successfully process REST API requests. Non-leader masters will respond with an HTTP 503 (Service Unavailable) error. Since the leader master can change at any time due to failures or cluster reconfigurations, applications should handle 503 errors by querying the /api/v1/leader endpoint to discover the current leader and retrying the request against the new leader master’s address.

Metadata Only

This REST API focuses exclusively on table DDL operations. Data read/write operations are not supported through this interface and should continue to use Kudu’s native client APIs.

Configuration

The REST API must be explicitly enabled using the --enable_rest_api flag on Kudu masters. The master’s webserver must also be enabled (via the --webserver_enabled flag) since the REST API is served through the webserver.

References

OpenAPI Specification: https://swagger.io/specification/
Kudu REST API OpenAPI specification: https://github.com/apache/kudu/blob/master/www/swagger/kudu-api.json
Kudu C++ Client API documentation: https://kudu.apache.org/cpp-client-api/index.html
Kudu Java Client API documentation: https://kudu.apache.org/apidocs/index.html

Apache Kudu 1.18.0 Released

2025-07-10T00:00:00+00:00

The Apache Kudu team is happy to announce the release of Kudu 1.18.0!

The new release adds several new features and improvements, including the following:

Segmented LRU Block Cache (Experimental): Helps protect hot data from eviction during large scans.
Embedded RocksDB for Metadata (Experimental): A new option to store metadata in embedded RocksDB for better scalability.
Auto-Incrementing Columns Enhancements: These columns are now correctly restored on backup/restore and initialized on bootstrap.
Improved Metrics & Observability: Added metrics for timeouts, tablet scans, RPC backlogs, and more as well as Prometheus support for tablet-level metrics.
Build and Platform Updates: Kudu now supports AArch64 Ubuntu/RedHat, Ubuntu24.04 and builds on macOS Sonoma (Xcode 15).
Java & Python Client Improvements:
- Python: simplified installation, UPSERT IGNORE, soft-deletes, immutable columns
- Java: configurable buffer size, better schema update handling
Security Enhancements:
- Improvements in JWT authentication
- Dedicated SPNEGO keytab for embedded webserver
- HTTP headers for enhanced security of the embedded webserver
Kudu CLI & Tooling Improvements: Better error messages and handling in kudu table copy, scan, and others.

The above is just a list of the highlights, for a more complete list of new features, improvements and fixes please refer to the release notes.

The Apache Kudu project only publishes source code releases. To build Kudu 1.18.0, follow these steps:

Download the Kudu 1.18.0 source release
Follow the instructions in the documentation to build Kudu 1.18.0 from source

For your convenience, binary JAR files for the Kudu Java client library, Spark DataSource, and other Java integrations are published to the ASF Maven repository and are now available.

The Python client source is also available on PyPI.

Additionally, experimental Docker images are published to Docker Hub.

Introducing the Externalized Kudu Connector

2025-05-13T00:00:00+00:00

Note: This is a cross-post from the Apache Flink blog Introducing the Externalized Kudu Connector

We are pleased to announce the revival of a connector that makes it possible for Flink to interact with Apache Kudu. The original connector existed as part of the Apache Bahir project, which was moved into the attic. Despite this, we saw interest to keep the Kudu connector updated, hence the community agreed to externalize it as a standalone connector in accordance with the current connector development model. For more information about the externalization process, see FLIP-439.

Highlights

The connector is built on the already existing Apache Bahir code.
The existing DataStream connector is updated to Sink V2 API.
New DataStream Source API connector implementation.
The Table API source and sink connectors are now using the new Schema stack.
The first released connector version is 2.0.0, and it supports Flink 1.19, and 1.20.

DataStream Source API

The Source API implementation is a net new addition to the externalized connector, and it brings some interesting features. Although Kudu itself is a bounded source, the Kudu Source implementation supports to configure boundedness, and can run in CONTINUOUS_UNBOUNDED mode. In CONTINUOUS_UNBOUNDED mode, the source operates similarly to a Change Data Capture (CDC) system. When the job starts, it takes a snapshot of the source table and records the snapshot timestamp. After that, it performs periodic differential scans, capturing only the changes that occurred since the last scan. The frequency of these scans is determined by the .setDiscoveryPeriod(Duration) setting. The following example demonstrates how to stream data from a Kudu table, capturing updates every one minute.

KuduSource<Row> source =
        KuduSource.<Row>builder()
                .setTableInfo(...)
                .setReaderConfig(...)
                .setRowResultConverter(new RowResultRowConverter())
                .setBoundedness(Boundedness.CONTINUOUS_UNBOUNDED)
                .setDiscoveryPeriod(Duration.ofMinutes(1))
                .build();

For more details and examples, see the DataStream connector documentation

Table API Catalog

The connector includes a catalog implementation designed to manage metadata for your Kudu setup and facilitate table operations. With the Kudu catalog, you can access all existing Kudu tables directly through Flink SQL queries. Such catalog can be defined in Flink SQL, as part of the Java application, or via a YAML catalog descriptor as well. The below example shows a minimal example in Filnk SQL.

CREATE CATALOG my_kudu_catalog WITH(
    'type' = 'kudu',
    'masters' = 'localhost:7051',
    'default-database' = 'default_database'
);

USE CATALOG my_kudu_catalog;

For other Table API related topics and examples, see the Table API connector documentation

Release Notes

Sub-task

[FLINK-34929] - Create “flink-connector-kudu” repository
[FLINK-34930] - Move existing Kudu connector code from Bahir repo to dedicated repo
[FLINK-34931] - Update Kudu DataStream connector to use Sink V2
[FLINK-35114] - Remove old Table API implementations, update Schema stack
[FLINK-35350] - Add documentation for Kudu connector
[FLINK-37389] - Add “flink-sql-connector-kudu” module

New Feature

[FLINK-36855] - Implement Source API in Kudu connector
[FLINK-37527] - Add KuduSource documentation
[FLINK-37664] - Integrate Kudu connector docs

Improvement

[FLINK-36839] - Update Kudu version to 1.17.1
[FLINK-37190] - Make Kudu FlushMode configurable in Flink SQL
[FLINK-37230] - Consolidate Kudu connector table options
[FLINK-37237] - Improve Kudu table creation based on Flink SQL CREATE TABLE

List of Contributors

Ferenc Csaky, Martijn Visser, Marton Greber

Resources

Connector GitHub repository: https://github.com/apache/flink-connector-kudu

Maven Central link: https://central.sonatype.com/artifact/org.apache.flink/flink-connector-kudu

Apache Kudu 1.17.1 Released

2024-11-13T00:00:00+00:00

The Apache Kudu team is happy to announce the release of Kudu 1.17.1!

This maintenance release fixes critical bugs and build issues discovered in Apache Kudu 1.17.0. Users of Kudu 1.17.0 are encouraged to upgrade to 1.17.1 as soon as possible.

The changes include the following:

Upgraded several thirdparty components either to address CVEs or fix build issues discovered with contemporary releases.
Fixed file descriptor leak in encryption-at-rest (see KUDU-3520).
Fixed bug in range-aware tablet replica placement causing Kudu master to crash (see KUDU-3532).

The above is just a few of the fixes, for a more complete list of improvements and fixes please refer to the release notes.

The Apache Kudu project only publishes source code releases. To build Kudu 1.17.1, follow these steps:

Download the Kudu 1.17.1 source release
Follow the instructions in the documentation to build Kudu 1.17.1 from source

For your convenience, binary JAR files for the Kudu Java client library, Spark DataSource, Flume sink, and other Java integrations are published to the ASF Maven repository and are now available.

The Python client source is also available on PyPI.

Additionally, experimental Docker images are published to Docker Hub.

Introducing Auto-incrementing Column in Kudu

2024-03-07T00:00:00+00:00

Introduction

Kudu has a strict requirement for a primary key presence in a table. This is primarily to help in point lookups and support DELETE and UPDATE operations on the table data. There are situations where users are unable to define a unique primary key in their data set and have to either introduce additional columns to be a part of the primary key or define a new column and maintain it to enforce uniqueness. Kudu 1.17 has introduced support for the auto-incrementing column to have partially defined primary keys (keys which are not unique across the table) during table creation. This way a user does not have to worry about the uniqueness constraint when defining a primary key.

Implementation Details

When a primary key is partially defined, Kudu internally creates a new column named “auto_incrementing_id” as a part of the primary key. The column is populated with a monotonically increasing counter. The system updates the counter value upon every INSERT operation and populates the “auto_incrementing_id” column on the server side. The counter is partition-local i.e. every tablet has its own counter.

Server Side

When a user writes data into a table with the auto-incrementing column, the server makes sure that no INSERT operations have the “auto_incrementing_id” column field value set and populates this column value. The highest value of the counter written into the “auto_incrementing_id” column until any particular point is stored in memory and this is used to set the column value for the next INSERT operation.

Client Side

When creating a table without an explicitly defined primary key, users will have to declare the key as non-unique. Internally, the client builds a schema with an extra column named “auto_incrementing_id” and forwards the request to the server where the table is created. For INSERT operations, the user shouldn’t specify the “auto_incrementing_id” column value as it will be populated on the server side.

Impala Integration

In Impala, the new column is not exposed to the user by default. This is due to the reason that it is not a part of the user table schema. The below query will not return the “auto_incrementing_id” column SELECT * FROM <tablename>

If the auto-incrementing column’s data is needed, the column name has to be specifically requested. The below query will return the column values: SELECT *, auto_incrementing_id FROM <tablename>

Examples

Create a table with two columns and two hash partitions:

default> CREATE TABLE demo_table(id INT NON UNIQUE PRIMARY KEY, name STRING) PARTITION BY HASH (id) PARTITIONS 2 STORED AS KUDU;
Query: CREATE TABLE demo_table(id INT NON UNIQUE PRIMARY KEY, name STRING) PARTITION BY HASH (id) PARTITIONS 2 STORED AS KUDU
+-------------------------+
| summary                 |
+-------------------------+
| Table has been created. |
+-------------------------+
Fetched 1 row(s) in 3.94s

Describe the table:

default> DESCRIBE demo_table;
Query: DESCRIBE demo_table
+----------------------+--------+---------+-------------+------------+----------+---------------+---------------+---------------------+------------+
| name                 | type   | comment | primary_key | key_unique | nullable | default_value | encoding      | compression         | block_size |
+----------------------+--------+---------+-------------+------------+----------+---------------+---------------+---------------------+------------+
| id                   | int    |         | true        | false      | false    |               | AUTO_ENCODING | DEFAULT_COMPRESSION | 0          |
| auto_incrementing_id | bigint |         | true        | false      | false    |               | AUTO_ENCODING | DEFAULT_COMPRESSION | 0          |
| name                 | string |         | false       |            | true     |               | AUTO_ENCODING | DEFAULT_COMPRESSION | 0          |
+----------------------+--------+---------+-------------+------------+----------+---------------+---------------+---------------------+------------+

Insert rows with duplicate partial primary key column values:

default> INSERT INTO demo_table VALUES (1, 'John'), (2, 'Bob'), (3, 'Mary'), (1, 'Joe');
Query: INSERT INTO demo_table VALUES (1, 'John'), (2, 'Bob'), (3, 'Mary'), (1, 'Joe')
..
Modified 4 row(s), 0 row error(s) in 0.41s

Scan the table (notice the duplicate values in the ‘id’ column):

default> SELECT * FROM demo_table;
Query: SELECT * FROM demo_table
..
+----+------+
| id | name |
+----+------+
| 3  | Mary |
| 1  | John |
| 1  | Joe  |
| 2  | Bob  |
+----+------+
Fetched 4 row(s) in 0.24s

Explicitly specify the auto-incrementing column name to fetch the column values:

default> SELECT *, auto_incrementing_id FROM demo_table;
Query: SELECT *, auto_incrementing_id FROM demo_table
..
+----+------+----------------------+
| id | name | auto_incrementing_id |
+----+------+----------------------+
| 3  | Mary | 1                    |
| 1  | John | 1                    |
| 1  | Joe  | 2                    |
| 2  | Bob  | 2                    |
+----+------+----------------------+
Fetched 4 row(s) in 0.24s

Update and Delete rows:

default> UPDATE demo_table SET name='Matt' WHERE id=3;
Query: UPDATE demo_table SET name='Matt' WHERE id=3
Modified 1 row(s), 0 row error(s) in 1.99s
default> UPDATE demo_table SET name='Liam' WHERE id=1;
Query: UPDATE demo_table SET name='Liam' WHERE id=1
Modified 2 row(s), 0 row error(s) in 2.15s
default> DELETE FROM demo_table where id=2;
Query: DELETE FROM demo_table where id=2;
Modified 1 row(s), 0 row error(s) in 1.40s

Scan all the columns of the table:

default> SELECT *, auto_incrementing_id FROM demo_table;
Query: SELECT *, auto_incrementing_id FROM demo_table
..
+----+------+----------------------+
| id | name | auto_incrementing_id |
+----+------+----------------------+
| 3  | Matt | 1                    |
| 1  | Liam | 1                    |
| 1  | Liam | 2                    |
+----+------+----------------------+
Fetched 3 row(s) in 0.20s

Limitations

Impala doesn’t support UPSERT operations on tables with the auto-incrementing column as of writing this article.

Kudu clients (Java, C++, Python)

Unlike in Impala, scanning the table fetches all the table data including the auto incrementing column. There is no need to explicitly request the auto-incrementing column.

There is also support for UPSERT operations but the user has to provide the entire primary key including the value for the auto-incrementing column. If the row is present, it will be considered a regular UPDATE operation. If the row is not present, it is considered an INSERT operation.

Examples

https://github.com/apache/kudu/blob/master/examples/cpp/non_unique_primary_key.cc

https://github.com/apache/kudu/blob/master/examples/python/basic-python-example/non_unique_primary_key.py

Backup and Restore

The Kudu backup tool from Kudu 1.17 and later supports backing up tables with the auto-incrementing column. The prior backup tools will fail with an error message - “auto_incrementing_id is a reserved column name” during backup and restore operations.

The backed up data (from Kudu 1.17 and later) includes the auto-incrementing column in the table schema and the column values as well. Restoring this backed up table with the Kudu restore tool will create a table with the auto-incrementing column and the column values identical to the original source table.

Apache Kudu 1.17.0 Released

2023-09-07T00:00:00+00:00

The Apache Kudu team is happy to announce the release of Kudu 1.17.0!

The new release adds several new features and improvements, including the following:

Kudu now supports encrypting data at rest. Kudu supports AES-128-CTR, AES-192-CTR, and AES-256-CTR ciphers to encrypt data, supports Apache Ranger KMS and Apache Hadoop KMS. See Data at rest for more details.
Kudu now supports range-specific hash schemas for tables. It’s now possible to add ranges with their own unique hash schema independent of the table-wide hash schema. This can be done at table creation time and while altering the table. It’s controlled by the --enable_per_range_hash_schemas master flag which is enabled by default (see KUDU-2671).
Kudu now supports soft-deleted tables. Kudu keeps a soft-deleted table aside for a period of time (a.k.a. reservation), not purging the data yet. The table can be restored/recalled back before its reservation expires. The reservation period can be customized via Kudu client API upon soft-deleting the table. The default reservation period is controlled by the --default_deleted_table_reserve_seconds master’s flag. NOTE: As of Kudu 1.17 release, the soft-delete functionality is not supported when HMS integration is enabled, but this should be addressed in a future release (see KUDU-3326).
Introduced Auto-Incrementing column. An auto-incrementing column is populated on the server side with a monotonically increasing counter. The counter is local to every tablet, i.e. each tablet has a separate auto incrementing counter (see KUDU-1945).
Kudu now supports experimental non-unique primary key. When a table with non-unique primary key is created, an Auto-Incrementing column named auto_incrementing_id is added automatically to the table as the key column. The non-unique key columns and the Auto-Incrementing column together form the effective primary key (see KUDU-1945).
Introduced Immutable column. It’s useful to represent a semantically constant entity (see KUDU-3353).
An experimental feature is added to Kudu that allows it to automatically rebalance tablet leader replicas among tablet servers. The background task can be enabled by setting the --auto_leader_rebalancing_enabled flag on the Kudu masters. By default, the flag is set to ‘false’ (see KUDU-3390).
Introduced an experimental feature: authentication of Kudu client applications to Kudu servers using JSON Web Tokens (JWT). The JWT-based authentication can be used as an alternative to Kerberos authentication for Kudu applications running at edge nodes where configuring Kerberos might be cumbersome. Similar to Kerberos credentials, a JWT is considered a primary client’s credentials. The server-side capability of JWT-based authentication is controlled by the --enable_jwt_token_auth flag (set ‘false’ by default). When the flat set to ‘true’, a Kudu server is capable of authenticating Kudu clients using the JWT provided by the client during RPC connection negotiation. From its side, a Kudu client authenticates a Kudu server by verifying its TLS certificate. For the latter to succeed, the client should use Kudu client API to add the cluster’s IPKI CA certificate into the list of trusted certificates.
The C++ client scan token builder can now create multiple tokens per tablet. So, it’s now possible to dynamically scale the set of readers/scanners fetching data from a Kudu table in parallel. To use this functionality, use the newly introduced SetSplitSizeBytes() method of the Kudu client API to specify how many bytes of data each token should scan (see KUDU-3393).
Kudu’s default replica placement algorithm is now range and table aware to prevent hotspotting unlike the old power of two choices algorithm. New replicas from the same range are spread evenly across available tablet servers, the table the range belongs to is used as a tiebreaker (see KUDU-3476).
Reduce the memory consumption if there are frequent alter schema operations for tablet servers (see KUDU-3197).
Reduce the memory consumption by implementing memory budgeting for performing RowSet merge compactions (i.e. CompactRowSetsOp maintenance operations). Several flags have been introduced, while the --rowset_compaction_memory_estimate_enabled flag indicates whether to check for available memory necessary to run CompactRowSetsOp maintenance operations (see KUDU-3406).

The above is just a list of the highlights, for a more complete list of new features, improvements and fixes please refer to the release notes.

The Apache Kudu project only publishes source code releases. To build Kudu 1.17.0, follow these steps:

Download the Kudu 1.17.0 source release
Follow the instructions in the documentation to build Kudu 1.17.0 from source

For your convenience, binary JAR files for the Kudu Java client library, Spark DataSource, and other Java integrations are published to the ASF Maven repository and are now available.

The Python client source is also available on PyPI.

Additionally, experimental Docker images are published to Docker Hub.

Apache Kudu 1.16.0 Released

2022-06-17T00:00:00+00:00

The Apache Kudu team is happy to announce the release of Kudu 1.16.0!

The new release adds several new features and improvements, including the following:

Kudu has a new /startup page on the web UI where admins can easily track startup progress of a Kudu server.
It is now possible to change the replication of an existing table, and to require a minimum replication factor across a Kudu cluster.

The above is just a list of the highlights, for a more complete list of new features, improvements and fixes please refer to the release notes.

The Apache Kudu project only publishes source code releases. To build Kudu 1.16.0, follow these steps:

Download the Kudu 1.16.0 source release
Follow the instructions in the documentation to build Kudu 1.16.0 from source

For your convenience, binary JAR files for the Kudu Java client library, Spark DataSource, Flume sink, and other Java integrations are published to the ASF Maven repository and are now available.

The Python client source is also available on PyPI.

Additionally, experimental Docker images are published to Docker Hub.

Apache Kudu 1.15.0 Released

2021-06-22T00:00:00+00:00

The Apache Kudu team is happy to announce the release of Kudu 1.15.0!

The new release adds several new features and improvements, including the following:

Kudu now experimentally supports multi-row transactions. Currently only INSERT and INSERT_IGNORE operations are supported. See here for a design overview of this feature.
Kudu now supports Raft configuration change for Kudu masters and CLI tools for orchestrating addition and removal of masters in a Kudu cluster. These tools substantially simplify the process of migrating to multiple masters, recovering a dead master and removing masters from a Kudu cluster. For detailed steps, see the latest administration documentation. This feature is evolving and the steps to add, remove and recover masters may change in the future. See KUDU-2181 for details.
Kudu now supports table comments directly on Kudu tables which are automatically synchronized when the Hive Metastore integration is enabled. These comments can be added at table creation time and changed via table alteration.
Kudu now experimentally supports per-table size limits based on leader disk space usage or number of rows. When generating new authorization tokens, Masters will now consider the size limits and strip tokens of INSERT and UPDATE privileges if either limit is reached. To enable this feature, set the --enable_table_write_limit master flag; adjust the --table_disk_size_limit and --table_row_count_limit flags as desired or use the kudu table set_limit tool to set limits per table.
It is now possible to change the Kerberos Service Principal Name using the --principal flag. The default SPN is still kudu/_HOST. Clients connecting to a cluster using a non-default SPN must set the sasl_protocol_name or saslProtocolName to match the SPN base (i.e. “kudu” if the SPN is “kudu/_HOST”) in the client builder or the Kudu CLI. See KUDU-1884 for details.
Kudu RPC now supports TLSv1.3. Kudu servers and clients automatically negotiate TLSv1.3 for Kudu RPC if OpenSSL (or Java runtime correspondingly) on each side supports TLSv1.3. If necessary, use the newly introduced flag --rpc_tls_ciphersuites to customize TLSv1.3-specific cipher suites at the server side. See KUDU-2871 for details.

The above is just a list of the highlights, for a more complete list of new features, improvements and fixes please refer to the release notes.

The Apache Kudu project only publishes source code releases. To build Kudu 1.15.0, follow these steps:

Download the Kudu 1.15.0 source release
Follow the instructions in the documentation to build Kudu 1.15.0 from source

For your convenience, binary JAR files for the Kudu Java client library, Spark DataSource, Flume sink, and other Java integrations are published to the ASF Maven repository and are now available.

The Python client source is also available on PyPI.

Additionally, experimental Docker images are published to Docker Hub, including for AArch64-based architectures (ARM).