Skip to main content
Skip to main content
Edit this page

Creating your first Kafka ClickPipe

In this guide, we will walk you through the process of creating your first Kafka ClickPipe.

Navigate to data sources​

Select the Data Sources button on the left-side menu and click on "Set up a ClickPipe".

Select a data source​

Select your Kafka data source from the list.

Configure the data source​

Fill out the form by providing your ClickPipe with a name, a description (optional), your credentials, and other connection details.

Configure a schema registry (optional)​

A valid schema is required for Avro streams. See Schema registries for more details on how to configure a schema registry.

Configure a reverse private endpoint (optional)​

Configure a Reverse Private Endpoint to allow ClickPipes to connect to your Kafka cluster using AWS PrivateLink. See our AWS PrivateLink documentation for more information.

Configure SSH tunneling (optional)​

You can use SSH tunneling if your Kafka broker isn't publicly accessible. Instead of connecting directly, ClickPipes establishes an SSH connection to a bastion host (a server in your network that is publicly accessible) and then forwards traffic through it to your Kafka broker on your private network.

  1. Enable the "SSH Tunnel" toggle.
  2. Fill in the SSH connection details:
    • SSH Host: The hostname or IP address of your bastion host — this is the publicly accessible server that acts as a gateway into your private network.
    • SSH Port: The port for SSH on the bastion host (default 22).
    • SSH User: The username to authenticate with on the bastion host.
  1. To use Key-based authentication, click on "Revoke and regenerate key pair" to generate a new key pair and copy the generated public key to your SSH server under ~/.ssh/authorized_keys.
  2. Click on "Verify Connection" to verify the connection.
Note

Make sure to whitelist ClickPipes IP addresses in your firewall rules for the SSH bastion host so that ClickPipes can establish the SSH tunnel.

Select your topic​

Select your topic and the UI will display a sample document from the topic.

Configure your destination table​

In the next step, you can select whether you want to ingest data into a new ClickHouse table or reuse an existing one. Follow the instructions in the screen to modify your table name, schema, and settings. You can see a real-time preview of your changes in the sample table at the top.

You can also customize the advanced settings using the controls provided

Configure permissions​

ClickPipes will create a dedicated user for writing data into a destination table. You can select a role for this internal user using a custom role or one of the predefined role:

  • Full access: with the full access to the cluster. It might be useful if you use Materialized View or Dictionary with the destination table.
  • Only destination table: with the INSERT permissions to the destination table only.

Complete setup​

Clicking on "Create ClickPipe" will create and run your ClickPipe. It will now be listed in the Data Sources section.