Pub/Sub IAM permissions

You can sign up for the Private Preview waitlist here.

This article describes the GCP IAM permissions ClickPipes requires to authenticate with Google Cloud Pub/Sub, consume data from your topics, and grant access to the service account used by ClickPipes.

Prerequisites

To follow this guide, you will need:

An active ClickHouse Cloud service
A GCP project containing the Pub/Sub topic you want to ingest from
IAM permissions in that project to grant roles
For service account key authentication, IAM permissions to create service accounts and keys

Authentication model

ClickPipes for Pub/Sub supports two authentication methods:

Service account key: You create a Google service account and upload its JSON key in the ClickPipes UI or provide it through the API. ClickPipes encrypts the key at rest.
GCP workload identity (Beta): ClickPipes uses short-lived credentials for a ClickPipes-managed Google service account. No customer service account key is stored. See the GCP workload identity guide for requirements.

With either method, the service account is used to:

list and read topics in your project,
create and delete the managed subscription ClickPipes uses to consume messages,
consume messages from that subscription, and
(optionally) read native Pub/Sub schemas from the schema registry.

Required permissions

ClickPipes requires the following IAM permissions on the GCP project that owns the topic. They cover the full pipe lifecycle: discovery (topic listing, validation, sampling), subscription management, steady-state ingestion, and cleanup.

Topic access (discovery and validation)

Permission	Purpose
`pubsub.topics.list`	List available topics in the project during discovery
`pubsub.topics.get`	Validate topic existence and retrieve schema settings
`pubsub.topics.attachSubscription`	Required on the topic when creating a subscription against it

Subscription lifecycle (discovery and ingestion)

Permission	Purpose
`pubsub.subscriptions.create`	Create the managed subscription (`clickpipes-{pipeID}`) and ephemeral discovery subscriptions
`pubsub.subscriptions.get`	Health checks (every 60s), follower polling, subscription validation
`pubsub.subscriptions.delete`	Clean up ephemeral discovery subscriptions and delete the managed subscription on pipe deletion
`pubsub.subscriptions.consume`	`Receive()`, `Ack()`, `Nack()`, and seek-to-timestamp operations

Schema access (optional — only for native Avro/Protobuf topics)

Permission	Purpose
`pubsub.schemas.get`	Retrieve native schema definitions from the Pub/Sub schema registry

Predefined roles

Role	Sufficient?	Notes
`roles/pubsub.editor`	Yes	Covers all required permissions. Broadest option.
`roles/pubsub.subscriber`	No	Missing `topics.list`, `topics.attachSubscription`, `subscriptions.create`, `subscriptions.delete`, and `schemas.get`.
`roles/pubsub.viewer`	No	Read-only — no subscription management or consumption.
Custom role (recommended)	Yes	Use the seven core permissions above (plus optional `schemas.get`) for least-privilege access.

Setup

Create a custom role (recommended)

For least-privilege access, create a custom role with exactly the permissions ClickPipes needs. You can do this with the gcloud CLI:

gcloud iam roles create clickpipes.pubsub.ingestion \
  --project=YOUR_PROJECT_ID \
  --title="ClickPipes Pub/Sub Ingestion" \
  --description="Permissions required by ClickHouse ClickPipes to ingest from Pub/Sub" \
  --permissions=pubsub.topics.list,pubsub.topics.get,pubsub.topics.attachSubscription,pubsub.subscriptions.create,pubsub.subscriptions.get,pubsub.subscriptions.delete,pubsub.subscriptions.consume \
  --stage=GA

Or, in the GCP Console, go to IAM & Admin → Roles → Create role and add the permissions listed in Required permissions.

Optional permissionsAppend pubsub.schemas.get to the --permissions list if you ingest from topics that use native Pub/Sub Avro or Protobuf schemas. Leave it out otherwise to keep the role minimal.

If you prefer to skip the custom role, you can grant roles/pubsub.editor instead.

Use GCP workload identity (Beta)

Retrieve the ClickPipes-managed service account from the service context by following the GCP workload identity guide. Bind the custom role you created, or roles/pubsub.editor, to that service account:

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:CLICKPIPES_SERVICE_ACCOUNT" \
  --role="projects/YOUR_PROJECT_ID/roles/clickpipes.pubsub.ingestion"

Create the ClickPipe with OpenAPI or Terraform, with authentication set to SERVICE_ACCOUNT_WORKLOAD_IDENTITY. Do not provide a service account key with this authentication method.

Create a service account

For service account key authentication, create a dedicated service account for the ClickPipe:

gcloud iam service-accounts create clickpipes-pubsub \
  --project=YOUR_PROJECT_ID \
  --display-name="ClickPipes Pub/Sub Ingestion"

Grant the role to the service account

Bind the role you created (or roles/pubsub.editor) to the service account at the project level:

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:clickpipes-pubsub@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="projects/YOUR_PROJECT_ID/roles/clickpipes.pubsub.ingestion"

Create and download a service account key

Create a JSON key for the service account and download it locally:

gcloud iam service-accounts keys create clickpipes-pubsub-key.json \
  --iam-account=clickpipes-pubsub@YOUR_PROJECT_ID.iam.gserviceaccount.com

You will upload this clickpipes-pubsub-key.json file in the ClickPipes UI when creating the pipe.

Treat the key as a secretService account keys grant access to your GCP project. Store the file securely, do not commit it to source control, and rotate it periodically. ClickPipes encrypts the key at rest after upload.

Notes

pubsub.topics.attachSubscription is required on the topic resource, not the subscription. This is commonly missed when granting only subscription-level permissions.
If your topic does not use a native Pub/Sub schema (Avro or Protobuf), the pubsub.schemas.get permission is not needed.
Managed subscriptions are named clickpipes-{pipeID} with a 60s ack deadline, 7-day message retention, and message ordering enabled.
Ephemeral discovery subscriptions are named clickpipes-discovery-{uuid} with a 10s ack deadline, 10-minute retention, and a 24-hour auto-expiry TTL.
ClickPipes treats PermissionDenied and Unauthenticated errors as non-retryable — if a permission is missing, the pipe fails fast rather than retrying indefinitely.

ClickPipes for Streaming

ClickPipes for Object Storage

ClickPipes for CDC

Programmatic access

Security and access

Networking

Prerequisites

Authentication model

Required permissions

Topic access (discovery and validation)

Subscription lifecycle (discovery and ingestion)

Schema access (optional — only for native Avro/Protobuf topics)

Predefined roles

Setup

Create a custom role (recommended)

Use GCP workload identity (Beta)

Create a service account

Grant the role to the service account

Create and download a service account key

Notes

​Prerequisites

​Authentication model

​Required permissions

​Topic access (discovery and validation)

​Subscription lifecycle (discovery and ingestion)

​Schema access (optional — only for native Avro/Protobuf topics)

​Predefined roles

​Setup

​Create a custom role (recommended)

​Use GCP workload identity (Beta)

​Create a service account

​Grant the role to the service account

​Create and download a service account key

​Notes

Prerequisites

Authentication model

Required permissions

Topic access (discovery and validation)

Subscription lifecycle (discovery and ingestion)

Schema access (optional — only for native Avro/Protobuf topics)

Predefined roles

Setup

Create a custom role (recommended)

Use GCP workload identity (Beta)

Create a service account

Grant the role to the service account

Create and download a service account key

Notes