Skip to main content
Connector Type: Input Connector Description Ingests messages from Google Cloud Pub/Sub subscriptions into Condense for processing and transformation. Brief Description The Google Cloud Pub/Sub Stream Input Connector enables Condense to consume messages from a Google Cloud Pub/Sub subscription and inject them into a real-time Kafka-native stream for further processing. This connector is critical for organizations using GCP infrastructure who wish to bridge Google-native data pipelines into real-time streaming use cases. Pub/Sub acts as a high-throughput, asynchronous messaging service that decouples senders and receivers. This connector allows Condense to tap into those streams, ingesting messages from Pub/Sub topics via secure, authenticated access using a service account. Links to Relevant Documents Connector Overview This connector uses the Streaming Pull model of Pub/Sub, enabling Condense to receive events as they arrive without polling. It authenticates using a Google Cloud Service Account, reads from the specified subscription, and forwards all messages into a configured Condense topic for real-time processing, enrichment, or routing. Typical use cases include: A. Streaming IoT or device telemetry from Pub/Sub into Condense B. Ingesting cloud-native events from GCP services (Cloud Functions, App Engine, etc.) C. Migrating from GCP pipelines to Kafka-native applications D. Processing real-time alerts, logs, or change events from GCP sources Core Functionality
  1. Streaming Ingestion from Google Cloud Pub/Sub using subscription ID
  2. Secure Auth via Service Account JSON keys
  3. Auto-Ack mechanism with offset management handled by Pub/Sub
  4. High-Throughput Pull Model using native GCP APIs
  5. Automatic Scaling aligned with Condense pipeline performance
How it Works in Condense
  1. Authentication: Condense uses a Service Account JSON file to authenticate with GCP.
  2. Subscription Connection: The connector connects to the specified Pub/Sub subscription.
  3. Message Ingestion: Messages are pulled continuously via the streaming pull API.
  4. Forwarding: Each message is published to a Kafka-native Condense topic in real-time.
  5. Pipeline Processing: Downstream Condense transforms or applications can process the events.
Configuration Field-by-Field Explanation 1. Title Purpose: Used to track, audit, and manage this connector instance. Example: gcp-pubsub-ingestor, alert-stream-from-gcp 2. Subscription Name Format: projects/{project_id}/subscriptions/{subscription_name} Only the subscription_name is needed. Note: The subscription must already be created and accessible by the service account. 3. Project ID Definition: The unique GCP project identifier (not the name). Example: A. zeliot-platform B. iot-alerts-prod Purpose: Used to scope API calls and locate the Pub/Sub subscription. 4. Service Account Key Contents: Raw JSON key for a GCP Service Account with Pub/Sub Subscriber role. How to obtain: A. Go to IAM & Admin → Service Accounts B. Create or select a service account C. Grant Pub/Sub Subscriber permission D. Generate a JSON key Security Note: This credential must be securely stored. Never expose it in logs. 5. Topic (Input) Definition: A Condense Kafka-native topic into which Pub/Sub messages will be published. Example: A. gcp-device-ingest B. raw-alerts-from-pubsub C. cloud-logs-stream Sample Configuration Example
Troubleshooting and Common Issues 1. Invalid Credentials A. Ensure the service account has proper permissions (Pub/Sub Subscriber). B. Verify the full JSON key is provided without format errors. 2. Permission Denied A. The service account must be added to the correct GCP project with the required IAM role. 3. No Messages Received A. Check if the subscription is correctly attached to the Pub/Sub topic. B. Confirm if messages are being published to the topic in GCP. 4. Message Duplication A. Pub/Sub guarantees at-least-once delivery. Downstream deduplication may be necessary. 5. API Quota Errors A. Monitor and increase GCP API quotas if the connector consumes at a high rate. Advanced Considerations 1. Message Acknowledgment A. Condense uses auto-acknowledgment after successful publishing to the Kafka topic. B. Failed publishing results in retry; no manual ack control is exposed. 2. Ordering Guarantees A. Pub/Sub does not guarantee message ordering unless explicitly configured with orderingKey and the subscription is set to enable message ordering. This connector does not expose ordering settings currently. 3. Load Management A. Condense auto-manages pull threads and concurrency. For high-throughput topics, use multiple connector instances. Best Practices
  1. Use Dedicated Service Accounts: Assign least-privilege IAM roles.
  2. Secure Key Management: Rotate keys regularly and store in secure vaults or environment secrets.
  3. Use Dead-Letter Topics in Pub/Sub: For resilient error handling in GCP.
  4. Benchmark with Real Load: Some latency may be observed with idle subscriptions.