Patterns
Messaging & Events
Cloud Infrastructure Google Cloud Platform DevOps & SRE Architecture

Messaging & Events

Comprehensive guide to Cloud Pub/Sub and event-driven architecture on Google Cloud Platform

Messaging & Events

Google Cloud Platform provides robust messaging and event-driven services to build scalable, decoupled applications. This section covers asynchronous messaging patterns and event-driven architecture implementations. This guide provides everything from basic messaging concepts to advanced event-driven patterns.

Prerequisites

Before working with messaging and events on GCP, ensure you have:

  • A GCP project with appropriate permissions
  • Understanding of asynchronous programming concepts
  • Knowledge of event-driven architecture patterns
  • Familiarity with message queuing systems
  • Understanding of error handling and retry logic

Cloud Pub/Sub

Asynchronous messaging for event-driven architectures. Cloud Pub/Sub is a fully managed messaging service for event-driven architectures. It provides reliable, many-to-many, asynchronous messaging between applications. Features include at-least-once delivery, message ordering, message filtering, and integration with other GCP services.

Overview

Cloud Pub/Sub is designed for event-driven architectures and streaming analytics use cases. It follows the publish-subscribe pattern where publishers send messages to topics and subscribers receive messages from subscriptions.

Core Concepts

Topics

  • Named resource to which messages are published
  • Can have multiple subscriptions
  • Support for message ordering
  • Schema validation support

Subscriptions

  • Named resource representing a message stream
  • Can be push or pull type
  • Independent message acknowledgment
  • Dead-letter queue support

Messages

  • Payload data (up to 10MB)
  • Attributes (key-value pairs)
  • Ordering keys for ordered delivery
  • Publishing timestamps

Subscription Types

Pull Subscriptions

  • Subscribers actively pull messages
  • Manual message acknowledgment
  • Suitable for batch processing
  • Fine-grained control over message consumption

Push Subscriptions

  • Pub/Sub pushes messages to endpoint
  • Automatic message acknowledgment
  • Lower latency for real-time processing
  • Requires HTTP/HTTPS endpoint

Key Features

  • At-least-once Delivery: Guaranteed message delivery
  • Message Ordering: Preserve message order within ordering keys
  • Message Filtering: Subscribe to subsets of messages
  • Dead-letter Queues: Handle message processing failures
  • Exponential Backoff: Automatic retry with backoff
  • Global Availability: Multi-region deployment

Use Cases

  • Building event-driven architectures
  • Decoupling microservices
  • Implementing asynchronous processing pipelines
  • Real-time data streaming
  • Fan-out patterns for broadcast scenarios

Pros

  • Highly scalable and reliable
  • Decouples producers and consumers
  • Built-in message retention
  • Supports fan-out patterns

Cons

  • Message ordering complexity
  • Requires careful error handling
  • Additional monitoring needed
  • Learning curve for pub/sub patterns

Pub/Sub Setup

# Create topic
gcloud pubsub topics create my-topic

# Create pull subscription
gcloud pubsub subscriptions create my-subscription \
  --topic=my-topic

# Create push subscription
gcloud pubsub subscriptions create my-push-subscription \
  --topic=my-topic \
  --push-endpoint=https://my-app.example.com/push

# Publish message
gcloud pubsub topics publish my-topic \
  --message='{"key": "value"}'

Publishing Messages

# publisher.py
from google.cloud import pubsub_v1

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path("my-project", "my-topic")

def publish_message(data):
    future = publisher.publish(
        topic_path,
        data=data.encode("utf-8"),
        ordering_key="my-key"
    )
    print(f"Published message ID: {future.result()}")

publish_message('{"message": "Hello, Pub/Sub!"}')

Consuming Messages (Pull)

# subscriber.py
from google.cloud import pubsub_v1

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path("my-project", "my-subscription")

def callback(message):
    print(f"Received message: {message.data}")
    message.ack()

streaming_pull_future = subscriber.subscribe(
    subscription_path, 
    callback=callback
)

print("Listening for messages...")
try:
    streaming_pull_future.result()
except KeyboardInterrupt:
    streaming_pull_future.cancel()

Consuming Messages (Push)

# push_handler.py (Flask example)
from flask import Flask, request

app = Flask(__name__)

@app.route('/push', methods=['POST'])
def handle_push():
    message = request.get_json()
    envelope = message.get('message')
    if envelope:
        data = envelope.get('data')
        print(f"Received push message: {data}")
    return '', 204

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

Event-Driven Architecture

Build applications that respond to events asynchronously. The Event-Driven Architecture pattern builds applications that respond to events asynchronously using services like Cloud Pub/Sub, Eventarc, and Cloud Functions. This decouples producers and consumers, enables loose coupling, and provides scalability.

Architecture Components

Event Producers

  • Generate events based on state changes
  • Publish events to event bus
  • Don’t need to know about consumers
  • Can be services, applications, or IoT devices

Event Bus

  • Central event routing mechanism
  • Cloud Pub/Sub as the backbone
  • Event filtering and routing
  • Message retention and delivery

Event Consumers

  • Subscribe to relevant events
  • Process events asynchronously
  • Scale independently
  • Handle failures gracefully

Event Patterns

Event Notification

  • Simple event broadcasting
  • Multiple consumers for same event
  • Fire-and-forget semantics
  • Best for state change notifications

Event-Carried State Transfer

  • Events contain all relevant data
  • Consumers don’t need to query source
  • Reduces coupling between services
  • Higher message size

Event Sourcing

  • Store all state changes as events
  • Rebuild state from event stream
  • Excellent audit trail
  • Complex to implement

CQRS (Command Query Responsibility Segregation)

  • Separate read and write models
  • Optimized for different workloads
  • Event-driven synchronization
  • Increased complexity

GCP Event Services

Eventarc

  • Event delivery from 60+ GCP sources
  • Cloud Events standard
  • Direct delivery to Cloud Run, GKE, Cloud Functions
  • Filter events by type and attributes

Audit Logs

  • Capture all GCP API calls
  • Event source for monitoring
  • Security and compliance tracking
  • Integration with Eventarc

Cloud Scheduler

  • Time-based event triggers
  • Cron-like scheduling
  • Integrates with Pub/Sub
  • Support for complex schedules

Integration Patterns

Cloud Run + Pub/Sub

# service.yaml
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: event-processor
spec:
  template:
    spec:
      containers:
      - image: gcr.io/my-project/event-processor
        env:
        - name: PUBSUB_SUBSCRIPTION
          value: "projects/my-project/subscriptions/my-subscription"

Cloud Functions + Eventarc

# Deploy Cloud Function with Eventarc trigger
gcloud functions deploy my-function \
  --runtime=python39 \
  --trigger-topic=my-topic \
  --region=us-central1

GKE + Pub/Sub

# subscription-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: pubsub-config
data:
  subscription: "projects/my-project/subscriptions/my-subscription"

Use Cases

  • Building applications that need to react to state changes asynchronously
  • Decoupling services through events
  • Implementing microservices communication
  • Real-time data processing
  • IoT device event processing

Pros

  • Loose coupling between services
  • Scalable architecture
  • Asynchronous processing
  • Natural fit for cloud-native applications

Cons

  • Complex error handling
  • Event ordering challenges
  • Debugging distributed systems
  • Event schema evolution

Event Schema Design

{
  "specversion": "1.0",
  "type": "com.example.order.created",
  "source": "/orders-service",
  "id": "A234-1234-1234",
  "time": "2024-01-01T12:00:00Z",
  "datacontenttype": "application/json",
  "data": {
    "orderId": "12345",
    "customerId": "67890",
    "amount": 100.50,
    "items": [
      {"productId": "abc", "quantity": 2}
    ]
  }
}

Best Practices

Event Design

  • Use Cloud Events standard
  • Include event type and source
  • Keep events immutable
  • Design for event evolution

Error Handling

  • Implement dead-letter queues
  • Use exponential backoff for retries
  • Monitor failed events
  • Circuit breaker for unhealthy consumers

Monitoring

  • Track message backlog
  • Monitor processing latency
  • Alert on delivery failures
  • Log event processing

Security

  • Use IAM for access control
  • Encrypt messages in transit
  • Validate event schemas
  • Implement event authentication

Advanced Patterns

Saga Pattern

  • Break transactions into local transactions
  • Use events to coordinate steps
  • Implement compensating actions
  • Handle failure scenarios

Event Collaboration

  • Multiple services collaborate via events
  • Event choreography vs orchestration
  • Distributed transaction management
  • Complex business processes

Event Replay

  • Store events for replay capability
  • Debugging and testing
  • System recovery
  • Feature development

Common Issues and Troubleshooting

Pub/Sub Message Delivery Failures

  • Verify topic and subscription configuration
  • Check subscription endpoint availability
  • Review message acknowledgment handling
  • Monitor dead-letter queue for failed messages

Event Ordering Problems

  • Verify ordering key configuration
  • Check message publication sequence
  • Review consumer processing logic
  • Monitor for duplicate message handling

Event Schema Compatibility

  • Validate event schema versions
  • Implement backward compatibility
  • Use schema registry if needed
  • Monitor schema evolution impact

Integration Issues

  • Verify Eventarc trigger configuration
  • Check Cloud Function execution logs
  • Review service account permissions
  • Validate event filter expressions

Cleanup Commands

# Delete Pub/Sub topic
gcloud pubsub topics delete my-topic

# Delete Pub/Sub subscription
gcloud pubsub subscriptions delete my-subscription

# Delete Eventarc trigger
gcloud eventarc triggers delete my-trigger --region=us-central1

# Clean up Cloud Functions
gcloud functions delete my-function --region=us-central1

# Remove event resources
gcloud scheduler jobs delete my-job --location=us-central1

Jump to other sections

Additional Resources