Introduction to Asynchronous Tasks Architecture

Design patterns, best practices and architectural insights for async task execution

Asynchronous tasks are commonplace in backend and infrastructure engineering. This is the first in a series of articles aiming to present suggestions and insights on architecting and maintaining effective asynchronous task frameworks and pipelines.

In the upcoming articles the following topics will be explored:

• Message Structure and Serialization Formats

  • Headers & Versioning

  • Schema Validation

  • JSON, YAML, Thrift, Avro, Protocol Buffers

  • Language-specific marshalling

  • Security: Signing and Encryption

• Essential Data Structures

  • FIFO/LIFO/Priority Queues

  • Sorted Sets

  • Sharded Queues

• Messaging Patterns

  • Event Notification

  • Topics & Streams

  • Task Queue

  • Pub/Sub

• Message Delivery

  • Fanout

  • Early / Late & Negative Acknowledgement

  • Failure Handling with Dead-Letter Queues

  • At-least-once/Exactly-once/At-most-once Guarantees

  • Deduplication

• Message Brokers

  • RabbitMQ

  • Redis

  • Kafka

  • AWS SQS

  • RDBMS (MySQL/PostgreSQL)

• Task Design

  • Idempotence & Optimal Parameter Selection

  • Timeouts

  • Error Handling & Retries

  • Acknowledgement Timeout

• Task Processing

  • Worker Pool Concepts & Types

  • Context & Tracing

  • Horizontal Scalability & Auto-Scaling

  • Interdependent Tasks - Directed Acyclic Graphs

  • Scheduled / Delayed Tasks

  • Circuit Breaker Pattern

  • Rate Limiting

• Frameworks, Services and Workflow Management Platforms

  • Resque, Sidekiq, Celery, Faktory

  • AWS Lambda

  • Apache Airflow, Luigi

The above topics should provide a comprehensive theoretical background but how applicable are these concepts to actual use cases? Throughout the article series we’ll be using a sample application comprising several processes, to help us visualize how each architectural pattern addresses real-world scenarios. Our application will be an implementation of a very common System Design question, the URL Shortener Service.

The main goal of the service is to provide one or more unique, shorter URLs that redirect to the target URL:

GET https://url-shortener-service.io/q2csXrZC

HTTP/1.1 301 Moved Permanently
location: https://controlflow.substack.com/async-task-architecture

This is a seemingly simple operation, but let’s try to lay out some of the features and functional requirements for a SaaS offering and determine where async task execution is applicable.

Authentication & Authorization

Email address verification during signup

An email containing the verification URL is sent to the email address of the new user account.

Password Change/Reset Requests

Users will be notified on password changes and must receive a link for password resets.

Redirect Setup

When a new redirect is configured, the change must be propagated to the redirect proxy service, which is likely using a cache of valid URLs. Likewise, when a redirect is disabled/removed, the cache must be updated to stop serving requests for this short URL.

Analytics

The service collects analytics for each redirect, such as total views, view count aggregated by referrer domain or user agent.

The redirect operation should have very low latency, thus the service must forward the HTTP request information to a queue or stream for async processing.

Paid Accounts

Our service will have paid features such as using a custom domain for shortened URLs and will offer a free tier that limits the total active redirect URLs or the total views per month.

Cost calculations occur asynchronously and warning notifications about reaching the end of the free tier quota are also sent via email in the background.

Billing

Assuming a usage-based billing plan, the corresponding amount must be calculated and charged on the account’s credit card each month. A new invoice is issued and the billing contact receives it as an email attachment.

The users may also want to configure billing alerts in order to be informed early about sudden increases in charges.

The above operations are not triggered by a user action, but instead are executed on regular intervals in the background.

---

What’s next?

The next article in the series will discuss Message Structure and Serialization Formats.