Skip to content

Schedule Service (ac-service-schedule) – Technical Design Document

Introduction

Purpose: This document describes the design, architecture, and operational model of the Schedule Service as it currently stands on develop, and records the design decisions taken.

Scope: The full service: the domain model, persistence, the HTTP API, the background scheduler, the HTTP and Kafka executors, the timezone model, concurrency guarantees, and the test strategy.

Source code: github.com/appcircleio/ac-service-schedule

System Overview

A DDD microservice that stores scheduled jobs and fires them according to a cron expression. It has a single bounded context, ScheduleContext. On its schedule, each job either issues an HTTP request or publishes a Kafka event. The minimum granularity is one minute (second-level scheduling is not supported).

Architecture

Technology Stack:

Layer Technology
Runtime .NET 8 (ASP.NET Core)
Persistence MongoDB
Distributed lock + cache Redis (RedLock, StackExchange.Redis)
Messaging Kafka (producer)
Identity Keycloak (organization name lookup)
Cron parsing Cronos (CronFormat.Standard)
Logging NLog

External Dependencies:

  • MongoDB - job store.
  • Redis - distributed lock (one instance per tick) and distributed cache.
  • Kafka - event publishing for Kafka-type jobs.
  • Keycloak - resolves the organization name on job creation (failure tolerated).

Layers:

  • Domain (ScheduleContext/ScheduledJobAgggreggate/) - ScheduledJob aggregate root, the ScheduledJobHttpExecutor / ScheduledJobKafkaExecutor value objects, ScheduledJobDto, ScheduledJobStoreObject (the MongoDB document), and ScheduledJobFactory.
  • Repository (Data/ScheduledJobRepository, IScheduledJobRepository) - MongoDB CRUD plus the atomic claim; ScheduleMongoUnitOfWork is the Unit-of-Work that fires domain events on commit.
  • Application (Services/) - ScheduledJobService (CRUD plus cron / next-run calculation) and ScheduledJobExecutorService (firing plus status notification).
  • Background (BackgroundServices/SchedulerService) - a BackgroundService that polls periodically.
  • Infrastructure - the Kafka producer (OutgoingMessageBrokerEventHandler).
  • Shared (ScheduleContext/ScheduleTimeZone) - the single source of truth for timezone resolution.

Key Components

ScheduledJob (Domain)

  • Purpose: Aggregate root representing a scheduled job and its executor(s).
  • Responsibilities: Holds schedule, timezone, executor type, and executor details; exposes IsHttpExecutor() / IsKafkaExecutor() guards and typed executor accessors.

ScheduledJobService (Application)

  • Purpose: CRUD entry point used by the controller.
  • Responsibilities: Validates executor type and timezone, computes the initial NextExecution from the cron expression and offset, resolves the organization name from Keycloak (tolerating failures), and persists through the repository.

SchedulerService (Background)

  • Purpose: Periodically finds due jobs and fires them.
  • Responsibilities: Polls MongoDB, acquires a per-job distributed lock, performs the atomic claim through the repository, and invokes the executor.

ScheduledJobExecutorService (Application)

  • Purpose: Executes a due job and reports its status.
  • Responsibilities: Runs the HTTP or Kafka executor and emits Started / Succeeded / Failed notifications (best-effort).

ScheduledJobRepository (Repository)

  • Purpose: All MongoDB access.
  • Responsibilities: CRUD plus TryClaimNextExecutionAsync, the optimistic-concurrency claim that guarantees a tick fires once.

Detailed Design

Database Design

Collection: ScheduledJobs.

Fields: Id, OrganizationId, OrganizationName, Key, Name, ScheduleExpression, Timezone, ExecutorType (http / kafka), the embedded HttpExecutor / KafkaExecutor, IsActive, NextExecution, LastExecution, CreateDate.

Serialization: DateTimeOffset is stored as BsonType.DateTime; GUIDs use CSharpLegacy; a camelCase element-name convention is applied.

Indexes (created at startup by EnsureIndexesAsync):

  • idx_isActive_nextExecution - serves the scheduler poll query.
  • idx_organizationId_key - unique.

API Surface (api/v1/schedules)

Method Route Function
GET /?organizationId= List by organization
GET /{id} Get one
POST / Create
PUT /{id} Update
PATCH /{id}/activate and /deactivate Activate / deactivate
DELETE /{id} Delete

ExecutorType must be http or kafka; otherwise a BusinessException (ScheduleServiceExecutorTypeInvalid) is thrown. An invalid timezone returns ScheduleServiceTimezoneInvalid.

Scheduler Algorithm

SchedulerService ticks every 45 seconds:

  1. Fetches jobs where IsActive == true && NextExecution <= now from MongoDB, capped at 100 per poll.
  2. Acquires a RedLock per job, keyed scheduled-job:{id}:{nextExecution:yyyyMMddHHmmss} (waitTime 0, expiry 2 minutes). Only one instance runs a given tick.
  3. If the lock is acquired, the new NextExecution is computed from the cron expression, then the repository's atomic claim is invoked: TryClaimNextExecutionAsync runs a conditional UpdateOne that advances NextExecution and LastExecution only if id + the old NextExecution still matches. If it returns false, another instance already handled the tick and this one exits.
  4. The job is fired through ScheduledJobExecutorService.ExecuteAsync; due jobs run concurrently via Task.WhenAll.

Double guard: RedLock (fast, prevents most contention) plus the MongoDB conditional update (the definitive guarantee). Together they enforce the invariant that a tick fires at most once.

Missed occurrences: the next occurrence is computed forward from now. If the service was down for a stretch, missed occurrences are skipped (the job fires once and resumes on schedule) instead of accumulating and catching up one occurrence per poll.

Executors

  • HTTP: Method / Url / Body / Headers (JSON). An Idempotency-Key of {jobId}:{lastExecution:O} is set by the executor and cannot be overridden by the job's own headers. The timeout is applied with a linked CancellationTokenSource. EnsureSuccessStatusCode() runs first, followed by the optional ExpectCode check.
  • Kafka: the event is published through MessageBrokerEventBuilderFactory. OrganizationId is taken from the job field first, then parsed from the message payload as a fallback; if both are empty the event is not published. A JSON message is sent as a JToken, otherwise as a raw string. Producer settings: Acks.Leader, MaxInFlight = 1, 30-second message timeout.
  • Status notifications (Started / Succeeded / Failed) are best-effort - a publish failure never fails the job (SafeNotifyAsync).

Timezone Model

Timezones are entered and stored as fixed UTC offsets: 01:00, +03:30, -05:00. Both the create/update flow and the scheduler resolve them through a single ScheduleTimeZone helper, so next-run is computed identically everywhere. An invalid or empty value falls back to UTC. IANA names (for example Europe/Istanbul) are not supported.

Security Considerations

  • Keycloak is consulted only at job creation to resolve the organization name; failures are tolerated and the name is left as N/A.
  • The Idempotency-Key header is owned by the executor and cannot be overridden by a job definition.

Integration and Interfaces

External interfaces: outbound HTTP requests to job-defined targets; Kafka event publishing to job-defined topics; Keycloak for organization name lookup.

Internal interfaces: the REST API under api/v1/schedules, consumed by other Appcircle components.

Concurrency and Distributed Execution

  • Multiple instances can run simultaneously; the single-fire guarantee comes from RedLock plus the atomic MongoDB claim.
  • DB I/O lives in the repository layer; SchedulerService invokes the claim through the repository.
  • Background work opens a fresh DI scope each tick via IServiceScopeFactory, so no scoped service leaks across ticks.

Testing and Quality Assurance

Test project: test/Appcircle.ScheduleService.WebApi.Tests (xUnit).

  • Unit: ScheduleTimeZone offset parsing and resolution, cron next-run against UTC offsets, and the ScheduledJob executor guards.
  • Integration (Testcontainers MongoDB, Category=Integration, requires Docker): the claim wins a due tick exactly once under concurrency, and a stale claim is rejected.

Configuration

Required environment variables: ASPNETCORE_PORT, ASPNETCORE_REDIS_ENDPOINT, SCHEDULE_SERVICE_DB_CONNECTION_STRING, ASPNETCORE_KEYCLOAK_SERVER_URL / _CLIENT_ID / _SECRET_ID, ASPNETCORE_KAFKA_SERVER_URLS.

The NuGet package tier is selected by NUGET_ENVIRONMENT (Development = alpha, Preproduction = beta, Production = stable, unset = local ProjectReference).

Design Decisions

  1. Timezone format is a fixed UTC offset (timespan). IANA support was removed; the create/update flow and the scheduler resolve through one ScheduleTimeZone helper in the same way.
  2. CreateDate is set on create.
  3. The by-key surface (upsert / set-active / delete by key) was removed. Only id-based and organization-based operations are supported.
  4. The optimistic-concurrency claim was moved into the repository (TryClaimNextExecutionAsync); SchedulerService calls it, keeping DB I/O out of the background handler.
  5. The single-fire guarantee is provided by RedLock plus the atomic MongoDB claim.
  6. Missed occurrences are skipped (no catch-up).
  7. A test project was added; integration tests run against MongoDB via Testcontainers / Docker.