A scheduled task often begins life as a single cron entry: run a script every hour, import some data, send a report or remove expired records. That approach is perfectly reasonable on one machine. It becomes much less comfortable when the task must survive deployments, temporary outages, traffic spikes, permission changes and a growing number of work items.
A useful AWS pattern is to separate when work should be created from how work is processed. EventBridge Scheduler creates a message at the appropriate time, Amazon Simple Queue Service (SQS) stores that message durably, and AWS Lambda processes it. The queue becomes a buffer between the schedule and the worker, rather than forcing a scheduled invocation to complete everything synchronously.
This design is not automatically reliable just because it uses managed services. You still need to choose retry policies, configure visibility timeouts, handle duplicate delivery, restrict IAM permissions and monitor failures. The benefit is that each concern has an explicit place in the architecture.
The architecture and when to use it
The workflow looks like this:
- EventBridge Scheduler evaluates a
rate,cronor one-timeatexpression. - It calls the Amazon SQS
SendMessageoperation using an execution role. - Amazon SQS holds the message until a consumer is ready.
- Lambda polls the queue through an event source mapping and receives one or more messages.
- The worker performs a small, repeatable unit of work and reports any failed messages.
- A dead-letter queue (DLQ) preserves messages that repeatedly fail, so they can be investigated rather than silently disappearing.
EventBridge Scheduler supports recurring and one-time schedules, retry policies and SQS dead-letter queues. It can also use a flexible time window when a task does not need to run at an exact second, allowing invocations to be spread across a period rather than concentrated at one instant. ([docs.aws.amazon.com](https://docs.aws.amazon.com/scheduler/latest/UserGuide/managing-schedule.html?utm_source=openai))
This pattern is a good fit for tasks such as synchronising an account, generating a customer report, polling an external API, processing an import or starting a batch operation. It is less suitable when a job must run continuously, needs a long-lived process, requires strict transactional coordination across many records or routinely exceeds Lambda’s execution limits. In those cases, consider ECS tasks, AWS Batch or Step Functions instead.
Start with a message contract
Do not put an ambiguous string such as run import on the queue. Treat the message as an API contract between the scheduler and the worker. Include an operation name, a version, a business identifier and a correlation identifier.
{
"version": 1,
"operation": "refresh-customer",
"customerId": "customer-123",
"scheduledFor": "2026-11-22T09:00:00Z",
"correlationId": "schedule-run-8b6d"
}
The scheduledFor value is useful for deciding whether a message is stale. The correlationId should appear in structured logs and downstream requests. Avoid putting secrets in the message. If a worker needs credentials, retrieve them from IAM-authorised services such as AWS Secrets Manager or Systems Manager Parameter Store.
Keep messages small and pass references to large data rather than embedding large documents. Amazon SQS has message-size limits, and large payloads also make retries more expensive. If the task needs a substantial input file, store it in Amazon S3 and put the bucket key, checksum and expected version in the message.
Create the queue and its failure path
For most independent scheduled tasks, begin with a standard SQS queue. Standard queues provide high throughput but may deliver a message more than once and do not provide strict ordering. That is normally acceptable if the worker is idempotent. Choose FIFO only when ordering or deduplication semantics are genuinely required; FIFO can reduce concurrency because a blocked message group prevents later messages in that group from progressing.
Create a separate DLQ rather than treating a growing source queue as your failure store. Configure a redrive policy on the source queue with a deliberate maxReceiveCount. AWS recommends giving Lambda several attempts before moving a message to a DLQ; its Lambda guidance recommends a value of at least five as a starting point. ([docs.aws.amazon.com](https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-configure.html?utm_source=openai))
The retention period of the DLQ must be long enough for somebody to notice, inspect and remediate a failure. A DLQ is not a substitute for alerting: publish an alarm based on the approximate number of visible messages, and include the queue name in the alert.
Think carefully about what should happen after remediation. Some messages can be safely redriven after a code fix. Others may represent invalid input and should be archived or marked as permanently rejected. A replay process should be controlled, observable and rate-limited so that fixing one failure does not create a second outage.
Configure the scheduled producer
The scheduler needs an IAM execution role that it can assume and that can send messages to the intended queue. The role should grant sqs:SendMessage only for that queue. Do not give the scheduler broad permissions such as sqs:* or access to every queue in the account.
The AWS CLI supports a templated SQS target. The following example creates a test schedule that sends a message every 15 minutes. Replace the placeholders with values from your account, and use a schedule-specific role rather than a general-purpose deployment role.
aws scheduler create-schedule \
--name customer-refresh-every-15-minutes \
--schedule-expression 'rate(15 minutes)' \
--target '{
"RoleArn": "arn:aws:iam::123456789012:role/EventBridgeSchedulerSendToCustomerQueue",
"Arn": "arn:aws:sqs:eu-west-2:123456789012:customer-refresh",
"Input": "{\"version\":1,\"operation\":\"refresh-customer\",\"customerId\":\"customer-123\"}"
}' \
--flexible-time-window '{ "Mode": "OFF" }'
EventBridge Scheduler supports retry settings for failed target delivery. Configure a maximum event age and a maximum retry count appropriate to the work. A short-lived infrastructure problem may justify retries, but retries cannot repair malformed input or a permanently denied permission. Pair the retry policy with a scheduler DLQ so that failed deliveries remain available for diagnosis. ([docs.aws.amazon.com](https://docs.aws.amazon.com/cli/latest/reference/scheduler/create-schedule.html?utm_source=openai))
For non-urgent periodic work, consider a flexible time window. If a report can start any time within ten minutes, use that flexibility to avoid making every account perform work at the same instant. Do not use a flexible window when the business requirement is genuinely time-sensitive, such as a deadline-driven notification.
For schedules tied to local business time, set the schedule time zone explicitly and test daylight-saving transitions. A cron expression evaluated in UTC is not equivalent to “every weekday at 09:00 in London” throughout the year. Make the time-zone decision part of the schedule definition, not an assumption hidden in application code.
Connect Lambda to SQS
Lambda uses an event source mapping to poll SQS and invoke the function with a batch of messages. The function does not need to call ReceiveMessage or DeleteMessage itself. When a batch succeeds, Lambda deletes the messages. If processing fails, messages become visible again after the queue’s visibility timeout.
The visibility timeout must be longer than the Lambda timeout. AWS recommends setting it to at least six times the function timeout, with the batching window added when one is configured. This gives Lambda room to retry when an invocation is throttled or a previous batch takes longer than expected. ([docs.aws.amazon.com](https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-configure.html?utm_source=openai))
For example, if the function timeout is 60 seconds and the batching window is zero, a visibility timeout of 360 seconds is a reasonable starting point. This is not a performance benchmark or a universal value: measure the real processing time, account for downstream latency and leave enough room for retries.
Set the batch size according to the work performed by each record. A larger batch can reduce invocation overhead, but it increases the amount of work that may be retried and makes memory and timeout calculations less obvious. Begin conservatively, then tune after observing duration, errors, queue age and concurrency.
Use reserved concurrency or the event source mapping’s maximum concurrency controls when the worker calls a fragile downstream system. Without a limit, a backlog can cause Lambda to scale the consumer aggressively and overload an API or database. Scaling the queue consumer is only useful if the dependency can absorb the resulting load.
Make the worker idempotent
SQS event source mappings provide at-least-once processing. The same message can be delivered more than once, including after the worker has completed its business action but before the message is deleted. AWS explicitly recommends designing the function to process the same message multiple times without unintended side effects. ([docs.aws.amazon.com](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html?utm_source=openai))
Idempotency means that repeating the operation produces the same intended result. For a “refresh customer” operation, an update keyed by customerId is usually safer than blindly inserting a new row. For an email operation, store a durable send key before calling the provider, or use a provider-supported idempotency key if available.
A common approach is an idempotency table with a conditional write:
- Derive a stable key from the operation and business identifier.
- Attempt to insert the key with a condition that it does not already exist.
- If the condition fails, treat the message as already claimed or completed according to your state model.
- Perform the operation and record a completed status.
- Allow retries to resume safely when the operation fails part-way through.
Be precise about the boundary. Writing “completed” before the external side effect can suppress a necessary retry. Writing it only afterwards can still allow a duplicate if the side effect succeeds but the status write fails. For high-value operations, model states such as started, completed and failed, and design reconciliation rather than pretending distributed systems provide one atomic transaction.
Return partial batch failures
By default, one failed record can cause the whole Lambda batch to become visible again. That means successfully processed records may be repeated because of one poison-pill message. Enable the ReportBatchItemFailures response type and return only the message identifiers that failed.
export const handler = async (event) => {
const batchItemFailures = [];
for (const record of event.Records) {
try {
const message = JSON.parse(record.body);
await processMessage(message);
} catch (error) {
console.error("message_failed", {
messageId: record.messageId,
error: error instanceof Error ? error.message : String(error)
});
batchItemFailures.push({
itemIdentifier: record.messageId
});
}
}
return { batchItemFailures };
};
Do not catch an exception and then return an empty failure list. That would tell Lambda that every record succeeded, causing failed work to be deleted. Conversely, if your function throws after processing some records, Lambda can treat the entire batch as failed. AWS documents this behaviour and also provides a Powertools batch-processing utility for supported runtimes. ([docs.aws.amazon.com](https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-errorhandling.html?utm_source=openai))
For a FIFO queue, stop processing after the first failure and report the failed and unprocessed records so that ordering is preserved. For a standard queue, independent records can normally continue, provided their operations do not depend on sequence.
Separate retryable and permanent errors
Not every exception deserves the same treatment. A timeout from a downstream API, a throttling response or a temporary network error is usually retryable. Invalid JSON, an unknown operation or a missing required identifier is normally permanent until somebody changes the message or the code.
Even with that distinction, sending a permanent error repeatedly through the source queue is wasteful. Validate the message near the start of processing, log the validation failure with the message ID and allow the configured redrive policy to move it to the DLQ. If the business requires a separate rejection workflow, publish a sanitised failure event to another queue or store the original message securely for investigation.
Never log access keys, tokens, passwords or complete personal records merely because a message failed. Log identifiers, error categories and safe diagnostic context. If the payload contains personal data, configure retention and access to logs and queues as carefully as the application database.
Observe the complete path
Monitoring only Lambda errors leaves a blind spot. The scheduler, queue and consumer can fail in different ways, so create visibility across all three layers.
- Alarm when the scheduler’s failed invocations or DLQ message count increases.
- Monitor the source queue’s approximate age of the oldest message, not just its total message count.
- Track Lambda errors, throttles, duration, concurrent executions and iterator or event-source lag metrics where available.
- Record structured fields such as
operation,messageId,correlationId, customer scope and outcome. - Trace the correlation identifier through downstream API calls and database writes.
- Alert on a queue that is silently accumulating work even when Lambda has no function errors.
Queue age is especially important. A queue can have a modest number of messages but still be unhealthy if they are old. Conversely, a large queue may be normal during a planned import if its age remains within the service-level objective.
Test the failure path deliberately. Send an invalid message, deny the worker’s access to a dependency in a non-production environment and force a downstream timeout. Confirm that the message is retried, that successful messages in the same batch are not needlessly repeated, that the DLQ receives exhausted failures and that somebody receives an actionable alert.
Security and cost considerations
Use separate IAM roles for the scheduler and Lambda. The scheduler should send only to its target queue. The Lambda execution role should read only the required source queue and write only to the services it needs. If queues or messages use a customer-managed KMS key, grant the corresponding encryption and decryption permissions explicitly.
Keep the queue and Lambda function in the same AWS Region. The Lambda SQS event-source documentation notes that they may be in different AWS accounts, but cross-account designs require additional resource policies and are harder to troubleshoot. Use cross-account delivery only when the ownership boundary justifies the complexity. ([docs.aws.amazon.com](https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-configure.html?utm_source=openai))
Cost is usually controlled by keeping messages small, batching compatible work and avoiding unnecessary retries. Do not set a large batch size for a slow operation simply to reduce invocation count. The resulting timeout, duplicate processing and downstream load may cost more than the invocations saved.
Encrypt queues where your requirements demand it, restrict who can receive or purge messages and treat queue permissions as production credentials. In particular, sqs:PurgeQueue can destroy pending work and should not be part of a general developer role.
A practical implementation checklist
- Define a versioned JSON message with stable identifiers and no secrets.
- Create a source queue and a separate DLQ with an appropriate redrive policy.
- Create an EventBridge Scheduler execution role limited to
sqs:SendMessageon the source queue. - Configure the schedule expression, time zone, retry policy and scheduler DLQ.
- Implement the Lambda worker as a small, idempotent operation.
- Set the queue visibility timeout in relation to the Lambda timeout and batching window.
- Enable partial batch responses and return failed message IDs accurately.
- Limit consumer concurrency when downstream services have strict capacity.
- Alarm on scheduler failures, DLQ messages, queue age, Lambda errors and throttles.
- Test duplicates, malformed messages, timeouts, permission failures and replay procedures.
The important change is conceptual: a schedule should create work, not hide an entire application inside a timer callback. EventBridge Scheduler gives you a dependable trigger, SQS gives the work somewhere durable to wait, and Lambda gives you an elastic consumer. Reliability comes from the decisions around those services: idempotent effects, bounded retries, correctly sized visibility timeouts, least-privilege IAM and a failure path that people can operate.