Scheduling and Monitoring Customer Workflows at RapidCanvas

At 2:03 AM a network fault lasting a few seconds drops a nightly pipeline’s connection to a customer’s data warehouse, and the run fails. By 9 AM, when the customer’s team opens the dashboards that pipeline feeds, their alert channel holds forty messages, thirty-nine of them identical. The one that matters is at the top of a long scroll. Had anyone re-run the pipeline at 2:04, it would have succeeded.

Scheduled workflows are how RapidCanvas customers operationalize their data pipelines: a customer attaches a cron expression to a workflow, and the platform runs it unattended from then on. When one fails, two things should happen. The platform should recover on its own if the failure is transient, and if it is not, the right people should be told once, with enough context to act. For a long time our scheduler did neither well. This post covers why we run scheduling ourselves, why the textbook fix did not fit our platform, and how we redesigned failure handling so that a transient fault and a genuine outage no longer produce the same alert.

TL;DR

• We added a monitoring pipeline beside our existing Kubernetes-based scheduler, with no new queue, no new datastore and only one small change to how workflows execute.
• Retry and alert behaviour is derived from each workflow’s cron cadence: frequent workflows get patience, rare ones get immediate retries and fast escalation.
• A failure is now a tracked state rather than a message: one alert per workflow that is threaded, escalates on its own, has an owner and resolves itself on the next success.

The problem: forty alerts, one root cause

Consider a customer with a dozen scheduled workflows. One is a data-quality check that runs every five minutes. Another is a nightly pipeline that runs at 2 AM and produces the dashboards their team reviews at 9. Both the customer and our platform team are notified of failures.

At 2:03 the brief network fault fails the nightly run. At 2:05 the quality check fails too, because the table it validates in the customer’s database was never refreshed. Both failures produce an identical Slack message. The check fails again at 2:10, 2:15 and 2:20, and keeps failing through the night.

02:03  nightly-pipeline   FAILED   → Slack: "Failed to run the Scheduler ..."
02:05  quality-check      FAILED   → Slack: "Failed to run the Scheduler ..."
02:10  quality-check      FAILED   → Slack: "Failed to run the Scheduler ..."
02:15  quality-check      FAILED   → Slack: "Failed to run the Scheduler ..."
  ...  (35 more identical messages)
05:20  quality-check      FAILED   → Slack: "Failed to run the Scheduler ..."

       40 messages. 1 root cause. 0 retries. Nobody owns any of it.

Every notification system balances two costs: alert on every failure and recipients stop reading; wait for certainty and the damage is already done. Our previous system never made that trade-off explicitly. Its symptoms (every failure treated as critical, no retries, identical handling for a five-minute and a daily job, no ownership, no cross-customer view) traced back to two underlying limitations.

Every failure was evaluated in isolation. The decision to send had no memory. It did not consider the previous run, the workflow’s cadence or when the next run would occur. A single transient fault and the third consecutive failure of a critical pipeline looked exactly the same.

Detection and notification were the same act. A failure was a Slack message. There was no intermediate object that could be counted, owned, escalated or resolved, so once the message was sent the platform had no further representation of the incident.

Why the obvious fix doesn’t work

The textbook fix sounds simple: the failing run inspects recent history, retries if budget remains and alerts otherwise. Three properties of the RapidCanvas execution model made that harder than it looks.

Runs are heavy, so the scheduler stays minimal

Every RapidCanvas project executes inside a custom environment: a long-lived Kubernetes pod provisioned with the customer’s own Python and Linux packages, their CPU, memory and optional GPU allocation, and a persistent disk holding project data. Building such an environment from scratch can take tens of minutes, so environments stay warm between runs and are stopped when idle.

That rules out the common orchestration pattern of launching a fresh container per task, which would spend most of its life installing dependencies. A scheduled run instead needs to wake the customer’s environment if it is stopped, authenticate as the user who published the schedule, execute the workflow’s recipe graph (a recipe is one pipeline step, such as a SQL query, a Python transform or a model invocation) inside that environment and tenant, enforce the configured timeout and record the result.

An off-the-shelf orchestrator could in principle drive this, since an Airflow or Temporal task can be a thin remote call that wakes an environment and runs a graph. But the trigger was never the difficult part, and Kubernetes already provided one. Adopting an orchestrator for the trigger alone would have meant operating a second scheduler, a second metadata store and a second tenancy model to gain a capability we already had. So we kept scheduling minimal and put the intelligence in the worker. Here is the system as it stood before monitoring was added.

Figure 1: The scheduler before monitoring was added.

• Control plane. The web application. On publish it creates one Kubernetes CronJob per schedule and hands over the clock; pausing a schedule deletes the CronJob. A manual “run now” bypasses Kubernetes and launches the worker directly through the same code path.
• Worker. The container Kubernetes starts on every tick. It authenticates as the schedule’s owner, wakes the customer’s environment if needed, starts a watchdog for the schedule’s timeout, executes the recipe graph inside the environment, records the outcome, ships its logs and exits.
• Run ledger. A table in the platform database with exactly one row per run, scheduled or manual: start time, terminal status, outputs and error. It is the customer’s run history and the one place both the worker and the control plane write.
• Reconciler. A background job that compares the CronJobs in Kubernetes against the schedules in the database and repairs drift: recreating missing CronJobs, correcting edited schedules and deleting CronJobs whose workflow no longer exists. The database is the source of truth, and a background process converges the world toward it. The monitoring pipeline follows the same principle.

Figure 2: The worker lifecycle on each tick.

A fifth element completed the original design: on failure, the worker posted a Slack message, the same one every time. That is the behaviour that generated the forty alerts.

The failure is discovered where no decision can safely be made

The worker knows a run failed, but it is a process about to exit. The control plane, the only component with a view across all runs, never sees inside the worker. Their only shared medium is the run ledger, so any retry or alert decision has to be made somewhere other than where the failure happens.

That somewhere is replicated. Several identical control-plane instances run at once, and if each independently observed a failure and launched a retry, we would get duplicate retries and duplicate notifications. Whatever reacts to a failure has to react exactly once across all replicas, without a coordinator we did not already have.

Multi-tenancy and thousands of live schedules

RapidCanvas is multi-tenant, so anything that processes runs in the background has to respect each customer’s isolation exactly as a request from that customer would. And the execution path could not change: thousands of schedules were live, so everything new had to sit beside the existing system rather than inside it.

Our first design, built around an in-process event and delayed retries, did not survive these constraints. The design that did rests on two ideas: the schedule itself tells us how patient to be, and a failure should become a state we track rather than a message we send.

How we built the monitoring pipeline

Figure 3: The monitoring pipeline, from run outcome to alert.

The pipeline adds eight components, and all of its state lives in the existing database: the run ledger, plus a new table for alerts. We describe them in the order a failed run encounters them.

Outcome polling: the run record is the event

The worker already writes each run’s terminal status to the ledger. That row is a durable event visible to every control-plane instance, so we poll it rather than propagate an in-process event. A poller wakes every thirty seconds and claims finished runs it has not yet processed using SELECT … FOR UPDATE SKIP LOCKED. Each instance locks the rows it takes and skips rows another instance already holds, so concurrent pollers receive disjoint batches with no leader election and no message broker.

Figure 4: Concurrent pollers claim disjoint batches with SKIP LOCKED.

One detail matters in production: the poller marks a run as processed at claim time, so a downstream fault logs an error and moves on rather than reprocessing forever; the next scheduled run is evaluated afresh.

Frequency profiling: urgency comes from the schedule

A workflow that runs every five minutes will run again in five minutes; a daily workflow will not run again until tomorrow. The cron expression already encodes how much patience a failure warrants. The profiler parses the schedule, computes the interval between consecutive fire times and derives three parameters: how many automatic retries the workflow gets, how many confirmed failures in a row (ticks whose retries are exhausted) must occur before anyone is notified, and whether the first alert is a Warning or a Critical.

• Frequent workflows get patience. No retry, since the next tick effectively is the retry, and several confirmed failures before notification, because a transient fault corrects itself before a human could read the alert.
• Infrequent workflows get urgency. Retries launch immediately on failure, and if they do not recover the run, the first confirmed failure is Critical.
• Intermediate cadences scale between the two. The rarer the workflow, the more aggressively we attempt recovery and the sooner we escalate.

Monitoring is enabled per workflow. Once it is on, these defaults apply with no further configuration, and customers can override any of them.

Retry orchestration: an honest run history

Figure 5: How run status flows through retries.

A retry serves three audiences with different needs. The customer wants one row per scheduled tick showing the final result. The alert evaluator wants to count one outcome per tick, and only after retries are exhausted. An engineer investigating an incident wants every attempt.

We made one change to the worker: on failure, it checks the profile’s retry budget for the current tick and writes the status pending retry instead of failed if budget remains. The poller routes every failed run through the retry engine. With no budget left, the run passes straight to evaluation; otherwise the engine launches a fresh worker container on the same code path, tagged with the tick it belongs to. Budget is tracked per tick, not per workflow, so retries used on one day do not shrink the allowance on the next.

When the final retry completes, the engine writes the verdict back onto the original row. If a retry succeeded, the tick is recorded as successful with a note that it recovered on retry. If every retry failed, the tick is recorded as failed with the final error. The customer sees one row per tick, the evaluator counts one outcome per tick, and the individual attempts remain available underneath.

Alert evaluation: state derived from history

Once retries are exhausted, the evaluator works out how many consecutive ticks have failed. A stored counter that increments on failure and resets on success drifts whenever data is corrected by hand or a retry rewrites a row. So instead, the evaluator reads recent run history newest-first and counts until it reaches a success. The answer is correct regardless of how the rows came to be. It then applies the workflow’s rule and returns either “alert, at this severity” or “not yet”.

Figure 6: Counting consecutive confirmed failures.

Here the count of consecutive confirmed failures is 3, which is then compared with the workflow’s threshold.

Alert lifecycle, escalation and ownership

Figure 7: The alert lifecycle.

The old system produced a message per failure. The new one maintains a state per workflow: is it currently broken? There is at most one open alert per workflow, and later failures update that alert’s count instead of creating another. Thirty-nine identical check alerts collapse into one alert with a counter.

Warnings that nobody acts on escalate on their own. An escalation clock, a separate background loop, examines every open Warning each minute. If the workflow has kept failing since the alert opened, or the alert has been open too long relative to the workflow’s cadence, it is promoted to Critical. The deadline is a single expression, escalate_at = opened_at + multiplier × cadence, and the cadence is captured when the alert opens so that editing the schedule mid-incident does not move it.

When the workflow next succeeds, whether on schedule or from a manual run, the alert resolves itself and records its duration. Every alert also carries an owner, engineering or the customer, with a note and a handoff history.

Notification and visibility

Because an alert is a state, its notifications form a single conversation: the first message opens a thread, and each later event (escalation, reassignment, recovery) is a reply within it. The notifier receives an alert and an event type and knows nothing about what triggered them, which is how we added email alongside Slack without upstream changes. A dashboard reads the same alert state across all customers and shows, on one screen, which workspaces have open alerts and who owns each.

How this compares with a typical orchestrator setup

None of these techniques is exotic. What is different is the set of choices taken together: each one removes a moving part rather than adding one, and each follows from the shape of the platform rather than from a generic orchestration playbook.

The same incident, replayed

Back to 2:03 AM. The connection drops and the run fails. The worker sees that a daily workflow has retry budget and writes pending retry. Within a minute the poller claims the run and launches a retry; by 2:06 it has succeeded, the fault long since cleared. The 2 AM row is recorded as recovered and no alert is opened. Meanwhile the five-minute check failed once against a profile that needs several consecutive failures, then passed at 2:10. Nobody is notified, and the dashboards are ready at 9.

Now suppose the failure is a genuine defect. Retries fail, and the row is recorded as failed with the actual error. The evaluator counts one confirmed failure against a daily profile and opens a Critical alert: one message in the customer’s channel and one in ours, with the failing step’s log attached.

The five-minute check keeps failing because its table is never refreshed. After several failures it opens its own Warning in its own thread, and as the failures continue that Warning escalates to Critical in the same thread. At 9:00 the customer fixes the defect and triggers a manual run. Both alerts resolve on the next success, the pipeline’s alert recording seven hours of downtime, and each thread gets its closing reply.

Forty messages become none in the common case and two alerts in the failure case, because the system now evaluates what it observes instead of relaying it.

Just as important, the pipeline needed no new queue, no new datastore and no change to how a workflow executes. That is what let us deploy it under thousands of live production schedules, with the previous alerting still running alongside until we were confident in the new behaviour.

Trade-offs and lessons

Every design choice here has a cost, and we accepted these deliberately.

Polling adds latency. A failure may wait up to thirty seconds before it is examined. For a system designed to wait before alerting, that is negligible.

Retries are immediate. Our original design delayed the second attempt to give a transient fault time to clear. Delayed retries need their own scheduler, a pending state and recovery logic after restarts, so we dropped them. As a result, a retry rescues faults shorter than the poll interval plus the time the retry takes to reach the failing step, a few minutes for the nightly pipeline. A warehouse that is unavailable for ten minutes is not treated as transient; it becomes a genuine alert, which we consider correct. Frequent workflows get a delayed retry for free, because their next tick fills that role.

The lessons we would pass on to anyone building something similar:

• When the data is already in the database, the database can be the queue. Introducing SQS or Kafka is not always worth the added complexity. A message broker earns its place when the system needs fan-out, independent consumers or replay of failures. Until then, using the database as the work queue keeps the architecture simpler and is often the more pragmatic choice.
• The schedule already expresses urgency. Deriving retry and alert behaviour from cadence produces sensible defaults without asking users to configure thresholds.
• Model the alert as a state, not an event. Deduplication, threading, escalation, ownership and automatic resolution all follow from that single decision.

Wrapping up

For RapidCanvas customers, the result is a scheduler that quietly absorbs transient faults and speaks up once, with context and an owner, when something is genuinely broken. If you run scheduled data workflows and want to see how this works on your own pipelines, book a discovery call with our team.