Template Builder is Now Serverless: Notes From Ten Years of Migrations

Template Builder is our data parsing service, and it sits right at the centre of everything we do at Fund Recs. Every year it processes millions of files, around 3 billion transactions and over 24 billion data points.

It works as a mini ETL pipeline. It ingests data from banks, custodians, administrators and counterparties in virtually every format imaginable — XLS(X), PDF, CSV, TXT, email, XML, SWIFT, BAI and JSON — and transforms those raw inputs into standardised, structured data that downstream reconciliation systems consume reliably.

Last week we hit a major milestone. Template Builder went serverless, across six regions, with zero downtime, making it the second mission-critical service within Fund Recs to adopt a serverless architecture.

As a result of this migration:

• A fleet of always-on servers has disappeared
• The system scales on a moment's notice. Scaling no longer means waiting for new tasks to initialise and pass health checks.
• Every file processes in total isolation, eliminating noisy-neighbour concerns.
• The service runs on demand as functions, incurring zero cost when idle — a significant cost reduction.
• A tiered architecture matches compute to each file, so small files no longer reserve the resources a large one needs.

Going serverless is the most recent migration we have completed for Template Builder, but it wasn't the first. Over ten years the service has changed its database, its framework and its processing model, and each of those changes took surprisingly little engineering effort given how critical and core the service is to our operations.

Most of the credit goes to a few design decisions made early on, and to a test suite we've kept investing in.

Keep the core small - UNIX Philosophy

The core engine is built from isolated, independent units that know nothing about each other and share one tiny contract. Adding new processing capabilities is as simple as implementing that contract and registering the new handler unit. High composability was built into the architecture from day one, keeping the core engine small and making extensions trivial to write, test and ship.

A lean core is cheap to reason about, cheap to move, and cheap to grow. It avoids the compounding technical debt and operational burden that typically drag down ageing systems.

Over the years there has been steady pressure, from business and engineering alike, to let Template Builder take on adjacent, convenient jobs. The data was already there, and those tasks would have been low-hanging fruit — short-term gain at the expense of the long-term health of the service.

Resisting that temptation kept the engine focused.

Every foreign responsibility avoided one less thing to break, and one less drag on our next migration.

Tests we trust

Small, isolated units with a fixed contract are easy to test deterministically. Over time we have built a thorough test suite and CI/CD pipeline around them, covering parsers, transformations and ETL operations.

On top of that, our UAT environment runs black-box end-to-end acceptance tests against synthetic production workloads. Those tests run the whole system under realistic load and verify everything from file ingestion and queueing to storage writes, IAM roles and environment-level controls.

This is what lets us move quickly. Structural changes are checked against thousands of real-world scenarios before they reach production. When a client needs a new file format or a custom ETL operation, we can build it, run it through the pipeline and ship it with a short turnaround.

And a change as large as moving the execution engine to serverless becomes manageable, because the pipeline shows the output is identical across thousands of edge cases.

How we got here

The following are some of the changes we made over several years, each one only when a real requirement, backed by data, called for it: growing volumes, larger files, resiliency needs or client requests. Each change also made the next one easier.

1. From a relational database to DynamoDB

As volumes grew, the relational model was adding complexity we didn't need. We moved to DynamoDB because it was the simplest storage option that met our requirements.

2. From Grails to Spring Boot

We replaced the underlying framework without touching the core processing engine.

3. Profiling in depth

When increasing file sizes began straining performance, we resisted the impulse to throw hardware, caching layers or parallel compute at the problem. Instead, keeping it simple, we performed systematic and rigorous profiling to uncover the exact computational hotspots. Replacing quadratic operations O(n × m) with linear, chunk-based streaming O(n + m), and optimising core data structures, removed bottlenecks and allowed Template Builder to scale throughput for years on existing hardware without inflating infrastructure costs.

4. Asynchronous processing

Moving to a queue-based model was a quiet change, but probably the one with the most impact. File surges stopped blocking execution, throughput went up, resiliency improved — and it was the precondition that made serverless possible at all.

5. AI integrations

We added AI-powered parsers to pull data reliably from complex, unstructured documents. By integrating frontier LLMs via AWS Bedrock, we empowered users to express data cleaning rules and pipeline intentions in plain language. Template Builder translates those natural-language statements into deterministic ETL operations, allowing users to focus on the business problem rather than learning the system. Users get an intuitive interface, while the underlying operations remain exact, safe and completely deterministic.

What's comes next

Resisting speculative complexity, and avoiding over-engineering before it was needed, ensured that when real bottlenecks showed up the codebase was still nimble enough to evolve. Keeping the core lean and simple allowed the exact same engine to scale from a few hundred thousand transactions a year to over 3 billion without sweating.

Ten years in, Template Builder leverages cutting-edge AI, runs on a serverless architecture, and remains more powerful, fault-tolerant and scalable than ever before, with remarkably low developer friction.

With a serverless, AI-enabled foundation firmly in place, the next chapter is already underway: building an agentic execution layer where autonomous agents reason through complex data anomalies, autonomously configure transformations, and handle edge cases with minimal human intervention.