How Splink Automated Production-to-Staging Database Refreshes Beyond the Limits of AWS DMS.

Logiciel Solutions engineered an on-demand database refresh pipeline for Splink to reliably synchronize its staging PostgreSQL environment with production data. GitHub Actions starts an Amazon EC2 instance that runs a purpose-built replication script, logs every run to Amazon CloudWatch, and reports success or failure back to the workflow.

The result is on-demand, repeatable, and observable refreshes without a continuously running DMS replication instance.

Build with Us See the Impact

On demand

Self-service staging refreshes

Full schema

Materialized views carried over

Always-on replication instances

5 steps

From trigger to pass/fail

Client profile

Who we built it for.

Company

Splink

Industry

CPG Trade Promotion / Retail Execution (convenience and wholesale channel)

Trigger event

Splink needed to refresh staging with production data on demand, but AWS DMS migrations failed on schemas that depend on materialized views.

The challenge

What stood in the way.

Splink’s engineering team needed to refresh its staging PostgreSQL database with production data on demand, so new features and fixes could be tested against realistic data before release.

AWS DMS was the obvious managed option, but it migrates tables and their data, not secondary objects such as materialized views. Several of Splink’s schemas depend on materialized views, so DMS migrations of the database failed.

DMS also bills a replication instance for every hour it is provisioned, a poor fit for a task that runs only occasionally.

On demand, no manual steps

Refresh staging with production data whenever the team needs it.

The full schema

Carry over materialized views and the objects that depend on them.

A clear result

A definite success or failure outcome, with logs for troubleshooting.

No always-on infrastructure

No migration instance billing between refreshes.

Why they chose Logiciel

A partner proven at scale.

Hands-on experience running production workloads on AWS: EC2, ECS, S3, IAM, DynamoDB, Lambda and CI/CD-driven deployments.

Prior knowledge of Splink’s platform, including its cost-optimized file exchange pipeline.

A proposal to script the refresh and handle the full schema, instead of working around DMS’s limits.

The solution

How we built it.

Layer 01

Triggered from GitHub Actions

A team member runs the workflow on demandThe workflow starts a dedicated Amazon EC2 instancePass or fail shows up next to the code and deployments

Layer 02

Scripted replication on EC2

Copies production data and schema into stagingRecreates materialized views and their dependents, which DMS skipsInstance is terminated after the run

Layer 03

Observable and reviewed like code

Every run logged to Amazon CloudWatchFailures diagnosed without logging in to the instanceWorkflow lives in the repository and goes through code review

Layer 04

AWS services used

Amazon EC2: runs the replication scriptAmazon RDS for PostgreSQL: production and staging databasesAmazon CloudWatch: logs for every runAWS IAM: workflow and instance permissionsAmazon EBS: working storage during a refreshGitHub Actions (non-AWS): trigger and result

Results & impact

What shipping it actually changed.

Refreshes that complete

Materialized views and dependent schemas come across intact, where DMS migrations failed.

Self-service for engineers

The team refreshes staging when it needs to, without waiting on DevOps.

Clear visibility

Status in GitHub Actions and full logs in CloudWatch for every run.

Realistic testing

New features and fixes are tested against current production data before release. Because the instance runs only during a refresh, Splink also avoids paying for a migration instance between refreshes.

Trade-offs we accepted

Planned for what a managed service handles.

Production data in staging

PII is masked after each restore, and staging access is restricted to Splink’s engineering team.

Load on production

Refreshes run off-hours, so they don’t compete with live traffic.

Failure handling

A failed refresh restores staging from a backup, so no one is left with a half-copied database.

Maintenance

Logiciel maintains the replication script and the EC2 instance.

Honest trade-offs

When AWS DMS is the better choice.

Continuous replication with change data capture.

Migrations between different database engines.

One-time migrations of schemas without materialized views or other unsupported objects.

DMS Serverless also removes the need to size and run a replication instance.

For Splink’s on-demand, same-engine refresh with materialized views, a scripted pipeline was the better fit.

Build to last

Automate the Work Your Managed Service Can’t.

If a managed service keeps hitting its limits, let’s look at a pipeline built for your workload.

Start Your Project