Moving Mountains of Data off S3 with Jeremy Daer

In this episode of RECORDABLES, we talk through the final and most nerve-racking part of our cloud exit — moving massive amounts of data out of Amazon S3. Principal Programmer Jeremy Daer shares how we moved billions of files with no downtime. He covers everything from dealing with bandwidth limits and AWS constraints to building custom tooling when off-the-shelf options won’t work.

The conversation gets into the human side of a project like this, including verification, anxiety, and the moment you finally hit delete. You’ll also hear how long it actually takes to move that much data and the tools we used to make it happen seamlessly.

*Timestamps*

00:00:00 – Introduction
00:02:05 – Why S3 was the last (and scariest) piece
00:08:34 – The volume of data to move
00:11:11 – Bandwidth limits and AWS constraints
00:13:12 – The custom-built Rails tool for copying and reconciliation
00:21:25 – The logistics of hard drives, write speeds, and network connections
00:28:05 – The intentional order of moving data
00:49:55 – Anxiety, verification, and the fear with deleting data you can’t get back
00:54:13 – Was there any downtime?
00:58:56 – Essential tools that made the migration possible
01:07:03 – What happens next

*Links*

Rclone – https://rclone.org/
DuckDB – https://duckdb.org/
S3 Fast List – https://github.com/aws-samples/s3-fast-list

For the full episode transcript, visit https://dev.37signals.com/