Uploading large satellite images while keeping the web app usable

Users originally uploaded satellite imagery with a prescribed rsync command. The images could be very large; some were up to 1 TB. A new product requirement was to let users start those uploads from the web interface.

I carried out three rounds of improvements to the browser uploader, from reviewing the system design to designing and implementing the frontend modules. My work also included error handling, API updates, and memory optimization.

Keeping the upload active while users worked

The upload needed to keep running as long as the user kept the service’s browser tab open. Users needed to monitor its progress and keep working in the application, so I aimed to minimize the uploader’s browser resource use.

Within this client-rendered application, moving between views did not require separate upload handling. Keeping uploads alive after the tab closed was outside the requirement.

Reading and uploading one chunk at a time

I did not read the entire file into memory. The browser’s main thread read the file as a byte stream. A Web Worker prepared the upload chunks, calculated checksums, and handled transmission to the backend.

I remember using upload chunks of about 64 KB, with infrastructure bandwidth constraints also part of the context. Each chunk was sent with its chunk number. Once that upload completed, I read and uploaded the next chunk.

Choosing sequential transfers for field debugging

Parallel uploads were possible. I chose to send chunks sequentially so I could focus on error handling during on-premises debugging.

Handling failures alongside the transfer

The uploader retried the failed chunk instead of restarting the file upload. Avoiding a restart mattered because the files were so large. This recovery logic covered transfer failures and checksum errors.

Moving chunk preparation, checksum calculation, and transmission into the Worker was intended to reduce their performance impact on the application.

Testing large transfers and retry behavior

Large-file uploads were tested on a test computer for every deployment. Integration tests checked recovery separately: the test arranged for the backend to return a chunk-upload failure response, then verified that the data sent again matched the previous packet.

The retry test checked the resent chunk’s data. I do not have throughput or memory measurements, or a record of the largest file used in the deployment checks.