If you are piping large file uploads through your Node.js backend, you are building a bottleneck. I've seen production servers crash because a single 2GB video upload exhausted the RAM or blocked the event loop. The solution isn't a bigger server; it's getting out of the way.
AWS S3 Multipart Upload with Presigned URLs is an architectural pattern where the client uploads file chunks directly to S3 using temporary, secure tokens generated by the server. This bypasses the backend entirely for data transfer, reducing server load to near zero.
The "Valet Parking" Analogy
Concept: Imagine a busy hotel (Your App).
The Old Way (Server-Side Upload): A guest arrives with huge luggage. The receptionist (Your Server) leaves the desk, carries the luggage all the way to the storage room (S3), and then comes back. During this time, the phone rings unanswered, and new guests wait in line.
The New Way (Presigned URL): The receptionist stays at the desk and simply hands the guest a temporary access key to the storage room. The guest carries their own luggage. The receptionist remains free to handle new requests instantly.
In technical terms, instead of streaming data Client -> Server -> S3, we orchestrate Client -> S3. The server only handles authentication and "signing" the permission slips.
Production Implementation (Node.js SDK v3)
To implement this securely, we use AWS SDK v3. The process has three distinct phases: Initiate, Sign, and Complete.
Why Multipart? For files larger than 100MB, standard uploads are unstable. Multipart allows you to upload chunks in parallel and retry only failed chunks, not the whole file.
// Required packages:
// npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner
import {
S3Client,
CreateMultipartUploadCommand,
UploadPartCommand,
CompleteMultipartUploadCommand
} from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
const s3 = new S3Client({ region: "us-east-1" });
const BUCKET = "my-production-bucket";
// 1. INITIATE: Call this first to get an UploadId
export const startUpload = async (fileName) => {
const command = new CreateMultipartUploadCommand({
Bucket: BUCKET,
Key: fileName,
});
return await s3.send(command); // Returns { UploadId, Key }
};
// 2. SIGN PARTS: Generate secure URLs for each chunk
export const getPartUrls = async (fileName, uploadId, partNumbers) => {
const promises = partNumbers.map((partNumber) => {
const command = new UploadPartCommand({
Bucket: BUCKET,
Key: fileName,
UploadId: uploadId,
PartNumber: partNumber,
});
// Link expires in 15 minutes (900 seconds)
return getSignedUrl(s3, command, { expiresIn: 900 });
});
return Promise.all(promises);
};
// 3. COMPLETE: Finalize after client uploads all parts
export const finishUpload = async (fileName, uploadId, parts) => {
// parts = [{ ETag: "...", PartNumber: 1 }, ...]
// Parts must be sorted by PartNumber!
const sortedParts = parts.sort((a, b) => a.PartNumber - b.PartNumber);
const command = new CompleteMultipartUploadCommand({
Bucket: BUCKET,
Key: fileName,
UploadId: uploadId,
MultipartUpload: { Parts: sortedParts },
});
return await s3.send(command);
};
The "Zombie Upload" Trap: If a user starts an upload but closes the tab halfway, the uploaded chunks sit in S3 forever, costing you storage fees. You MUST configure an S3 Lifecycle Rule to "Abort incomplete multipart uploads" after 3-7 days.
Frequently Asked Questions
Q. How do I handle CORS errors?
A. Since the browser uploads directly to S3, your bucket needs a CORS configuration. In the S3 Console > Permissions > CORS, allow your frontend domain (e.g., https://myapp.com) and expose the ETag header, which is required for the completion step.
Q. What is the optimal chunk size?
A. The minimum chunk size for S3 Multipart is 5MB (except for the last part). A standard practice is to use 5MB-10MB chunks for reliable parallelization on mobile networks.
Q. Is this secure?
A. Yes. The Presigned URL is valid only for a short time (e.g., 15 minutes) and can be restricted to a specific file name and operation. Your private AWS credentials never leave the server.
Post a Comment