Next.js 14 AWS S3 file management: the ultimate file handling guide

Have you ever considered the power you have when integrating robust cloud storage solutions with a modern web framework? Not only for your side-project, but for your entire development career?

As a developer, you're no stranger to the complexities of file management in a dynamic Next.js application. By incorporating Amazon S3 into your Next.js 14 toolkit, you're not just storing files; you're unlocking a new realm of efficiency and scalability.

In this guide, we'll take a deep-dive into the landscape of S3 file management, from crafting a user interface that handles file uploads with an amazing user experience to securing your digital assets like a seasoned pro. With my guide you'll piece together the puzzle of seamless file synchronization, ensuring your users' data is always at their fingertips.

And by the end of this journey, the seamless integration of S3 with your Next.js app will seem less like a challenge and more like a great way to manage files in any of your current or future projects. Lets get to it and discover the true potential of your application.

The Next.js 14 application we'll be building and which tools we'll be using

To demonstrate the power of S3 file management in Next.js, we'll be building a social media application that allows users to upload files to their posts. I'll show you how to set up your S3 bucket and integrate it within the Next.js application. Here are some of the topics we'll be covering:

Topics covered in this tutorial

1. Best practices for file organization in Next.js (directories, components, pages, utilities)

2. Efficient file streaming or chunking mechanisms to anticipate heavy traffic and manage timeouts and other network issues

3. Integrating a database using Prisma and PostgreSQL to keep track of the files' metadata and associations with user accounts or posts

4. Drag and drop file upload in Next.js for a better user-experience

5. Securing file uploads with authentication and authorization using Clerk

Stack & tools used in this tutorial

1. Next.js 14
2. Clerk authentication
3. AWS S3 file uploads
4. Postgres database with Prisma ORM
5. ShadCN/UI and Tailwind CSS

Enough talk, let's get to the code!

Creating our Next.js 14 social media boilerplate

Lets get started and create a new Next.js application with the following command:

npx create-next-app@latest nextjs14-aws-s3-example --typescript --tailwind --eslint

You'll be asked a few questions, these are the answers I've used if you want to follow along:

✔ Would you like to use `src/` directory? … No
✔ Would you like to use App Router? (recommended) … Yes
✔ Would you like to customize the default import alias (@/*)? … No

Once you've created your Next.js application, let's start setting up shadcn/ui for our UI elements. To initialize shadcn/ui, run the following command:

npx shadcn-ui@latest init

You'll be asked a few questions to configure your components.json file:

Would you like to use TypeScript (recommended)? yes
Which style would you like to use? › Default
Which color would you like to use as base color? › Slate
Where is your global CSS file? › › app/globals.css
Do you want to use CSS variables for colors? › yes
Are you using a custom tailwind prefix eg. tw-? (Leave blank if not) …
Where is your tailwind.config.js located? › tailwind.config.ts
Configure the import alias for components: › @/components
Configure the import alias for utils: › @/lib/utils
Are you using React Server Components? › yes
Write configuration to components.json. Proceed? › (Y/n) Y

An important thing to note here is that by default shadcn/ui will assume your tailwinf.config.js is located in the root of your project. Since we're using TypeScript, we'll need to change this to tailwind.config.ts.

I use Inter as the default font. Inter is not required. You can replace it with any other font. If you'd like to learn more, check out the shadcn/ui fonts section of the documentation.

Setting up Clerk authentication

Before I dive into the code, I'd like to explain why I've chosen Clerk as the authentication provider for this project. Clerk is a developer-first authentication solution that allows you to build secure applications with ease. It's a great fit for this project because it allows us to focus on the file management aspect of the application, without having to worry about authentication and authorization.

I've written a detailed guide on how to set up Clerk with Next.js 14, which you can find here: Next.js 14 authentication with Clerk, so I won't go into too much detail here. If you'd like to follow along, you can create a free Clerk account here.

Let's clean up our main page and prepare for our Clerk integration.

const Home = () => {
  return (
    <main className="grid place-content-center min-h-screen">
      AWS S3 file upload
    </main>
  );
};
 
export default Home;

Next go to Clerk.js and create your free account. Once you've logged into your Clerk account, set up a project. We'll use the Next.js quickstarts, so select Next.js and copy the API keys into your .env file. The reason I'm not using the .env.local file is to save some time later so we won't have to point our Prisma client to use .env.local instead of .env.

NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=pk_test_ZHJpdmVuLXJlaW5kZWVyLTQuY2xlcmsuYWNjb3VudHMuZGV2JA
CLERK_SECRET_KEY=sk_test_••••••••••••••••••••••••••••••••••••••••••

Next click on Continue in docs or following along with my tutorial.

Before we integrate anything into our application, we'll need to install the Clerk SDK. To do this, run the following command:

npm install @clerk/nextjs

Now it's time to wrap our app in <ClerkProvider>. Since we're using App Router we'll wrap our <html /> element with ClerkProvider. By doing so, our active session and user context is accessible anywhere within our app. We will also give our layout some default styling by adding some classes to the body and using our { cn } import to add some utility classes. This is provided by shadcn/ui and uses a package called tailwind-merge. Don't forget to add suppressHydrationWarning to your <html /> element, otherwise you'll get a warning in your console.

import { ClerkProvider } from "@clerk/nextjs";
import "./globals.css";
 
import { cn } from "@/lib/utils";
 
export default function RootLayout({
  children,
}: {
  children: React.ReactNode,
}) {
  return (
    <ClerkProvider>
      <html lang="en" suppressHydrationWarning>
        <body
          className={cn(
            "bg-white text-black dark:bg-black dark:text-white",
            inter.className
          )}
        >
          {children}
        </body>
      </html>
    </ClerkProvider>
  );
}

Now that our entire application is protected by the ClerkProvider, we have to add a middleware.ts file to help decide which pages are protected and require authentication, as well as which pages are public and don't require authentication. You have to create the middleware.ts file in the root of your directory, as this is a Next.js practice and has nothing to do with Clerk.

import { authMiddleware } from "@clerk/nextjs";
 
// This example protects all routes including api/trpc routes
// Please edit this to allow other routes to be public as needed.
// See https://clerk.com/docs/references/nextjs/auth-middleware for more information about configuring your Middleware
export default authMiddleware({});
 
export const config = {
  matcher: ["/((?!.+\\.[\\w]+$|_next).*)", "/", "/(api|trpc)(.*)"],
};

Now that we've set up our Clerk authentication, try accessing your app by visiting http://localhost:3000. The middleware will instantly redirect you to the Sign In page, provided by Clerk's Account Portal. If you'd like to adjust your middleware and add any other pages to this application that you'll want to make publicly available, check out the authMiddleware page in the Clerk documentation.

Now that we have set up our Clerk authentication, let's start setting up a simple skeleton UI we can use for our social media application.

Creating a basic UI for our social media app

Let's start by creating a simple layout for our application. We'll use the Layout component to wrap our entire application. This will allow us to add a navigation bar and footer to our application.

In our <body> tag, let's add the navigation to start with.

<body className={inter.className}>
  <header>
    <Navigation />
  </header>
  <main className="m-auto max-w-lg">{children}</main>
</body>

As we haven't created our <Navigation /> component yet, this will give us an error. So let's go ahead and create our <Navigation /> component.

For the sake of simplicity, we'll stick with a pretty basic navigation. You're free to build out this application or make a pull request (PR) if you'd like to add more features to this application.

Creating our navigation/navbar

In our components folder, not within the ui folder, let's create a new file called navigation.tsx.

Now let's add the following code to our navigation.tsx file:

import { UserButton } from "@clerk/nextjs";
 
import Link from "next/link";
 
const Navigation = () => {
  return (
    <nav className="sticky h-14 inset-x-0 top-0 z-30 w-full bg-white/75 backdrop-blur-lg transition-all">
      <div className="mx-auto w-full max-w-lg px-2.5">
        <div className="flex h-14 items-center justify-between">
          <Link href="/" className="flex z-40 font-semibold">
            <span>AWS S3 Upload Example</span>
          </Link>
 
          <UserButton afterSignOutUrl="/" />
        </div>
      </div>
    </nav>
  );
};
 
export default Navigation;

Now don't forget to import your <Navigation /> component inside your app/layout.tsx file.

import Navigation from "@/components/navigation";
 
// Rest of your default layout code here
 
export default function RootLayout({
  children,
}: {
  children: React.ReactNode,
}) {
  return (
    <ClerkProvider>
      <html lang="en">
        <body className={inter.className}>
          <header>
            <Navigation />
          </header>
          {children}
        </body>
      </html>
    </ClerkProvider>
  );
}

Your page should now look like this:

Now that we have set up our basic navbar, lets start setting up a simple componen that will allow us to upload files to our application.

Setting up our file upload component

Now that we have created our <Navigation /> component and integrated our Clerk <UserButton /> we can start creating a simple social media feed, in which we can display the posts and files we attach to them when we upload these to our AWS S3 bucket.

As we'll be using Gravatar images for our social feed, we'll need to whitelist the hostname to avoid getting an unconfigured host error later.

Go into your next.config.js file and add the following code:

/** @type {import('next').NextConfig} */
const nextConfig = {
  images: {
    remotePatterns: [
      {
        protocol: "https",
        hostname: "www.gravatar.com",
      },
    ],
  },
};
 
module.exports = nextConfig;

Now that we have whitelisted our gravatar remote pattern, we can start creating our <CreatePostForm /> component. Head over to your components folder and create a new file called create-post-form.tsx.

Since we'll be using useState for our content, the loading status, and status messages, we'll need to make this a client component, by adding "use client" at the top of our new component. If you'd like to learn more about Client Components make sure to check out the Next.js documentation here.

Now go back into your create-post-form.tsx file and paste the boilerplate component that I've created:

"use client";
 
import { useState } from "react";
import { Paperclip } from "lucide-react";
 
import Image from "next/image";
 
import { cn } from "@/lib/utils";
 
interface CreatePostFormProps {
  user: {
    name?: string | null,
    image?: string | null,
  };
}
 
export default function CreatePostForm({ user }: CreatePostFormProps) {
  const [content, setContent] = useState("");
 
  const [statusMessage, setStatusMessage] = useState("");
  const [loading, setLoading] = useState(false);
 
  const postButtonDisabled = content.length < 1 || loading;
 
  const handleSubmit = async (e: React.FormEvent<HTMLFormElement>) => {
    e.preventDefault();
 
    setStatusMessage("Creating post");
    setLoading(true);
 
    // Process the image upload etc...
 
    setStatusMessage("Post created");
    setLoading(false);
  };
 
  return (
    <>
      <form
        className="border border-neutral-200 rounded-lg px-6 py-4"
        onSubmit={handleSubmit}
      >
        {statusMessage && (
          <p className="bg-yellow-100 border border-yellow-400 text-yellow-700 px-4 py-3 mb-4 rounded relative">
            {statusMessage}
          </p>
        )}
 
        <div className="flex gap-4 items-start pb-4 w-full">
          <div className="rounded-full h-12 w-12 overflow-hidden relative">
            <Image
              className="object-cover"
              src={user.image || "https://www.gravatar.com/avatar/?d=mp"}
              alt={user.name || "avatar"}
              priority={true}
              fill={true}
            />
          </div>
 
          <div className="flex flex-col gap-2 w-full">
            <div>{user.name}</div>
 
            <label className="w-full">
              <input
                className="bg-transparent flex-1 border-none outline-none"
                type="text"
                placeholder="Create a new post..."
                value={content}
                onChange={(e) => setContent(e.target.value)}
              />
            </label>
 
            {/* File preview */}
 
            <label className="flex">
              <Paperclip
                className="w-5 h-5 hover:cursor-pointer transform-gpu active:scale-75 transition-all text-neutral-500"
                aria-label="Attach media"
              />
 
              <input
                className="bg-transparent flex-1 border-none outline-none hidden"
                name="media"
                type="file"
                accept="image/jpeg,image/png,image/webp,image/gif,video/mp4,video/webm"
              />
            </label>
          </div>
        </div>
 
        <div className="flex justify-between items-center mt-5">
          <div className="text-neutral-500">Characters: {content.length}</div>
          <button
            type="submit"
            className={cn(
              "border rounded-xl px-4 py-2 disabled",
              postButtonDisabled && "opacity-50 cursor-not-allowed"
            )}
            disabled={postButtonDisabled}
            aria-disabled={postButtonDisabled}
          >
            Post
          </button>
        </div>
      </form>
    </>
  );
}

We can now import this into our app/page.tsx file and render it on our page. This way we can keep an eye on the changes we make. Go back to your app/page.tsx file and add the following code:

import CreatePostForm from "@/components/create-post-form";
 
const testUser = {
  name: "John Doe",
  image: "https://www.gravatar.com/avatar/?d=mp",
};
 
const Home = () => {
  return (
    <div className="py-12 max-w-lg mx-auto">
      <CreatePostForm user={testUser} />
    </div>
  );
};
 
export default Home;

Explaining our boilerplate code for the CreatePostForm component

Let's stop for a second and briefly explain the boilerplate component that I created for this tutorial, our <CreatePostForm />. This component is essentially a form that allows users to create a new post. It's designed with a pretty simple interface, so if you're up for it try to go to town and make your own design.

The form includes fields for the user's name and profile image which we pass down as a prop. It also includes a field for the post content, a field for uploading files, and a submit button that's disabled by default until we have more than 1 character, or if the form is submitting and we trigger the loading state.

We use the useState hook from React to manage the state variables like content, statusMessage, and loading. These states help us give feedback to the user during the post creation process and make it a better user-experience.

The core functionality of the form lies in the handleSubmit function, triggered when the user submits the form. It simulates the process of creating a post, updating the status message and loading state accordingly. In a real-world application, this is where you would handle tasks like image uploads and post creation on the server, so this is where we'll be adding our S3 file upload logic later on.

Preparing our CreatePostForm component for file uploads

Before we can start setting up our S3 bucket, bucket permissions and CORS configuration, we need to make some changes to our <CreatePostForm /> component. First of all, let's make sure we can keep track of the file in state:

const [file, setFile] = (useState < File) | (null > null);
// ...
const handleFileChange = (e: React.ChangeEvent<HTMLInputElement>) => {
  const file = e.target.files?.[0] ?? null;
};
// ...
<input
  type="file"
  // ...
  onChange={handleFileChange}
/>;
 
const handleSubmit = async (e: React.FormEvent<HTMLFormElement>) => {
  e.preventDefault();
 
  setStatusMessage("Creating post");
  setLoading(true);
 
  // Process the image upload etc...
  console.log(content, file);
 
  setStatusMessage("Post created");
  setLoading(false);
};

Pay attention: make sure you do not replace your text input's onChange effect, but find the file input below our <PaperClip /> icon that does not have an onChange effect yet.

Now that we have added our file state and handleFileChange function, we can display the file to the user.

const [previewUrl, setPreviewUrl] = (useState < string) | (null > null);
 
const handleFileChange = (e: React.ChangeEvent<HTMLInputElement>) => {
  const file = e.target.files?.[0] ?? null;
  setFile(file);
  if (previewUrl) {
    URL.revokeObjectURL(previewUrl);
  }
  if (file) {
    const url = URL.createObjectURL(file);
    setPreviewUrl(url);
  } else {
    setPreviewUrl(null);
  }
};

Now replace our {/* File preview */} comment with the following code, so that we can actually display the preview file video or image. Note: normally I would use the next/image component for this, but for the sake of simplicity I've used a regular <img /> tag.

{
  previewUrl && file && (
    <div className="mt-4">
      {file.type.startsWith("image/") ? (
        // eslint-disable-next-line @next/next/no-img-element
        <img src={previewUrl} alt="Selected file" />
      ) : file.type.startsWith("video/") ? (
        <video src={previewUrl} controls />
      ) : null}
    </div>
  );
}

You should now be able to upload a file and see the preview of the file you've uploaded.

Now that we've set up our <CreatePostForm /> component, let's start go back to what we really want to do, which is upload the file to S3. There's two things that we want to have happen here:

1. Upload the file to our S3 bucket
2. Store any (meta)data about the file in our database

Understanding AWS S3 file storage

Grasping the fundamentals of S3 file storage is crucial for managing uploads and retrievals within your Next.js social media app. Understanding S3 storage costs is essential; you'll pay for the storage space used and the transfer of data in and out of S3. It's a pay-as-you-go service, so optimizing your files for size can save you money.

Implementing file versioning in S3 is a smart move. It ensures that every change to a file is tracked, allowing you to restore previous versions if needed. This feature is a lifesaver when dealing with accidental deletions or unintended overwrites.

To enhance user experience, integrating S3 with a CDN (Content Delivery Network) is key for faster file delivery. This reduces latency since files are served from the closest geographical location to your users.

Don't overlook security; implementing file encryption in S3 protects your data at rest. It's a straightforward process that adds an additional layer of security to your files.

Lastly, implementing file access control in S3 is about managing who can view or edit your files. Properly configured access policies prevent unauthorized access, keeping your users' data private and secure.

Creating an AWS S3 bucket

Now that we have a basic understanding of AWS S3 file storage, let's start setting up our S3 bucket. Head over to your AWS console and search for S3. Once you're in the S3 dashboard, click on Create bucket.

Now give your bucket a name and select the region closest to you. For this tutorial, I've selected Europe (Ireland) eu-west-1.

Uncheck Block all public access and check "I acknowledge that the current settings might result in this bucket and the objects within becoming public."

Once you've done that, scroll all the way down and click on Create bucket. This will take us to the overview page where you'll now see the newly created bucket. Click the name and you'll be taken to the bucket overview page.

Setting up permissions for our S3 bucket

To get started, click on the Permissions tab. Now it's time to set up our Bucket Policy. Scroll down to the Bucket policy section, click on Edit and paste in the following policy, replacing BUCKET_NAME with the name of your bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicReadGetObject",
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::BUCKET_NAME/*"]
    }
  ]
}

It's important to note that after doing this, you'll have to create an access key. So click View user to open the newly created user.

Under the Summary click on Create access key. If you can't find it, I've attached a screenshot down below to point out where you can create your access key.

Copy the access key and secret key as environment variables, we'll need them later.

AWS_ACCESS_KEY=
AWS_SECRET_ACCESS_KEY=
AWS_BUCKET_NAME=
AWS_BUCKET_REGION=

This policy allows anyone (Principal: *) to read (Action: s3:GetObject) any object in the bucket (Resource: arn:aws:s3:::BUCKET_NAME/_). The "Effect" is set to "Allow," indicating that this permission is granted. The policy applies to all objects within the specified bucket.

Your Bucket policy should now look like this, with your bucket name in there.

Now that we have set up the policy for our S3 bucket, we can focus on the CORS configuration. If you left the permissions tab, go back to Permissions, otherwise scroll down to the Cross-origin resource sharing (CORS) section. Click on Edit and paste in the following CORS configuration:

[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["PUT", "GET"],
    "AllowedOrigins": ["*"],
    "ExposeHeaders": [],
    "MaxAgeSeconds": 3000
  }
]

Hit save, and we're good to go.

Creating an IAM user for our S3 bucket

Now that we have set up our S3 bucket, we need to create an IAM user that has access to our S3 bucket. But before we do, here's a short explanation of what an IAM user is and why we need one.

What is an IAM user and why do we need one?

An IAM (Identity and Access Management) user is a resource within AWS (Amazon Web Services) that represents an individual or an application that interacts with AWS services. IAM users are used to securely manage access to AWS resources and services.

IAM users have unique credentials (username and password or access keys) that are used to authenticate and authorize their access to AWS resources. Each IAM user can have specific permissions and policies assigned to them, which determine what actions they can perform and what resources they can access within AWS.

IAM users are typically used to grant access to different individuals or entities within an organization, allowing them to manage and interact with AWS resources based on their specific roles and responsibilities. By using IAM users, you can enforce security best practices, limit access to sensitive resources, and track actions performed by different users within your AWS environment.

In the context of this project, creating an IAM user for our S3 bucket will allow us to grant specific permissions to that user, enabling them to interact with the bucket and perform actions such as uploading, downloading, and managing files within the bucket.

Creating our IAM user and policy

Now that you know what an IAM user is and why we need one, we need to create an IAM user that has access to our S3 bucket. To do this, go to your AWS console and search for IAM.

Before we can continue, we'll have to set up a policy that we can attach to our IAM user. In the IAM dashboard, click on Policies under the Access management dropdown and then click on Create policy.

Make sure you change the Visual policy editor to JSON so you can copy and paste my policy.

Down below you'll find the policy that we'll be using. Make sure to replace BUCKET_NAME with the name of your bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualFileEditor0",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::BUCKET_NAME/*"
    }
  ]
}

Lets break this policy down a little more:

• Version - The version of the policy language that you want to use.
• Statement - The policy statement that lists the permissions that you want to grant.
• Sid - An identifier for the policy statement.
• Effect - Whether the statement allows or denies access. In our case this is set to "Allow", indicating that the defined actions are permitted.
• Action - Two actions, "s3:PutObject" and "s3:DeleteObject," are specified. This means that any entity associated with this policy is allowed to upload (PutObject) and delete (DeleteObject) objects within the specified S3 bucket.
• Resource - The "Resource" field narrows down the scope of this policy to the objects within a specific S3 bucket. The Amazon Resource Name (ARN) arn:aws:s3:::BUCKET_NAME/* denotes all objects within the BUCKET_NAME bucket.

In essence, this IAM policy empowers users or entities associated with it to perform two crucial actions—putting and deleting objects—within the confines of the BUCKET_NAME S3 bucket. This level of granularity ensures that users have the necessary permissions for their tasks without granting unnecessary access to other resources.

Now that IAM is all clear, let's continue.

Give your policy a name, such as next-s3-file-upload-policy, and optionally a description if you'd like to. After creating your policy, you'll be redirected to the IAM overview page again.

Once you're in the IAM dashboard, click on Users under the Access management dropdown and then click on Create user. Enter a username, e.g. next-s3-file-upload and click on Next. Don't enable "Provide user access to the AWS Management Console". After you click on Next, it's time to make a new policy and attach it.

Finally, we're ready to start using the AWS SDK and integrate this into our Next.js 14 app.

Uploading files to S3 using the AWS SDK

Giving the client direct access to S3 is a bad idea. It's a security risk as the user would be able to upload whatever they want and it's not scalable. Instead, we need to use the AWS SDK to generate a presigned URL that we can use to upload the file. This way we can control the file size and type, our server can verify who the user is, and we can also set an expiration date for the URL if needed.

Setting up a server action to generate our presigned URL

Let's create a server action that wil generate a presigned URL that the client can use to upload the file. In the root of your project create a new folder called actions, and inside this folder create a new file called index.ts.

Now add the following code to your actions/index.ts file:

import { auth } from "@clerk/nextjs";
 
type SignedURLResponse = Promise<
  | { error?: undefined; success: { url: string } }
  | { error: string; success?: undefined }
>;
 
export async function getSignedURL(): Promise<SignedURLResponse> {
  const { userId } = auth();
 
  if (!userId) {
    return { error: "User is not authenticated" };
  }
 
  return { success: { url: "" } };
}

In this action we import { auth } from @clerk/nextjs. This allows us to check if the user is authenticated. If the user is not authenticated, we return an error. If the user is authenticated, we return a success object with an empty string as the URL. We'll replace this empty string with the actual URL later on.

We can now go ahead and call this action from the client in our <CreatePostForm /> component.

import { getSignedURL } from "@/actions";
// ....
const handleSubmit = async (e: React.FormEvent<HTMLFormElement>) => {
  e.preventDefault();
  // ...
  if (file) {
    const signedURLResult = await getSignedURL();
    if (signedURLResult.error !== undefined) {
      console.error(signedURLResult.error);
      return;
    }
 
    const { url } = signedURLResult.success;
    console.log({ url });
  }
};

With that in place, we can install the required packages to use the AWS SDK. Run the following command in your terminal:

npm i @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

After installing these packages, update your actions/index.ts file with the following code:

"use server";
 
import { auth } from "@clerk/nextjs";
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
 
const s3Client = new S3Client({
  region: process.env.AWS_BUCKET_REGION!,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
  },
});
 
type SignedURLResponse = Promise<
  | { error?: undefined; success: { url: string; id: number } }
  | { error: string; success?: undefined }
>;
 
export async function getSignedURL(): Promise<SignedURLResponse> {
  const { userId } = auth();
 
  if (!userId) {
    return { error: "User is not authenticated" };
  }
 
  const putObjectCommand = new PutObjectCommand({
    Bucket: process.env.AWS_BUCKET_NAME!,
    Key: "s3-test-file",
  });
 
  const url = await getSignedUrl(
    s3Client,
    putObjectCommand,
    { expiresIn: 60 } // 60 seconds
  );
 
  return { success: { url, id: 123 } };
}

We're now ready to test this code and see what's being console.logged. If you run into any error, make sure you copied your AWS variables into your .env file correctly.

If you're not sure of what your <CreatePostForm /> should look like, check out this gist with the correct create-post-form.tsx file: view create-post-form.tsx gist here

After attaching a file and entering a short message, you can click on post and a URL should be logged in your console.

Great! We can now go ahead and make sure that the client code actually sends the file to our S3 bucket.

Client side uploading of the file to our S3 bucket

To make sure the client can upload our file to S3, we have to update our handleSubmit function inside of our <CreatePostForm /> component.

const handleSubmit = async (e: React.FormEvent<HTMLFormElement>) => {
  e.preventDefault();
  // ...
  if (file) {
    const signedURLResult = await getSignedURL();
    if (signedURLResult.error !== undefined) {
      console.error(signedURLResult.error);
      return;
    }
 
    const { url } = signedURLResult.success;
    console.log({ url });
    await fetch(url, {
      method: "PUT",
      headers: {
        "Content-Type": file.type,
      },
      body: file,
    });
  }
};

Now go back to your S3 bucket, click the name of your bucket and you'll see one object there, called test-file. You should be able to view it using the URL.

🎉 Yay, it's working - but don't lean back yet, we need to make some improvements. Right now we're doing nothing to ensure that the user actually uploads an image or video, and they could upload any file of any size. Our form input currently allows:

accept = "image/jpeg,image/png,image/webp,image/gif,video/mp4,video/webm";

There is no server side validation, so we have to add a couple of checks on the server to:

1. Make sure the file is one we accept (image or video)
2. Set a limit for the maximum file size that we're accepting

3. Use a hash of the file to make sure there's no issues during transport from the client to S3.

Let's go back to the function that generates our signed url, in our actions folder.

const allowedFileTypes = [
  "image/jpeg",
  "image/png",
  "video/mp4",
  "video/quicktime"
]
 
const maxFileSize = 1048576 * 10 // 1 MB
 
type GetSignedURLParams = {
  fileType: string
  fileSize: number
  checksum: string
}
export async function getSignedURL({
  fileType,
  fileSize,
  checksum,
}: GetSignedURLParams): SignedURLResponse {
  const { userId } = auth();
 
  if (!userId) {
    return { error: "User is not authenticated" };
  }
 
  // check for allowed file types
  if (!allowedFileTypes.includes(fileType)) {
    return { error: "File type is not allowed" };
  }
 
  // check against the maximum file size
  if (fileSize > maxFileSize) {
    ret
 
  const putObjectCommand = new PutObjectCommand({
    Bucket: process.env.AWS_BUCKET_NAME!,
    Key: "test-file",
    ContentType: fileType,
    ContentLength: fileSize,
    ChecksumSHA256: checksum,
    // metadata for our user
    Metadata: {
      userId: userId,
    },
  });
 
// ...
}

We now need to make sure that we pass a hash from the client. Let's go back into our <CreatePostForm /> component and implement some code to create our hash.

const computeSHA256Hash = async (file: File) => {
  const buffer = await file.arrayBuffer();
  const hashBuffer = await crypto.subtle.digest("SHA-256", buffer);
  const hashArray = Array.from(new Uint8Array(hashBuffer));
  const hashHex = hashArray
    .map((b) => b.toString(16).padStart(2, "0"))
    .join("");
  return hashHex;
};
 
const signedURLResult = await getSignedURL({
  fileSize: file.size,
  fileType: file.type,
  checksum: await computeSHA256Hash(file),
});

The computeSHA256Hash function takes the file object as input and asychronously computes its SHA-256 hash. Our function then reads the file content as an array buffer, and uses the Web Crypto API's crypto.subtle.digest method to generate the hash in a buffered form. The hash is converted into a hexadecimal representation, providing a unique identifier for the file's content.

Let's upload a file again and check our changes. You should notice that there is still only one file, but the user ID has been added to the metadata and we can only upload files that are allowed. Let's go back to our test-file key and update this to be something else!

Generating a unique name for each upload

Considering the eventual visibility of this URL to the public, we'll adopt a strategy inspired by platforms like Discord. In private direct messages on Discord, the image URLs are accessible to anyone on the internet, yet they remain secure due to their unguessable nature. Following this lead, we'll leverage the power of crypto to dynamically generate a random and unguessable string. This ensures that our URLs, while publicly accessible, maintain a robust layer of security, aligning with industry best practices and offering a level of safety akin to widely-used platforms.

Go back into your actions/index.ts file and add the following changes:

import crypto from "crypto";
// ...
const generateFileName = (bytes = 32) =>
  crypto.randomBytes(bytes).toString("hex");
// ...
const fileName = generateFileName();
 
const putObjectCommand = new PutObjectCommand({
  Bucket: process.env.AWS_BUCKET_NAME!,
  Key: fileName,
  ContentType: fileType,
  ContentLength: fileSize,
  ChecksumSHA256: checksum,
  // metadata for our user
  Metadata: {
    userId: userId,
  },
});

If you're running this as an edge function, you can instead use the following code:

const generateFileName = (bytes = 32) => {
  const array = new Uint8Array(bytes);
  crypto.getRandomValues(array);
  return [...array].map((b) => b.toString(16).padStart(2, "0")).join("");
};

This has been quite a lot to take in, so I'm going to cut this tutorial short here. In the next blog post, published next week, we'll start implementing a database, add the code that allows us to display the posts and add functionality that allows users to delete their posts.

If you have any questions or run into any issues, hit me up on X (Twitter) or LinkedIn and I'll try to help you out!