Migrating uploads (images, videos, files) from a Linux server to AWS S3

I’ve a considerably large amount of content in the CMS already in use. The upload_file table in the database contains the URL of the file and the provider. If I am to move all the content to S3 from Linux, and change the provider in the provider column, will it work or break? How do I deal with the formats column?

1 Like

So I been working on this myself from local to S3.
So the steps you will need.

  1. Copy all images from server to S3
  2. The URL etc will still be uploads but it changes from /uploads to uploads I THINK so might need to have a migration on the database or depending on how you consume it in the database.

The issue is, I can replace URLs, but the thumbnails in admin panel stop loading. So wondering how to make this work!

Need to extend multiple files, I been working on it a while now myself.

But like card, card preview and edit form in the uploads components needs to be extended.

I would possible wait until 4 and make a plugin with it.

Ok, thank you! I look forward to the plugin! :slightly_smiling_face:

Well as I said, it might be.

One thing btw, I think I explained the url changes, but that’s only needed with cloud front.

If it’s just s3 I think the images can be stored a bit differently.

Any inputs on moving local to S3? Thank you!

@DMehaffy Can you please give some insights into moving media of the existing projects from local storage to the upload provider storage?

I’m guessing moving the images you can use aws cli. I mean is it on a server?
Ftp? Web interface Dowload all upload all? Are they in a docker container?

Zip all images download and upload them?

Issue is not with uploading images. I can move all images from local to AWS S3. The database for Strapi has a table ‘upload_file’. This table has columns - ‘format’, ‘url’, ‘provider’. I can modify the ‘url’ and ‘provider’ column with a query, by appending S3 link to all urls and by changing the ‘provider’ string. However, I need to fix the ‘format’ column to make the thumbnails work on CMS work. How do I do that?

Should not need to modify them :slight_smile:
It would still store it in there as /uploads/ etc for the S3 bucket or uploads/
The provider and format I think is internally for Strapi to use, or info itself not how it’s rendered.

Strapi itself will prefix the URL with the backend etc, unless you want to extend files using cloud URL :slight_smile:
I had to rewrite at work to use cloudurl when this was enabled. But for normal Uploads for S3 it should work just fine.

Hey there!
I know that this is old, but I am having the same issue where I host the uploaded images in an S3 bucket, and use a PostgreSQL database. URLs in the upload_file table are the absolute URLs, and they contain the bucket URL as a prefix. My issue is that I want to move the images to a new bucket with a different URL. Simply changing the configuration of Strapi will only change the URL for the new uploads (Correct me on this if I am wrong), but won’t change the pre-existing images’ URLs. How do I change the existing images’ URL to be prefixed with the new bucket URL?

My current approach is:

  1. Move the images to a new bucket
  2. Configure Strapi to upload on the new bucket
  3. Use SQL queries to modify the url and formats columns in the upload_file table

Is there any better way to do this?

2 Likes

I’m also facing the same issue while migrating assets to s3, is there any better approach to change the URL of already existing media files?

Hello there!

I wrote a SQL sentence to update the media URL inside the database (MySQL in my case).

  1. Move the images to a new bucket
  2. Configure Strapi to upload on the new bucket
  3. Use SQL queries to modify the url and formats columns in the upload_file table

Update url column.

UPDATE files
SET url = CONCAT('<YourBucketURL>', url);

Update formats column with image sizes.

UPDATE files 
SET formats = JSON_SET(formats, '$.large.url', CONCAT('<YourBucketURL>', JSON_UNQUOTE(JSON_EXTRACT(formats, '$.large.url')))),
formats = JSON_SET(formats, '$.small.url', CONCAT('<YourBucketURL>', JSON_UNQUOTE(JSON_EXTRACT(formats, '$.small.url')))),
formats = JSON_SET(formats, '$.medium.url', CONCAT('<YourBucketURL>', JSON_UNQUOTE(JSON_EXTRACT(formats, '$.medium.url')))),
formats = JSON_SET(formats, '$.thumbnail.url', CONCAT('<YourBucketURL>', JSON_UNQUOTE(JSON_EXTRACT(formats, '$.thumbnail.url'))))
WHERE formats IS NOT NULL;

Hope it helps! :grinning:

1 Like

Thanks @JaviRuiz for the insights with the formats field. I had to adapt the request to my PostgreSQL database, so in case anyone needs it someday here are the requests I used:

Update other formats urls (couldn’t manage to do it in one request so here are all of them). YOUR_BUCKET_URL must end with /

UPDATE files 
SET formats = jsonb_set(formats, '{thumbnail,url}', concat('"', YOUR_BUCKET_URL, 'thumbnail_', hash, ext, '"')::jsonb)
WHERE formats IS NOT NULL;

UPDATE files 
SET formats = jsonb_set(formats, '{small,url}', concat('"', YOUR_BUCKET_URL, 'small_', hash, ext, '"')::jsonb)
WHERE formats IS NOT NULL;

UPDATE files 
SET formats = jsonb_set(formats, '{medium,url}', concat('"', YOUR_BUCKET_URL, 'medium_', hash, ext, '"')::jsonb)
WHERE formats IS NOT NULL;

UPDATE files 
SET formats = jsonb_set(formats, '{large,url}', concat('"', YOUR_BUCKET_URL, 'large_', hash, ext, '"')::jsonb)
WHERE formats IS NOT NULL;

Update url field:

UPDATE files SET url=concat(YOUR_BUCKET_URL, hash, ext);

Update provider field:

UPDATE files SET provider='aws-s3';

With those 3 steps, you should be good to go!

1 Like