Need help with reviving strapi-plugin-import-content

Questions and Answers Strapi Backend
1

Hi all,

I’m a new user so I can only put 2 links in this message, so I couldn’t add the various github links and I prefixed them with github →

Currently there’s no easy way to import large datasets into Strapi but there is a very interesting guide on the Strapi website: How to create your own plugin on strapi (1/4)

This blog post is also accompanied by a matching repo: github → jbeuckm/strapi-plugin-import-content

However, as Strapi development moved forward since then, this plugin is not longer working. There are some forks already containing fixes, namely github → akoharu/strapi-plugin-import-content (solves the id vs _id problem) and github → khoazero123/strapi-plugin-import-content (solving the path changes and using the strapi-plugin-helper where possible).

By combining both fork’s changes and comparing the structure with a manually generated plugin (using strapi generate:plugin) I’m using my own fork to come up with a changeset that makes this plugin usable again.

So far I’ve had very mixed results: when following the tutorial and applying the changes from both forementioned forks, I get the plugin to run, EXCEPT the media import from url.
When using the plugin package directly, nothing works and even after applying the fixes, the plugin kinda works but the UI is broken since it’s still using sass instead of the newer styled components (webpack even refuses to parse the scss files)

I’m in dire need to have the media items imported properly from a url and so far this plugin is the only code I’ve seen that tries to do that. Most guides I find involve creating a FormData object and adding the uploaded file in it, to then send it off the the regular endpoints provided by the content-manager. This makes sense when using the frontend to upload images to the media library, however, this plugin is meant to parse files, containing urls, in an async way, so there’s no browser context. Creating FormData objects and sending requests to the api IN BULK doesn’t seem like a good approach here…

The call that consistently fails is the call to the uploadProvider upload method as seen here: strapi-plugin-import-content/importMediaFiles.js at master · jbeuckm/strapi-plugin-import-content · GitHub on lines 22-26

As I am not a Strapi developer, I’m feeling overwhelmed to study the Strapi internals just to revive the import plugin. Has anyone with a more profound knowledge of Strapi the solution to make this plugin functional again?

Kind regards,

Erik aka 4levels

1 Like
2

Hi, Erik
You don’t describe your task.
Do you need to do import once - or you need to do it regularly ?

3

Hi Gayrat,

this is supposed to be a one-off operation, to migrate and existing site from Wordpress to Strapi / Gatsby.

On the other hand, being able to import data via CSV / RSS, either via a file upload, or file url, or even a textarea, seems still very valuable to me and almost a core functionality of any CMS.

Besides that, the approach explained in the blog post is also very interesting as it takes you through the internals of Strapi plugin development without becoming too abstract.

I’ve also opened a Github issue with the original plugin repo, asking the owner for help and his future involvement in this plugin: Plugin outdated - not longer working · Issue #31 · jbeuckm/strapi-plugin-import-content · GitHub

4

Oh, and not to mention that the blog post even explains how to add a history and undo import feature, very impressive as those still work as intented.

5

I’ve added my changes, taken from both forks and with my own enhancements found by comparing with a newly generated plugin, to my own fork and made a PR as POC to see if the original plugin author cares to help out as well. PR link: [WIP] First attempt to revive this plugin by 4levels · Pull Request #32 · jbeuckm/strapi-plugin-import-content · GitHub

6

I’m did this import don’t using strapi.

  1. connect Strapi to MySQL or Postgress database (don’t use mongo because Strapi uses relation tables) (I’m use Mysql)
  2. If you wish you can turn on MySQL log - that will show all sql queries that strapi send to database
  3. Try to upload any picture using admin media library

You can see - that picture resized and stored in public\uploads directory and picture data stored in upload_file table.

I’m put all my pictures to public\uploads and write information about it to MySQL upload_file table.
If you want to resize your pictures you can use gulp, for example

You can also transfer your content using SQL queries from wordpress to strapi MySQL database (you need previously create some Strapi collections)

1 Like
7

Hi Gayrat,

thanks for elaborating on your vision to import media items in bulk. Working with the database directly is definitely easier for me since I know SQL quite well.

However, I do have some concerns with this:

  1. how would I connect Strapi to my database? AFAICT the database is part of Strapi and not something I can easily access directly
  2. investigating via SQL logs how the queries are supposed to look like feels rather cumbersome and and hacky, I’d rather see some code that generates the SQL statements (via the Strapi ORM) and work from there.
  3. how to upload the images (without adding them to the git repository) directly to Strapi

And finally, let’s not forget the elephant in the room here and the main subject of this thread: how to revive the strapi-plugin-import-content project that does all the above using Strapi’s tooling instead of the reverse-engineering approach you’re suggesting?

So I’ll start a new topic with as title “How to upload images programmatically without browser context”. Basically finding the answer to that question will enable me to revive the aforementioned plugin as that’s the only part currently failing, I’ve managed to solve the other issues already in my own fork (see the Github PR link above).

So thanks again for your insights on how to approach a bulk media import, but this solution is not what I’m currently looking for. I’m sure there’s a lot of Strapi users out there that would benefit from a neat way to import CSV/XML files that can contain URL’s to images as well. Spending time on reverse engineering the database and media internals of Strapi feels like a waste and will not result in a decent repeatable solution.

8

Hi Gayrat,

did you see this topic here? Upload image url
This is the approach that I also doubt as it involves fetching the images in bulk and creating formdata objects in bulk to post them to the regular upload endpoints. As I mentioned before, this approach doesn’t sit right with me either as I’m pretty sure the Node process will just eat up way to many resources when fetching the image, creating the FormData object and posting them to itself. I also need a way to get authorization working then and I’m not sure at all on how to access the authorization info in an async server-side process.

Thanks again!

9

Create an empty MySQL database.
Create Strapi project from scratch: run without --quickstart parameter

npx create-strapi-app my-project

You shall be asked about the database - select MySQL and input connection secrets.
Run Strapi and create collections - that you need.
Finally, you receive strapi MySQL database with full access.

10

I’ve been looking around here in the forums and the topic of having a way to upload images from a server-side process already exists. I’ll refer to these here as well as they are a key part of the solution

11

I’m used Openserver (Windows) and simply see log
But you only need finaly database scheme

I’m described steps that I’m do byself

12

Hi Gayrat,

don’t get me wrong here: I’ve been working with Linux and OpenSource IT for over 20 years so I know HOW to access a database, I just don’t see how creating a new project, doing the SQL manipulations you mention and having a local MySQL database is going to help get the images online. I really don’t want to start storing images in the git repo I’m using, that’s not what a git repository is meant for. I also still don’t see how I then get the local database into production.

FYI I’m using a Gatsby frontend and Platform.sh as service provider.

13

Strapi have several upload providers - fro example - Cloudinary, AWS S3

The standart upload provider - is filesystem.
At server side you need to get file from source and write it to public/uploads directory

At client side (from brauser) you can follow documentation:
https://strapi.io/documentation/developer-docs/latest/development/plugins/upload.html#upload-files

14

You can store database at any place.
In my case - I’m do backup database at local and restore at prod

Database connection is defined at file: config/database.js like this:

module.exports = ({ env }) => ({
  defaultConnection: 'default',
  connections: {
    default: {
      connector: 'bookshelf',
      settings: {
        client: 'mysql',
        host: env('DATABASE_HOST', '127.0.0.1'),
        port: env.int('DATABASE_PORT', 3306),
        database: env('DATABASE_NAME', 'strapi3'),
        username: env('DATABASE_USERNAME', 'root'),
        password: env('DATABASE_PASSWORD', 'root'),
        ssl: env.bool('DATABASE_SSL', false),
      },
      options: {}
    },
  },
});

secrets stored in .env file in project root

15

Hi Gayrat,

I appreciate your efforts here, but this is not what I’m looking for, the topic title clearly reads “reviving the strapi-plugin-import-content” and all you keep suggesting is a workaround that doesn’t add any value to the plugin nor for anyone else looking into importing lots of data with media, be it once or on a regular basis. The plugin does exactly that, except that the media import is not working.

The plugin uses an uploadProvider as seen here strapi-plugin-import-content/getUploadProvider.js at master · jbeuckm/strapi-plugin-import-content · GitHub, but I’m sure the way the provider is being called is deprecated. I also cannot find any documentation about the global strapi.store method. As mentioned by others, the upload documentation only shows how to use it from a browser context, not server side.

16

You can see this tutorial:

and this git :

17

Hi Gayrat,

nice find! I’ll be looking into that community-content version of the plugin.

FYI this plugin doesn’t work either out of the box, it doesn’t even show up in the admin.
I’m still curious on what is different from the original version though.

Looking at the package.json, it’s also depending on Strapi version 3.0. Last commit is also more than a year ago, so here’s to hoping…

Anyway, I’ll try to apply my new knowledge about Strapi plugin development to get the plugin to at least show up in the admin.

Regarding the first tutorial you linked (about developing your own plugin): I did follow that one a few weeks ago to be able to compare how a new plugin looks like vs. the existing import-content plugin. Actually, this is the tutorial that I followed to the letter, but still, media import not working. I even contacted Joe Beuckman directly via email to ask for help… that’s how desperate I’m becoming from all this. What a struggle, for a feature that should IMHO be built-in any CMS…

1 Like
18

I still don’t understand what you want :slight_smile:

You want to import media files on the server side.
Ok - it means - that must be the source from you want to import.

If source - user input from browser form - it is described in the standard documentation.

If source - a set of files from anywhere - you can simply copy it to public/upload

You can store any plugin data in collection - or in plugin store (undocumented) - worked at server side

Store value:
 const pluginStore = strapi.store({
      environment: strapi.config.environment,
      type: 'plugin',
      name: 'unique-string-store-id' 
    })

    const result = await pluginStore.set({ key: 'pk', value: pk })
....
Retrive value:
 const pluginStore = strapi.store({
      environment: strapi.config.environment,
      type: 'plugin',
      name: 'unique-string-store-id' 
    })

  const pk = await pluginStore.get({ key: 'pk' })
19

Hi Gayrat,
I figured I was still on my own here

I wont repeat everything I explained so far, sad I can’t seem to express myself well enough so you understand what I’m trying to achieve here

So I went ahead, started from scratch and wrote the plugin by following the tutorial (without using any code from the jbeuckm repo), adjusting along the way to make it work with the latest and greatest Strapi version. I wrote everything as functional components and hooks, no more class components and traditional react lifecycle handling.

To start off with the bad news: no matter what I’ve tried, I can’t seem to get the media import feature to work as expected. If you would still be able to investigate why, I’d appreciate a lot! More info down below.

The good news:
I managed to get the import to work with the i18n features in a way I can import a single record holding values in multiple languages, translating that to an internal linked list of records (as that’s how Strapi handles translations). I even added some language detection in the UI based on the import field names to ease the pain of selecting 8*x language dropdowns during testing with 8 languages :smile:

The amazing news:
I managed to import relations as well. Even on localized models. WIth optional creation of missing related records. Select the related record field to match, the separator to use, :partying_face:

The unbelievable news
I added options to allow updating existing records instead of creating new ones so I can re-import the same files and update records along the way. I mainly need this to handle better import files and update custom content parsing methods.

The approach is very optimistic, there’s still tons of things to improve, re-organize, etc etc, but so far it’s working for me. Somehow it’s very slow on large imports: as soon as I try with import files containing hundreds of records, analysing takes very long and crashes eventually.
Feel free to have a look at it here on Github:

The part specifically for the media import is here: and here’s the link to the section in the original blog post as well, documenting the approach:

I hope someone can still help with the media import. I’m afraid though that this topic has become a one-on-one conversation with no solution and no further involvement from others…

20

And now I just discovered what was breaking the media import all along!
This line, this very line is the culprit of me failing to import a Buffer, streamed from a URL straight into Strapi, from a service file running on the server. As I just discovered this, I’m still unsure what the full impact is of that line, but it seems also the reason that the current version of Strapi cannot handle media import from url anymore. I’ll create a new topic for that issue once confirmed.

I’ve created a fork to be able to show you the diff:

As soon as I change that particular line, to allow Buffers to be passed on the fs.readFile, it started working and the images appear in the Media library.

I’m not sure if this is what actually broke the original plugin as well, it seems possible to me as the error messages where pretty similar as far as I can remember. I’ll still work a little further on this as I’m so close to completing the import now, I’ll keep you posted.

Oh, and I never posted the link to the my own take on the plugin, I’ve published it here: GitHub - kiddicolour/strapi-plugin-import-content

1 Like