MongoDB compatibility delayed on v4

Discussions
1

Update! Please see this message:

Context

Back in 2014, the very first version of Strapi was using an ORM called Waterline. The goal of an ORM is to provide a generic query language to perform requests on different databases. For the developer experience, it’s great, it makes things easier but it can create a lot of issues: performance, lack of native features support according to the database, etc.

When we re-developed from scratch Strapi, in 2017, we decided to use more specific ORMs. One for SQL database called Knex, and one for NoSQL MongoDB called Mongoose. Why? MongoDB was huge at that time. The MEAN (Mongo Express Angular Node) stack was rising, Node.js was often associated with MongoDB. We had to support it! We were also heavy users of software based on MySQL, and Postgres was getting a lot of traction too. We had to support them too!

Note: In 2017, our definition of Strapi was that “we are a data management framework with an administration panel”. We wanted to manage data without defining ourselves as a CMS. Data isn’t content. We now assume that we are a CMS. Nothing else.

Since then, Strapi has always supported SQL databases such as MySQL, Postgres, SQLite, MariaDB, etc, …and MongoDB.

Two years ago, we introduced the quickstart command to create a project in no time without any configuration. In order to achieve that goal, we used SQLite. Thus, MongoDB wasn’t the recommended starting database to test and try Strapi. It has been replaced by SQLite. We started to see a decrease in MongoDB usage from that moment until now (thanks to the Telemetry system)

  • In March, MongoDB usage represents less than 0.4% of all the projects.

Mongo seems to represent a very low amount of projects.

Problem

Today, having to support two different connectors (SQL + MongoDB) slows down our developments. It makes also the engineering part more complicated. We have to make sure a feature has the same behavior and the API returns the same response according to the database. We have to do twice the job.

As we don’t use MongoDB as we should do to keep a consistent experience across databases, we have performance issues on many-to-many relations for example. But there are many more (see)

Motivation

As the usage of MongoDB as a database continues to decrease, as MongoDB Inc. told us they won’t maintain the current connector, as the success of Strapi isn’t linked with MongoDB and as the content modeling promotes the relational model that fits better with SQL database, I would suggest removing the official support of MongoDB and only support SQL databases natively.

From a product perspective, we could have a strong importers strategy to compensate for the lack of MongoDB (& other databases) support. If it’s easy to migrate content from a CMS using MongoDB to Strapi thanks to importers, it sounds good to me.

Drupal, WordPress, and many other CMSs restrict to one database, often SQL one and these are successful projects.

Consequences & Impacts

  • Documentation
    • We would have to remove all the documents & guides that relate to MongoDB.
    • We would have to inform our users that we don’t officially support MongoDB natively.
  • Support
    • We should engage with customers and work with them to migrate to a supported database, we will not be able to support them long term as they will be unable to handle the jump to v4 when it’d released.
  • Engineering
    • We would have to remove the strapi-connector-mongoose package from the monorepo and create a new one.
    • We would have to change the README and clearly indicates that we don’t support it officially.
    • Find a community member, company or partner who would like to maintain the connector.

Last but not least, we already had a debate within the Core team, and we are all aligned by making this decision. We want to gather your feedback and discuss it with you (the community). It’s important for us to be more transparent about the decisions we would like to make as it impacts each one of you.

I’m looking forward to reading your answers, let’s have a free and respectful debate :pray:

12 Likes
3

Please do not drop MongoDB. It’s part of javascript community & you do not have to follow others who uses just MySQL. I do not know if Prisma will solve your problems with MongoDB; If you have conclude droping MongoDB at least give us a clear option how to integrate our database we want instead of being opinionated with stack.

Thanks for your hard work.

12 Likes
4

The issue is not with the ORM, mongoose is capable enough, but just that we continue to hit road blocks every time we develop a feature and most of the time the issue happens on the MongoDB database and how different it is from the SQL databases we support.

Also a large chunk of our GitHub bug reports revolve around Mongo specific issues that don’t impact the other databases and it’s eating away at our time to try and debug, fix, and release for a database that is only an extremely niche usage from our users.

If we want to build Strapi better, more robust, and faster we need to cutout the bottleneck.

7 Likes
5

I use mongo + strapi. I have not used any project with strapi without mongodb :astonished:

13 Likes
6

I just started using Strapi with MongDB, still in the testing/learning phase so the change will be minimal for me. I understand the hard decisions about allocating resources though and the strong importers strategy will help ease the burden transition from MongoDB to SQL db.

Thanks for all the hard work!

4 Likes
7

That is middle finger for 8 months of development. Provide a way to migrate data to postgre if you have any sense. This goes against what you said you stand for

6 Likes
8

Dropping support for something as substantial as DB engine when the product this far in its roadmap sucks. But the rationale sounds reasonable. I think everyone will benefit from the core team’s better velocity and sharper focus on other very important aspects of the product.

If you provide good importers strategy as mentioned, it should help existing impacted customers to migrate.

I am not sure how architecture v4 will pan out and what the rework of database layers entails.
But I would suggest considering a separation layer between the core of Strapi and concrete data storage. Introduce the point where a database adapter can be plugged. Give developers an option to use custom DB adapter as long as it conforms to expected interface.

This way you give community a way to take ownership of MongoDB support (should they choose to accept that mission). And you are clearly off the hook to provide maintenance. If any problems happen with MongoDB or adapter, it is not Strapi core team who will be providing support and maintenance, so it aligns with your rationale for dropping the Mongo.

12 Likes
9

Completely agreed with @aveprik . In that way, both sides have a solution and also give Strapi the ability to adapt to new and upcoming database technologies well without doing complete rewrites.

3 Likes
10

Bad decision in my opinion, you can’t decide based on the Telemetry system data, because it can be turned off. many users might have disabled it, you should do some kind of poll.

8 Likes
11

This is why we have the telemetry system, is for making decisions like these. If users disable it, that’s completely fine but they are choosing to have their data excluded from the way we make decisions.

It’s entirely anonymous and it’s sole purpose is to make decisions on the project. If people aren’t using feature x but are using feature y, is there something wrong with feature x or are people not aware of it?

We won’t go around the telemetry system and build out an entirely different information gathering platform when we already have one. (We do for “non-project decisions” like Discord migration, but for Strapi code it’s based on telemetry and RFCs).

6 Likes
12

We use MongoDB right now, but as the application becomes more complex, I struggle to make efficient complex database queries. I already was looking for ways to switch from MongoDB to a SQL database. If you decide to remove MongoDB support, I think there must be a good migration functionality / guide from MongoDB to a SQL database. If that’s the case I am willing to switch databases.

5 Likes
13

Certainly within our plan either some kind of good migration guide or maybe some tooling. Will largely depend on all the changes we are making to the database layer.

1 Like
14

@DMehaffy makes total sense.

As you know I’m a big advocate for MongoDB and personally would choose it every time. But you can clearly see in Strapi that it doesn’t fit well. Strapi has been designed for a relational database and that’s pretty clear, in the way that embedded documents just isn’t something you can do with Strapi + MongoDB, at which point you’ve lost almost all the power and flexibility it provides.

4 Likes
15

Indeed you are correct @IPWright83 and it’s just not something we even want to implement as the work required to do so would take so much time and almost no benefit (in terms of users). This is why we directly asked the MongoDB if they were interested in taking over the maintainership of the connector as their team would be widely more capable to handle it and they declined.

1 Like
16

Elaborate please, So I can understand you right.

3 Likes
17

We don’t use nested documents that are talked about in that and what they call relations are not relations, it’s just nested json.

My opinion is that being able to embed related documents within the primary documents being returned is more intuitive than duplicating rows for every relationship found in a relational join.

This is exact opposite of what a relationship is and they are joined by a shared key store either on one side or the other. MongoDB instead decides to handle this by using nested documents (which Strapi has never used) and we instead handle relationships the way they are intended, by creating another collection and storing the ID on the related model.

For example from that article:

'cast': ['Charles Kayser', 'John Ott'],

They store the raw name so every single time I need to add Charles Kayser to an entry I have to save the full string to each document, in a normal relationship I would have whatever collection this cast key belongs to and instead have a cast collection where I put both of these names (each with their own ObjectID) and instead just store an array of IDs and use joins (which mongo does support, but is extremely slow)

aggregations are not relations

Point being made here was that we (Strapi) are not following the standards that MongoDB have implemented because its far too difficult to handle them in an application that also uses SQL without a very powerful abstraction layer that really hasn’t existed and we have not had the time to implement.

Our goal with dropping mongo is to never build this abstraction layer if we can avoid it to cut the time needed to constantly maintain and update it as new features are built or we fall into a constant cycle of refactoring the abstraction layer to make sure we support all of the ORMs being used.

The second side to this is the response from each type of database varies far too much. In SQL if I don’t set a value, the “key” (column) exists but is null, in mongo it’s undefined (or null, which are very different). Likewise without strictly setting constraints (that SQL have by default) I can just as easily write an integer to a string field or a boolean to an integer field. On top of that if I do it, I have very little in the way of migration tooling.

SQL is easy, I know the column type, I want to change it, I run a migration using standardized tools (something we are planning to implement to make renaming or modifications of the content-types much easier)

1 Like
18

What are other alternative solutions the Strapi team come up with beside asking MongoDB Inc to maintain the connecter; I mean did try to find someone expert who can maintain this connecter beside MongoDB Inc?

If it is private thing you are not obligated to answer me thank you.

1 Like
19

If someone in the community is ready to help by maintaining the MongoDB support we are open to doing a smooth transition. MongoDB Inc wasn’t interested, they took time to understand how Strapi works and they had seen that our use case/product is a perfect fit for SQL databases, not MongoDB. Also, the number of projects and the market opportunity was too low for them.

6 Likes
20

We are still discussing how we plan to implement connectors in the future (v4) and if we will maintain the current strapi-database package (it’s the abstraction layer between Strapi and the database ORMs/connectors).

There are many within the team that still wish to continue to allow new connectors and open the connector ecosystem to allow the community to build and maintain their own but there is still a cost to this that being discussed internally.

To be clear, I don’t have an answer to this and nor does anyone else in the team as it’s being discussed at an engineering level (I say this and 3 seconds later @Aurelsicoko answers lol)

However I am one of those who would like to keep this layer and eject the current strapi-connector-mongoose package to it’s own repo and offer the option for someone else to take over maintainership of the code and npm package. Many others in the Strapi team also suggested this. Something else we are possibly looking to do is create different connectors for each database to allow for smaller connectors with narrow scopes and the ability to use different orms (or none) to interact with specific supported databases. eg:

  • @strapi/connector-mysql
  • @strapi/connector-postgres
  • @someCompany/connector-mongodb

(Yes btw we are moving all Strapi packages to scoped packages to allow for easier forks of our own packages: Init scoped packages by alexandrebodin · Pull Request #10183 · strapi/strapi · GitHub)

4 Likes