Discussion for filtering, sorting, paging, and general parameters rewrite

This discussion has been migrated from our Github Discussion #4906


Describe the topic
I believe that there should be a discussion related to the current implementation of Strapi’s parameter system and am purposing a total rewrite to better allow for standardization not just within Strapi’s ecosystem (REST and GraphQL as well as all the connectors) but to make it easier to upstart potential new users and those migrating from other platforms and services.

Currently as it stands, Strapi is using a very uncommon suffix system with regards to filters, while this doesn’t matter on parameters such as:

  • _start
  • _limit
  • _sort

It does matter for any of the other filter mechanisms, some examples being:

  • username=testuser & username_eq=testuser being the same filter (can cause confusion, especially related to the documentation plugin)
  • Lack of filtering for polymorphic relations (components/groups/dynamic zones)
  • General overall incompatibility with the OpenAPI and Swagger REST standards thus rendering much of the feature set for documentation plugin unusable.

My goal with this discussion is to get community feedback about implementing a proper API parameter standard and plan out the best way to migrate to this and to eventually submit an RFC to hopefully make these changes within 2020.


Your suggestions for this topic
Doing a bit of research into other popular standards such as that of OpenAPI among other discussions across the web has brought up this idea that Strapi’s current implementation is not really sustainable in the long term.

What I feel we really need is a better structure related to accepted delimiters, a great example I saw on stackoverflow brings up using something like & for $AND based filters and | for $OR based filters, and :: for splitting key/value pairs. Something similar to the Oasis/odata specification

As an example: localhost:1337/blogposts?filter=title_contains::welcome&author.name::todd|author.name::chris

Alternatively for $OR we could do something similar to how the _in filter currently operates in that we could provide an array of values (although some more brain storming would need to happen for partial searches)

Along with the above, I think splitting filters from other parameter options such as sorting and paging would also help cut out some of the confusion and make implementation of proper paging also much easier, such as an example like this:

localhost:1337/blogposts?sort=publication_date::asc,title::desc&limit=10&page=5

In the above we would be filtering by publication_date then by the title (in different orders), limiting the page to 10 entries, and starting on page 5 (in SQL terms say ID 51)


Additional context
I welcome any alternative options and suggestions from those skilled in other platforms to weigh in on parameter filtering, sorting, and paging systems you liked and disliked and any suggestions on how something would best be implemented.

Our end goal it make this system as user friendly and usable on as many frontend “channels” as possible.

1 Like

Responses to the discussion on Github


derrickmehaffy

As a sub-point to the above topic, I would like to bring up the fact that in my opinion until a solid parameter system is implemented Strapi should not go into a stable release. During the Alpha and Beta phase of any project is when core structural changes are happening, this is when major breaking changes are expected to happen. I bring this up as previously in some of the Strapi blog posts they are intending to go into a stable release within Q1 of 2020 (not sure if this date has been moved).

To state that clearly: In my opinion I do not think Strapi is currently ready for a stable release.

I do have other reasons to state this, however those are not part of this discussion.


lauriejim

Ping @Aurelsicoko @alexandrebodin


alexandrebodin

Thanks for opening up the discussion @derrickmehaffy,

We should start by identifying the features we want and need to include first. Once we have that we can see if a standard already exists for them and go into more details about implementation.


derrickmehaffy

Thanks for opening up the discussion @derrickmehaffy,

We should start by identifying the features we want and need to include first. Once we have that we can see if a standard already exists for them and go into more details about implementation.

Absolutely I 100% agree. Defining the parameters we have currently and what we think we should have vs what we want to see.


derrickmehaffy

So doing a bit more research and comparing current parameters vs some fairly common ones out in the wild, I’ve made a list and some examples

General

  • _sort: ?_sort=[email::desc,username:asc]
  • _populate: ?_populate=[header.url,tag,comments]
  • _q: This is unchanged and is more so a “hidden” parameter that is used by the AdminUI for just global searching

Pagination

  • _limit: ?_limit=15
  • _start: ?_start=100
  • _page: ?_page=5 (uses _limit * _page number to get a _start value to make it easier and cleaner)

Querying

(prefixed by _filter=)

  • _eq: ?_filter=[_eq[username]::bob] (we could opt to remove or keep the standard matching also ?_filter=[username::bob])
  • _ne: ?_filter=[_ne[username]::jim]
  • _lt: ?_filter=[_lt[created_at]::2020-01-10 00:017:00]
  • _gt: ?_filter=[_gte[powerLevel]::9000]
  • _lte: ?_filter=[_lte[likes]::10]
  • _gte: ?_filter=[_gt[created_at]::2020-01-10 00:06:00]
  • _in: ?_filter=[_in[id]::[1,2,3,4]]
  • _nin: ?_filter=[_nin[tag.name]::[technology,art,science]]
  • _ct: ?_filter=[_ct[username]::alex] (previously _contains)
  • _cts: ?_filter=[_cts[city]::New] (previously _containss)
  • _nct: ?_filter=[_nct[username]::admin] (previously _ncontains)
  • _ncts: ?_filter=[_ncts[firstName]::John] (previously _ncontainss)
  • _null: ?_filter=[_null[isPublished]::false]
  • _regex: ?_filter=[_regex[phoneNumber]::{'((\(\d{3}\) ?)|(\d{3}-))?\d{3}-\d{4}': '123-456-7890'}] (In this example it will match (123) 456-7890 or 123-456-7890)

Logical

  • AND Operator: ?_filter=[_gte[powerLevel]::9000, _lte[powerLevel]::9500]
  • OR Operator (Single Field): ?_filter=[_nct[firstName,lastname]::miller]
  • OR Operator (multiple fields/filters): ?_filter=[_or::[_eq[firstName]::pierre, _eq[lastName]::burgy]]
  • OR Operator (multiple values): ?_filter=[_nin[tag.name]::[technology,art,science]]
  • _moment: ?_filter=[_gte[updated_at]::$currentYear] or ?_filter=[_gte[updated_at]::$moment.subtract.3.days]

Summery

These were just a few example I created with a similar format I’ve seen in a few places now. The main additions are:

  • _populate (has been requested many times in the past to custom populate without needing the controller to have custom code)
  • _page to support a more standard pagination system that removes the calculation logic from the client
  • Renaming the contains and not contains parameters to be cleaner and match up with other parameters naming system
  • Adding regex support to make certain OR based queries easier (say profile information like phone numbers in which you need to handle multiple countries ect)
  • Keeping the current $AND based system of just throwing the filters together in a single array for _filter
  • Allowing for $OR operations multiple fields, single value
  • Allowing for $OR operations single field, multiple values
  • Allowing for $OR operations between an array of other filters
  • Adding support for easier Moment.js type queries to make asking for something like greater than or equal to Last Year or just general moment type stuff. (Since Strapi already uses Moment.js for conversions)

The above examples would fall within the guidelines of OpenAPI specification, will need to look at this for something like GraphQL though. It would now populate the swagger UI with the following options:

  • _filter
  • _sort
  • _populate
  • _limit
  • _start
  • _page

lauriejim

Thank you @derrickmehaffy for these details.


cadavre

First of all – I’m not a contributor to the Strapi. But I wanted to give my point of view on current implementation vs proposed changes.

First of all we’re already in beta which should mean that there won’t be huge breaking BC changes. Unfortunately for Strapi being a CMS – this kind of change would lead to breaks not only in backend itself but also related frontend clients.

I cannot see a huge improvement here on changing param names. I think it’s arbitrary what syntax should be used, and to be honest, I haven’t also met before with syntax enlisted by @derrickmehaffy. But I may be the only one. What just pops on my mind is that probably every migrating user will have many more changes to be made than simple query params rewrite. On the other hand it really hurts every already-using-Strapi user.

Me and my company decided to trust Strapi when entered beta stage and still I can see a tons of changes that shouldn’t require us to migrate, like we had a version ago with rename of groups to components.

From my perspective (as a CTO, PM and software architect) – you should do more theoretical work before actually starting implementation. I know we’re 2020s agile/scrum, but what I can see is a lack of bigger picture and purpose what Strapi should become. Why am I writing this? Because this issue is “future general discussion” and I think there are way more important topics to talk about with community.


alexandrebodin

@cadavre thank you for sharing your perspective.

If this wasn’t clear enough from the previous messages this issue is only a starting discussion and is not going to happen anytime soon. It will go through the RFC process before even thinking of implementing it. This would go into an eventual v4.

If you read some of our latest articles you will see that we will go through a rc and have a stable v3 soon. Then we will start the work for future releases and be following semver standards closely.

We are considering breaking changes as much as we can but some of them are unavoidable to make Strapi stable both in term of features and bugs without bumping major version every weeks or waiting for 6months to release :confused:

I can assure you we take time to think about the future of the project and have a clear picture of what we want Strapi to become. We try to share this vision as much as we can when publishing our monthly updates article.

Please share the topcis you feel are way more important with us so we can take them into account.
You can submit ides through product board for example :slight_smile:

Thanks again for sharing. We hope that strapi will continue helping you build great products in the future and make your life easier while doing it :wink:


nonameolsson

I have been faced with some issues when trying to use the OpenAPI json file generated by Strapi, with other tools like SwaggerHub, openapi-generator and Postman. Since there is a problem with how the parameters are defined, mostly with. = for equal, it is not possible to use any of those tools :sad:

Related discussions:
#7219
https://strapi.slack.com/archives/C0BNGCDNH/p1598770368000200

Any suggestion on how to make a quick workaround so that Strapi would work with OpenAPI standard tools?


derrickmehaffy

No workarounds known as of yet, I do think we plan to rework the query parameter system with the database refresh over the rest of Q3 and Q4 but we will provide RFCs before we put anything into effect

1 Like

I am confused by the intended use of the _in filter. The way it is described in the docs, appears that the syntax should look something like id_in=3,6,8, but the example further down the page is ?id_in=3&id_in=6&id_in=8, and I can’t tell how that’s different from a regular OR operation.

Slightly unrelated to this post @acalvino this was a discussion that was migrated here to talk about options we could take to adjust the parameters system in the future.

Probably best to start up a convo in another thread about this.

@DMehaffy - Any news on the rewrite? We’d also love to use tools like (Swagger Codegen) to build out clients. If the re-write is a long way off, we might consider writing some sort of adapter for the current strapi filter spec to [try and] enable codegen.

It will most likely be around the same time we refactor the database layer in Q3

Since the filter is done at the database layer, I think we already have some light plans but nothing set in stone yet and not to a level we are ready to talk about. The engineering team are still reviewing what we have currently and discussing options. The timeframe though should still be on point.

1 Like

Hi, now that we are in the middle of Q3 could you give an update if this is being worked on? Thanks a lot for the help.

Yes the filtering, sorting, and paging is being completely overhauled in the v4, we have a general RFC that goes over it here:

Is there already a solution to search dynamic zones or searching custom JSON arrays?

Yes @pedrosimao see this RFC: rfcs/xxxx-v4-rest-api.md at v4/rest-api · strapi/rfcs · GitHub

(This has already been built and is live in our v4 beta: Strapi v4 beta is now available)
You can also hop on our Discord Server: https://discord.strapi.io and we have a v4-discussions channel

I’ve been trying to make basic type of sorting work without any success so far.
I’ve added these ones to my query string:

?sort[0]=sortOrder
response:
{“statusCode”:400,“error”:“Bad Request”,“message”:“Your filters contain a field ‘sort’ that doesn’t appear on your model definition nor its relations”}

?sort=sortOrder
response:
{“statusCode”:400,“error”:“Bad Request”,“message”:“Your filters contain a field ‘sort’ that doesn’t appear on your model definition nor its relations”}

?_sort=sortOrder
response:
{“statusCode”:500,“error”:“Internal Server Error”,“message”:“An internal server error occurred”}

If I try to ‘filter’ same happens:
response:

{“statusCode”:400,“error”:“Bad Request”,“message”:“Your filters contain a field ‘filters’ that doesn’t appear on your model definition nor its relations”}

It looks as if ‘sort’ or ‘filter’ were interpreted as fields from the content-type and not the built-in strapi function for sorting and filtering.

Can you please advise?

is “sortOrder” a column in you table?

You can refer to this link: Sort and Pagination | Strapi Documentation

yes it is.

I followed the documentation you just linked. There, it is written to set up the query just as I did above in my previous post. (the first one should be the correct, the rest are just desperate attempts)

Am I missing something obvious here?

Not anything obvious. Make sure you are not filtering a component or relation. I sometimes have issues with filtering and sorting with relations. I recommend that you use postman to check your API response without the filters/sort. You should post a sample data request and response.

An example response: (which works as expected without any parameter)
[{"id":7,"title":"nice title","sortOrder":8}]

I found it, eventually:
?sort[0]=sortOrder is Strapi v4 format.
?_sort=sortOrder is Strapi v3 format.