Discussion regarding the complex response structure for REST & GraphQL (Developer Experience)

This topic is not about population, please discuss that topic here: Discussion regarding default population of REST

Current Status Post: Discussion regarding the complex response structure for REST & GraphQL (Developer Experience) - #35


Hello Everyone,

As mentioned in a news post in our Discord last week we wanted to setup this discussion around how the response structure is handled (nested data & populate) in Strapi v4.

If you haven’t already please read the following documents, pull requests, or issues before commenting:


Currently purposed suggestions:

  • Revert to v3 style in GraphQL => Not possible at this moment
  • Remove Attributes and throw everything under data in GraphQL => Not possible due to breaking changes
  • Recommend usage of something like normalizr for REST Users + potentially integrate a normalization library in the Strapi SDK JS

Current status on v4:

At this time we are not prepared or planning to modify this data structure in the response, the time for that type of modification has passed as it should have been addressed during the RFC process. We do understand that it was not clear at the time during our RFC process this was the case and we are already doing retrospectives to improve this process.

In the short term we would rather explore options to optimize using this response structure on the client side and see how we can assist. An above mentioned suggestion of including normalization libraries in best practices or even sponsored SDKs (like the Strapi-SDK-JS is) is one example of what we believe could benefit everyone.


Why is Strapi using this structure?

The primary goal behind this new structure was to pick some kind of standard for a response structure, ideally not reinvent the wheel and try to use an already existing standard. In our case we picked JSON API: https://jsonapi.org/

image
(Image src: xkcd: Standards)

We have not entirely implemented the standard yet but our goal with the first release of Strapi v4 was to lay the groundwork so that we could continue to build on it without breaking changes. This meant unifying the REST and GraphQL data responses, error handling, filtering/paging system, ect.

2 Likes

Thank you very much for opening a space for a discussion. I think it should be clear that you had the best intentions with the change.

However, the GraphQL response in v4 feels very complicated and cumbersome. I understand there shouldn’t be a breaking change and I’m happy you follow the semver conventions.

What about a non-breaking change, which still reverts part of it: you could output the data under attributes and the root object at the same time. Instead of changing the schema, it would extend it and therefore be a non-breaking change:

query posts {

    // current v4
    data {
      attributes {
          title
      }
    }

    meta {}

    // desired v4 alternative
    data {
      title
    }
}

At some point the question was raised where to put the meta information in this flat case and somebody mentioned it could just be repeated with every entity:

query posts {
    data {
      title
    }

    meta {}
}

would lead to:

[
  data: {
    {
        title: ''
    }
  },

  meta: {}
]

I don’t know what the implications would look like from an implementation perspective, but from a users perspective this wouldn’t be bad.

Is this something worth considering?

1 Like

@DMehaffy I just realized I made a mistake. Take this example from the Github GraphQL API, which I’d say shows the desired behavoir:

{
  repositories {
    nodes {
      id
      name
      issues {
        nodes {
          author {
            login
            ... on Organization {
              id
              name
            }
          }
        }
      }
    }

    pageInfo {
      hasNextPage
    }
  }
}

To re-build that in strapi you’d have to create the following query:

{
  repositories {
    data {
      id
      attributes {
        name
        issues {
          data {
            attributes {
              author {
                data {
                  attributes {
                    login
                    ... on Organization {
                      data 
                        id
                        attributes {
                          name
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }

    meta {
        pagination {
          pageCount
        }
    }
  }
}

As you can see the strapi response is much deeper, because attributes is always nested under data. I guess the desired behavoir would be that everything that right now lives in attributes would be available under data (more common is nodes though, because GraphQL is about nodes, but let’s ignore that for now) too (to make this a non breaking change).

{
  repositories {
    data {
      id
      name
      issues {
        data {
          author {
            data {
              login
                ... on Organization {
                data 
                  id
                  name
                }
              }
            }
          }
        }
      }
    }

    meta {
        pagination {
          pageCount
        }
    }
  }
}

This would make the situation already much better.

5 Likes

Personally I don’t really get the issue of the data response being to ‘deep’.

I do however mainly use REST and the new structure of the API in my opinion is much more structured and more consistent. This gives me the opportunity to write resolvers that can be used for multiple entities. It also makes data response more predictable.

Looking at the usage for GraphQL, yeah, it feels a little bit tedious to write data > attributes over and over again, but thats just the structure of the response/Strapi. You just need to write it once anyway. You could also opt for using fragments for your entities (highly recommended anyway).

In terms of using your data and finding your data set in your response, I’d go for a function in a frontend framework where I’d put my response in and it returns my data from the attributes object.

2 Likes

I see what you’re getting at. I understand in your case it would be painful.

You do have a solid argument in regards to hydration. I haven’t thought of that for now, but it could have a serious impact on the Frontend.

It’s hard to think of a solution when the structure will be the same.

I do not know if the GraphQL should have the same response as REST in terms of consistency but maybe it’s a solution to just change the response for the GraphQL plugin?

1 Like

When working with the new response for a few days, it somehow started to bug me to mistype attributes every single time and missing the ‘old’ response. I decided to write a normalizer function which looks into deep objects & arrays. In case somebody want’s to work with this, here’s the code. It’s not implemented into the API, but into Nuxt in the frontend.

const normalize = (data) => {
  const isObject = (data) =>
    Object.prototype.toString.call(data) === '[object Object]'
  const isArray = (data) =>
    Object.prototype.toString.call(data) === '[object Array]'

  const flatten = (data) => {
    if (!data.attributes) return data

    return {
      id: data.id,
      ...data.attributes
    }
  }

  if (isArray(data)) {
    return data.map((item) => normalize(item))
  }

  if (isObject(data)) {
    if (isArray(data.data)) {
      data = [...data.data]
    } else if (isObject(data.data)) {
      data = flatten({ ...data.data })
    } else if (data.data === null) {
      data = null
    } else {
      data = flatten(data)
    }

    for (const key in data) {
      data[key] = normalize(data[key])
    }

    return data
  }

  return data
}
17 Likes

+1 on the complexity of the structure. Really dislike it and hope there is a solution such as the one @gustavpursche or @abdonrd suggested

I’ve attached a picture of a pretty gross before and after converting one of my queries from v3 to v4. This is not anything unusual. I would say most of my queries have become this uglified

7 Likes

As requested by @DMehaffy , I’m copy-pasting my github issue related to this discussion here:

-----------------------------------------------------------8<-----------------------------------------------------------

Bug report

Describe the bug

Because of id and attributes split in API schema the GraphQL cache can’t be updated and raise a warning in the browser console (among obvious cache handling issue).

The warning message from the console is:

invariant.ts?6cf4:42 Cache data may be lost when replacing the attributes field of a ExampleEntity object.

To address this problem (which is not a bug in Apollo Client), either ensure all objects of type Example have an ID or a custom merge function, or define a custom merge function for the ExampleEntity.attributes field, so InMemoryCache can safely merge these objects:

  existing: {"__typename":"Example","createdAt":"2021-12-25T23:43:56.648Z","firstName":"John","lastName":"Do","message":"I'm asleep","status":"NEW","dueDate":null,"referer":"https://example.com","contactDetails":[],"user":{"__typename":"UsersPermissionsUserEntityResponse","data":null}}
  incoming: {"__typename":"Example","dueDate":null,"firstName":"John","lastName":"Do","message":"I'm asleep","referer":"https://example.com","ipAddress":"12.34.56.78","createdAt":"2021-12-25T23:43:56.648Z","updatedAt":"2021-12-25T23:43:56.648Z","contactDetails":[],"communications":[]}

For more information about these options, please refer to the documentation:

  * Ensuring entity objects have IDs: https://go.apollo.dev/c/generating-unique-identifiers
  * Defining custom merge functions: https://go.apollo.dev/c/merging-non-normalized-objects

This is because the GraphQL schema is the following:

  • example query:
example(id: ID): ExampleEntityResponse
type ExampleEntityResponse {
  data: ExampleEntity
}
id: ID

and

data: ExampleEntity
type ExampleEntity {
  id: ID
  attributes: Example
}

Steps to reproduce the behavior

  1. Using GraphQL API, request the same asset in a query and in the result of a mutation (for example)
  2. See warning message in the browser console

Expected behavior

The id and attributes split in the GraphQL API schema is incomprehensible for me. I can’t understand why there are not in the same object (like Apollo Client is obviously expecting to find it). In addition, I also can’t understand why we need to wrap everything in a data object while fetching a single asset. I guess there is strong reasons for such things, but I can’t actually believe them :exploding_head:

IMHO, the expected schema should be like this:

example(id: ID!): Example
type Example {
  id: ID!
  createdAt: DateTime!
  updatedAt: DateTime!
  // other fields...
}

Screenshots

N/A

Code snippets

Regardless the GraphQL cache issue, I’m currently forced to write such code in every query/mutation (and I really hate it :angry:):

function Example({ id, attributes }) {
  Object.assign(this, { id, ...attributes });
}

this.$apollo
  .query({ query: examplesQuery, variables: { ids: this.ids } })
  .then(({ data: { examples: { data } } }) => {
    const examples = data.map(e => new Example(e));
    // finally some useful code
  }

Strapi v3 use by far more “dev friendly” in this respect (no useless data wrapper for single asset, no incomprehensible id and attributes split).

System

  • Node.js version: v16.13.1
  • NPM version: v8.1.2
  • Strapi version: v4.0.5
  • Database: MariaDB
  • Operating system: Archlinux
    -----------------------------------------------------------8<-----------------------------------------------------------
6 Likes

I have nested objects as well and have ported over to the new graphql format already but now am adapting the front end code and it is unreadable, prone to errors, really hard to debug.

Not very usable without a workaround, fix or plugin

For my use case:
Strapi 3: this.companies[i].investments[0].fund.id
Strapi 4: this.companies.data[i].attributes.investments.data[i].attributes.fund.data.id

Authoring the GraphQL queries isn’t too bad but using them on the front end is tough. If this were the approach in Strapi 3 I’d have gone with something else.

I’m happy to help w/ feedback, test etc. rather than just complain. Know you are all balancing design/usability etc.

2 Likes

In addition to my github issue and after reading carefully the whole discussion, I have notice some points which make me (ironically) laughing:

  • At this time we are not prepared or planning to modify this data structure in the response, the time for that type of modification has passed as it should have been addressed during the RFC process. We do understand that it was not clear at the time during our RFC process this was the case and we are already doing retrospectives to improve this process.” => so the priority should be on fixing the issue instead of speaking about how to improve for next time (no worries, you will have months or even years for that), especially since “next time” is somewhere, far away, in the futur.

  • The primary goal behind this new structure was to pick some kind of standard for a response structure, ideally not reinvent the wheel and try to use an already existing standard” => good goal! We are lucky, there is already an existing javascript reference implementation named “Apollo”, it’s even written black on white in the github of the project. Unfortunately, strapi v4 is incompatible with that standard (caching issue). At the end, it seems that strapi v4 is just creating the last (15) competing standards… congrats…

  • In our case we picked JSON API: https://jsonapi.org/” => Good choice for REST API, not at all for GraphQL which is different by design and have nothing to do with JSON API

  • This meant unifying the REST and GraphQL data responses, error handling, filtering/paging system, ect.” => again, there is nothing to unify, REST and GraphQL are different by design and even it’s the reason why GraphQL was created. Going against that difference means missing the primary goal of GraphQL. Sorry but if strapi goal is to unify GraphQL with REST then the only way is to drop GraphQL support

  • Remove Attributes and throw everything under data in GraphQL => Not possible due to breaking changes” => I guess it’s a joke?!? Seriously guys, who is using strapi v4 in production right now? Strapi v4 is more a beta than a stable product atm and only early adopters are trying to migrate on it (no offense, it’s a perfectly normal situation). Even the migration guide is missing for now! If your concern is about semver, a very easy solution is to release a new graphql plugin (for example “graphql-apollo-plugin”) with an initial v4.0.6 version and that’s it! Simple, easy. Once v5 will be ready, simply drop the current graphql-plugin and replace it with graphql-apollo-plugin (renamed graphql-plugin, ofc)


Now let me provide a bit of context:
I’m currently working on a huge migration both frontend (vue v2.x => v3.x) and backend (strapi v3.x => v4.x) with a strong deadline of mid-feb. I have already spent a lot of time to migrate DB schema from v3 to v4 manually.

Now, no doubt, GraphQL API in v4 is just unusable as is and so my mid-feb release is fucked because I will have to switch back on strapi v3 everything which is already migrated at frontend side.

Not only that, but since strapi v3 is in maintenance mode, I will have to plan another migration for backend side (aka building my own or switching on another strapi-like product) so it’s the whole roadmap which is now totally fucked.

Tomorrow, I will have to explain that to my boss and, if I’m not fired immediately, I will have to buy a lot of redbull, in order to develop 24/7 during the next months. I guess everyone here could very well understand why I’m really upset.

I’m really sorry if my words aren’t as kind as they should be, I’m just very disappointed because of that horrible situation. However, I’m still confident about a brillant futur for strapi and love all guys which works hard to make it better.

10 Likes

I am no one to complain but this deeply nested data/attributes is a deal breaker for me. Seriously what problem it was trying to solve in the first place?

6 Likes

I created a simple middleware to transform V4’s REST responses into a better format. Basically remove the existence of data and attributes
Hope this helps.

7 Likes

As a tiny update to this, we have scheduled an internal meeting to discuss the issue at length on tuesday. We are taking into account all of the feedback gathered here and elsewhere.

When I have more information to share on this, I will :slight_smile:

Please do feel free to keep the feedback coming.

6 Likes

Yup I was just re-reading through our notes from the meeting we did this morning and intended to share what we discussed and what we are thinking.

Actions we are taking right now:

  • Doing a bit more research into other existing implementations such as Facebooks Relay: https://relay.dev/ and how they are doing things as functionally our setup is very similar to theirs
  • Researching a bit into getting caching working (it is possible but not just as simple as apollo’s native config for it) as it’s a huge area of concern from everyone

Actions we plan to take in the next month or so:

  • Have a community call (or steal part of an existing one) to cover some of the background with this
  • Record a video or write a lengthy blog-post explaining some of the future needs we have from the response structure to support the following three features:
    • Content Versioning
    • Publication Workflows
    • Better meta information about an entity’s i18n locale
  • Breaking down our focus towards keeping things stable and some bits of the longer term plan

What we discussed was the following (keep in mind we aren’t taking a full decision right now because some people are on vacation next week and we want to do more research before we just rush into things).
I hope everyone can understand we don’t want to just take a decision on this lightly, especially now that we have extended the v3 EOL til Sept 2022.

  • Removing data {} is not possible now or in the future because it would conflict with high level meta {}
  • attributes {} could be removed in theory now but it would conflict with our long term plans in the future with meta {} (merged, we will probably touch on this in the video/blog post)
  • We are willing to make a breaking change for GraphQL plugin but would require a long and complex RFC before we would do that, not a short term solution but yes a plugin could technically be ejected from the versioning system the core uses. There would need to be a lot of validation from the community around the RFC that we didn’t have when we first issued the RFC (largely the problem on our side because it wasn’t clear what feedback we wanted, and it wasn’t shared enough)
  • Researching Relay => why does it work so well => why is it so accepted by their community => what learnings can we take from that => their caching layer?
  • Can SDKs help the end user? Is this stuff we can build to help?

We also discussed a bit on the RFC side and what went wrong and how to prevent it in the future:

  • RFC needs more readers AND interactions (comments, discussions, debates, not preferences but breaks downs on why an alternative is better with trade offs, ect)
  • More data in the RFC, too much internal information and context wasn’t shared
  • RFC format wasn’t clear enough, not enough descriptions
  • RFC process was too fast // Strapi v4 Beta process was too short (won’t happen again, we were under a time crunch due to poor timeline planning in v4)
  • Ideally we really should publish RFCs long before we start development and engineering team needs to be more active reading responses and responding to RFCs. Small team problems that we are fixing slowly but simply we need more devs.

Funnily enough one of the original respondents to the RFC when we published it was a community member who now works for us so we were able to rapidly gather some cool insights and in the future we would like to have some of these discussions in person such as in community calls going forward as it was really helpful to get that point of view.


TLDR: Nothing major happening right now, it’s clear we need to do some research but we need to make decisions carefully, we can’t “just revert” back to v3 style. Keep in mind that we did what we did as we were building pre-reqs for what we want to build in the future (ideally this year). That was never described to any of you and it’s what we need to do now.

Give us a bit of time we are listening and we do plan to engage with the community to find a working resolution to this. Your opinions are very valuable to us, time crunches just didn’t make things easy and too much got rushed.

5 Likes

Dear Strapi team,

Why so many ‘data/attributes’ flying around?

I really like Strapi (there are so many more pros than cons) but have been struggling with your recent changes in v4 related to the ‘Unified response format,’ even wondering if there might be an alternate headless solution that might prove to be more effective.

If one is trying to consume a single endpoint that has 2 or 3 levels hierarchy, which easily occurs when one defines any relations, media fields, components, or dynamic zones, it seems likely that the developer (who uses Strapi to build the hierarchy) would be the one responsible for any necessary meta or pagination that runs so deeply within the model they’re constructing, making the data/attribute wrapping redundant for a majority of the use cases beyond the 2nd level.

In your documentation, you say a single entry returns an object with id, attributes, and meta. This seems logical for level 1 ( of which I would be in accord), as I would consider the endpoint to be one query (regardless of filtering parameters); any subsequent level queries are not directly part of the interface.

As an initial test, I have spent some time creating a transformation layer to remove this wrapping (in one instance, trimming by 4 data/attribute levels), but have yet to come up with a standard and clean approach to use across all my models.

In my opinion, I shouldn’t really need to create a transformation layer specifically to eliminate what in all current practices feels like an arbitrary rule for distant future-proofing.

I’ve considered several solutions: 1) restrict my models to a single dynamic zone across one level (which isn’t ideal in all cases for data-entry, and defeats the flexibility of Strapi’s types), thus keeping the response flatter; 2) create a middleware service to transform the response (as others have suggested), but this seems too cumbersome and not the best place to perform such a transformation; 3) create a plugin to handle transformations, which is not ideal but would in theory be helpful for migrations; or 4) find another headless CMS with a cleaner response format.

I would hate to have gone into production with v3 (the version and response structure that originally won me over) and be faced with adding a transformation layer to migrate to v4.

The following would be ideal for dev-friendly front-end consumption:

Apps should be able map the object with ease without transformation:
page.content.type’ instead of ‘data.attributes.page.content.type.data.attributes.name.’

For instance, with a Vue component, passing data attributes from the response should map sensibly without the front-end dev having to unnecessarily transform the object to remove verbosity (at least, that is what I wish to provide her/him):

<component v-bind:is="$compName(attrs.__component)" :attributes="attrs" /> 

instead of;

<component 
v-bind:is="$compName(data.attributes.page.content.__component)" 
:attributes="data.attributes.page.content.attrs" 
/>

I hope you will at least consider removing the data/attributes wrapper (from both REST and GraphQL APIs) on nested objects that do not require meta/pagination, or include them only when directly called by an endpoint return one level. I hope this feedback is helpful.

10 Likes

Thanks a lot pal, exactly what I needed. It’s so relaxing to see recursion doing amazing stuff.
Kudos!

1 Like

Thanks for the kind words!

I’ve been using the transformer plug-in in my last few projects and it been a blast (Transformer plugin). I’d suggest you check it out.

I personally am not using GraphQL for any new projects as it doesn’t fit my needs at the moment in combination with Strapi v4.

4 Likes

Currently purposed suggestions:

  • Revert to v3 style in GraphQL => Not possible at this moment
  • Remove Attributes and throw everything under data in GraphQL => Not possible due to breaking changes

@DMehaffy does the telemetry show significant adoption and use of the v4 in production already? Where is this fear of breaking change coming from?
It sounds like a sunk cost fallacy to me (The Sunk Cost Fallacy - The Decision Lab)

With regards to

the structure specified is needed for some of our upcoming features (primarily content-versioning)

Which part of the structural change in the graphql response is specifically needed for this future content-versioning feature?

Is there any design draft or anything that explains what the content versioning feature is and how technically it will be implemented?

Could, perhaps, the design for content-versioning be reconsidered instead so to not impact the graphql schema in this way?

Say, if you needed to have meta for each data so you could specify some version info there, maybe just have that info available among other attributes? Just inject those fields in addition to the content-type based ones? Just poking in the dark here :slight_smile:
I don’t currently see any good justification for splitting id and attributes for example. And split for data + meta could have been just "usual flat response + _meta field injected.

For the vast majority of users there aren’t too many reasons you need to reach for GraphQL :sweat_smile: Since we introduced the field selection and populate or even the nested relational filtering/sorting/ect. If anything you lose out on the ability to cache responses at the network level by using GraphQL.

(don’t worry GraphQL users, it’s not going anywhere :wink: but we are recommending to customers to use our REST APIs over GraphQL)

Unfortunately, this is not viable for many who, like I, have committed to GraphQL on the frontend a while back. For a ReactJS application written with Apollo Client heavily using useMutation or useQuery hooks, there is no way to switch to REST without a massive rewrite.

A bit of an off topic:
I just looked at this chart yesterday: Strapi Usage Statistics

Maybe the downward slope is a coincidence, but it could also be related to the v3 → v4 shock. And I guess initially announced quick sunset timeline for the v3 support just scared people off.

But let’s admit it, the migration is painful, at least this is my experience. Basically, everything has changed. Migrating from v3 to v4 feels like a full rewrite.
I can convince myself this is necessary evil because there are benefits I can gain on the backend.

But the need to do the massive rework of the frontend as the result of the migration is unjustified because it brings no benefits to the frontend and in fact is somewhat poisonous to it instead.

6 Likes

Thank you for your feedback. The Strapi team’s goal is to ensure that we continue to create the best possible CMS we can.

Changes, like going from Srapi v3 to v4, brought some new challenges. But we are listening to the community, even when some comments are negative.

We are also human beings, and we are working at a company where we believe in the product and want to continue making it better.

We will be holding a Livestream around Strapi v3 vs. v4 to discuss some changes made and why. Also, it would be a great place to ask questions.

You can join us here Difference Between Strapi v3 and v4, what changed, and why?

I know the migration process was difficult, specifically when migrating a v3 GraphQl Strapoi Project to v4 since we fundamentally rewrote how that is handled in version 4.

But if you have any suggestions to help us improve the process. Please feel free to let me know.

Thanks, Toby.

I know changes can be frustrating. But unfortunately, we “dropped the ball” on communicating why we made the changes.

And what is coming next to help with some of the issues we are currently facing.

Thank you all for your input.

3 Likes

We all hope to make strapi better, Even if we are some complaint users.

I don’t doubt graphql v4 has many improvements, and also has some good aspects for strapi’s future.

Besides that, here is what user want:

  1. if you want keep your graphql roadmap, that’s fine, but “enforce” the users to adopt new way to query is not a smart way.
  2. I do suggest you can have two ways to handle graphql(of course it needs a lot of work). First, enable the user use a normal graphql world query language; second, use your new query to show the task that we can’t complete when we use old graphql query way.

We should also heed the general lesson implicit in the injunction of Ben Franklin in Poor Richard’s Almanack : “If you would persuade, appeal to interest and not to reason.”

I believe tell the user your roadmap, your thinking of graphql are all “reason”, not “interest”.

The user already see the “bad” part of it, can we see the “interest” part of it ? Do we need to endure the bad part to accept the “interest” part?

In terms of engineering, keep the response as same format as restful v4 api is fine, we can easily flat the response to adopt, but the query part is hard to handle(need to clear and easy to read).

2 Likes