I need to update Strapi tables from my node program
I have a system that updates information about companies. The system queries many external APIs and scrapes web pages to get the data. Data synced to external systems like Hubspot, Insightly, Firestore, and WordPress.
I now need to add the data to Strapi. I have tried to add the data in the following ways:
- Using rest and graphql to update data.
But it creates stress on the Strapi server. Plus it is less effective as the communication is HTTP requests.
- I implemented the system that accesses external APIs, scrapes webpage, and updates data in external systems as a Strapi Plugin. This created even more stress on Strapi.
- Implemented the system that fetches and updates data as an external system and uses knex to access tables defined by Strapi.
The use of queues and workers (BullMQ connected to Redis and workers running in docker containers) reduces the load on the system and scales a lot better. But accessing Strapi tables using knex is complex because Strapi stores repeatable fields (stored in a separate table).
So: How can I use the Entity Service API and the Query Engine API from my node program without running the Strapi server?