Load testing using K6.io results in High CPU usage

Discussions
1

—copying discussion from github —

Hi Team, I’ve been performing some load testing using k6.io The models I’m testing are really simple (no relationships). I’ve run this locally and via docker and I’m always seen a 100% CPU spike. Any ideas what I could tune/change?

Strapi Version 3.1.6
Environment: Production
Node: v12.16.3
Npm: 6.14.4
Db: Postgres

Example k6 runs to see the spike

k6 run strapi.js --vus 5 --iterations 20000
k6 run strapi.js --vus 10 --iterations 10000

[@derrickmehaffy] Can you provide your sample strapi.js K6 file, running with just 5 or 10 virtual users shouldn’t be all that taxing but I’m not sure about the iterations.

import papaparse from 'https://jslib.k6.io/papaparse/5.1.1/index.js';
import http from 'k6/http';
import { check, sleep } from 'k6';


// Open requests from a csv file
const csvData = papaparse.parse(open('strapi.csv'), {
  header: true,
}).data;

// For each request in csv
// 1. Grab a random query and call it
export default function () {
  // Grab a random request
  const requestItem = csvData[Math.floor(Math.random() * csvData.length)];

  // Call the selected endpoint
  const res = http.get(requestItem.request);
  check(res, {
    'succeeded': (r) => r.status === 200 || r.status === 404,
  });
}

I actually created a sample nodejs api express server and tested communications to postgres and CPU was performant. I’m thinking it could be some middleware that could be causing it. I am running default project with no extra middlewares though.

The Node app I made uses these modules express, pg and sequelize and performs a basic GET. CPU was 4% using k6 call above. Would love to get to the bottom of this.

There are various other calls that are most likely slowing the response down, in your Strapi application, in ./config/database.js or if you are using a NODE_ENV then ./config/env/${NODE_ENV}/database.js within the options object put debug: true .

From here you will see all the RAW SQL queries being made, I know off the top of my head there will be at least one to the users-permissions plugin to check if the user has permissions to do what they are asking to (Including public/non-auth’ed users).

I take it your express application is not doing any processing or checking of permissions of some kind, you are simply taking the request payload, parsing it, and dumping it to the database yeah?

[@derrickmehaffy] good point, I’ve enabled debugging there are 6 select calls being made. Note I did check Postgres CPU and it was at 1%.

re: the express app just doing a basic select on the table that’s correct. I’ll add the extra selects to my test app but I doubt its db related.

I’m going to try playing with commenting code out, divide and conquer time

I actually went a step further and added APM (Newrelic) to pinpoint the issue. Looks like most time is spent in KOA

Screen Shot 2020-10-01 at 11 27 28 am

I then went down the koa line investigating what was happening there.

DEBUG=koa* node ./node_modules/strapi/bin/strapi.js start
2

my next steps are to make a sample test using koa, koa-router

3

I plan on doing some tests outside of docker myself as I’m wondering if that can cause some performance issues also

4

the issue is KOA @DMehaffy.

Here is a very simple koa script talking to postgres, run this and load test using k6.io you will get about 80% cpu

const Koa = require('koa');
const Router = require('koa-router');
const Logger = require('koa-logger');
const { Pool } = require('pg');

const app = new Koa();
const router = new Router();

app.pool = new Pool({
  user: 'postgres',
  host: 'localhost',
  database: 'postgres',
  password: '', 
  port: 5432, 
});


router.get('/:id', async (ctx) => {
  const { rows } = await ctx.app.pool.query('select * from public.badges ORDER BY random()')
  ctx.body = rows[0];
});


// Development logging
app.use(Logger());
// Add routes and response to the OPTIONS requests
app.use(router.routes()).use(router.allowedMethods());

// Listen the port
app.listen(1337, () => {
  console.log('Server running on port 1337');
});
5

Did you get a chance to look at this @DMehaffy? What do you think next steps are for Strapi?

In the interim we’ve decided to write our own API that contains the strapi data models in PG. CPU usage is now at max 5% under heavy load testing.

6

I have not, been quite busy as of late. For the moment we are planning a database refresh this quarter that may assist in some of the load issues but the bulk of it is just down to complexity of the project in general, there isn’t a “fix all” for something like this and will need to be taken in steps with direct reproducible tests.

Simply running a handful of RPS tests doesn’t tell us where the issue comes from.