Investigate extremely slow requests captures on Edge environment

[alex-fedotyev]

Observing the app performance overtime on Edge environment, I noticed that it sometimes has rare but extremely slow transactions (2-5 minutes slow).

Details don't capture why this could be happening.

I suggest to investigate a little to better understand what could be happening.
cc @trentm

[trentm]

Thanks. I poked a little bit -- mostly me learning some about opbeans-node and opbeans-loadgen -- and I don't see any obvious normal reason why the "GET /api/customers" endpoint code could have a long delay at the end:

opbeans-node/server/routes.js

Lines 167 to 180 in a052ed7

	app.get('/customers', function (req, res) {
	redis.get('customers', function (err, obj) {
	if (err) apm.captureError(err)
	else if (obj) return res.json(obj)
	const limit = req.query.limit || 1000
	const sql = 'SELECT * FROM customers LIMIT $1'
	db.pool.query(sql, [limit], function (err, result) {
	if (err) return error(err, res)
	res.json(result.rows)
	})
	})
	})

• There is the separate handler to redirect "/api" requests 50% of the time that I'm not at all familiar with:

  opbeans-node/server.js

  Lines 86 to 109 in a052ed7

  	app.use('/api', function (req, res, next) {
  	if (Math.random() > opbeansRedirectProbability) {
  	return next()
  	}
  	const service = opbeansServiceUrls[Math.floor(Math.random() * opbeansServiceUrls.length)]
  	const opts = {
  	method: req.method,
  	hostname: service.hostname,
  	port: service.port,
  	path: req.originalUrl
  	}
  	req.log.debug('proxying request: %s => %s:%s', req.originalUrl, opts.hostname, opts.port + opts.path)
  	const clientReq = http.request(opts)
  	clientReq.on('response', clientRes => {
  	res.writeHead(clientRes.statusCode, clientRes.headers)
  	clientRes.pipe(res)
  	})
  	req.pipe(clientReq)
  	})

• and the app.get('*', ... routing pattern that I'm not familiar with:

  opbeans-node/server.js

  Line 113 in a052ed7

  app.get('*', function (req, res, next) {

• I noticed that some of the other opbeans-$lang services also have some slow outliers as well.
• The slow transactions on edge today are all "GET /api/customers".

tl;dr: I don't have much useful info. @alex-fedotyev If something changes to either make this a higher prio or if you notice a behaviour change, please mention it here.