There is a memcached bug where this error appears:

level=debug ts=2022-06-23T10:28:27.220696365Z caller=memcached_client.go:406 name=frontend-cache msg="failed to store item to memcached" key=1@b2ae91c4319dafc4 sizeBytes=86848 server=10.70.1.208:11211 err="memcache: unexpected response line from \"set\": \"SERVER_ERROR out of memory storing object\\r\\n\""

The above is logged from here.

Background (from related issue):

Memcached 1.5 and above uses a segmented LRU by default (blog post). Items can be evicted by a background routine if they’re expired, or directly during a mset operation if the cache memory is full. The latter operation is called a “direct reclaim”.

Query-frontend caches results with 7d TTL. Since the load test has run for less than 7d, presumably all evictions are caused by direct reclaims. Using the stat command on memcached shows:
0 reclaimed (Number of times an entry was stored using memory from an expired entry)
0 crawler_reclaimed (Total items freed by LRU Crawler)
983890 direct_reclaims (Times worker threads had to directly reclaim or evict items)

We would like the above “direct_reclaims” metric to be exported in Prometheus.