Suboptimal memory allocations in metadata IO path leads to avoidable metadata_errors

[jfckm]

Description

When memory allocation fails in metadata IO we have no reasonable fallback and wind up in metadata_error territory which makes cache unusable.

Expected Behavior

OCF handles all IO gracefully.

Actual Behavior

We encounter memory allocation error (by e.g. exhausting MIO pool, but other paths are also handled the same way) and report metadata_error rendering cache unusable.

Steps to Reproduce

1. Set up WB OCF bdev in spdk
2. Expose it over vhost
3. Start VM consuming this bdev
4. Create FS
5. Copy over big file using rsync

Possible Fix

Couple of possible paths exist:

• extend mpools - stopgap, doesn't address the underlying problem
• mitigate the issue by limiting in-flight requests to match with maximal potential mpool allocation needs - better stopgap, there are non-mempool allocations there as well
• implement some kind of fallback - if allocation fails, wait for resources to free up or try limiting the needed allocations to minimum to ensure steady stream of requests progressing either way