INV processes eating machine's vCPU

[nigini]

I woke up this morning, and my blog was really slow. Then I tried to SSH to it, but it was even slower. All that I could see was that a couple of these INV processes were eating the CPU:

The blog processes eventually died, and when I looked into the logs, it was covered in failed (timed-out) requests for federated user keys.

  File "/mnt/volume_sfo3_01/microblog.pub/./app/httpsig.py", line 125, in _get_public_key
    actor = await ap.fetch(key_id, disable_httpsig=True)
                  │  │     └ 'https://bcast.guru/actor#main-key'
                  │  └ <function fetch at 0x7fcae5127f40>
                  └ <module 'app.activitypub' from '/mnt/volume_sfo3_01/microblog.pub/./app/activitypub.py'>

##...

  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
    │    │   │     │    │      └ <traceback object at 0x7fcae1b542c0>
    │    │   │     │    └ ConnectTimeout(TimeoutError())
    │    │   │     └ <class 'httpcore.ConnectTimeout'>
    │    │   └ <method 'throw' of 'generator' objects>
    │    └ <generator object map_httpcore_exceptions at 0x7fcae1ba6880>
    └ <contextlib._GeneratorContextManager object at 0x7fcae1c982b0>
  File "/root/.cache/pypoetry/virtualenvs/microblogpub-QDkw_Xeb-py3.10/lib/python3.10/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
          │          └ ''
          └ <class 'httpx.ConnectTimeout'>

My working hypothesis is that it happened after I got a couple of new follow(er)s... I think this is related to some Federation protocol that is running havoc. That would not be the first time I heard of these cases. I will keep monitoring and investigating...

[nigini]

Someone seem to have had a similar issue: https://todo.sr.ht/~tsileo/microblog.pub/162/

[nigini]

I have talked to other people working with the ActivityPub protocol and the major implementations out there. It seems that this problem is related to the broadcasting of delete posts from all over the Fediverse! That would explain the fact that the server is requesting keys from all over, regardless of the local social network.

What this person did was simply disregard these delete requests. :/ (Maybe we can do a bit better and have a quick check for "is this content relevant"?