[Dune-devel] GitLab Server issues

[Simon Praetorius]

Hi everyone,

as you might have noticed, and as discussed already in the matrix chat, 
we have some issues with our GitLab server. It is partially 
non-responsive or very slow. While the computing and storage resources 
are actually fine (we still have a small buffer in storage space where 
we could ask for an increase, but this is not the reason for the slow 
down of the server), the server seems to be under heavy load.

It might be related to an increase in concurrent server requests. Ansgar 
measured sometimes >500 requests per second. This already lead to a 
flooding of our log files (a partial solution for this is implemented). 
We have several Bots, crawling our server, 
e.g., YandexRenderResourcesBot, meta-externalagent, Applebot, GPTBot and 
others. I discussed with Ansgar a little what we could do about it.

There are a few options easy to install, but maybe not effective at all:
- a robots.txt file to explicitly exclude Bots, (only "friendly bots 
respect that file),
- a server configuration with a access rate limit,
- a tool like fail2ban which monitors logs and bans IPs automatically, ...

Other ideas are a bit more involved, like
- a reverse proxy or CDN in front of our gitlab server (e.g., Akamai, 
Cloudflare, AWS, maybe a european/german CDN provider like MyraCDN - 
costs money)
- very restrictive: putting the server behind a VPN. (e.g. development 
in our gitlab only, for users the github mirror)
- In the matrix chat also some crawler firewall was suggested, like 
https://github.com/TecharoHQ/anubis (Anubis uses a proof-of-work 
challenge to validate clients. I think something like this is what 
gitlab.com uses: sometimes you are asked to check a box that you are not 
a robot before landing on the webpage, but it might do this also mostly 
automatically)

Also internal configurations of the server might be improved.
- I have seen issues with too many open files in the puma backend, such 
that no new requests can be handled.

All in all, it is a very technical problem. And we have to do something 
about it. I have only very limited knowledge about all that. Most ideas 
what to do I got from chatgpt - so take it with caution. I have the 
impression that Ansgar has only very limited time to do the 
administration of our server (@Ansgar, correct me if I am wrong).

- Is anyone able and willing to dive into this problem and find a solution?
- we need more than a single person, who could fix server issues - I can 
only click on the restart button of the VM
- are there alternatives, e.g., spend some money to either hire an 
expert, or switching to a different server solution?

Best,
Simon

-- 
Dr. Simon Praetorius
Technische Universität Dresden
Institute of Scientific Computing
phone: +49 351 463-34432
mail: simon.praetorius@​tu-dresden.de