[Dune-devel] Future of the Heidelberg-hosted Dune infrastructure

[Dominic Kempf]

Dear Dune developers,TL;DR: Steffen is leaving academia at the end of
the year. The Heidelberg group needs your help in
reconsidering/redistributing maintenance work previously carried out
by him.This email is sent to you by the Heidelberg Dune Core
Developers: Peter, Steffen and Dominic.If you have seen Steffen's
email to the list, you know that he will be leaving academia at the
end of the year.For the Heidelberg group this is a drastic change, as
we do not only lose a valuable mentor, but also need to reorganize our
group processes w.r.t. server/software maintenance.Discussion of the
topic has shown that the Heidelberg group does not have the manpower
and expertise to continue to provide all the services we currently
do.We therefore think that this is a good point in time to discuss
these services with the aim of either distributing parts of the
workload to other people or - even better - simplify the maintenance
overhead of our infrastructure by switching to simpler, external
solutions.This is a summary of services we currently supply and the
opinion of the Heidelberg group on how to proceed with
them.gitlab.dune-project.org-----------------------Currently, the Dune
GitLab instance is running on our webserver and has a few
customizations applied to it (whitespace hook and better formatting
for merge commit messages).The current GitLab setup in Heidelberg is
too complicated to maintain with our resources in the future. We see
two options:- Find another volunteer that does the hosting- Move to
gitlab.comIf we move to another hosted instance that we can run
exclusively for Dune, the move is straightforward and we can simplydo
a backup of the old server and a restore of the new one.Moving to a
shared hosting (either somewhere on-premise or on gitlab.com) would
have some implications:- It would of course change the URLs for all
repositories, e.g:    https://gitlab.dune-project.org/core/dune-common
-> https://gitlab.com/dune-project/core/dune-common- gitlab.com
enforces quotas for all projects, and these apply to both free and
enterprise projects. Currently they  are set at 10 GiB per project,
which is shared across the actual Git repository, LFS, the wiki,
uploaded attachments and  build artefacts. The container registry
currently has no restrictions. At the moment, all of our repositories
fit  within those restrictions.- We can export the projects from our
server and import them into gitlab.com. This preserves the LFS data as
well as all issues  and merge requests, but the import reassigns all
issues, merge requests and comments to the user who performed the
import. The  gitlab.com importer adds a line to each comment with the
original author, but that information is missing from the issue and
merge request descriptions. It would however be possible to manually
patch it in with a bit of PostgreSQL scripting. You  can take a look
at https://gitlab.com/dune-project/test/dune-common for what it looks
like without this additional patching
  (of course we would use a dedicated Importer-User for the actual
import and not Steffen's account).- We would have to look at the
current build process of the website, which runs up against a stricter
gitlab.com quota regarding  build artefacts.- All projects must be
moved one at a time, but this could be scripted. We would, however,
probably not move personal projects  and instead give people time to
move them themselves to wherever they want.Heidelberg opinion:In the
last years, Gitlab has grown to be one of the major players in the
field.Moving our repositories to its cloud-based version at gitlab.com
would remove all our maintenance overhead while maintaining our
issue/MR history(with the restrictions regarding authorship listed
above) and workflows.Using Gitlab.com at Community Level (the one we
also had on our own instance) is free.We may even get an upgraded
feature set as Gitlab hands out Enterprise Edition licenses for free
to open source projects.Mailserver for
@dune-project.org--------------------------------The mailserver that
receives mail addressed to @dune-project.org is running on our
webserver.This is mainly used for admin mails from Gitlab and signups
at services like Let's encrypt, Cloudflare etc.Personalized email
addresses @dune-project.org have been given out in the past, but are
rarely used.Heidelberg opinion:Those email addresses seem to be rarely
used and could be removed without harm. Alternatively, we could move
themail services to some external, cloud-hosted platform if there is
someone willing to take care of it.Website
dune-project.org------------------------The website is currently
hosted on the group webserver and the deployment process is managed by
Gitlab CI.Heidelberg opinion:We can continue hosting the website, but
it might also be a good idea to offload that task to an external
serviceto improve reliability and uptime.One idea (if moving to
gitlab.com) would be to use Gitlab pages, which allows to serve static
websites(cf. https://about.gitlab.com/blog/2016/04/07/gitlab-pages-setup)
and is free.The deployment process could stay in Gitlab CI and we can
continue to provide the runner thatperforms this specific task (we
need it anyway because our group website works similar).Continuous
Integration----------------------Currently, several groups contribute
hardware to the Dune CI system by registering runners to our Gitlab
instance.The Heidelberg group also contributes 4 servers to this
system.Heidelberg opinion:The distributed nature of the CI system is
well-suited for the future.We will continue to provide this hardware
but restructure our servers to remove some moving parts.Docker in
Docker Continuous
Integration---------------------------------------The build process
for Docker images from the repository
https://gitlab.dune-project.org/docker/ciis a very special CI job, as
it requires a Docker in Docker setup on the runner side.Such setups
are prone to security issues as malicious code could escape the Docker
sandbox.We currently provide this runner in Heidelberg.Heidelberg
opinion:While we can continue to provide this service for the
immediate future, we would love to not have to maintainthe DinD-setup
anymore.We consider it worthwhile to move the building of Docker
images to a dedicated service. There are multiple optionshere that
would work and that would need discussing. Using external services at
the Dune scale is not free,but neither is maintaining the process at
university.DNS for dune-project.org------------------------The domain
dune-project.org is registered to Peter with the registrar
domaindiscount24.de. Theactual DNS servers are delegated to
Cloudflare. This was done to use DNS validation for obtainingLet's
Encrypt certificates, but could be moved back to domaindiscount24.com
again. Peter will continueto provide the domain and pay the yearly
fee, but we would like to have an additional person with a bitof
knowledge about DNS available when there are problems like the recent
mailing list outage, which requiredchanges to the DNS server.We hope
that this email serves as a basis for fruitful discussion.If
discussion becomes unwieldy, we propose to have a telephone conference
to resolve these issues more easily.Best regards from
Heidelberg,Peter, Steffen and Dominic
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20191121/41b46289/attachment.htm>