[Dune-devel] Gitlab: Migrate artifacts, packages, (dependency proxy) to S3-compat object storage

[Ansgar Burchardt]

Hi,

the virtual disk for Gitlab data is fairly full:

root at dune-gitlab:~# df -h /srv/data
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       300G  264G   32G  90% /srv/data

A large chunk is CI job artifacts and packages:

root at dune-gitlab:~# du -shc /srv/data/gitlab/gitlab-rails/shared/*
157G	/srv/data/gitlab/gitlab-rails/shared/artifacts
3.3G	/srv/data/gitlab/gitlab-rails/shared/cache
0	/srv/data/gitlab/gitlab-rails/shared/ci_secure_files
0	/srv/data/gitlab/gitlab-rails/shared/dependency_proxy
0	/srv/data/gitlab/gitlab-rails/shared/encrypted_settings
0	/srv/data/gitlab/gitlab-rails/shared/external-diffs
2.7G	/srv/data/gitlab/gitlab-rails/shared/lfs-objects
47G	/srv/data/gitlab/gitlab-rails/shared/packages
0	/srv/data/gitlab/gitlab-rails/shared/pages
0	/srv/data/gitlab/gitlab-rails/shared/registry
0	/srv/data/gitlab/gitlab-rails/shared/terraform_state
0	/srv/data/gitlab/gitlab-rails/shared/tmp
209G	total

For comparison the Git repository data and Gitlab database are much
smaller:

root at dune-gitlab:~# du -shc /srv/data/gitlab/git-data /srv/data/gitlab/postgresql
14G	/srv/data/gitlab/git-data
15G	/srv/data/gitlab/postgresql
29G	total

Instead of resizing the disk, I would like to migrate some more data to
S3-compatible object storage. We already use object storage for the
Docker registry.

I would migrate the following objects:

1. Job artifacts (CI/CD job artifacts including logs).
2. Project packages (PyPI, ...)
3. Dependency proxy (currently unused, just configure it to use object
storage).

I would keep Git LFS objects on the disk as the S3-compat storage has
redundancy (to cover disk failure), but no full backups (when something
deletes too much). I don't think losing CI/CD job artifacts or packages
is too bad, just like with Docker images.

The migration should be possible while Gitlab runs. It is also possible
to migrate back to local storage.

See the Gitlab documentation for
details: https://docs.gitlab.com/administration/object_storage/

Regards,
Ansgar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20260611/4a2bf7d3/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6599 bytes
Desc: not available
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20260611/4a2bf7d3/attachment.bin>