[Dune-devel] Gitlab: Migrate artifacts, packages, (dependency proxy) to S3-compat object storage

[Ansgar Burchardt]

Hi,

the migration is done. I also had to allow the shared CI runner to
access the S3 domain.

Disk usage now looks like this:

# df -h /srv/data
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       300G   63G  231G  22% /srv/data

And on the S3 side:

| Bucket name                  | Number of objects | Size        |
|------------------------------|-------------------|-------------|
| dune-gitlab-artifacts        |            667385 | 171.222 GiB |
| dune-gitlab-dependency-proxy |                 0 |   0.000 GiB |
| dune-gitlab-packages         |              7970 |  44.667 GiB |
| dune-gitlab-registry         |            118906 | 170.384 GiB |
|------------------------------|-------------------|-------------|
| ∑                            |            794259 | 386.263 GiB |

We currently have a quota of 2.5 TB.

I also configured a policy that objects in S3 will only be marked for
deletion and only be removed after 90 days. This should be sufficient
in case Gitlab deletes data due to a software malfunction.

Some inconsistencies were found (and still exist):

~7 GiB of CI artifacts on disk, but missing in database (not moved to
S3).

~40 packages in database, but missing from disk.

Ansgar

On Thu, 2026-06-11 at 15:34 +0000, Ansgar Burchardt wrote:
> Hi,
> 
> the virtual disk for Gitlab data is fairly full:
> 
> root at dune-gitlab:~# df -h /srv/data
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/sdb1       300G  264G   32G  90% /srv/data
> 
> A large chunk is CI job artifacts and packages:
> 
> root at dune-gitlab:~# du -shc /srv/data/gitlab/gitlab-rails/shared/*
> 157G	/srv/data/gitlab/gitlab-rails/shared/artifacts
> 3.3G	/srv/data/gitlab/gitlab-rails/shared/cache
> 0	/srv/data/gitlab/gitlab-rails/shared/ci_secure_files
> 0	/srv/data/gitlab/gitlab-rails/shared/dependency_proxy
> 0	/srv/data/gitlab/gitlab-rails/shared/encrypted_settings
> 0	/srv/data/gitlab/gitlab-rails/shared/external-diffs
> 2.7G	/srv/data/gitlab/gitlab-rails/shared/lfs-objects
> 47G	/srv/data/gitlab/gitlab-rails/shared/packages
> 0	/srv/data/gitlab/gitlab-rails/shared/pages
> 0	/srv/data/gitlab/gitlab-rails/shared/registry
> 0	/srv/data/gitlab/gitlab-rails/shared/terraform_state
> 0	/srv/data/gitlab/gitlab-rails/shared/tmp
> 209G	total
> 
> For comparison the Git repository data and Gitlab database are much
> smaller:
> 
> root at dune-gitlab:~# du -shc /srv/data/gitlab/git-data /srv/data/gitlab/postgresql
> 14G	/srv/data/gitlab/git-data
> 15G	/srv/data/gitlab/postgresql
> 29G	total
> 
> Instead of resizing the disk, I would like to migrate some more data
> to S3-compatible object storage. We already use object storage for
> the Docker registry.
> 
> I would migrate the following objects:
> 
> 1. Job artifacts (CI/CD job artifacts including logs).
> 2. Project packages (PyPI, ...)
> 3. Dependency proxy (currently unused, just configure it to use
> object storage).
> 
> I would keep Git LFS objects on the disk as the S3-compat storage has
> redundancy (to cover disk failure), but no full backups (when
> something deletes too much). I don't think losing CI/CD job artifacts
> or packages is too bad, just like with Docker images.
> 
> The migration should be possible while Gitlab runs. It is also
> possible to migrate back to local storage.
> 
> See the Gitlab documentation for
> details: https://docs.gitlab.com/administration/object_storage/
> 
> Regards,
> Ansgar
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20260615/1f0815d1/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6599 bytes
Desc: not available
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20260615/1f0815d1/attachment.bin>