[Dune-devel] Migration to gitlab.dune-project.org

[Steffen Müthing]

Hi Bernd,

> Am 30.10.2015 um 08:09 schrieb Bernd Flemisch <bernd at iws.uni-stuttgart.de>:
> 
> Hi Steffen,
> 
> ---- Steffen Müthing schrieb ----
> > I have also taken a look at the Flyspray bugs - I think we’ll be able to move them over to
> 
> 
> > GitLab without major problems.
> Since we recently also moved to GitLab, it would be very interesting if you could share some details on how to move the Flyspray tasks.

moving the issues wasn’t as bad as I feared. Here is a quick outline:

- You need access to the PostgreSQL databases of both Flyspray and GitLab and the Flyspray filesystem with the stored
  attachments.

- There shouldn’t be any issues in the target project before starting the conversion. That way, the bug numbers in Flyspray and GitLab will
  match. That’s why we created the flyspray group with the FS repository (and because flyspray/FS#1234 are nicely readable links in GitLab).

Step 1: Upload the attachments

Attachments in GitLab are always associated with a project and are stored with some hash-like URL. Unfortunately, GitLab doesn’t have API calls
for creating attachments, so this is a little fiddly, you have to reuse the session from your browser to do the uploads.

So I did the following:

- Enable the wiki on the FS project
- Go to the wiki section. You can directly edit the default wiki page, and below the text field, there is an upload button.
- Before doing anything else, turn on the developer tools and network request logging in your browser.
- Reload the page
- Upload a small file with the upload button. As a result, you will see a link pasted into the text field.
- In the network logs of the developer tools, you should see an XHR request to a URL ending in uploads. That has two important headers, one for
  the session cookie and some X-CSRF-Token. You need the URL, the session cookie and the token
- I’ve pasted all three of those into a Python script that also knows how to access the Flyspray database and the directory with the uploaded Flyspray
  attachments.
- The script will then upload all attachments and dump some information about the uploaded files into a state file that will be used in step 2.

Step 2: Create tasks and comments

- This can be done through the GitLab API
- As an administrator, look up the private token in your GitLab profile
- You need to create a map from Flyspray email addresses to GitLab user names
- You will also need a GitLab user that acts as author for users that are in Flyspray, but not in GitLab
- Optional: maps from things like Flyspray task types, operating systems and resolution types to GitLab issue labels. For this, you’ll have to poke around
  in the Flyspray SQL tables.

I’ve written another Python script that creates all of the issues and comments through the GitLab API. For the ~1700 issues and 9.000 comments from the
DUNE Flyspray instance, this took about an hour.

The script again dumps some metadata for step 3.

Step 3: Fix dates:

GitLab doesn’t let you forge creation dates through the API, so we have to fiddle with SQL to work around that. My third Python script thus creates lots
and lots of SQL UPDATE statements that have to be run against the GitLab database to fix up the creation, closing, and update dates.

I’ve attached the three scripts for reference. The scripts all require Python 3.4 and the python packages requests, arrow and postgres (best installed in a virtualenv
through pip). Looking at the scripts, you’ll also have to adjust some GitLab project ids, GitLab URLs and paths to state files with metadata. The scripts aren’t pretty,
but they did the job. ;-)

Best,

Steffen

-------------- next part --------------
A non-text attachment was scrubbed...
Name: flyspray_attachments.py
Type: text/x-python-script
Size: 1969 bytes
Desc: not available
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20151102/089825e1/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: flyspray_fix_tasks.py
Type: text/x-python-script
Size: 2033 bytes
Desc: not available
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20151102/089825e1/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: flyspray_tasks.py
Type: text/x-python-script
Size: 14333 bytes
Desc: not available
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20151102/089825e1/attachment-0002.bin>
-------------- next part --------------




> 
> Thank you. Kind regards
> Bernd
> 
> _______________________________________________
> 
> 
> > Dune-devel mailing list
> 
> 
> > Dune-devel at dune-project.org
> 
> 
> 
> > http://lists.dune-project.org/mailman/listinfo/dune-devel
> 
> _______________________________________________
> Dune-devel mailing list
> Dune-devel at dune-project.org
> http://lists.dune-project.org/mailman/listinfo/dune-devel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20151102/089825e1/attachment.sig>