[Dune-devel] Migration to gitlab.dune-project.org

[Bernd Flemisch]

Hi Steffen,

thank you for the instructions and scripts!

Kind regards
Bernd

On 11/02/2015 03:44 PM, Steffen Müthing wrote:
> Hi Bernd,
>
>> Am 30.10.2015 um 08:09 schrieb Bernd Flemisch <bernd at iws.uni-stuttgart.de>:
>>
>> Hi Steffen,
>>
>> ---- Steffen Müthing schrieb ----
>>> I have also taken a look at the Flyspray bugs - I think we’ll be able to move them over to
>>
>>> GitLab without major problems.
>> Since we recently also moved to GitLab, it would be very interesting if you could share some details on how to move the Flyspray tasks.
> moving the issues wasn’t as bad as I feared. Here is a quick outline:
>
> - You need access to the PostgreSQL databases of both Flyspray and GitLab and the Flyspray filesystem with the stored
>    attachments.
>
> - There shouldn’t be any issues in the target project before starting the conversion. That way, the bug numbers in Flyspray and GitLab will
>    match. That’s why we created the flyspray group with the FS repository (and because flyspray/FS#1234 are nicely readable links in GitLab).
>
> Step 1: Upload the attachments
>
> Attachments in GitLab are always associated with a project and are stored with some hash-like URL. Unfortunately, GitLab doesn’t have API calls
> for creating attachments, so this is a little fiddly, you have to reuse the session from your browser to do the uploads.
>
> So I did the following:
>
> - Enable the wiki on the FS project
> - Go to the wiki section. You can directly edit the default wiki page, and below the text field, there is an upload button.
> - Before doing anything else, turn on the developer tools and network request logging in your browser.
> - Reload the page
> - Upload a small file with the upload button. As a result, you will see a link pasted into the text field.
> - In the network logs of the developer tools, you should see an XHR request to a URL ending in uploads. That has two important headers, one for
>    the session cookie and some X-CSRF-Token. You need the URL, the session cookie and the token
> - I’ve pasted all three of those into a Python script that also knows how to access the Flyspray database and the directory with the uploaded Flyspray
>    attachments.
> - The script will then upload all attachments and dump some information about the uploaded files into a state file that will be used in step 2.
>
> Step 2: Create tasks and comments
>
> - This can be done through the GitLab API
> - As an administrator, look up the private token in your GitLab profile
> - You need to create a map from Flyspray email addresses to GitLab user names
> - You will also need a GitLab user that acts as author for users that are in Flyspray, but not in GitLab
> - Optional: maps from things like Flyspray task types, operating systems and resolution types to GitLab issue labels. For this, you’ll have to poke around
>    in the Flyspray SQL tables.
>
> I’ve written another Python script that creates all of the issues and comments through the GitLab API. For the ~1700 issues and 9.000 comments from the
> DUNE Flyspray instance, this took about an hour.
>
> The script again dumps some metadata for step 3.
>
> Step 3: Fix dates:
>
> GitLab doesn’t let you forge creation dates through the API, so we have to fiddle with SQL to work around that. My third Python script thus creates lots
> and lots of SQL UPDATE statements that have to be run against the GitLab database to fix up the creation, closing, and update dates.
>
> I’ve attached the three scripts for reference. The scripts all require Python 3.4 and the python packages requests, arrow and postgres (best installed in a virtualenv
> through pip). Looking at the scripts, you’ll also have to adjust some GitLab project ids, GitLab URLs and paths to state files with metadata. The scripts aren’t pretty,
> but they did the job. ;-)
>
> Best,
>
> Steffen
>
>
>
>
>
>
>> Thank you. Kind regards
>> Bernd
>>
>> _______________________________________________
>>
>>
>>> Dune-devel mailing list
>>
>>> Dune-devel at dune-project.org
>>
>>
>>> http://lists.dune-project.org/mailman/listinfo/dune-devel
>> _______________________________________________
>> Dune-devel mailing list
>> Dune-devel at dune-project.org
>> http://lists.dune-project.org/mailman/listinfo/dune-devel


-- 
_______________________________________________________________

Bernd Flemisch                         phone: +49 711 685 69162
IWS, Universität Stuttgart             fax:   +49 711 685 60430
Pfaffenwaldring 61            email: bernd at iws.uni-stuttgart.de
D-70569 Stuttgart            url: www.hydrosys.uni-stuttgart.de
_______________________________________________________________

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.dune-project.org/pipermail/dune-devel/attachments/20151102/e2bbe831/attachment.htm>