Kick the Bitbucket – magnushoff.com

On June 1, Atlassian will indiscriminately delete all Mercurial repositories from Bitbucket, becoming another Git-only site. In the beginning, Bitbucket exclusively supported Mercurial, so this change is unexpected to say the least.

In addition to supporting Mercurial, the main distinguishing feature that kept me using Bitbucket was its generous offer of unlimited private repositories for free. After Microsoft acquired GitHub in late 2018, it didn't take long for GitHub to start offering this as well. Faced with this competition, I guess Atlassian figured out it would only make sense to remove their last remaining distinguishing feature.

To add insult to injury, they have really made zero effort in assisting with migration, putting the entire burden on their users. I'm not a paying customer, but there's no indication that they are treating them any differently either. In their own words:

We realize that there is no one-size-fits-all solution. That’s why we’ve created the following resources to best equip you with the knowledge and tools for a seamless transition:

A Community thread to discuss conversion tools, migration, tips, and offer troubleshooting help

A Git tutorial that covers anywhere from the basics of creating pull requests to rebasing and Git hooks

They have created a fricken forum post! The absolute minimal acceptable effort here would be to create a migration tool that let you switch SCMs in-place, so issues, wikis, hooks and so on would keep on working. This is now all an exercise for the reader.

That being said, I have really been looking for a good way off Bitbucket ever since it was acquired by Atlassian in 2010. I never trusted them to do anything good for the service, and I have felt it getting worse slowly, but surely, all that time. This latest move by Atlassian really cements my distrust of the company, and it is the push I needed to finally move away.

Moving away

So I made a script. The ones I found were old enough to be based on obsolete APIs and they focused on migrating one repository. My goal was to migrate all my stuff from Bitbucket over to GitHub.

The script iterates through all repositories you own on Bitbucket and for each one:

Creates a corresponding repository on GitHub with the same name. If the name is already taken, the script skips to the next repository in the list.
Migrates the source code with git --mirror. If the source repository is Mercurial, conversion is performed with git-remote-hg.
If there is an associated wiki, it is migrated as well. Weirdly, you cannot push to the wiki of a GitHub repository before creating the first page via the web interface. I implemented a work-around that requires you to click a link and a button before the wiki is migrated.
Optionally the source repository is deleted, leaving a UI level redirect to the new GitHub repository.

Note that the script does not migrate issues, downloads, hooks and probably more things. This was sufficient for my needs. I do hope others will pick up the torch and implement more features as they see the need.

I have reproduced the script below. You can also download it directly: escape-bitbucket.sh or go to my gist. Feel free to fork the gist to add your own features!

Happy exodus!

#!/usr/bin/env bash

# Originally published on https://magnushoff.com/blog/kick-the-bitbucket/

# Copyright (c) 2019 Magnus Hovland Hoff
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

set -eo pipefail

# Check for installed dependencies
for CMD in curl git hg git-remote-hg jq
do
    if ! which "$CMD" > /dev/null
    then
        echo -e "\e[1m\e[31mError: Missing command $CMD\e[0m"
        exit 1
    fi
done

# Specify -1 on the command line to only migrate one repository
ONE_REPO=false
if [ "x$1" == "x-1" ]
then
    echo -e "\e[1m\e[32m-1 specified. Will only migrate one repository\e[0m"
    ONE_REPO=true
fi

if [ -z "$BB_USERNAME" ] ; then read -p  "Bitbucket username: " BB_USERNAME; fi
if [ -z "$BB_PASSWORD" ] ; then read -sp "Bitbucket password: " BB_PASSWORD; echo; fi
if [ -z "$GH_USERNAME" ] ; then read -p  "GitHub username: " GH_USERNAME; fi
if [ -z "$GH_PASSWORD" ] ; then read -sp "GitHub password: " GH_PASSWORD; echo; fi

BB_CURL="curl -s --user $BB_USERNAME:$BB_PASSWORD"
GH_CURL="curl -s --user $GH_USERNAME:$GH_PASSWORD -H Accept:application/vnd.github.v3+json"

echo
echo -e "This script will migrate the \e[36msource\e[0m code and \e[36mwikis\e[0m from Bitbucket to GitHub."
echo -e "NOTE: Nothing else will be migrated! \e[33mIssues\e[0m, \e[33mdownloads\e[0m and more will be lost."
while true; do
    read -p "Delete repos from Bitbucket after successful migration (yes/no)? " DELETE
    case "$DELETE" in
        yes) break;;
        no) break;;
        *) echo "Please answer yes or no";;
    esac
done

TMP=$(mktemp -d "${TMPDIR:-/tmp/}$(basename $0).XXXXXXXXXXXX")
function cleanup {
    rm -rf "$TMP"
}
trap cleanup EXIT


# Prefetch all the data from Bitbucket to avoid problems with deleting while iterating
echo -en "\e[1m\e[32mFetching metadata from Bitbucket"
ALL_REPOS="$TMP/all-repos.json"
echo > "$ALL_REPOS"
PAGE="https://api.bitbucket.org/2.0/repositories/$BB_USERNAME"
while [ "x$PAGE" != xnull ]
do
    $BB_CURL "$PAGE" -o "$TMP/page.json"
    echo -n "."
    jq -c '.values[]' "$TMP/page.json" >> "$ALL_REPOS"
    PAGE=$(jq -r .next "$TMP/page.json")
done
echo -e "\e[0m"

SIZE="$(jq -r '.size' "$TMP/page.json")"


I=0

# jq iteration due to https://starkandwayne.com/blog/bash-for-loop-over-json-array-using-jq/
for REPO in $(jq -r '. | @base64' "$ALL_REPOS")
do
    echo "$REPO" | base64 --decode > "$TMP/repo"

    I=$(( I + 1 ))
    SELF="$(jq -r '.links.self.href' "$TMP/repo")"
    CLONE_URL="$(jq -r '.links.clone[] | select(.name == "ssh") | .href' "$TMP/repo")"
    SCM="$(jq -r '.scm' "$TMP/repo")"
    SLUG="$(jq -r '.slug' "$TMP/repo")"
    HAS_WIKI="$(jq -r '.has_wiki' "$TMP/repo")"

    echo -e "\e[1m\e[32mMigrating \e[36m$SLUG\e[32m ($I/$SIZE)\e[0m"

    case "$SCM" in
        git)
            CLONE_ARG="$CLONE_URL"
            ;;
        hg)
            CLONE_ARG="hg::$CLONE_URL"
            ;;
        *)
            echo -e "    \e[1m\e[31mAborting due to unknown SCM: \e[36m$SCM\e[0m" >&2
            continue
    esac

    echo -e "    \e[1m\e[32mCloning \e[36m$SCM\e[32m repository from Bitbucket\e[0m"
    rm -rf "$TMP/cloned-repo.git"
    git clone --mirror "$CLONE_ARG" "$TMP/cloned-repo.git"

    echo -e "    \e[1m\e[32mCreating new repository on GitHub\e[0m"
    $GH_CURL https://api.github.com/user/repos -d "$(
            jq -n -c \
                --slurpfile p "$TMP/repo" \
                '{
                    name: $p[0].slug,
                    private: $p[0].is_private,
                    description: $p[0].description,
                    has_wiki: $p[0].has_wiki,
                    homepage: $p[0].website
                }'
        )" > "$TMP/to-repo.json"

    if [ "$(jq -r .ssh_url $TMP/to-repo.json)" == null ]
    then
        echo -e "    \e[1m\e[31mUnable to create repository \e[36m$SLUG\e[31m, does it already exist?\e[0m" >&2
        continue
    fi

    echo -e "    \e[1m\e[32mPushing repository to GitHub\e[0m"
    GH_WEB_URL="$(jq -r .html_url $TMP/to-repo.json)"

    GH_REPO_URL="$(jq -r .ssh_url $TMP/to-repo.json)"
    git -C "$TMP/cloned-repo.git" push --mirror "$GH_REPO_URL"

    if [ "x$HAS_WIKI" == "xtrue" ]
    then
        echo -e "    \e[1m\e[32mCloning wiki repository from Bitbucket\e[0m"
        rm -rf "$TMP/cloned-wiki-repo.git"
        git clone --mirror "$CLONE_ARG/wiki" "$TMP/cloned-wiki-repo.git"

        echo -e "    \e[1m\e[32mRenaming files for new naming convention\e[0m"
        rm -rf "$TMP/cloned-wiki-repo"
        git clone "$TMP/cloned-wiki-repo.git" "$TMP/cloned-wiki-repo"
        export TMP
        find "$TMP/cloned-wiki-repo" -depth -name "*.wiki" -exec sh -c 'git -C "$TMP/cloned-wiki-repo" mv "$1" "${1%.wiki}.creole"' _ '{}' \;
        git -C "$TMP/cloned-wiki-repo" commit -m 'Update naming scheme from Bitbucket to GitHub'
        git -C "$TMP/cloned-wiki-repo" push

        echo -e  "    \e[1m\e[35mAction requred:"
        echo -e  "        \e[32mVisit \e[36mhttps://github.com/$GH_USERNAME/$SLUG/wiki/_new"
        echo -en "        \e[32mCreate a wiki page (contents irrelevant), then press ENTER to continue\e[0m"
        read

        echo -e "    \e[1m\e[32mPushing wiki repository to GitHub\e[0m"
        GH_WIKI_URL="${GH_REPO_URL%.git}.wiki.git"
        git -C "$TMP/cloned-wiki-repo.git" push --mirror "$GH_WIKI_URL"
    fi

    if [ "$DELETE" == "yes" ]
    then
        echo -e "    \e[1m\e[32mDeleting Bitbucket repository\e[0m"
        $BB_CURL -X DELETE "$SELF?redirect_to=$GH_WEB_URL" > /dev/null
    fi

    echo -e "    \e[1m\e[32mSuccessfully migrated \e[36m$SLUG\e[32m ✔\e[0m"

    if [ $ONE_REPO == true ]
    then
        break
    fi
done

Magnus Hovland Hoff, 2019