9.4 KiB
| title | excerpt | date | tags | |
|---|---|---|---|---|
| Manually mirroring a GitLab repository to GitHub | DIY mirroring the native mirroring feature | 2020-04-21 |
|
GitLab has a native mirroring feature and supports both pull and push directions; in pull, it periodically fetch any changes (including addition or removal of branches) from another location and push them to your GitLab repository, whereas in push, it periodically updates of your GitLab repository to the mirrored repository.
I've been using this feature mirror Hexojs repository to my GitLab account as a backup. It works well so far, especially when the repository has multiple branches. However, what if you has a simpler repository setup, like my blog which only has two branches and you want to get your hands dirty. Let's find out.
Manually push to mirror
The easiest to sync origin location (GitLab repo in this case) to another location (GitHub repo) is to add another remote,
$ git remote add mirror https://github.com/<user>/<project>
after you push commits to origin, push to mirror as well.
$ git push origin master
# Or `git push -u`
$ git push mirror master
# Enter your GitHub credential
Managing multiple SSH identities
The previous guide assumes HTTP authentication, but what if you (want to) use SSH authentication for both GitLab and GitHub. If you haven't set up SSH authentication yet, read on.
Generate and add a new SSH key to GitLab and GitHub
For this guide, I'll be using ED25519 keypair, instead of the default RSA. This SSH keypair will be your account SSH key and has the same privilege as a personal access token; it covers all the repo that your account has access to, which is different from deploy key also known as per-repository key.
$ ssh-keygen -t ed25519 -C 'email@example.com'
# Save the key to /home/user/.ssh/id_gitlab_username
# The email address is only used for label, not authentication
# Enter a password when prompted, that password will be used
# to encrypt the private key
Two files id_gitlab_username.pub and id_gitlab_username will be generated in "~/.ssh" folder; they are public and private key respectively, remember to back them up.
Add the public key to your GitLab account in the "SSH Keys" setting. Repeat this step to generate another keypair for your GitHub account, name the file accordingly. The private key is the credential that you will use to authenticate your account.
Configure SSH
Create a new file in "~/.ssh/config" with the following content,
#github account
Host github.com-username
HostName github.com
User git
IdentityFile ~/.ssh/id_github_username
Preferredauthentications publickey
IdentitiesOnly yes
#gitlab account
Host gitlab.com-username
HostName gitlab.com
User git
IdentityFile ~/.ssh/id_gitlab_username
Preferredauthentications publickey
IdentitiesOnly yes
The config and private keys must have chmod 600 permission, adjust accordingly.
Add GitLab and Github as trusted hosts
When connecting to the SSH server for the first time, you will encounter the following message, The authenticity of host 'gitlab.com (35.231.145.151)' can't be established.. This is because SSH also verifies the authenticity of the server. To avoid that message, you can import the respective key fingerprints into "~/.ssh/known_hosts".
$ ssh-keyscan gitlab.com 2>&1 | grep -vE '^#' > ~/.ssh/known_hosts
$ ssh-keyscan github.com 2>&1 | grep -vE '^#' >> ~/.ssh/known_hosts
2>&1 | grep -vE '^#' is to remove the comment "# github.com:22 SSH-2.0-babeld-76c80caa".
Update remote repository location
The final step is to update the origin and mirror remote repository locations.
$ git remote set-url origin git@gitlab.com-username:username/project.git
$ git remote set-url mirror git@github.com-username:username/project.git
It's a good practice to verify the values,
$ git remote -v
mirror git@github.com-username:username/project.git (fetch)
mirror git@github.com-username:username/project.git (push)
origin git@gitlab.com-username:username/project.git (fetch)
origin git@gitlab.com-username:username/project.git (push)
Next time you push or pull from the repository, you will be prompted with a password that you previously set during ssh-keygen, to decrypt your private key and authenticate your account.
Use multiple origin location (alternative)
Alternatively, a neat trick I found is that you could also add multiple locations to the origin, so that whenever you push the commits, git will automatically push to all locations. This shortcut also applies to SSH authentication.
$ git remote set-url origin --push --add https://gitlab.com/<username>/<project>
$ git remote set-url origin --push --add https://github.com/<username>/<project>
The first command will replace the default push location of origin and only the second command onward will add a new location, so make sure you add both GitLab and GitHub.
As always, verify it.
$ git remote -v
origin https://gitlab.com/curben/blog.git (fetch)
origin https://gitlab.com/curben/blog.git (push)
origin https://github.com/curben/blog.git (push)
Push to mirror using GitLab CI
Part of this guide is also found in my previous post, Using Git and Git+SSH in GitLab CI, this time I add more explanation for clarity.
The first thing is to set up a deploy key, a public key set in the targeted GitHub repository, in which its private key counterpart will be used by GitLab to authenticate to the GitHub. Deploy key is a per-repository key, thus applies only to a repository; this effectively isolate the privilege of the key, compromise of a deploy key would not affect other repositories, unlike account key or personal access token which affects all of your repo. Nothing stops you from using the same deploy key on multiple repos, but that practice is highly discouraged.
In this use case, since deploy key will only be used in a CI environment, it is not necessary to save it "~/.ssh" folder--you would use your account SSH key to authenticate from your workstation. I use "~/Desktop" folder here as it would be easier to locate the keypair when configuring the CI. As always, backup your SSH key.
$ ssh-keygen -t ed25519 -C 'email@example.com'
# Save the key to /home/user/Desktop/id_github_repo
# The email address is only used for label, not authentication
Two files id_github_repo.pub and id_github_repo will be generated in "~/Desktop" folder; they are public and private key respectively, remember to back them up.
On your GitHub repository (that will mirror your GitLab repository), go to Settings -> Deploy keys -> Add deploy key, paste the content of id_github_repo.pub and tick Allow write access.
On your GitLab repository, navigate to Settings -> CI / CD -> Variables, add a new Var variable named GH_PRIVATE_KEY and add the content of id_github_repo, make sure Protect variable is ticked so that the key is only imported to the "master" branch (which is a protected branch by default).
Add another Var variable named SSH_KNOWN_HOSTS and the output of ssh-keyscan github.com 2>&1 | grep -vE '^#'. I explained the necessity of this step in previous section.
Add a new job named mirror in your repository's .gitlab-ci.yml. Optionally, you could move Import SSH key step to before_script if preferred, it does not make any practical difference. However, if you want to use Update GitHub mirror step in after_script, Import SSH key has to be moved there as well--before_script and script are executed in the same shell, while after_script is executed in a different shell.
image: node:alpine
mirror:
before_script:
# Install prerequisites
- 'which ssh-agent || (apk update && apk add openssh-client git)'
script:
## Import SSH key
# Launch ssh-agent
- eval $(ssh-agent -s)
# Add private key from CI variable
- echo "$GH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
# Create ~/.ssh folder if not exist
- mkdir -p ~/.ssh
# Restrict the folder privilege
- chmod 700 ~/.ssh
# Add GitHub key fingerprint
- echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
## Update GitHub mirror
# ssh connect to GitHub
- ssh -T git@github.com || ":" # workaround) force exit code 0
# Add a new remote location called "mirror"
- git remote add mirror git@github.com:curbengh/blog.git
# Discard changes before checking out branch
- git reset HEAD --hard
# Push "master" branch
- git checkout master && git push mirror master
# Uncomment below to push other branch(es), change "branchX" according to your need
#- git checkout branchX && git push mirror branchX
rules:
# Only trigger through push event in master branch
- if: '$CI_COMMIT_REF_NAME == "master" && $CI_PIPELINE_SOURCE == "push"'
when: always
# Only trigger through "Run pipeline" in master branch
- if: '$CI_COMMIT_REF_NAME == "master" && $CI_PIPELINE_SOURCE == "web"'
when: always