In this post, I’ll first walk through hosting static content with basic authentication. Then, we’ll look into deploying to Heroku using GitLab Pipelines, more specifically deploying a certain sub-directory within the project instead of pushing the whole project. Also, I’ll share what I’ve learned about GitLab CI/CD Pipelines while trying to find a solution.
Part 1: Basic Auth for Static Website
Before starting I want to explain where this problem came from and my hours long process of finding a solution. Lately, for my master’s thesis I’ve been working on a bookdown project. Bookdown is an R package that helps us build docs as pdf or HTML, you can consider it like a static-site-generator. So, the output of my project is static HTML files. I can serve these files using any static site hosting provider like GitLab Pages, GitHub Pages, Netlify, Surge… I’ve already tried many of them before and actively using GitLab for this blog. There are many free alternatives but none of them, as far as I know, provides basic authentication for the website.
In GitLab, the git repo can be private and the pages can be either public or open to only project members. So, you can make GitLab pages private to you or the users you have added to the project. This is very good because last time I checked GitHub didn’t have support for such feature. Still, it’s not enough. Because, what I need is a private site with basic authentication so that we can share link with people that doesn’t have GitLab account. There is an open issue for this topic and it seems it’ll stay open because alternatives like Netlify and Surge provides basic auth within paid plans.
I did some research hoping to find a work around… The best I could find is that we can set up basic auth with Heroku and a simple repo that’s enough to demonstrate the concept. Not the best solution, because it’ll require a little setup for Heroku but that’s affordable.
So, in my project after I compile the source docs (yes docs =P) the result is bunch of html files in a directory called
ROOT_DIR/public/index.html. I just moved the public folder 1 level below and added the files from the heroku-static-provider repo. Example:
ROOT_DIR/heroku_site/public/index.html. The directory will look like this:
Mode LastWriteTime Length Name ---- ------------- ------ ---- d---- 24/05/2022 6:57 PM public -a--- 04/05/2022 8:54 PM 26 .gitignore -a--- 04/05/2022 8:54 PM 137 package.json -a--- 04/05/2022 8:54 PM 20 Procfile -a--- 04/05/2022 8:54 PM 650 Readme.md -a--- 04/05/2022 8:54 PM 463 server.js
Then created a new git project at
ROOT_DIR/heroku_site, created new-app in heroku, add heroku as remote then just git push. The site is deployed to Heroku and we can set up basic auth using Heroku-CLI.
heroku config:set -a name-of-the-heroku-app USER=... heroku config:set -a name-of-the-heroku-app PASS=...
If we don’t count the searching phase, implementing this solution didn’t take long. Yet, the implementation I presented above is far from ideal. Because now, I have 2 git projects and I don’t actually care about source controlling for the second project, it’s just the output of the first one. By the way, the second project is excluded in the original one;
heroku_site added into the
.gitignore. So, I have to compile the project locally to generate the outputs
heroku_site/public, commit the changes and push to heroku
git push heroku master. I want to automate this using Gitlab CI/CD Pipelines. But how?
Part 2: Deploy To Heroku using GitLab CI/CD Pipelines
I had CI/CD configuration already for the original project; it was already hosted via GitLab Pages.
All I need to do is to edit the configuration.
It wasn’t so simple though. Because, I didn’t know much about configuring a pipeline in gitlab.
The previous configuration was simple; compile the source, put the static site content into a folder named
public under root directory.
This is the standart way for using gitlab pages.
Since things are a little bit more complicated than the standart way, I created an empty git project to play around and learn.
Started with a basic
.gitlab-ci.yml that does nothing but printing some texts like “hello world”.
Here is the docs to gitlab-ci configuration. By trial and error, and reading docs ofc, I’ve learned a lot.
You can find my notes about GitLab Pipelines in the project’s readme file. Hey, don’t go there yet! We haven’t finished here. I’ll now explain the
.gitlab-ci.yml that you can find in my test project. Example below is based on this configuration.
A Quick Tutorial for Newbies
In our example, the pipeline has two stages; build and deploy. Stages will be executed in order. If
deploy won’t even start.
image: busybox stages: - build - deploy
build-gitlab-pages is a job in the
build stage. It spins up a container from
busybox image then executes the bash commands under
script. The files generated within the job will not be available for the next job. To make them available we must use the keyword
pages is a job in the
deploy stage. So, it runs after
build stage is completed. Since, we set
public directory as artifact in the first job, it’s available for the
pages job. I run a dummy command in the
script part, because a job must have script keyword. Normally, you build your static website here using jekyll, hugo etc.
pages is a reserved keyword for gitlab pages. It must pass
public folder as artifact if we want Gitlab Pages to work.
build-gitlab-pages: stage: build script: - mkdir public && echo "<html><h1>GITLAB CI PIPELINE</h1><br><p></p></html>" > ./public/index.html artifacts: paths: - public expire_in: 30 mins pages: stage: deploy script: - echo "hellöğğ" artifacts: paths: - public only: - master
This will host the
index.html we have created under
Deploy to Heroku
Now, let’s come to the part we are interested in. The folder structure will be like this:
│ .gitlab-ci.yml │ ... │ source_files... └───heroku_deps package.json Procfile Readme.md server.js
In the pipeline:
- we’ll compile the source files and generate a the output in a directory
- copy the files under
- deploy the
websitedirectory to heroku.
Notice that at the end of step 2, the folder structure will be the same as we had at the end of Part 1 but all happens in a container within the pipeline.
Again, we have two stages; build and deploy. The build job creates the
website directory, copies the dependencies for heroku project. Lastly it adds a static html file for testing. Normally, you would generate the static content by compiling your project. Pass the directory as artifact for downstream jobs.
build-heroku: stage: build script: - mkdir -p website/public - cp heroku_deps/* website - echo "<html><h1>GITLAB CI PIPELINE</h1><br><p></p></html>" > website/public/index.html artifacts: paths: - website
Now, we’ll simply cd into the directory we have created. This is our root directory for a new git project. I used
ruby image for this job because I knew
git is included in it. You can use any base image you want, just make sure git is installed. Otherwise, you would have to install git too; e.g.
apt install git if the image is ubuntu based. If we don’t specify the image, gitlab will use ruby too.
Anyway… What we do is pretty simple; initialise a new project, commit changes, add heroku as a remote, then push. We don’t care about version history here. All we want is to deploy the static content to heroku. Everytime this job runs, there will be only one commit and it’ll override the commit in the remote (heroku). That’s why we use
git push with
--force. Beside heroku, I’ve added another remote that points to gitlab for testing, you can check it here.
deploy-job: stage: deploy image: ruby:latest script: - cd website - git config --global init.defaultBranch ci_branch - git init - git config user.email email@example.com - git config user.name mertbakir - git add . - git commit -m "Commit gerenated from project $CI_PROJECT_NAME:$CI_COMMIT_SHA, $CI_JOB_STAGE:$CI_JOB_NAME at $(date)." - git remote add heroku https://heroku:$HEROKU_API_KEY@git.heroku.com/$HEROKU_APP_PRODUCTION.git - git remote add gitlab https://$GITLAB_ACCESS_TOKEN_NAME_TEST:$GITLAB_ACCESS_TOKEN@gitlab.com/$GITLAB_APP_PRODUCTION.git # - git push --force https://heroku:$HEROKU_API_KEY@git.heroku.com/$HEROKU_APP_PRODUCTION.git ci_branch:master # Push directly, no need to remote add first. - git remote -v - git push --force gitlab ci_branch:master - git push --force heroku ci_branch:master dependencies: - build-heroku
You are probably wondering about the variables;
$GITLAB_ACCESS_TOKEN etc. These are configured in the project settings:
Settings > CI/CD > Variables. It’s a convenient way to store variables; they are separated from the repo. If we ever need to change these variables, we don’t need to update the source code and make irrelevant commit. Another obvious benefit is that secrets are kept hidden and secure.
That’s all but before finishing, I want to shortly talk about 2 other methods I’ve come across.
- DPL seems really useful for deployment in CI/CD pipelines. I’m sure there are alternatives but this is the one I tried. Dpl supports so many other providers not only heroku. Yet, I wasn’t able to deploy only the subdirectory that is generated in the pipeline.
- The other way is again git based.
git subtree push --prefix=path/to/subdir origin testThis is what I’ve first found while googling. Subtree generates a new git history for the subdirectory we’ve given as prefix. Yet, it didn’t work for me, because the sub-directory must be already known by git; in our case the directory
websitewill be created in the pipeline and it’s not included in the git project. Also, it takes some because git extracts the history for the files in the sub-directory and creates a new branch from that history. Still, it’s an interesting command you can try it locally
git subtree split --prefix path/to/subdir -b new_branch_name.