Host Static Content with Basic Authentication on Heroku using GitLab CI/CD Pipelines

In this post, I’ll first walk through hosting static content with basic authentication. Then, we’ll look into deploying to Heroku using GitLab Pipelines, more specifically deploying a certain sub-directory within the project instead of pushing the whole project. Also, I’ll share what I’ve learned about GitLab CI/CD Pipelines while trying to find a solution.

Part 1: Basic Auth for Static Website

Before starting I want to explain where this problem came from and my hours long process of finding a solution. Lately, for my master’s thesis I’ve been working on a bookdown project. Bookdown is an R package that helps us build docs as pdf or HTML, you can consider it like a static-site-generator. So, the output of my project is static HTML files. I can serve these files using any static site hosting provider like GitLab Pages, GitHub Pages, Netlify, Surge… I’ve already tried many of them before and actively using GitLab for this blog. There are many free alternatives but none of them, as far as I know, provides basic authentication for the website.

In GitLab, the git repo can be private and the pages can be either public or open to only project members. So, you can make GitLab pages private to you or the users you have added to the project. This is very good because last time I checked GitHub didn’t have support for such feature. Still, it’s not enough. Because, what I need is a private site with basic authentication so that we can share link with people that doesn’t have GitLab account. There is an open issue for this topic and it seems it’ll stay open because alternatives like Netlify and Surge provides basic auth within paid plans.

I did some research hoping to find a work around… The best I could find is that we can set up basic auth with Heroku and a simple repo that’s enough to demonstrate the concept. Not the best solution, because it’ll require a little setup for Heroku but that’s affordable. So, in my project after I compile the source docs (yes docs =P) the result is bunch of html files in a directory called public. Example: ROOT_DIR/public/index.html. I just moved the public folder 1 level below and added the files from the heroku-static-provider repo. Example: ROOT_DIR/heroku_site/public/index.html. The directory will look like this:

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----          24/05/2022  6:57 PM                public
-a---          04/05/2022  8:54 PM             26 .gitignore
-a---          04/05/2022  8:54 PM            137 package.json
-a---          04/05/2022  8:54 PM             20 Procfile
-a---          04/05/2022  8:54 PM            650 Readme.md
-a---          04/05/2022  8:54 PM            463 server.js

Then created a new git project at ROOT_DIR/heroku_site, created new-app in heroku, add heroku as remote then just git push. The site is deployed to Heroku and we can set up basic auth using Heroku-CLI.

heroku config:set -a name-of-the-heroku-app USER=...
heroku config:set -a name-of-the-heroku-app PASS=...

If we don’t count the searching phase, implementing this solution didn’t take long. Yet, the implementation I presented above is far from ideal. Because now, I have 2 git projects and I don’t actually care about source controlling for the second project, it’s just the output of the first one. By the way, the second project is excluded in the original one; heroku_site added into the .gitignore. So, I have to compile the project locally to generate the outputs heroku_site/public, commit the changes and push to heroku git push heroku master. I want to automate this using Gitlab CI/CD Pipelines. But how?

Part 2: Deploy To Heroku using GitLab CI/CD Pipelines

I had CI/CD configuration already for the original project; it was already hosted via GitLab Pages. All I need to do is to edit the configuration. It wasn’t so simple though. Because, I didn’t know much about configuring a pipeline in gitlab. The previous configuration was simple; compile the source, put the static site content into a folder named public under root directory. This is the standart way for using gitlab pages.

Since things are a little bit more complicated than the standart way, I created an empty git project to play around and learn. Started with a basic .gitlab-ci.yml that does nothing but printing some texts like “hello world”. Here is the docs to gitlab-ci configuration. By trial and error, and reading docs ofc, I’ve learned a lot. You can find my notes about GitLab Pipelines in the project’s readme file. Hey, don’t go there yet! We haven’t finished here. I’ll now explain the .gitlab-ci.yml that you can find in my test project. Example below is based on this configuration.

A Quick Tutorial for Newbies

In our example, the pipeline has two stages; build and deploy. Stages will be executed in order. If build fails, deploy won’t even start.

image: busybox

stages:
  - build
  - deploy

build-gitlab-pages is a job in the build stage. It spins up a container from busybox image then executes the bash commands under script. The files generated within the job will not be available for the next job. To make them available we must use the keyword artifacts.

pages is a job in the deploy stage. So, it runs after build stage is completed. Since, we set public directory as artifact in the first job, it’s available for the pages job. I run a dummy command in the script part, because a job must have script keyword. Normally, you build your static website here using jekyll, hugo etc. pages is a reserved keyword for gitlab pages. It must pass public folder as artifact if we want Gitlab Pages to work.

build-gitlab-pages:
  stage: build
  script:
    - mkdir public && echo "<html><h1>GITLAB CI PIPELINE</h1><br><p></p></html>" > ./public/index.html
  artifacts:
    paths:
      - public
    expire_in: 30 mins

pages:
  stage: deploy
  script:
    - echo "hellöğğ"
  artifacts:
    paths:
      - public
  only:
    - master

This will host the index.html we have created under username.gitlab.io/project_name; example.

Deploy to Heroku

Now, let’s come to the part we are interested in. The folder structure will be like this:

│   .gitlab-ci.yml
│   ...
│   source_files...
└───heroku_deps
        package.json
        Procfile
        Readme.md
        server.js

In the pipeline:

1. we’ll compile the source files and generate a the output in a directory website/public.
2. copy the files under heroku_deps to website.
3. deploy the website directory to heroku.

Notice that at the end of step 2, the folder structure will be the same as we had at the end of Part 1 but all happens in a container within the pipeline.

Again, we have two stages; build and deploy. The build job creates the website directory, copies the dependencies for heroku project. Lastly it adds a static html file for testing. Normally, you would generate the static content by compiling your project. Pass the directory as artifact for downstream jobs.

build-heroku:
  stage: build
  script:
    - mkdir -p website/public
    - cp heroku_deps/* website
    - echo "<html><h1>GITLAB CI PIPELINE</h1><br><p></p></html>" > website/public/index.html
  artifacts:
    paths:
      - website

Now, we’ll simply cd into the directory we have created. This is our root directory for a new git project. I used ruby image for this job because I knew git is included in it. You can use any base image you want, just make sure git is installed. Otherwise, you would have to install git too; e.g. apt install git if the image is ubuntu based. If we don’t specify the image, gitlab will use ruby too.

Anyway… What we do is pretty simple; initialise a new project, commit changes, add heroku as a remote, then push. We don’t care about version history here. All we want is to deploy the static content to heroku. Everytime this job runs, there will be only one commit and it’ll override the commit in the remote (heroku). That’s why we use git push with --force. Beside heroku, I’ve added another remote that points to gitlab for testing, you can check it here.

deploy-job:
  stage: deploy
  image: ruby:latest
  script:
    - cd website
    - git config --global init.defaultBranch ci_branch
    - git init
    - git config user.email 2335694-mertbakir@users.noreply.gitlab.com
    - git config user.name mertbakir
    - git add .
    - git commit -m "Commit gerenated from project $CI_PROJECT_NAME:$CI_COMMIT_SHA, $CI_JOB_STAGE:$CI_JOB_NAME at $(date)."
    - git remote add heroku https://heroku:$HEROKU_API_KEY@git.heroku.com/$HEROKU_APP_PRODUCTION.git
    - git remote add gitlab https://$GITLAB_ACCESS_TOKEN_NAME_TEST:$GITLAB_ACCESS_TOKEN@gitlab.com/$GITLAB_APP_PRODUCTION.git
    # - git push --force https://heroku:$HEROKU_API_KEY@git.heroku.com/$HEROKU_APP_PRODUCTION.git ci_branch:master # Push directly, no need to remote add first.
    - git remote -v
    - git push --force gitlab ci_branch:master
    - git push --force heroku ci_branch:master
  dependencies:
    - build-heroku

You are probably wondering about the variables; $HEROKU_API_KEY, $HEROKU_APP_PRODUCTION, $GITLAB_ACCESS_TOKEN etc. These are configured in the project settings: Settings > CI/CD > Variables. It’s a convenient way to store variables; they are separated from the repo. If we ever need to change these variables, we don’t need to update the source code and make irrelevant commit. Another obvious benefit is that secrets are kept hidden and secure.

That’s all but before finishing, I want to shortly talk about 2 other methods I’ve come across.

• DPL seems really useful for deployment in CI/CD pipelines. I’m sure there are alternatives but this is the one I tried. Dpl supports so many other providers not only heroku. Yet, I wasn’t able to deploy only the subdirectory that is generated in the pipeline.
• The other way is again git based. git subtree push --prefix=path/to/subdir origin test This is what I’ve first found while googling. Subtree generates a new git history for the subdirectory we’ve given as prefix. Yet, it didn’t work for me, because the sub-directory must be already known by git; in our case the directory website will be created in the pipeline and it’s not included in the git project. Also, it takes some because git extracts the history for the files in the sub-directory and creates a new branch from that history. Still, it’s an interesting command you can try it locally git subtree split --prefix path/to/subdir -b new_branch_name.