Running Blast locally

If you want to develop on Blast, for most things, you will have to run Blast locally to see the effect of your code changes. This pages walks you through how to install Blast and get it running on your machine.

Install the Docker Desktop app

The recommended option for installing and running Blast locally is to use Docker. It is so strongly recommended in fact, that I’m not going to write documentation on how to install and run Blast any other way. The first step is to install the Docker Desktop application, which can be found here for Mac, Windows, and Linux. Make sure you have Docker Compose version 1.28.0 or later. We recommend allocating at least 32 GB of memory in the Docker settings.

Clone the Blast repository

Clone the Blast repository.

git clone https://github.com/scimma/blast.git

Setup environment file

Blast needs some environment variables to run. All of these are defined with their default values in the env/.env.default file. If you want to override any of the default values or add additional environment variables, define these in a file named env/.env.dev following the same format as env/.env.default.

You will likely want to change the BLAST_IMAGE variable in your env/.env.dev to be blast_base instead of blast_latest. This will force Docker to build the image locally AND the app source code directory will be mounted in the container to facilitate live code editing. Otherwise the static published image will be downloaded and used, and local code changes will be ignored.

If you need to ingest real transient data from the Transient Name Server (TNS), you will need to populate the TNS variables with TNS API bot credentials (see https://www.wis-tns.org/bots).

Run the Blast app

You are encouraged to use the Blast control script (run/blastctl) to start and stop the application. This script uses a combination of docker compose and docker commands to properly manage the container lifecycles.

To launch the full Blast stack, invoke the run/blastctl script with the full_dev Docker Compose profile:

bash run/blastctl full_dev up

Alternatively, if you are only interested in running the web server and database, which is usually sufficient for front end web development, you can use the slim_dev Docker Compose profile:

bash run/blastctl slim_dev up

Then go to http://0.0.0.0:8000/ in your web browser after all the containers have started, and Blast should be running.

To watch container logs run

bash run/blastctl $PROFILE logs

Running Blast in these two modes means you can edit most code and you will see the resulting live changes in the web interface.

Tip

Multiple instances of Blast can run concurrently using the different profiles. It is possible, for example, to be running full_dev and simultaneously run the ci profile to execute unit tests.

To terminate Blast and remove the containers, open a new terminal window and run:

bash run/blastctl $PROFILE down

where $PROFILE is the active Docker Compose profile as described above (for example, slim_dev).

Warning

When you stop the Blast container make sure all services are stopped. You can see which services are running in the Docker Desktop app and stop services manually there (or run docker ps --all).

Persistent data volumes

There are three Docker volumes created for persistent data storage: (1) blast-data, (2) blast-db, and (3) django-static. When you run Blast for the first time, these are created automatically by Docker, and then the initialization script populates each volume.

The blast-db volume stores the Django SQL database, and it is provisioned by standard Django migration commands. The django-static volume stores the Django static files, which are generated by standard Django commands as well.

The blast-data volume stores astronomical data. During initialization, all required data files are downloaded and installed.

The Docker volume names are prepended by the Docker Compose project name (e.g. blast-dev) and joined with an underscore. You can list them like so:

$ docker volume ls
...
local     blast-data
local     blast-dev_blast-db
local     blast-dev_django-static

Due to the size of the initial data files, the blast-data volume is shared between the different profiles to avoid redundant downloads and because this data will rarely be in a conflicted state. The static files and database files, however, are specific to each profile.

To restart the application with a clean Django database, use the purge-db launcher action as shown below. Alternative options include purge-data (delete only astro data) and purge-all (delete astro data AND Django database). These options are mutually exclusive, and only one can be used.

# Stop and remove services and internal networks
bash run/blastctl $PROFILE purge-db

The initialization process is idempotent, meaning that it is safe to repeatedly restart the services with or without existing application data.

The initialization process generates temporary files on the astro data volume (/mnt/data/.initializing_db and /mnt/data/.initializing_data) to support the scenario where multiple replicas of service containers are running concurrently. These files are automatically removed when data initialization is complete, but they can also be forcefully purged (in the case of a failed installation for example) by setting the env var FORCE_INITIALIZATION to true.

Testing the Blast app

To run tests with the Blast app, while the full_dev or slim_dev containers are up, in a separate terminal run

$ docker exec -it blast-dev-app-1 bash
root@491edb948cfb:/app# coverage run manage.py test \
    --exclude-tag=download -v2 host.tests api.tests users.tests

This allows you to run the tests without stopping the containers. Some of the tests are excluded in this mode because the assumption is that you are iterating on the unit tests themselves and probably do not want a slow iteration cycle, for example when cutout data is downloaded.

To run all tests from scratch in an dedicated container that does not mount any host paths, run

bash run/blastctl ci up
bash run/blastctl ci purge-db

Building the Blast app image

The Blast app image is a multi-stage build, where the base image is built separately because it is so large and does not often need to change, and it minimizes the duration of the CI pipeline that automatically builds and pushes the official image. To build the base image, use the following pattern, where $YYYYMMDD is some immutable tag:

docker build app/ \
    -f app/Dockerfile.deps \
    -t registry.gitlab.com/ncsa-blast/kubernetes/blast/deps:latest \
    -t registry.gitlab.com/ncsa-blast/kubernetes/blast/deps:$YYYYMMDD

Update app/Dockerfile with the new tag $YYYYMMDD. Then rebuild the app image – incorporating the updated dependencies – by launching the slim_dev Compose profile:

bash run/blastctl slim_dev up
bash run/blastctl slim_dev purge-db

And, finally, run the unit tests as described in the previous section before opening a pull request.