Running Blast locally ===================== If you want to develop on Blast, for most things, you will have to run Blast locally to see the effect of your code changes. This pages walks you through how to install Blast and get it running on your machine. Install the Docker Desktop app ------------------------------- The recommended option for installing and running Blast locally is to use Docker. It is so strongly recommended in fact, that I'm not going to write documentation on how to install and run Blast any other way. The first step is to install the Docker Desktop application, which can be found `here `_ for Mac, Windows, and Linux. Make sure you have Docker Compose version 1.28.0 or later. We recommend allocating at least 32 GB of memory in the Docker settings. Clone the Blast repository -------------------------- Clone the Blast repository. .. code:: bash git clone https://github.com/scimma/blast.git Setup environment file ---------------------- Blast needs some environment variables to run. All of these are defined with their default values in the :code:`env/.env.default` file. If you want to override any of the default values or add additional environment variables, define these in a file named :code:`env/.env.dev` following the same format as :code:`env/.env.default`. You will likely want to change the :code:`BLAST_IMAGE` variable in your :code:`env/.env.dev` to be :code:`blast_base` instead of :code:`blast_latest`. This will force Docker to build the image locally **AND** the :code:`app` source code directory will be mounted in the container to facilitate live code editing. Otherwise the static published image will be downloaded and used, and local code changes will be ignored. If you need to ingest real transient data from the Transient Name Server (TNS), you will need to populate the TNS variables with TNS API bot credentials (see ``_). Run the Blast app ----------------- You are encouraged to use the Blast control script (:code:`run/blastctl`) to start and stop the application. This script uses a combination of :code:`docker compose` and :code:`docker` commands to properly manage the container lifecycles. To launch the full Blast stack, invoke the :code:`run/blastctl` script with the :code:`full_dev` Docker Compose profile: .. code:: bash bash run/blastctl full_dev up Alternatively, if you are only interested in running the web server and database, which is usually sufficient for front end web development, you can use the :code:`slim_dev` Docker Compose profile: .. code:: bash bash run/blastctl slim_dev up Then go to `http://localhost:8000/ `_ in your web browser after all the containers have started, and Blast should be running. To watch container logs run .. code:: bash bash run/blastctl $PROFILE logs Running Blast in these two modes means you can edit most code and you will see the resulting live changes in the web interface. .. tip:: Multiple instances of Blast can run concurrently using the different profiles. It is possible, for example, to be running :code:`full_dev` and simultaneously run the :code:`ci` profile to execute unit tests. To terminate Blast and remove the containers, open a new terminal window and run: .. code:: bash bash run/blastctl $PROFILE down where :code:`$PROFILE` is the active Docker Compose profile as described above (for example, :code:`slim_dev`). .. warning:: When you stop the Blast container make sure all services are stopped. You can see which services are running in the Docker Desktop app and stop services manually there (or run ``docker ps --all``). Persistent data volumes ----------------------- There are three Docker volumes created for persistent data storage: (1) :code:`blast-data`, (2) :code:`blast-db`, and (3) :code:`django-static`. When you run Blast for the first time, these are created automatically by Docker, and then the initialization script populates each volume. The :code:`blast-db` volume stores the Django SQL database, and it is provisioned by standard Django migration commands. The :code:`django-static` volume stores the Django static files, which are generated by standard Django commands as well. The :code:`blast-data` volume stores astronomical data. During initialization, all required data files are downloaded and installed. The Docker volume names are prepended by the Docker Compose project name (e.g. ``blast-dev``) and joined with an underscore. You can list them like so: .. code:: bash $ docker volume ls ... local blast-data local blast-dev_blast-db local blast-dev_django-static Due to the size of the initial data files, the :code:`blast-data` volume is shared between the different profiles to avoid redundant downloads and because this data will rarely be in a conflicted state. The static files and database files, however, are specific to each profile. To restart the application with a clean Django database, use the :code:`purge-db` launcher action as shown below. Alternative options include :code:`purge-data` (delete only astro data) and :code:`purge-all` (delete astro data AND Django database). These options are mutually exclusive, and only one can be used. .. code:: bash # Stop and remove services and internal networks bash run/blastctl $PROFILE purge-db The initialization process is idempotent, meaning that it is safe to repeatedly restart the services with or without existing application data. The initialization process generates temporary files on the astro data volume (:code:`/mnt/data/.initializing_db` and :code:`/mnt/data/.initializing_data`) to support the scenario where multiple replicas of service containers are running concurrently. These files are automatically removed when data initialization is complete, but they can also be forcefully purged (in the case of a failed installation for example) by setting the env var :code:`FORCE_INITIALIZATION` to true. Testing the Blast app --------------------- To run tests with the Blast app, while the :code:`full_dev` or :code:`slim_dev` containers are up, in a separate terminal run .. code:: bash $ docker exec -it blast-dev-app-1 bash root@491edb948cfb:/app# coverage run manage.py test \ --exclude-tag=download -v2 host.tests api.tests users.tests This allows you to run the tests without stopping the containers. *Some of the tests are excluded in this mode* because the assumption is that you are iterating on the unit tests themselves and probably do not want a slow iteration cycle, for example when cutout data is downloaded. To run all tests from scratch in an dedicated container that does not mount any host paths, run .. code:: bash bash run/blastctl ci up bash run/blastctl ci purge-db Building the Blast app image ---------------------------- The Blast app image is a multi-stage build, where the base image is built separately because it is so large and does not often need to change, and it minimizes the duration of the CI pipeline that automatically builds and pushes the official image. To build the base image, use the following pattern, where :code:`$YYYYMMDD` is some immutable tag: .. code:: bash docker build app/ \ -f app/Dockerfile.deps \ -t registry.gitlab.com/ncsa-blast/kubernetes/blast/deps:latest \ -t registry.gitlab.com/ncsa-blast/kubernetes/blast/deps:$YYYYMMDD Update :code:`app/Dockerfile` with the new tag :code:`$YYYYMMDD`. Then rebuild the app image -- incorporating the updated dependencies -- by launching the :code:`slim_dev` Compose profile: .. code:: bash bash run/blastctl slim_dev up bash run/blastctl slim_dev purge-db And, finally, run the unit tests as described in the previous section before opening a pull request.