Eric J Ma's Website

Better conda environments on GitHub actions

written by Eric J. Ma on 2021-12-30 | tags: continuous integration github actions til

I recently figured out two tips to make GitHub actions play nicely with conda installations. Here they are.

Ensure bash is in login mode

The first is to use the following block:

    # https://github.com/marketplace/actions/setup-miniconda#use-a-default-shell
    defaults:
      run:
        shell: bash -l {0}

What this does is ensure that every shell command is run in login mode. As detailed in this StackOverflow answer:

  • -l to insure (sic) a login bash, where the environment is correctly set;
  • {0}, a template placeholder,

replaced at pipeline execution time by the actual script command to execute.

Counterfactually, I would have had to use the deprecated source activate <env_name>, which always made me a bit nervous. Now, I can instead switch over to using conda activate <env_name> before executing environment-specific commands, thereby providing longevity for the build.

Use mambaforge

The other tip is to use the mambaforge installer to get a conda installation onto GitHub actions. The block I recently used for my causality repo is as follows:

      # See: https://github.com/marketplace/actions/setup-miniconda
      - name: Setup miniconda
        uses: conda-incubator/setup-miniconda@v2
        with:
          auto-update-conda: true
          miniforge-variant: Mambaforge
          channels: conda-forge
          python-version: 3.9
          activate-environment: causality
          environment-file: environment.yml
          use-mamba: true

This configuration guarantees the use of mamba to solve the environment, which means we will have blazingly fast builds. Previously, I used to use a different GitHub action (s-weigand/setup-conda@v1), original conda (rather than mamba), and a convoluted build that involved environment caching. You can take a look at an example that I copied over from my nxviz project repository by expanding the details below.


name: Build documentation

on:
  push:
    branches:
      - master


jobs:
  build-environment:
    runs-on: ubuntu-18.04
    name: Build conda environment
    steps:
      - uses: actions/checkout@v2
        name: Checkout repository


  # See: https://github.com/marketplace/actions/setup-conda
  - name: Setup anaconda
    uses: s-weigand/setup-conda@v1
    with:
      conda-channels: "conda-forge"

  # Build cache of environment
  - name: Cache conda environment
    id: cache-environment
    uses: actions/cache@v2
    with:
      path: nxviz.tar.gz
      # Note: Remember that whatever files the environment build depends on
      # should be hashed and added to the key.
      key: ${{ runner.os }}-env.${{ hashFiles('environment.yml') }}

  - name: Build environment
    if: steps.cache-environment.outputs.cache-hit != 'true'
    run: |
      conda env create -f environment.yml
      python -m pip install .

  - name: Install conda-pack
    if: steps.cache-environment.outputs.cache-hit != 'true'
    run: conda install -c conda-forge conda-pack

  - name: Run conda-pack
    if: steps.cache-environment.outputs.cache-hit != 'true'
    run: conda pack -n nxviz -o nxviz.tar.gz

  # See: https://github.com/actions/upload-artifact
  - name: Upload environment
    uses: actions/upload-artifact@v2
    with:
      name: nxviz-tarball
      path: nxviz.tar.gz




docs:
    name: Build static site docs
    runs-on: ubuntu-latest
    needs: build-environment


steps:
  - name: Checkout repository
    uses: actions/checkout@v2

  # https://github.com/actions/download-artifact
  - name: Download environment tarball
    uses: actions/download-artifact@v2
    with:
      name: nxviz-tarball

  - name: Unpack environment and activate it
    run: |
      bash scripts/ci/unpack_environment.sh

  - name: Build docs
    run: |
      source /tmp/nxviz/bin/activate
      python -m ipykernel install --user --name nxviz
      make docs

  - name: Deploy website
    uses: peaceiris/actions-gh-pages@v3
    with:
      # https://github.com/peaceiris/actions-gh-pages#%EF%B8%8F-set-personal-access-token-personal_token
      personal_token: ${{ secrets.GHPAGES_TOKEN }}
      publish_dir: ./site
      publish_branch: gh-pages
      # destination_dir: manuscript
      allow_empty_commit: false
      keep_files: false
      force_orphan: true
      enable_jekyll: false
      disable_nojekyll: false

By contrast, the new build is much smaller and easier to maintain:


name: Build documentation

on:
  push:
    branches:
      - master


jobs:
  build-docs:
    runs-on: ubuntu-latest
    name: Build conda environment


# https://github.com/marketplace/actions/setup-miniconda#use-a-default-shell
defaults:
  run:
    shell: bash -l {0}

steps:
  - uses: actions/checkout@v2
    name: Checkout repository

  # See: https://github.com/marketplace/actions/setup-miniconda
  - name: Setup miniconda
    uses: conda-incubator/setup-miniconda@v2
    with:
      auto-update-conda: true
      miniforge-variant: Mambaforge
      channels: conda-forge
      python-version: 3.9
      activate-environment: nxviz
      environment-file: environment.yml
      use-mamba: true

  - name: Build environment
    run: |
      conda activate nxviz
      python -m ipykernel install --user --name nxviz
      python -m pip install .
      make docs

  - name: Deploy website
    uses: peaceiris/actions-gh-pages@v3
    with:
      # https://github.com/peaceiris/actions-gh-pages#%EF%B8%8F-set-personal-access-token-personal_token
      personal_token: ${{ secrets.GHPAGES_TOKEN }}
      publish_dir: ./site
      publish_branch: gh-pages
      # destination_dir: manuscript
      allow_empty_commit: false
      keep_files: false
      force_orphan: true
      enable_jekyll: false
      disable_nojekyll: false

Cite this blog post: 
@article{
    ericmjl-2021-better-actions,
    author = {Eric J. Ma},
    title = {Better conda environments on GitHub actions},
    year = {2021},
    month = {12},
    day = {30},
    howpublished = {\url{https://ericmjl.github.io}},
    journal = {Eric J. Ma's Blog},
    url = {https://ericmjl.github.io/blog/2021/12/30/better-conda-environments-on-github-actions},
}

I send out a newsletter with tips and tools for data scientists. Come check it out at Substack.

If you would like to sponsor the coffee that goes into making my posts, please consider GitHub Sponsors!

Finally, I do free 30-minute GenAI strategy calls for teams that are looking to leverage GenAI for maximum impact. Consider booking a call on Calendly if you're interested!