Worker Deployment (WSL)

The following pipelines can be run via the docker containers: Calcium_Imaging_Worker, Spike_Sorting_Worker, and Standard_Worker.

For the DeepLabCut worker, we will run this from the provided dlc notebook.

More on proper usage in the How To section.

An important note about worker deployment

Once workers are up, they will begin populating the tables with data. Each worker will populate the following tables:

  • Standard_Worker: BehaviorIngestion, EphysRecording, CuratedClusting, WaveformSet, LFP, ScanInfo, MotionCorrection, Segmentation, Fluorescence, and Activity (all tables without large computational resources needed)

  • Spike_Sorting_Worker: Clustering table (runs Kilosort)

  • Calcium_Imaging_Worker: Processing table (in Imaging schema)

Workers will continually operate and will work if new data appears in your inbox directory. If no data is added, and data is not added to LabBook, the workers will not do anything.

If there are any changes to the code, you will need to rebuild the images before running the workers again.

How to “up” the workers

We will be working in WSL/Ubuntu 20.04 CLI for this section.

Standard Worker

  1. First, create a local .env file in ./docker/standard_worker/ based on the standard example.env.

You will need to fill in DJ_PASS (e.g. Datajoint Database Password), ROOT_DATA_DIR (e.g. local /Inbox data directory), PROCESSED_DATA_DIR (e.g. local /Outbox data directory).

  1. Next, we will need to navigate to the ./docker/standard_worker/dist/debian directory and build the worker image:

cd ./docker/standard_worker/dist/debian

Now that we’re in the right place, we will need to build the image:

docker compose --env-file=../../.env -f docker-compose-standard_worker.yaml -p sabatini-datajoint-pipeline_standard build --no-cache

The build should complete without any errors. If you receive errors please check your .env file and make sure you have the correct paths. Note: you only have to “build” the image once. If you make changes to the code, you will need to rebuild the image.

  1. Now that the image is built, we can run the worker by running the following command:

docker compose --env-file=../../.env -f docker-compose-standard_worker.yaml -p sabatini-datajoint-pipeline_standard up -d

The worker should now be running in the background. You can check the status of the worker by running the following command:

docker ps -a

If you wish to follow the container logs:

docker logs -f [container-ID] -f

  1. To “down” the worker, run the following command:

docker compose --env-file=../../.env -f docker-compose-standard_worker.yaml -p sabatini-datajoint-pipeline_standard down

Spike Sorting Worker

The spike sorting worker is a bit more complicated than the standard worker. This is because the spike sorting worker requires our MATLAB license to run and the GPU to run. Please do not hesitate to contact the database manager for help with this section.

  1. First, create a local .env file in ./docker/spike_sorting_worker/ based on the spike sorting example.env.

You will also need to edit the network.txt file to include our HMS MATLAB server. For both the .env and network.txt files, please contact the database manager for help.

  1. We will then need to build the images. First, we will need to navigate to the ./docker/spike_sorting_worker/dist/debian directory and build the worker image:

cd ./docker/spike_sorting_worker/dist/debian

Now that we’re in the right place, we will need to build the image:

docker compose --env-file=../../.env -f docker-compose-spike_sorting_worker.yaml -p sabatini-datajoint-pipeline_spike_sorting build --no-cache

Check that the build completed without any errors. If you receive errors please check your .env and network.txt file and make sure you have the correct information. Note: you only have to “build” the image once. If you make changes to the code, you will need to rebuild the image.

  1. After the image has been built, we can run the worker by running the following command:

docker compose --env-file=../../.env -f docker-compose-spike_sorting_worker.yaml -p sabatini-datajoint-pipeline_spike_sorting up -d

The worker should now be running in the background. You can check the status of the worker by running the following command:

docker ps -a

If you wish to follow the container logs:

docker logs -f [container-ID] -f

  1. Once you have finished using the worker, you can “down” the worker by running the following command:

docker compose --env-file=../../.env -f docker-compose-spike_sorting_worker.yaml -p sabatini-datajoint-pipeline_spike_sorting down

Calcium Imaging Worker

The calcium imaging worker is similar to the standard worker. We will be running Suite2P for this worker and the paramaters can be set using the provided calcium_imaging notebook.

  1. First, create a local .env file in ./docker/calcium_imaging_worker/ based on the calcium example.env.

You will need to fill in DJ_PASS (e.g. Datajoint Database Password), ROOT_DATA_DIR (e.g. local /Inbox data directory), PROCESSED_DATA_DIR (e.g. local /Outbox data directory).

  1. Next, we will need to build the images. First, we will need to navigate to the ./docker/calcium_imaging_worker/dist/debian directory and build the worker image:

cd ./docker/calcium_imaging_worker/dist/debian

Now that we’re in the right place, we will need to build the image:

docker compose --env-file=../../.env -f docker-compose-calcium_imaging_worker.yaml -p sabatini-datajoint-pipeline_calcium_imaging build --no-cache

Check that the build completed without any errors. If you receive errors please check your .env file and make sure you have the correct paths. Note: you only have to “build” the image once. If you make changes to the code, you will need to rebuild the image.

  1. After the image has been built, we can run the worker by running the following command:

docker compose --env-file=../../.env -f docker-compose-calcium_imaging_worker.yaml -p sabatini-datajoint-pipeline_calcium_imaging up -d

The worker should now be running in the background. You can check the status of the worker by running the following command:

docker ps -a

If you wish to follow the container logs:

docker logs -f [container-ID] -f

  1. Once you have finished using the worker, you can “down” the worker by running the following command:

docker compose --env-file=../../.env -f docker-compose-calcium_imaging_worker.yaml -p sabatini-datajoint-pipeline_calcium_imaging down