Custom Environments

Utility Commands for Custom Kernels

After loading an environment

List your current kernels

jupyter kernelspec list

Remove a kernel

jupyter kernelspec uninstall {KERNEL_NAME}

Custom Python Environments in Jupyter

Load the desired version of python

module load python/3.10.5

Create a directory for your desired environment:

mkdir myenv
cd myenv
python -m venv .

. bin/activate

Once you are in the environment, install ipykernel with the following command:

pip install ipykernel

Add the environment to your jupyter kernels:

python -m ipykernel install --user --name=myenv --display-name "myenv" --env PATH $PATH

Follow the instructions to launch jupyter. You should now be able to select the environment we just added:

Custom Conda Environments in Jupyter

For information on how to create custom conda environments, please visit our guide:

Miniconda Environment Set Up - Storrs HPC - UConn Knowledge Base

Begin by activating/creating your desired environment:

conda create -n myenv

conda activate myenv

Once you are in the environment, install ipykernel with the following command:

conda install ipykernel

Add the environment to your jupyter kernels:

python -m ipykernel install --user --name=myenv --display-name "myenv" --env PATH $PATH

Follow the instructions to launch jupyter. You should now be able to select the environment we just added:

Initiating a JupyterLab with your own Conda Environment (Any Python Version)

These instructions will outline how to start an interactive session on a compute node, load a conda environment with any version of python, and initiate a jupyter lab with that conda environment on that compute node. This task requires ssh tunneling and a conda environment that has jupyter lab installed. For more info on installing conda and setting up your conda environment, please see our other page, Miniconda Environment Setup.

Please note that you will need to install jupyter lab in your conda environment. The command for installing JupyterLab with conda can be found here.

These are the general steps.

1. Start an interactive session on the cluster. This is just the basic command I recommend for people trying to start an interactive session with 1 core. You can adjust this command to fit your needs. More info on flags can be found here.

   1. srun -n 1 --mem=5G -p general --constraint=cpuonly --pty bash

2. Once you are allocated resources, then you’ll have to activate your conda environment. Replace “myenv” with the name of your environment.

   1. conda activate myenv
      python3 -m pip install jupyterlab

3. Now use the hostname command to see what node you are on. Take note of what it says.

   1. hostname

4. Navigate to the folder where your data is. In jupyter lab, you can go into subfolders but you can’t go higher up in folders than where you started. Once you get where you want to be, initiate your jupyter lab.

   1. jupyter-lab --no-browser --ip='*'

5. When you do that a whole bunch of stuff will pop up. It’ll look something like this:

6. 

7. We need to find two things in that mess, the name of the node and our port access number.

   1. The name of the node is the same as the output “hostname" command we used earlier. In this case, the name of the node is cn506.

   2. Port access numbers are 4 digits long and normally start 88##. In this case, the port access number is 8888.

8. We now have to log in to the cluster a second time from a new terminal on your computer, but this time we’re going to log in directly to the node where our jupyter notebook was initiated and enable ssh tunneling. The command will look like this

   1. ssh -NL localhost:8888:cn506:8888 netID@hpc2.storrs.hpc.uconn.edu

   2. The only things you’ll have to update are your netID, the port access numbers, and the name of the node in the command above.

   3. Sometimes in very rare instances, the port binding from the SSH tunnel command above might be denied due to a conflict within a conda environment and provide the following error message:

      bind [127.0.0.1]:8888: Permission denied.

      To fix the issue, it is recommended to remove and recreate the conda environment that is looking to be used. The Python packages within the Conda environment would need to be reinstalled. Once the environment is recreated and the Python packages reinstalled, the new Conda environment would have a clean state to launch Jupyter lab/notebook instances.

9. After you successfully log in to the node where your jupyter lab was initiated, you can copy the link that starts with numbers (127.0. etc.) and paste that into a browser. It should open a jupyter lab session that looks like the JupyterHub pictures above. From there you can click on the “Notebook” button to open a jupyter notebook or click on a previous notebook saved in your directory from the menu of your files on the left-hand side of the screen.

10. And from that notebook, you should be able to use your own conda environment. It should already be activated, but if it isn’t working you can look into installing nb_conda in all of your conda environments which allows users to switch between conda environments from within a jupyter notebook.