Jupyter Notebooks: Tricks when on a remote server

In Machine Learning and Data Science, we often find ourselves stuck with the limited computing resources that are required for a vast dataset. Especially training a deep learning model can sometimes take hours or even days. However, this process can be speeded up with the help of cloud services like AWS, Azure, etc that offer GPU machines at a reasonable price. In this article we will see some tricks of working with Jupyter Notebooks on remote server

This will be a short article, wherein we will see how we can set up the Jupyter Notebook when working on a remote machine and how we can access it through our local machine using SSH tunneling. The remote instance just provides the terminal, and it becomes quite challenging to code everything there without the IDE. Please note that the following steps apply to when working with a Linux based instance.

Setting up Jupyter Notebook on Remote Server

So first things first, we will make a virtual environment for our project. It’s always advisable to use virtual environments to isolate the libraries needed by our project from those at the system level. This will ensure no conflicts concerning versions, and every project can be independent of each other.

For the creation of a virtual environment, we will make use of “venv” command which works with Python>=3.3.

• Go to the project folder and type

  $ python3 -m venv myenv

   

  This will create a directory with the name myenv in the project directory.

  Note that myenv can be replaced with any name of your choice.

• Now activate your virtual environment,

  $ source myenv/bin/activate

• The virtual environment will be activated. Next, install Jupyter Notebook in it.

  (myenv)$ pip install jupyter

• In the virtual environment created, install the necessary libraries for your project. Now when the Jupyter Notebook is run, it automatically does so on the IPython kernel using the system level python environment. The IPython kernel is nothing but a computational engine for running and executing the code. To use our virtual environment in the Jupyter Notebook, we have to add an additional kernel in it that would make use of our venv. This can be done by first installing the ipykernel

  (myenv)$ pip install ipykernel
  
  

• And then adding our virtual environment as a kernel by,

  $ python -m ipykernel install --user --name myenv --display-name "Python MyEnv"

So now you have successfully added your python environment in the Jupyter Notebook as shown in the screenshot.

Access Jupyter Notebook on Remote Server

The next question is how to access the Jupyter Notebook running on the remote system through your local system. Let’s see this right away.

• First, you have to ensure that Jupyter is installed both on the local system and the remote system.
• Secondly, on the remote side go to the project directory and run Jupyter Notebook using the following command.

  $ jupyter notebook --no-browser --port=8889
  
  

  You must be wondering why –no-browser is used here. As we will be accessing the notebook remotely we won’t be needing any browser for opening the notebook. The port specified can also be changed.

• Now to access this running notebook from our local machine, we will use port forwarding using SSH tunneling. Once the port used by remotely running Jupyter is forwarded to a port on the local machine, it can be easily accessed commonly as any Jupyter notebook is accessed.
  So to establish the SSH connection, use the following command in the terminal of your local system:

  $ ssh -N -f -L localhost:8888:localhost:8889 remote_user@remote_host

  The first localhost specifies the port on the local system and the second of the remote system.

  remote_user is the user name in the remote host and remote_host is the address of the remote server.

• After entering this command, the terminal will prompt you to enter the password. Once you enter the password, the connection will be established.
• Now just go to any browser, and type

  localhost:8888

Woah, you just accessed the Jupyter Notebook running remotely in your local system. These were some little tricks with Jupyter Notebooks on remote server that can make your coding part easy. Now you can make any changes and dive right into the coding!!