PMIP3 wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Working with ipython notebooks ======

===== What does this mean? =====

There are **many** python distributions, **many** python versions (''2.7.xx'', ''3.nn'') and... **many** ways to use python:
  * //interactively// or by running a //script// (or running a script with ''python -i script.py'' and then typing interactive commands)
  * using python in a //Linux terminal//, the [[https://www.spyder-ide.org/|Spyder]] GUI (Graphical User Interface) a //Windows// //cmd// terminal or powershell, a //Mac// terminal, or whatever GUI the distribution you have installed on Windows or Mac provides
  * using the default ''python'' interpreter or the ''ipython'' interpreter \\ <code> > python
Python 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 09:07:38)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> "python interpreter in a terminal".upper()
'PYTHON INTERPRETER IN A TERMINAL'
>>>

 > ipython
Python 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 09:07:38)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.3.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: "ipython interpreter in a terminal".upper()
Out[1]: 'IPYTHON INTERPRETER IN A TERMINAL'

In [2]:
Do you really want to exit ([y]/n)? y</code>
  * using python interactively in a //ipython notebooks//, which means that
    * you will use an ipython interpreter
    * in a web browser (rather than a terminal)
    * a notebook is the content of the web page where you have interacted with python. You can save it (''.ppynb'' file) and share it, and it will display both the commands you have used and the output/results of your commands
    * the notebooks have cells with python input, python output (including graphics), and cells that can be used to format the document (titles, formatted text, equations, images, etc...)
    * the notebooks support several other languages (e.g. **R**)

===== How does it work? =====

Using an ipython notebooks implies the following steps:

  - determine if you will use python on your local computer or a remote computer and //go// to that computer
    * the selected computer mostly depends on the data you need to work with. The idea is to move the computation (i.e. python) near the data, and not move/replicate the data
  - choose/initialize the distribution and version of python you need to use on the selected computer
    * e.g. you can [[other:python:starting#ultra_quick-start_on_the_lsce_servers|use the CDAT distribution on the LSCE servers]]
  - start the ipython notebook server with the appropriate parameters (on the selected computer, with the selected python)
    * this will start a temporary web server that you can connect to, in order to use the notebooks
      * the server will display the URL that you have to use. The URL has an authentication token and only you (or the person who has a copy of the full URL) can connect to your server
    * the basic way to start a server is to just  type ''jupyter notebook'', but only do this if you want to work locally on your desktop/laptop! See the next section for working on a remote computer
      * the ''jupyter notebook'' command will start the notebook server, and it will automatically start a web browser on the same computer. That's OK on your local computer, but on a remote server, the browser will probably seem to be slow (and probably out of date) and will use unnecessary resources of the server. This will be frustrating for you and the other users of the server
    * if you have to work on a remote computer, <wrap hi>the idea is to start the ipython server on the remote computer and use a web browser on your local computer</wrap> (thanks to an //ssh tunnel//)
  - in a web browser, open the temporary URL displayed by the ipython server and start using the ipython notebooks
  - when you are finished, <wrap em>do not forget to kill the ipython server</wrap> by typing ''^C^C'' (2x ''CTRL-C'') in the terminal where you started the server. Otherwise the server and the python processes started by the server will keep on using resources on the computer
    * Warning! Closing the browser, or using the ''Logout'' button in the browser will not stop the ipython server. You can stop the server by clicking on the ''XXXX'' FIXME button

===== Working correctly with the notebooks =====

==== Reminder ====

  * Make sure you read and understand the [[#how_does_it_work|How does it work?]] section above
  * If you have a python installed **on your own desktop/laptop**, you can just start a local notebook server and browser with\\ ''jupyter notebook''
  * **On a remote and shared server**, only start the notebook server and use a local browser to connect to the notebook server through an //ssh tunnel//.

==== Prerequisites ====

You need to meet the following prerequisites in order to create an ssh tunnel to the remote server where your notebook server will run

  * We assume that:
    * **you have a local web browser** installed!
    * **you have an account on the remote server**: replace ''<LOGIN>'' below with your account name
    * **you have an assigned a //port number//** on the remote server: replace ''<PPPP>'' with your specific port number
      * List of assigned port numbers at **LSCE**: check the [[http://wiki.ipsl.jussieu.fr/IGCMG/Outils/IPython_Notebook#Shared_TCP_ports|Shared TCP ports]] table
        * Ask your advisor for a port number at LSCE if you do not have one
      * List of assigned port numbers on **ciclad**: check the ''/ciclad-home/jservon/Evaluation/CliMAF/climaf_installs/climaf_1.2.10/bin/cmip6.notebook.txt'' file
        * Ask Jérôme Servonnat for a port number on ciclad if you do not have one
  * System dependent requirements:
    * **Linux** and **Mac**: you should already be able to open a local terminal, and ''ssh'' is installed by default
    * **Windows**: you either need to have [[other:win10apps#putty_pageant|Putty]] installed, or you need to know how to open a Windows PowerShell (''CTRL''+''Esc'' to open the start menu, then type ''PowerShell'' and start a PowerShell)
      * **Windows 7**: you have to install [[other:win10apps#putty_pageant|Putty]] in order to use ''ssh'':
        * In the instructions below, replace ''ssh'' with ''plink'' if you are using Windows 7, because //plink// is the name of the Putty ssh command-line executable
      * **Windows 10**: ssh is already installed, but you may still want to install [[other:win10apps#putty_pageant|Putty]] in order to use predefined sessions
        * if you have installed [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:win10wsl|Windows Subsystem for Linux and Ubuntu]], you can also use a Ubuntu terminal

==== Digging ssh tunnels ====

The ssh tunnels will allow you to connect a web browser running **locally** on your computer to a notebook server running on a **remote** server

=== From a local LSCE computer to obelix ===

ssh -L 7225:localhost:7225 jypeter@obelix

=== From a computer outside LSCE to obelix ===

In that case, we have to go through the ssh1 gateway before connecting to obelix

<code>plink -L 7225:localhost:7225 jypeter@obelix


plink -L 7225:localhost:7225 jypeter@obelix

ssh -X -t -L7100:localhost:7100 jypeter@ssh1.lsce.ipsl.fr ssh -L7100:localhost:7100 obelix

plink -X -t -L7100:localhost:7100 jypeter@ssh1.lsce.ipsl.fr ssh -L7100:localhost:7100 obelix

plink -ssh -X -t -L 7100:localhost:7100 jypeter@ssh1.lsce.ipsl.fr ssh -L7100:localhost:7100 obelix

plink -X -t -L 7100:localhost:7100 jypeter@ssh1.lsce.ipsl.fr ssh -L7100:localhost:7100 obelix

ssh -X -t -L 7100:localhost:7100 jypeter@ssh1.lsce.ipsl.fr ssh -L 7100:localhost:7100 obelix

(cdatm_py3) jypeter@obelix3 - ...jypeter - 44 >jupyter notebook --no-browser --port=7100
[I 23:37:26.017 NotebookApp] [nb_conda_kernels] enabled, 14 kernels found
[I 23:37:26.977 NotebookApp] [nb_conda] enabled
[I 23:37:26.978 NotebookApp] Serving notebooks from local directory: /home/users/jypeter
[I 23:37:26.978 NotebookApp] The Jupyter Notebook is running at:
[I 23:37:26.978 NotebookApp] http://localhost:7100/?token=256f8fa26c8478f136720e0c0c24c4995b6682d9febf6387
[I 23:37:26.978 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 23:37:26.992 NotebookApp]

    To access the notebook, open this file in a browser:
        file:///home/users/jypeter/.local/share/jupyter/runtime/nbserver-1442-open.html
    Or copy and paste one of these URLs:
        http://localhost:7100/?token=256f8fa26c8478f136720e0c0c24c4995b6682d9febf6387</code>

==== Extra resources ====

  * Jérôme's script for starting notebooks on the ciclad cluster: ''/ciclad-home/jservon/Evaluation/CliMAF/climaf_installs/climaf_1.2.10/bin/climaf-notebook''
  * [[http://wiki.ipsl.jussieu.fr/IGCMG/Outils/IPython_Notebook|Patrick's page 1]]
  * [[https://nbviewer.jupyter.org/github/PBrockmann/IPy_Notebooks/blob/master/introduction/Presentation_01.ipynb|Patricks' page 2]]
  * JYP's [[https://wiki.lsce.ipsl.fr/pmip3/doku.php/other:uvcdat:cdat_conda:climaf_test|Testing CLIMAF on ciclad]] page

/* standard page footer */

\\ \\ \\ 
----
[ [[pmip3:|PMIP3 Wiki Home]] ] -
[ [[pmip3:wiki_help|Help!]] ] -
[ [[wiki:syntax|Wiki syntax]] ]