• You can do what you want on your local laptop/desktop (except get viruses)

• Be very careful when you use remote interactive servers shared with other users
  • Others should not even notice that you(r processes) are there!
    Yes, you can use computing resources cleanly and efficiently…
  • Note: notebook servers are shared servers!
  • If you have some heavy duty processing to do:
    • Use shared interactive servers only for quick tests (ideally with a subset of your huge dataset)
    • Use dedicated batch/computing clusters for heavy processing
      • heavy =
        • will run for a long time
        • and/or use a lot of CPU (one or many CPUs)
        • and/or use a lot of RAM
        • and/or do a lot of disk reading/writing,
        • …
      • resources you use in High Performance Computing centers will probably be deducted from a (shared) project

• Disk space (and backup space) is also an important resource!
  • When possible, avoid duplicating data, and run your computations near the data
    • e.g. use the mirror data stored in /bdd/ on the spirit[x] servers
  • Use scratch disks, when available, for temporary data and do not forget to remove temporary data
    • LSCE: /home/scratch01/$USER
    • spiritx: /scratchx/$USER
    • TGCC: $SCRATCH
  • Do not store temporary data, or replicated data, or data that can be easily re-created, on backed-up disks

• Ask your experienced colleagues when you are not sure about something (sometimes even if you are sure…)
  • you can get better domain specific answers from your colleagues than from chatbots
  • There are no stupid questions (especially for beginners)
  • There are always different ways to do things, some are much less efficient than others

• Learn how to monitor resources usage:
  • CPU and RAM: top
  • Disk: quota, du, df
  • Listing and killing processes: ps, kill
  • …

• Read (some of) the local documentation
  RTFM (Read The Fine Manual)
  • Some Linux related documents (by JYP)
    • Learn at least how to use the man(ual) command !!
  • LSCE documentation
  • IPSL Computing and Data Center documentation
  • TGCC documentation
  • Jean Zay @ IDRIS documentation
  • …

• We use ssh and ssh clients to connect securely to the servers we work with
  • Use the PuTTY ssh client to easily open a terminal on a remote server from a Windows computer

• Some computing resources can also be accessed with a web browser and a JupyterHub server

• Servers can usually only access data on local (in the same computer room) disks. If you need to work locally on data available on a remote server, you first have to download the remote data
  • Notable exceptions :
    • spiritx servers in Palaiseau campus can access spirit servers disks at Sorbonne Université, in read-only mode, and the other way around (details)
    • some of the data available in /bdd on the spirit[x] servers are actually distributed from HPC centers
  • Reminder: when possible, avoid duplicating data, and perform the data analysis near the (original) data
  • When possible, reduce the data before download, in order to transfer only the subset you need: e.g. select only the regions and variables you need, pre-compute spatial or temporal means, etc…
    You can easily reduce NetCDF data with:
    • cdo (Climate Data Operators)
    • NCO (netCDF Operators)
  • It is of course a good idea to document (in a simple text file) where the files come from, how they were pre-processed, …

• We use transfer programs based on the ssh protocol to move data securely
  • scp (secure copy): easily copy files and directories recursively
    • Use the WinSCP scp client to easily copy files to/from a Windows computer
  • rsync: mirror directories (more complex)

• Hint: when needed (and after checking the file size), you can check data integrity by comparing the checksum (with sha256sum or md5sum) of the original and copied files. It is more efficient and secure to compare this unique fingerprint than to transfer a file again
  • e.g. with a 14 Gb file on spirit

    $ cd /bdd/CMIP6/CMIP/IPSL/IPSL-CM6A-LR/piControl/r1i1p1f1/Omon/thetao/gn/latest
    
    $ md5sum thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc
    5378634e97136d75e4517cb72e0cda9a  thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc
    
    $ sha256sum thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc
    90cf48c540216897bf4d9841acd7034f85cc10e233eed74b1aa27fd77a8da1e6  thetao_Omon_IPSL-CM6A-LR_piControl_r1i1p1f1_gn_185001-194912.nc

[ PMIP3 Wiki Home ] - [ Help! ] - [ Wiki syntax ]