Exercise materials for "Practical machine learning for spatial data" at CSC

Content of this repository

This repository contains all Jupyter Notebooks and other code used in the course. Data is not inlcuded here, data download links are provided in data preparations Notebooks. Each exercise has its own folder:

• 01_vector_data_preparation
• 02_raster_data_preparation
• 03_shallow_regression
• 04_shallow_classification
• 05_deep_regression
• 06_deep_classification
• 07_samgeo
• 08_cnn_segmentation
• 09_object_detection

Course exercise enviroment

During the course exercises are done in LUMI, which is EuroHPC supercomputer. Accessing LUMi requires LUMI project. Finnish users get access to LUMI via CSC. For course the course participants are added to the course project.

LUMI webinterface

• Open https://www.lumi.csc.fi
• Log in with:
  • HAKA, if you have (Finnish universities and some research institutes)
  • CSC account, you need your CSC username and password

Copy exercise materials

Open Login node shell

cd /scratch/project_462001167/students/
mkdir $USER
cd $USER
git clone https://github.com/csc-training/GeoML.git

Jupyter

• Click "Jupyter" on dashboard

• Select following settings:

  • Project: project_462001167 during course, own project later
  • Partition: interactive
  • CPU cores: 4
  • Local disk: 0
  • Time: 4:00:00 (or adjust to reasonable)
  • Working directory: /scratch/project_462001167 during course, own project's scratch later*
  • Python: geoconda OR custom depending on the exercise
    • Exercises 1 - 6, 9 data preparation: geoconda
      • No virtual environment
    • Exercises 7 - 9: pytorch
      • Before opening Jupyter the first time, you need to create virtual environment with some extra packages, see below.
      • Check, Enable virtual environment
      • Virtual environment path: /scratch/project_462001167/students/$USER/geoml
      • Check, Enable packages under ~/.local/lib on venv start
  • (Do not select any of the check-boxes below.)

• Click launch and wait until granted resources

• Click "Connect to Jupyter"

• Open Terminal and clone exercise materials

• Open in JupyterLab folder students/<your_username>/GeoML

Adding deep learning librares to pytorch module

The Pytorch module does not include all Python packages required by these exercises. To add custom packages, the best option is to use venv (virtual environment).

Open Login node shell and add the venv to scratch:

cd /scratch/project_462001167/students/$USER
module use /appl/local/csc/modulefiles/
ml pytorch
python3 -m venv --system-site-packages geoml
source geoml/bin/activate
pip install torchgeo # CNN exercise
pip install sahi ultralytics folium==0.13 mapclassify # Object detection exercise
pip install segment-geospatial[samgeo] addict yapf pycocotools supervision #SAM
pip install groundingdino-py # SAM

Optional, QGIS

CSC Dosc: QGIS

Exercises on own computer

Exercises 1-7 Jupyter notebooks can be run as is on any computer. Exercises 8 - 9 (CNN and object detection) require GPU availability for execution in reasonable time.

To get started:

• Get the exercise materials from Github
  • If you have git intalled: git clone https://github.com/csc-training/GeoML.git
  • OR download the repository as a zip-file
• Install all needed packages for running the notebooks:
  • Install mini-conda or some other tool supporting conda .yml files.
  • Open Miniforge promt
  • Go to the folder where you saved the downloaded Github materials
  • Create new conda environment based on the environmet.yml
    • conda env create --name geo-ml --file environment.yml
    • See conda docs, envs for more information.
• Activate the created environment:
  • conda activate geo-ml
• Launch JupyterLab:
  • jupyter lab
• Adapt the main path in beginning of each notebook to your environment.
  • On Windows, it would look something like: base_directory= os.path.join('C:\\temp\\geoml-course','GeoML')
• Have fun going through the notebooks and add an issue to this repository if something is not working.

Extra material

• Links to further resources
• Optional tools for machine learning with spatial data

Authors

Iida Hakulinen, Kylli Ek, Samantha Wittke, Johannes Nyman

Acknowledgement

These materials have been developed with Location Innovation Hub (LIH) and Geoportti funding.

Please acknowledge CSC, Location Innovation Hub (LIH) and Geoportti in your publications, it is important for project continuation and funding reports. As an example, you can write "The authors wish to thank CSC - IT Center for Science, Finland (urn:nbn:fi:research-infras-2016072531), Location Innovation Hub and the Open Geospatial Information Infrastructure for Research (Geoportti, urn:nbn:fi:research-infras-2016072513) for computational resources and support".