Organizing Python Projects

Organizing Python Projects#

A complex research project often relies on many different programs and software packages to accomplish the research goals. An important part of data science or scientific computing is deciding how to organize and structure the code you use for data analysis and research. A well-structured project can make you a more efficient and effective researcher. It is also a key component of scientific reproducibility.

Just putting all of your code into git repositories won’t magically turn a mess of scripts into a beautiful, well-organized project. More deliberate effort is required.

Types of Projects#

Not all projects are created equal. There are three common “research code” scenarios in geosciences:

  1. Exploratory analyses: When exploring a new idea, a single notebook or script is often all we need.

  2. A Single Paper: The “paper” is a standard unit of scientific output. The code related to a single paper usually belongs together.

  3. Reusable software elements: In the course of our computing, we often identify specialized routines that we want to package for reuse in other projects, or by other scientists. This is where “scripts” become “software.”

This page outlines some suggested practices for each category.

Working through this notebook

The first half of this page (project organization and reproducibility) is reading material — no code to run.

The Modules section below has runnable code that imports a small gcdistance.py module. To follow along, either:

  • Download this notebook using the ⬇ icon in the top-right, and also grab gcdistance.py from the course repo. Put both files in the same folder in your environment.

  • Or copy-paste: open a fresh notebook in your environment and copy each code cell over from this page. The contents of gcdistance.py are shown below in the Modules section — paste those into a sibling text file named gcdistance.py.

Either way, run the cells in JupyterLab on LEAP or Colab.

In-class assignment — 10 points

The “Try it” exercises in this notebook are part of your in-class assignment for this section. Complete them in your own copy of the notebook, push it to your week folder, and post the notebook link on the matching Courseworks assignment. (One 10-point assignment covers all the lecture notebooks in this section.)

Exploratory Analysis#

When starting something new, you’re often motivated to just start coding and get results quickly. This is fine! Jupyter notebooks are an ideal format for open-ended exploratory analysis since they’re self-contained — text, code, and figures all in one place.

If you find something cool or useful, preserve it. The simplest option is a single notebook in a folder of your project repo (e.g., the GitHub repo from Assignment 1).

For one-off notebooks you want to share with others, GitHub’s Gist mechanism is also a lightweight option — gists are mini repos you can drag-and-drop a notebook into at https://gist.github.com/.

A Single Paper#

Scientific Reproducibility#

Reproducibility is a cornerstone of the scientific process. Today, almost all earth science relies on some form of computation — from simple statistical analysis to advanced numerical simulation. In principle, computational science should be highly reproducible. In practice, it often isn’t.

The audience for a reproducible project isn’t just other scientists — it’s you, a year from now, when you need to repeat or build on this work. Extra time spent on reproducibility now pays off later.

A useful framing from one of the earliest papers on the topic:

An article about computational science … is not the scholarship itself, it’s merely scholarship advertisement. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.

Donoho, D. et al. (2009), Reproducible research in computational harmonic analysis, Comp. Sci. Eng. 11(1):8–18, doi: 10.1109/MCSE.2009.15

A few practical principles that follow from this:

  1. Keep track of how each result was produced — which script, which inputs, which parameters.

  2. Version-control all custom scripts (Git is doing this for you).

  3. Avoid manual data-manipulation steps — write a script or notebook cell instead of editing the data by hand.

  4. Provide public access to code, raw data, and analysis steps, so others (and future you) can re-run them.

Further reading: Sandve et al. (2013) lists ten specific recommendations for computational reproducibility, and the Barba-group Reproducibility Syllabus collects best-practice references.

These principles suggest a certain structure for a project.

Project Layout#

A reproducible single-paper project directory structure might look something like this:

README.md
LICENSE
environment.yml
data/intermediate_results.csv
notebooks/process_raw_data.ipynb
notebooks/figure1.ipynb
notebooks/figure2.ipynb
notebooks/helper.py
manuscript/manuscript.tex

A great example of such a paper is Cesar Rocha’s Upper Ocean Seasonality project: crocha700/UpperOceanSeasonality.

Reuseable Software Elements#

Scientific software can perhaps be grouped into two categories: single-use “scripts” that are used in a very specific context to do a very specific thing (e.g.~to generate a specific figure for a paper), and reuseable components which encapsulate a more generic workflow. Once you find yourself repeating the same chunks of code in many different scripts or projects, it’s time to start composing reusable software elements.

Modules#

The basic element of reusability in python is the module. A module is a .py file which contains python objects which can be imported by other scripts or notebooks. Let’s illustrate how modules work with a simple example.

A common task in geoscience is to calculate the great-circle distance between two points on the globe. There are several pacakges that could do this for you, but let’s write our own as an example of a module.

The formula for great circle distance is

great circle distance formula

(Note that this formula requires 64-bit precision for adequate accuracy.)

Let’s write a module to do this calculation. Open a file called gcdistance.py in a text editor. (The file should be in the same directory as the notebook you are working in now.) Populate it with the following code:

"""
A python module for computing great circle distance
"""
import numpy as np

# approximate radius of Earth
R = 6.371e6

def great_circle_distance(point1, point2):
    """Calculate great-circle distance between two points.
    
    PARAMETERS
    ----------
    point1 : tuple
        A (lat, lon) pair of coordinates in degrees
    point2 : tuple
        A (lat, lon) pair of coordinates in degrees
        
    RETURNS
    -------
    distance : float
    """
    
    # unpack coordinates
    lat1, lon1 = point1
    lat2, lon2 = point2
    
    # unpack and convert everything to radians
    phi1, lambda1, phi2, lambda2 = [np.deg2rad(v) for v in 
                                    (point1 + point2)]
    
    # apply formula
    # https://en.wikipedia.org/wiki/Great-circle_distance
    return R*np.arccos(
        np.sin(phi1)*np.sin(phi2) + 
        np.cos(phi1)*np.cos(phi2)*np.cos(lambda2 - lambda1))

The module begins with a docstring explaining what it does. Then it contains some data (just a constant R) and a single function.

Now let’s import our module

import gcdistance
help(gcdistance) 

Help on module gcdistance:

NAME
    gcdistance - A python module for computing great circle distance

FUNCTIONS
    great_circle_distance(point1, point2)
        Calculate great-circle distance between two points.
        
        PARAMETERS
        ----------
        point1 : tuple
            A (lat, lon) pair of coordinates in degrees
        point2 : tuple
            A (lat, lon) pair of coordinates in degrees
            
        RETURNS
        -------
        distance : float

DATA
    R = 6371000.0

FILE
    /home/runner/work/summer_2026/summer_2026/lectures_DS/core_python/gcdistance.py

And let’s try using it to make a calculation

gcdistance.great_circle_distance((60, 0), (50, 15))

np.float64(1460007.189049398)

We could just import the function we need

from gcdistance import R, great_circle_distance
R

6371000.0

If we change the module, we need to either restart our kernel or else reload the module. (Note that functions imported via from module import func cannot be reloaded.)

from importlib import reload
reload(gcdistance) 

<module 'gcdistance' from '/home/runner/work/summer_2026/summer_2026/lectures_DS/core_python/gcdistance.py'>

Modules are a simple way to share code between different scripts or notebooks in the same project. Module files must reside in the same directory as any script which imports them! This is a big limitation; it means you can’t share modules between different projects.

Once you have a piece of code that is general-purpose enough to share between projects, you need to create a package.

Try it

In a fresh code cell, use gcdistance.great_circle_distance to compute the distance between three pairs of cities you’re interested in. Lat/lon coordinates are easy to find online (Google “latitude longitude of ”). The result is in meters, so divide by 1000 for kilometers. Cross-check one of your answers against the distance shown on Google Maps.

Aside: Python Style#

There are few absolute rules for python code style, but there is a detailed recommended style guide. Some especially relevant points are:

  • Line length should not exceed 79 characters

  • Module names should be lowercase

  • Function and variable names should be lower_case_with_underscores

  • Class names should be CamelCase

Beyond modules: packaging, testing, CI#

Once your code is general enough to share between projects or with collaborators, the next step is turning it into a package — a directory with a particular layout that can be installed with pip and used from anywhere. A parallel ecosystem of testing (with pytest) and continuous integration (with GitHub Actions) grows naturally alongside it.

This is beyond the scope of the weekly assignments, but see Going Further with Computing in the Appendix: Further Topics for pointers on packaging, testing, and continuous integration whenever you’re ready.

Recap#

This notebook was about structure rather than syntax:

  • Project organization — why a deliberate layout (separating data, code, and notebooks) makes research reproducible and your future self’s life easier.

  • Modules — moving reusable functions into a .py file and importing them, so the same code can serve many notebooks and projects.

  • Style and beyond — readable, consistent code (PEP 8), and where the path leads next: packaging, testing, and continuous integration.

That wraps up the Core Python section. Next we move into the scientific stack — NumPy and Matplotlib — where these language foundations get put to work on real numerical data.