Datasets

21st Century Weather host a series of datasets on su28. You can find them in the following table along with the available documentation.

If you need to store a dataset please contact the Research Software Engineer team through Cumulus.

Show code cell source

Hide code cell source

import itables
import yaml
import pandas as pd
import os
from pathlib import Path
itables.init_notebook_mode()

files = list(Path("./").glob("*.md"))

# List to hold data from each file
data_list = []

for file_path in files:
    with open(file_path, 'r') as f:
        content = f.read()
    
    if content.startswith("---"):
        parts = content.split('---')
        if len(parts) > 2:
            yaml_header = parts[1]
            try:
                header_data = yaml.safe_load(yaml_header)
                header_data["filename"] = file_path.name  
                data_list.append(header_data)
            except yaml.YAMLError as e:
                print(f"YAML parse error in {file_path.name}: {e}")


df = pd.DataFrame(data_list)
df['filename'] = df['filename'].str.replace('.md', '.html', regex=False)
df["title"] =  "<a href='" + df["filename"] + "'>" + df["title"] + "</a>"
df  = df.drop("filename", axis=1)
df.columns = ['Name', 'Location on Gadi', 'Mantainer', 'Description']

itables.show(df, html=False, allow_html = True, columnDefs=[{"className": "dt-left", "targets": "_all"}])

/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/itables/typing.py:308: SyntaxWarning: These arguments are not documented in ITableOptions: {'html'}. You can silence this warning by setting `itables.options.warn_on_undocumented_option=False`. If you believe ITableOptions should be updated, please make a PR or open an issue at https://github.com/mwouts/itables
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/itables/typing.py:308: SyntaxWarning: These arguments are not documented in DTForITablesOptions: {'html'}. You can silence this warning by setting `itables.options.warn_on_undocumented_option=False`. If you believe ITableOptions should be updated, please make a PR or open an issue at https://github.com/mwouts/itables
  warnings.warn(

ⓘName	Location on Gadi	Mantainer	Description
MMLEAv2	/g/data/su28/MMLEAv2	Sam Green, Nicola Maher	Multi-Model Large Ensemble Archive version 2 includes 18 models from the CMIP project in a 2.5 degree grid.
ERA5-based Weather Objects	/g/data/su28/weatherfeatures.era5/	Sam Green, Chenhui Jin	Weather objects (anticyclones, cyclones and warm-conveyor belts)
ERA5 daily	/g/data/su28/ERA5/daily	TBD	Reanalysis. Variables in daily frequency remaped to 1 degree
Energy Demand	/g/data/su28/energy_demand	Sam Green, Doug Richardson	Predictions of daily energy demand for the Australian National Electricity Market (1940-2023)
ERA5 Drought	/g/data/su28/drought_era5	Pao Corrales	Monthly drought indices from 1940 to present derived from ERA5 reanalysis
GSMaP	/g/data/su28/GSMaP/	Sam Green	Daily averaged precipitation derived from satellite.
MESACLIP	/g/data/su28/MESACLIP	Sam Green	MESoscale Atmosphere-Ocean Interactions in Seasonal-to-Decadal CLImate Prediction
Cloud Type from Himawari	/g/data/su28/himawari-ahi	Sam Green	Cloud type product is derived from Himawari AHI