DefaultReaders

Overview

The DefaultReaders class serves as the central coordinator for all data reading operations in the macro_data package. It manages the initialization, configuration, and coordination of various specialized readers for different types of economic data.

Specialized Readers

The following specialized readers are used internally by DefaultReaders:

Core Economic Data Readers

ICIOReader

Handles Inter-Country Input-Output tables, providing methods for:

• Reading and processing ICIO data
• Calculating trade flows and proportions
• Managing industry-level data
• Computing value added and consumption

View ICIOReader documentation →

EuroStatReader

Processes Eurostat data for EU countries, including:

• Economic indicators
• Industry statistics
• Social and demographic data
• Financial statistics

View EuroStatReader documentation →

WorldBankReader

Manages World Bank data for global economic indicators:

• GDP and growth rates
• Population statistics
• Trade data
• Development indicators

View WorldBankReader documentation →

OECDEconData

Handles OECD economic data:

• Economic indicators
• Industry statistics
• Policy data
• Social indicators

View OECDEconData documentation →

IMFReader

Processes IMF data:

• Economic forecasts
• Financial statistics
• Balance of payments
• Exchange rates

View IMFReader documentation →

Financial and Population Data Readers

ExchangeRatesReader

Handles currency exchange rate data:

• Bilateral exchange rates
• Currency conversions
• Rate time series

View ExchangeRatesReader documentation →

PolicyRatesReader

Processes central bank policy rates:

• Interest rates
• Monetary policy data
• Rate time series

View PolicyRatesReader documentation →

HFCSReader

Manages Household Finance and Consumption Survey data:

• Household wealth
• Income distribution
• Financial assets
• Liabilities

View HFCSReader documentation →

CompustatFirmsReader

Processes firm-level financial data:

• Balance sheets
• Income statements
• Financial ratios
• Industry classification

View CompustatFirmsReader documentation →

CompustatBanksReader

Handles bank-level financial data:

• Bank balance sheets
• Financial statements
• Banking indicators
• Risk metrics

View CompustatBanksReader documentation →

Environmental Data Readers

EmissionsReader

Manages environmental data:

• Emission factors
• Energy consumption
• Environmental indicators
• Climate data

View EmissionsReader documentation →

Usage Example

from macro_data.readers.default_readers import DataReaders
from pathlib import Path

# Initialize the central data reader
readers = DataReaders.from_raw_data(
    raw_data_path=Path("path/to/raw/data"),
    country_names=["FRA", "DEU", "USA"],
    simulation_year=2018,
    scale_dict={"FRA": 1.0, "DEU": 1.0, "USA": 1.0},
    industries=["C10T12", "C13T15"],
    aggregate_industries=True
)

# Access data through the unified interface
france_gdp = readers.world_bank.get_current_scaled_gdp("FRA", 2018)
german_trade = readers.icio[2018].get_trade("DEU", "FRA")
us_emissions = readers.emissions.get_emissions_data("USA")

Best Practices

1. Data Source Management

   • Keep raw data in a consistent location
   • Document data source versions
   • Maintain data update schedules

2. Reader Configuration

   • Use appropriate aggregation levels
   • Configure proxy relationships
   • Set up proper error handling

3. Performance Optimization

   • Cache processed data
   • Use efficient data structures
   • Implement parallel processing when possible

4. Data Quality

   • Validate input data
   • Handle missing values appropriately
   • Document data transformations

This module provides centralized data reader management for the macro_data package. It handles the initialization and coordination of various data readers for different types of economic data, including ICIO tables, Eurostat data, World Bank indicators, and more.

The module centers around two main classes: 1. DataPaths: Manages file paths for all data sources 2. DataReaders: Coordinates multiple data readers and provides unified access

Key features: - Centralized data source management - Automatic reader initialization - Data validation and preprocessing - Exchange rate handling - Industry aggregation - Country-specific data processing

DataReaders dataclass

Centralized management of all data readers for the model.

This class coordinates multiple data readers and provides unified access to various types of economic data. It handles initialization, data validation, and preprocessing for all data sources.

Attributes:

Name	Type	Description
icio	dict[int, ICIOReader]	ICIO table readers by year
wiod_sea	WIODSEAReader	WIOD SEA data reader
oecd_econ	OECDEconData	OECD economic data reader
world_bank	WorldBankReader	World Bank data reader
hfcs	dict[str, HFCSReader]	Household Finance and Consumption Survey readers
eurostat	EuroStatReader	Eurostat data reader
ons	ONSReader	ONS data reader
policy_rates	PolicyRatesReader	Policy rates reader
imf_reader	IMFReader	IMF data reader
exchange_rates	ExchangeRatesReader	Exchange rates reader
goods_criticality	GoodsCriticalityReader	Goods criticality reader
ecb_reader	ECBReader	ECB data reader
compustat_firms	CompustatFirmsReader	Compustat firms data reader
compustat_banks	CompustatBanksReader	Compustat banks data reader
emissions	EmissionsReader	Emissions data reader
regions_dict	Optional[dict[Country, list[Region]]]	Regional disaggregation mapping

icio: dict[int, ICIOReader] instance-attribute

wiod_sea: WIODSEAReader instance-attribute

oecd_econ: OECDEconData instance-attribute

world_bank: WorldBankReader instance-attribute

hfcs: dict[str, HFCSReader] instance-attribute

eurostat: EuroStatReader instance-attribute

ons: ONSReader instance-attribute

policy_rates: PolicyRatesReader instance-attribute

imf_reader: IMFReader instance-attribute

exchange_rates: ExchangeRatesReader instance-attribute

goods_criticality: GoodsCriticalityReader instance-attribute

ecb_reader: ECBReader instance-attribute

compustat_firms: CompustatFirmsReader instance-attribute

compustat_banks: CompustatBanksReader instance-attribute

emissions: EmissionsReader instance-attribute

regions_dict: Optional[dict[Country, list[Region]]] = None class-attribute instance-attribute

__init__(icio: dict[int, ICIOReader], wiod_sea: WIODSEAReader, oecd_econ: OECDEconData, world_bank: WorldBankReader, hfcs: dict[str, HFCSReader], eurostat: EuroStatReader, ons: ONSReader, policy_rates: PolicyRatesReader, imf_reader: IMFReader, exchange_rates: ExchangeRatesReader, goods_criticality: GoodsCriticalityReader, ecb_reader: ECBReader, compustat_firms: CompustatFirmsReader, compustat_banks: CompustatBanksReader, emissions: EmissionsReader, regions_dict: Optional[dict[Country, list[Region]]] = None) -> None

from_raw_data(raw_data_path: Path | str, country_names: list[Country | Region], simulation_year: int, scale_dict: dict[Country, int], industries: list[str], aggregate_industries: bool = True, imputed_rent_year: int = 2014, exog_data_range: Tuple[int, int] = (2010, 2018), prune_date: Optional[date] = None, force_single_hfcs_survey: bool = False, single_icio_survey: bool = False, proxy_country_dict: dict[Country, Country] = None, use_disagg_can_2014_reader: bool = False, use_provincial_can_reader: bool = False, regions_dict: dict[Country, list[Region]] = None) classmethod

get_investment_fractions(country_names: list[Country], eurostat: EuroStatReader, proxy_country_dict: dict[Country, Country], year: int) -> dict[Country, dict[str, float]] classmethod

Calculate investment fractions for each country.

This method computes the distribution of investment across different sectors (firms, households, government) for each country, using either direct data for EU countries or proxy data for non-EU countries.

Parameters:

Name	Type	Description	Default
country_names	list[Country]	List of countries to process	required
eurostat	EuroStatReader	Eurostat data reader instance	required
proxy_country_dict	dict[Country, Country]	Mapping of countries to their proxy countries	required
year	int	Reference year for the data	required

Returns:

Type	Description
dict[Country, dict[str, float]]	dict[Country, dict[str, float]]: Investment fractions by country and sector

get_exogenous_data(country_name: Country) -> Optional[dict[str, Any]]

Retrieve exogenous economic data for a country.

This method collects various exogenous economic indicators for a country, including inflation, sectoral growth, unemployment rates, and other key economic metrics.

Parameters:

Name	Type	Description	Default
country_name	Country	Country to get data for	required

Returns:

Type	Description
Optional[dict[str, Any]]	Optional[dict[str, Any]]: Dictionary of economic indicators, or None if data is unavailable

get_benefits_inflation_data(country_name: Country, year_min: int, year_max: int, exogenous_data: dict[str, Any]) -> pd.DataFrame

Calculate benefits and inflation data for a country.

This method processes unemployment benefits, other benefits, and inflation data for a country over a specified time period. It interpolates quarterly values and merges with inflation and unemployment rate data.

Parameters:

Name	Type	Description	Default
country_name	Country	Country to analyze	required
year_min	int	Start year for the analysis	required
year_max	int	End year for the analysis	required
exogenous_data	dict[str, Any]	Dictionary containing exogenous economic data	required

Returns:

Type	Description
DataFrame	pd.DataFrame: DataFrame containing benefits and inflation data with quarterly frequency

get_total_benefits_lcu(country_name: Country, year: int) -> float

Calculate total benefits in local currency units.

This method computes the total benefits (including unemployment and other benefits) for a country in its local currency units.

Parameters:

Name	Type	Description	Default
country_name	Country	Country to analyze	required
year	int	Reference year	required

Returns:

Name	Type	Description
float	float	Total benefits in local currency units

get_total_unemployment_benefits_lcu(country_name: Country, year: int) -> float

Calculate total unemployment benefits in local currency units.

This method computes the total unemployment benefits for a country in its local currency units.

Parameters:

Name	Type	Description	Default
country_name	Country	Country to analyze	required
year	int	Reference year	required

Returns:

Name	Type	Description
float	float	Total unemployment benefits in local currency units

get_govt_debt_lcu(country: Country, year: int) -> float

Calculate government debt in local currency units.

This method computes the total government debt for a country in its local currency units.

Parameters:

Name	Type	Description	Default
country	Country	Country to analyze	required
year	int	Reference year	required

Returns:

Name	Type	Description
float	float	Government debt in local currency units

get_export_taxes(country: Country, year: int) -> float

Calculate export taxes for a country.

This method computes the total export taxes for a country based on its exports and exchange rates.

Parameters:

Name	Type	Description	Default
country	Country	Country to analyze	required
year	int	Reference year	required

Returns:

Name	Type	Description
float	float	Total export taxes

get_national_accounts_growth(country: Country) -> pd.DataFrame

Calculate national accounts growth rates.

This method combines growth rate data from both IMF and OECD sources, with IMF data taking precedence for overlapping indicators.

Parameters:

Name	Type	Description	Default
country	Country	Country to analyze	required

Returns:

Type	Description
DataFrame	pd.DataFrame: Combined growth rate data from IMF and OECD sources

expand_weights_by_income(year: int, country: str | Country)

Expand consumption weights by income quantile.

This method calculates consumption weights for each industry and income quantile, using OECD data for income distribution and ICIO data for consumption shares.

Parameters:

Name	Type	Description	Default
year	int	Reference year	required
country	str | Country	Country to analyze	required

Returns:

Type	Description
	pd.DataFrame: Consumption weights by industry and income quantile

DataPaths dataclass

Manages file paths for all data sources used in the model.

This class provides a centralized way to manage and access file paths for various data sources, including ICIO tables, Eurostat data, World Bank indicators, and more.

Attributes:

Name	Type	Description
goods_criticality_path	Path	Path to goods criticality data
exchange_rates_path	Path	Path to exchange rates data
eurostat_path	Path	Path to Eurostat data directory
hfcs_path	Path	Path to Household Finance and Consumption Survey data
icio_paths	dict[int, Path]	Paths to ICIO tables by year
icio_pivot_paths	dict[int, Path]	Paths to pivoted ICIO data by year
wiod_sea_path	Path	Path to WIOD SEA data
oecd_econ_path	Path	Path to OECD economic data
oecd_econ_mapping_path	Path	Path to OECD economic data mappings
policy_rates_path	Path	Path to policy rates data
country_codes_path	Path	Path to country codes mapping
imf_path	Path	Path to IMF data
ons_path	Path	Path to ONS data
world_bank_path	Path	Path to World Bank data
ecb_path	Path	Path to ECB data
compustat_firms_annual_path	Path	Path to annual Compustat firms data
compustat_firms_quarterly_path	Path	Path to quarterly Compustat firms data
compustat_banks_path	Path	Path to Compustat banks data
emissions_path	Path	Path to emissions data

goods_criticality_path: Path instance-attribute

exchange_rates_path: Path instance-attribute

eurostat_path: Path instance-attribute

hfcs_path: Path instance-attribute

icio_paths: dict[int, Path] instance-attribute

icio_pivot_paths: dict[int, Path] instance-attribute

wiod_sea_path: Path instance-attribute

oecd_econ_path: Path instance-attribute

oecd_econ_mapping_path: Path instance-attribute

policy_rates_path: Path instance-attribute

country_codes_path: Path instance-attribute

imf_path: Path instance-attribute

ons_path: Path instance-attribute

world_bank_path: Path instance-attribute

ecb_path: Path instance-attribute

compustat_firms_annual_path: Path instance-attribute

compustat_firms_quarterly_path: Path instance-attribute

compustat_banks_path: Path instance-attribute

emissions_path: Path instance-attribute

__init__(goods_criticality_path: Path, exchange_rates_path: Path, eurostat_path: Path, hfcs_path: Path, icio_paths: dict[int, Path], icio_pivot_paths: dict[int, Path], wiod_sea_path: Path, oecd_econ_path: Path, oecd_econ_mapping_path: Path, policy_rates_path: Path, country_codes_path: Path, imf_path: Path, ons_path: Path, world_bank_path: Path, ecb_path: Path, compustat_firms_annual_path: Path, compustat_firms_quarterly_path: Path, compustat_banks_path: Path, emissions_path: Path) -> None

default_paths(raw_data_path: Path, icio_years: Iterable[int]) classmethod

Create default paths for all data sources.

Parameters:

Name	Type	Description	Default
raw_data_path	Path	Base path for raw data	required
icio_years	Iterable[int]	Years to include for ICIO data	required

Returns:

Name	Type	Description
DataPaths		Configured paths for all data sources

get_investment_fractions(country_names: list[Country | Region], eurostat: EuroStatReader, proxy_country_dict: dict[Country, Country], year: int) -> dict[Country, dict[str, float]]