Methodology and Source Notes

Integrated Research Design

The project uses a mixed-method design combining qualitative policy research, comparative case studies, legal and institutional analysis, and quantitative energy-security modelling. This design reflects the fact that EU energy security is simultaneously a measurable condition and a political process. Statistical indicators can identify changes in import exposure, energy intensity, domestic production, and renewable capacity, but they cannot by themselves establish whether greater coordination between energy policy and the Common Foreign and Security Policy (CFSP) is legally feasible, politically acceptable, or institutionally effective.

The two components therefore perform different but connected functions. The qualitative component identifies the mechanisms through which energy dependence may affect EU external action, including asymmetric interdependence, supplier concentration, divergent national preferences, institutional fragmentation, and the strategic use of energy relationships. The quantitative component operationalises selected aspects of this vulnerability, evaluates their historical development, and constructs conditional forecasts under a baseline, an IMEC scenario, and a Piano Mattei scenario. The combined design supports an assessment of whether the proposed policy direction is consistent with both the institutional evidence and observable energy-security trends. It does not treat a statistical association or a scenario difference as proof that institutional integration causes a particular energy outcome.

Qualitative policy, legal, and case-study component

The qualitative evidence base prioritises primary documents: EU treaties and legislation, European Council and Council conclusions, European Commission communications and strategies, official implementation reports, international agreements, and speeches or statements by relevant institutional actors. National-government and international-organisation documents provide supplementary evidence, while the academic literature is used to derive theoretical expectations, identify contested interpretations, and test rival explanations.

Documents are selected purposively when they meet at least one of three criteria: direct relevance to the relationship between energy and external security; formal institutional or legal authority; or evidence about actor preferences, implementation, competence allocation, or policy effects. The legal analysis distinguishes measures that can be pursued through existing EU energy competences, CFSP coordination, secondary legislation, external agreements, or voluntary intergovernmental action. This prevents political desirability from being treated as equivalent to legal feasibility.

The documentary analysis uses a common thematic framework covering energy dependence, diversification, solidarity, strategic autonomy, external influence, institutional competence, decision-making procedures, and the distribution of costs and benefits among Member States. It also records whether energy is framed primarily as an economic commodity, an environmental concern, a security interest, or an instrument of external influence. Changes in framing are traced across major disruptions, including interruptions of Russian energy supplies, the annexation of Crimea, Russia's full-scale invasion of Ukraine, and the subsequent diversification of EU supplies.

Four cases are examined through a structured, focused comparison:

1. EU-Russia energy relations as a case of asymmetric interdependence, supplier concentration, and potential coercive vulnerability.
2. The Energy Union as an internal attempt to pool capabilities, strengthen solidarity, and coordinate national energy interests.
3. The India-Middle East-Europe Economic Corridor (IMEC) as an infrastructure, connectivity, and diversification strategy.
4. Piano Mattei as an Italian-led Mediterranean and African partnership with possible EU-level energy and foreign-policy spillovers.

The cases are selected for analytical variation rather than statistical representativeness. Each is assessed using the same questions: what vulnerability is identified; which actors possess decision-making authority; through which mechanism could the initiative affect energy security; how are its expected benefits and costs distributed; and which political, legal, implementation, or ethical constraints limit it? Claims are triangulated across primary documents, statistical evidence, and independent scholarship. Market conditions, technological change, climate policy, and national industrial interests are considered as rival explanations before an observed development is attributed to foreign-policy coordination.

Quantitative design and unit of analysis

The principal quantitative dataset is an EU27 country-year panel covering 2015-2024, followed by conditional forecast paths for 2025-2030. Eurostat observations are cleaned and merged by country and year, restricted to the current EU27 Member States, and aggregated from the underlying national contributions. EU-level ESS values are calculated from summed numerators and denominators rather than from an unweighted average of national ratios, so Member States with very different economic and energy-system sizes do not receive identical weights.

The quantitative workflow proceeds in five stages:

1. retrieve and harmonise official Eurostat energy, GDP, industry, digital, PRODCOM, and partner-trade inputs;
2. construct the historical ESS indicators and related comparison measures;
3. validate formulas, units, coverage, and source provenance;
4. select and estimate empirical baseline forecast models using rolling-origin backtests; and
5. apply documented scenario and shock assumptions only after the baseline has been produced.

Eurostat remains the exact source for the production ESS targets. Comext supplies product-partner trade evidence. World Bank, OECD, Wikipedia, GDELT, and other external sources are used only as contextual evidence or experimental explanatory features, as described below; they do not replace the Eurostat variables used in the core formulas.

Core ESS Formulas Used In This Project

This repository applies an EU-focused adaptation of the ESS framework using the following indicators:

• ESS1_oil = sum_i(TOI_i * TOC_i * TEC_i) / sum_i(TOC_i * TEC_i * GDP_i)
• ESS1_gas = sum_i(TGI_i * TGC_i * TEC_i) / sum_i(TGC_i * TEC_i * GDP_i)
• ESS1_ensemble = (ESS1_oil + ESS1_gas) / 2
• ESS2 = sum_i(IEC_i) / sum_i(GDP_i)
• ESS3 = sum_i(DEP_i) / sum_i(TEC_i)
• Displayed paper composite: ESS4 = (ESS1 + ESS2 + ESS3) / 3

Where each i is a country-year contribution before EU-level aggregation by year.

Following Erdal et al. (2015, pp. 136-137), ESS1 is an energy import-dependency indicator tied to the accessibility dimension of supply security; ESS2 is energy intensity, defined as energy use relative to GDP and linked to both accessibility and affordability; and ESS3 is the domestic production rate of energy, an accessibility indicator for domestic supply coverage. The paper's ESS4 is a multidimensional composite calculated as the arithmetic mean of ESS1, ESS2, and ESS3.

The EU dashboard adapts this framework by splitting ESS1 into oil and gas import-dependency channels, then averaging those channels as ESS1_ensemble for a project-level fossil-import exposure signal. The main policy dashboard now displays ESS4 by averaging the EU-adapted ESS1_ensemble, ESS2, and ESS3. Because lower ESS1 and ESS2 values are favorable in this implementation while higher ESS3 values are favorable, interpret ESS4 as a composite trajectory to read alongside its three components rather than as a standalone ranking.

Measurement Units And Harmonization

For the energy-side ESS inputs, the project uses Eurostat units as follows:

• TOI: refreshed from Eurostat international oil trade in THS_T (thousand tonnes)
• TGI: refreshed from Eurostat and converted from TJ_GCV to ktoe
• TOC, TGC, TEC, T_REN_C, T_REN_PROD, DEP, IEC: refreshed from Eurostat in KTOE

GDP remains in Eurostat CP_MEUR monetary units and is not converted to an energy unit. The adapted ESS1 oil/gas channels and ESS2 values are therefore index-style implementations of the paper logic, not direct physical percentages. For publication, interpret magnitudes as internally consistent index levels and trends rather than as unit-free dependency shares.

Academic Reference For ESS Logic

The ESS family is adapted from:

Erdal, L., Pektas, A. O., Ozkan, O., & Ozkan, F. (2015). Security of energy supply in Japan: a key strategy and solutions. International Journal of Global Warming, 7(1), 128-159. https://doi.org/10.1504/IJGW.2015.067416

Reference pages used for metadata verification:

• Inderscience page: https://www.inderscience.com/info/inarticle.php?artid=67416
• Secondary abstract page: https://www.energy-xprt.com/articles/security-of-energy-supply-in-japan-a-key-strategy-and-solutions-471484

Adaptation Scope

The paper is Japan-focused, while this repository is EU-focused. Use this wording in research outputs:

an EU-focused adaptation of the ESS framework proposed by Erdal et al. (2015)

The primary source now stored locally in the project is:

• Security_of_energy_supply_in_Japan_a_key.pdf

Pages 136-137 of the local PDF provide the base logic for the import-dependency/accessibility, energy-intensity/accessibility-affordability, domestic-production/accessibility, and composite ESS indicators that are adapted in the repository for EU-focused use.

Reality And Validation Layer

To keep interpretation transparent, the pipeline now exports:

• DATA/generated/validation_checks.csv: formula consistency and forecast sanity checks.
• DATA/generated/publication_readiness_checks.csv: generated publication-readiness checklist with blocking failures and non-blocking limitations.
• DATA/generated/historical_metrics.csv: historical metric table used for trend projection.
• DATA/generated/reality_benchmarks.csv: side-by-side ESS channels vs conventional dependency ratios.
• DATA/generated/multisource_ess_target_history.csv, DATA/generated/multisource_ess_features.csv, DATA/generated/multisource_ess_model_features.csv, DATA/generated/multisource_ess_source_ablation_backtest.csv, DATA/generated/multisource_ess_source_ablation_summary.csv, DATA/generated/multisource_ess_model_selection.csv, DATA/generated/multisource_ess_robustness.csv, DATA/generated/multisource_ess_forecast_intervals.csv, DATA/generated/multisource_ess_augmented_backtest.csv, DATA/generated/multisource_ess_augmented_forecasts.csv, DATA/generated/multisource_ess_augmented_summary.csv, and DATA/generated/multisource_ess_api_influence.csv: exact production target history, source feature matrix, lagged/delta model features, source-ablation tests, final full-history regularized model selection, exploratory robustness rules, empirical forecast intervals, nested baseline-vs-augmented forecast evidence, and source-influence summary behind the multi-API ESS trend experiment.
• DATA/generated/multisource_ess_evidence.csv and DATA/generated/multisource_ess_source_coverage.csv: source-role audit tables behind the same dashboard.

These files are generated by python PIPELINE/run_all.py and displayed in Streamlit under Refresh Status.

The multi-API ESS trend dashboard preserves the publication distinction between exact formula targets and experimental features. Eurostat remains the exact source for ESS1, ESS2, ESS3, and renewable-production target series. The multi-API experiment uses the exact production-history target series in DATA/generated/multisource_ess_target_history.csv; the longer direct-EU Eurostat API series is retained as an aggregate-proxy forecast-quality context source, not as a replacement for the country-sum ESS1 formula. Comext, World Bank/DataBank, OECD, Wikipedia, and GDELT are converted into annual feature variables and then transformed into lagged and one-year-change model inputs. External API features are carried forward only after they first appear; missing early years are not filled backward or interpolated from later observations. World Bank WDI proxy zeros that appear after a positive historical run are treated as unavailable observations when they are known publication artifacts rather than plausible EU energy values. Wikipedia and GDELT event features include counts, evidence-weighted totals, direction counts, and magnitude-weighted channels so historical facts can affect the augmented model when they improve the rolling backtest. Phase 1 tests each source family separately, all sources together, and all-sources-minus-one-source cases. The reported augmented validation uses nested rolling-origin selection: for each cutoff, the model family is chosen only from earlier completed backtests. The final forecast line may use the best all-APIs regularized model selected from the full available history, and it is labelled separately from the nested validation evidence. Robustness is reported as publication-safe only when nested improvement holds across multiple horizons and error criteria; source-ablation robustness remains an exploratory candidate-grid diagnostic.

Forecast Quality Backtest Data Provenance

The Forecast Quality Backtest deliberately uses a different historical data construction from the main production ESS dataset.

The main ESS dataset is built from the project input workbooks refreshed through Eurostat (GDP.xlsx, TEC.xlsx, IEC.xlsx, TOC.xlsx, TGC.xlsx, T_REN_C.xlsx, T_REN_PROD.xlsx, TOI.xlsx, TGI.xlsx, and DEP.xlsx). These files are merged at country-year level, filtered to EU27 member states, and then used to build the current historical metrics and scenario forecasts.

The Forecast Quality Backtest instead uses DATA/generated/extended_eurostat_history.csv, an extended EU27 aggregate-proxy history fetched directly from Eurostat API series. This separate history is used because the backtest needs a longer sequence of past observations in order to simulate rolling-origin forecasts and compare each forecast with what actually happened afterward.

The extended backtest source uses direct EU27 aggregate Eurostat series, including:

• NAMA_10_GDP for GDP.
• NRG_BAL_C for total final energy consumption, industrial final energy consumption, oil consumption, gas consumption, and renewable consumption.
• NRG_BAL_PEH for renewable gross electricity production.
• NRG_TI_OIL for oil import volume.
• NRG_TI_GAS for gas import volume, converted from TJ_GCV to ktoe.
• NRG_BAL_S for domestic primary production.

This means the backtest is a validation layer, not a replacement for the main ESS construction. It tests whether the project trend logic would have performed well against a longer Eurostat historical record, while ESS1 production-history targets and ESS1 scenario forecasts continue to use the country-sum project formula.

Forecasting Methodology

Forecast objective and analytical distinction

The forecasting component estimates prospective EU energy-security paths from 2025 to 2030. It makes an explicit distinction between an empirical baseline and conditional policy scenarios. The baseline extends the historical series using the candidate model that performs best in historical testing. IMEC, Piano Mattei, and geopolitical or policy shock adjustments are applied only after that baseline has been constructed. The output is therefore a set of conditional projections, not a causal estimate of either initiative and not a claim about what will necessarily occur.

The directly forecasted variables include the ESS1 oil and gas exposure channels, ESS2 industrial energy intensity, ESS3 domestic production coverage, oil and gas import volumes, total energy consumption, renewable consumption and production, and the digital-intensity series used in the corridor scenarios. Derived indicators are recalculated from their projected components instead of being forecast independently:

• ESS1_ensemble_t = (ESS1_oil_t + ESS1_gas_t) / 2
• ESS4_raw_t = (ESS1_ensemble_t + ESS2_t + ESS3_t) / 3
• renewables_share_t = 100 * T_REN_C_t / TEC_t
• T_REN_RATE_t = T_REN_PROD_t / TEC_t
• T_REN_RATE_2_t = T_REN_PROD_t / T_REN_C_t

The displayed directional ESS4 security index is also recomputed from the projected ESS1, ESS2, and ESS3 values using the historical reference values. Lower ESS1 and ESS2 values raise the security index, while higher ESS3 values raise it.

Candidate baseline models

The engine evaluates five deliberately transparent candidates for each baseline metric instead of imposing one trend specification on every series:

1. Long linear trend: Y_hat(T+h) = alpha + beta * (T+h), estimated using the full available history.
2. Recent linear trend: the same linear specification estimated from the most recent observations, allowing greater responsiveness to structural change.
3. Average annual-change drift: Y_hat(T+h) = Y_T + h * mean(delta_Y).
4. Recent weighted annual-change drift: Y_hat(T+h) = Y_T + h * weighted_mean(delta_Y), with larger weights on recent annual changes.
5. Last observed value: Y_hat(T+h) = Y_T.

These parsimonious specifications are appropriate to the limited length of the exact country-sum history and remain interpretable during sensitivity and policy review. They reduce the risk that a highly parameterised model produces an apparently precise but unstable long-range projection.

Rolling-origin backtesting and model selection

Candidate models are evaluated through rolling-origin backtesting. At each historical cutoff, a model is estimated using only information that would have been available at that date and forecasts the following one to five years. Each simulated forecast is compared with the observation that subsequently occurred. This reproduces the information constraint of a genuine forecast and prevents future observations from leaking into model estimation.

The primary selection criterion is mean absolute percentage error:

MAPE = (100 / N) * sum_t(abs((Y_t - Y_hat_t) / Y_t))

The audit tables also report mean absolute error, root mean squared error, percentage bias, and directional accuracy. Directional accuracy records whether the predicted movement has the same sign as the observed movement. These supplementary measures matter because a model can produce a low average level error while failing to identify changes in direction.

The candidate with the lowest rolling-backtest MAPE is selected separately for each metric. Forecast-quality reporting uses nested selection at each cutoff: the model family used for a simulated forecast is chosen only from backtests completed before that cutoff. After final selection, the winning specification is re-estimated on all observations available for the relevant target and used for the 2025-2030 forward baseline. Thus, the final forecast may use the complete history, while the reported validation evidence remains genuinely out of sample.

Most metrics use the longer Eurostat EU27 aggregate-proxy history, generally beginning in 1995, when it is available. This history provides enough completed forecast windows to evaluate multiple horizons. It is a validation layer rather than a replacement for the production ESS dataset. ESS1 oil and gas exposure continue from the exact 2015-2024 country-sum formula history so the historical and forecast values refer to the same target displayed in the dashboard.

The core model-selection evidence is exported to:

• DATA/generated/forecast_model_backtest.csv: all candidate-model rolling forecasts and errors.
• DATA/generated/forecast_model_selection.csv: the selected baseline model and performance measures for each metric.
• DATA/generated/selected_baseline_forecasts.csv: the selected baseline paths before scenario and shock adjustments.
• DATA/generated/forecast_quality_backtest.csv and DATA/generated/forecast_quality_summary.csv: rolling-origin forecast evidence by metric and horizon.

Scenario construction after the baseline

For metric m, scenario s, and forecast year t, the central scenario path follows the operational logic:

Forecast_m,s,t = Baseline_m,t * (1 + ScenarioAdjustment_m,s + ShockAdjustment_m,s,t)

Scenario adjustments are calibrated from the historical distribution of year-over-year changes. The historical mean change is the reference rate. For metrics where lower values are favourable, such as import exposure and industrial energy intensity, an optimistic scenario normally draws on the lower part of the historical change distribution. For metrics where higher values are favourable, such as domestic and renewable production, it draws on the upper part. The exact selector, historical distribution, corridor component, and implied level adjustment are recorded in DATA/generated/scenario_calibration.csv.

The baseline represents a continuation of the selected historical dynamic without an additional IMEC or Piano Mattei acceleration. The IMEC scenario emphasises diversification, logistics efficiency, infrastructure connectivity, renewable capacity, and digital spillovers. The Piano Mattei scenario gives comparatively greater weight to gas-security and Mediterranean-African partnership channels, with more moderate changes elsewhere. Literature on corridors and foresight supports the proposed transmission mechanisms, but the numerical magnitudes remain author-calibrated assumptions because neither initiative supplies an observed historical treatment effect for the EU-level ESS targets.

Geopolitical and policy shocks

The separate shock registry defines the affected variable, direction, active window, persistence, decay function, evidence level, and scenario-specific magnitude of each event. Step, linear, and exponential decay functions allow an effect to end immediately, decline at a constant rate, or fade more rapidly after the active period. Official policy sources, historical evidence, Wikipedia, and screened GDELT results establish context or help detect potentially relevant events; they do not independently estimate the numerical effect.

Shock magnitudes are therefore scenario assumptions. Evidence levels affect uncertainty rather than converting contextual evidence into a causal coefficient. Manual entries whose source note contains placeholder are excluded from publication forecasts by default. Screened news or historical facts that fail the relevance rules remain diagnostic evidence and do not move the forecast.

Forecast uncertainty

Every central forecast is accompanied by a lower and upper bound. The uncertainty ratio combines:

1. empirical error from the rolling backtest at the closest available forecast horizon;
2. additional uncertainty proportional to the size of the scenario adjustment; and
3. uncertainty associated with the magnitude and evidence level of active shocks.

The operational bounds are:

• lower_m,s,t = max(forecast_m,s,t * (1 - uncertainty_m,s,t), 0)
• upper_m,s,t = forecast_m,s,t * (1 + uncertainty_m,s,t)

The non-negativity constraint prevents mechanically negative forecasts for physical quantities and ratios. Bounds for ESS1, ESS4, and renewable ratios are then derived consistently from the component bounds. These ranges are transparent scenario-uncertainty bands; they are not conventional probability-based confidence intervals and do not represent every possible geopolitical outcome.

Forecast validation, sensitivity, and interpretation

Automated checks confirm that the historical ESS formulas reproduce the stored series, the directional ESS4 index is correctly based at 2015 = 100, all central forecasts are non-negative, and every forecast satisfies lower <= central <= upper. Sensitivity analysis varies scenario parameters, while decomposition analysis identifies the components responsible for the difference between the baseline and each policy scenario.

The same rolling-backtest selection principle is applied to the related GDP, industrial-production, PRODCOM sold-production, and macroeconomic forecast views. Their selection and forecast files include:

• DATA/generated/scenario_gdp_model_backtest.csv and DATA/generated/scenario_gdp_model_selection.csv.
• DATA/generated/industrial_production_model_backtest.csv and DATA/generated/industrial_production_model_selection.csv.
• DATA/generated/eurostat_sold_production_model_backtest.csv, DATA/generated/eurostat_sold_production_model_selection.csv, and DATA/generated/eurostat_sold_production_forecasts.csv.
• DATA/generated/eurostat_macro_interaction_model_backtest.csv, DATA/generated/eurostat_macro_interaction_model_selection.csv, and DATA/generated/eurostat_macro_interaction_forecasts.csv.

This design keeps forecast improvements empirical and auditable: an alternative model or external feature can improve the published result only when it reduces error in a historical rolling test. The scenarios should be interpreted through their direction, robustness, component decomposition, and distance from the baseline, rather than as precise estimates of the causal impact of IMEC or Piano Mattei.

Energy Partner Source Hierarchy

The repository now separates energy-partner analysis into three provenance layers:

• Raw Comext: preferred scholarly replication layer when the local extract is populated
• Official API: reproducible annual quantity route for coherent project-wide partner analysis
• Workbook: published fallback and comparison layer, especially for workbook-specific 2025 figures

The operational outputs behind these layers are:

• DATA/generated/eu_energy_partner_official_api_annual.csv
• DATA/generated/eu_energy_partner_shares.csv
• DATA/generated/eu_energy_partner_shares_from_comext.csv when populated

For academically scrutinized work, the recommended sequence is:

1. use the raw Comext layer for product-partner evidence when publication-grade replication is required
2. use the official API annual layer for reproducible quantity-focused partner charts
3. keep workbook figures for published 2025 comparison and workbook-specific communication figures

Notes On Scenario Assumptions

The policy scenarios now use a documented calibration that combines historical evidence with corridor-informed scenario scaling instead of fixed ex ante percentage shifts.

• The baseline forecast is selected per metric from the rolling-backtest candidate set described above.
• Scenario differentials are calibrated from the historical year-over-year distribution of each forecasted metric.
• The reference rate is the historical mean year-over-year change for that metric.
• IMEC uses optimistic historical selectors, typically the 25th percentile for metrics where lower values are desirable (ESS1 channels, ESS2, import volumes, TEC) and the 75th percentile where higher values are desirable (ESS3, renewable and digital metrics).
• Piano Mattei uses a mixed profile: stronger historical improvement on gas-security and resilience channels, with more moderate selectors on the other metrics.
• A corridor layer informed by A recursive networking economic analysis of international economic corridors: IMEEC and INSTC is then applied to scenario-sensitive channels.
• In that layer, corridor benefit scores derived from the paper's reported post-corridor OPC improvements are translated into additional metric-specific adjustments for trade exposure, resilience, digital spillovers, and GDP overlays.
• The calibrated selectors, corridor components, and implied final level shifts are exported to DATA/generated/scenario_calibration.csv.
• Scenario GDP overlays are exported to DATA/generated/scenario_gdp_paths.csv.

The Mattei scenario should be read as an Italian-led Mediterranean partnership case with modeled EU spillovers. Corridor and shock sources support context and transmission mechanisms; effect magnitudes remain author-calibrated scenario assumptions rather than source-estimated treatment effects.

The scenario-method sources now stored locally in the project are:

• reference foresight scenarios on the global standing-KJNA31544ENN.pdf
• Attachment_0.pdf
• 978-3-030-63505-3.pdf
• A recursive networking economic analysis of international economic corridors IMEEC and INSTC.pdf

Shock magnitudes in DATA/geopolitical_shocks.csv remain research-facing assumptions with evidence tags (high, medium, low) and should still be reported as scenario overlays rather than causal estimates. The publication registry uses citable source notes for the current manual shocks. Any future manual row whose source_note contains placeholder is excluded from publication forecast impacts by default and exported to DATA/generated/excluded_shocks.csv; set INCLUDE_PLACEHOLDER_SHOCKS=1 only for internal exploratory runs.

Publication Interpretation Guardrails

• Scenario paths are exploratory policy scenarios, not point predictions.
• The raw Comext route is populated and rebuilt from DATA/raw/comext_energy_imports.csv; it is the preferred partner replication layer when discussing product-partner evidence.
• The official API partner layer remains the reproducible annual quantity route for partner charts when raw Comext value/mass scope is not the focus.
• Workbook partner charts are withheld when DATA/generated/energy_partner_validation.csv contains failed checks.
• Q1 2025 Eurostat news article rows in the workbook validation file are diagnostic context only because they are not annual 2025 checks.
• Digital-corridor effects are sensitivity assumptions. The historical digital-intensity panel now comes from Eurostat ISOC_E_DIIN2 for 2021-2025, combining DII version 3 observations for 2021, 2023, and 2025 with DII version 4 observations for 2022 and 2024, so version comparability should be disclosed in publication text.

Energy Partner Methodology

Scope

This project's Energy Partners view is built from Eurostat's workbook:

• Energy_update_MAR_2026.xlsx

The workbook is a published Eurostat communication artifact derived from Eurostat's international trade in goods statistics (Comext).

Directly observed components

The following dataset elements are treated as direct observations because they are extracted without transformation from the workbook's partner tables:

• by_product + trade_value_share_pct
• by_product + net_mass_share_pct

Products covered:

• petroleum_oil
• natural_gas_gaseous
• natural_gas_liquefied
• coal

These come from the sheets:

• Petroleum oil - partners
• Natural gas (gas) partners
• Natural gas (liq) partners
• Coal partners

Derived components

The following dataset elements are derived and must be described as such in any scholarly output:

• general + trade_value_share_pct
• general + net_mass_share_pct
• estimated_partner_value_billion_eur
• estimated_partner_net_mass_million_tonnes

They are computed by combining:

• product-level partner shares from the partner sheets
• annual product totals from Figure 3 and Extra-EU imports

Product coverage

The energy products represented are those explicitly listed by Eurostat in the June 26, 2025 news release:

• petroleum oils: 27090010, 27090090
• natural gas liquefied: 27111100
• natural gas in gaseous state: 27112100
• solid fuels: 2701, 2702, 2703, 2704

In the project, the partner-side product group names are simplified to:

• petroleum_oil
• natural_gas_liquefied
• natural_gas_gaseous
• coal

coal is used as a convenience label for the solid-fuels block shown in the workbook partner tables. The annual totals include coal, lignite, peat, and coke from the Extra-EU imports sheet.

Reliability classification

High reliability

• by-product partner shares directly extracted from workbook tables

Medium reliability

• estimated partner values and quantities computed from direct shares and direct annual totals

Medium-low reliability

• general all-energy partner rankings, because they aggregate across heterogeneous products and rely on the workbook's grouped presentation

Main limitations

1. The workbook is a secondary published artifact, not the raw Comext extraction itself.
2. The general aggregate is not a directly published Eurostat partner table.
3. Quantity aggregation across unlike energy products is descriptive, not physically harmonized in a common energy unit.
4. The partner category Other is an aggregated residual category.
5. Scholarly publication should distinguish direct observations from derived indicators.

Minimum citation language

Recommended wording:

By-product partner shares are directly extracted from Eurostat's March 2026 energy update workbook. Aggregate all-energy partner rankings and estimated partner quantities are author-derived indicators computed from Eurostat's published product-level shares and annual totals.

Validation approach in this repository

The script:

• PIPELINE/validate_energy_partner_dataset.py

now separates same-scope workbook validation from external context checks.

Blocking validation checks are based on the workbook-derived annual dataset itself:

• by-product partner shares must sum to 100% for each product, year, and metric
• product totals used for estimated values and quantities must be present
• derived general partner shares must sum to 100% by year and metric

The Eurostat June 26, 2025 news article on EU energy imports is retained as diagnostic context only. That article reports first-quarter 2025 partner shares, while this project's workbook extraction uses annual 2025 partner shares. Those rows are therefore marked not_comparable in DATA/generated/energy_partner_validation.csv and are not treated as validation failures.

That validation is meant to support transparency, not to substitute for a full raw-Comext replication.

Raw replication path

The repository now includes:

• PIPELINE/fetch_comext_energy_imports.py
• PIPELINE/build_energy_partner_from_comext_raw.py

The fetcher downloads the filtered official Comext extract from Eurostat dataflow DS-045409 and writes the local raw landing file expected by the builder. The builder is the preferred route for a scholarly-grade partner-share rebuild because it starts from the filtered raw trade records rather than the communication workbook.

Main command:

.\.venv\Scripts\python PIPELINE\run_all.py --refresh-eurostat --refresh-comext --comext-start-year 2014 --comext-end-year 2025 --skip-external-news

Expected raw input

The script expects a CSV with at least these columns:

• reporter
• partner
• flow
• year
• product_code
• trade_value_eur
• net_mass_kg

Column aliases are supported for common variants such as:

• declarant
• geo
• partner_country
• cn8
• value
• net_mass

Product mapping used in the raw replication script

• 27090010 -> petroleum_oil
• 27090090 -> petroleum_oil
• 27111100 -> natural_gas_liquefied
• 27112100 -> natural_gas_gaseous
• 2701 -> coal
• 2702 -> coal
• 2703 -> coal
• 2704 -> coal

Filtering logic

The raw replication script:

1. keeps import flows only
2. keeps EU27 reporters or EU27_2020
3. excludes EU partners to retain extra-EU trade only
4. aggregates to partner shares by product and in general

Scholarly recommendation

For academic writing, prefer:

• the raw-Comext pipeline when possible
• the workbook-derived pipeline only as an interim or descriptive layer

Recommended wording after the raw pipeline is in use:

Energy partner shares were replicated from an official Eurostat Comext extract using Combined Nomenclature energy-product codes and an explicit extra-EU filtering pipeline implemented in the project repository.

Energy Partner Data Routes

This project now has three distinct routes for energy-partner analysis.

1. Workbook-derived route

Output:

• DATA/generated/eu_energy_partner_shares.csv

Strength:

• direct extraction of Eurostat-published partner share tables

Weakness:

• not fully raw-source reproducible

2. Official Eurostat API annual route

Output:

• DATA/generated/eu_energy_partner_official_api_annual.csv

Builder:

• PIPELINE/fetch_energy_partner_official_api.py

Strength:

• directly reproducible from the public Eurostat API
• no manual workbook parsing required

Weakness:

• annual nrg_ti_* API currently anchors the official annual route to 2024 data
• quantity-focused, not the same as full raw Comext trade-value replication

3. Raw Comext scholarly route

Expected input:

• DATA/raw/comext_energy_imports.csv

Builder:

• PIPELINE/build_energy_partner_from_comext_raw.py

Strength:

• strongest scholarly route in this repository
• explicit CN-code mapping and extra-EU filtering logic

Weakness:

• requires the official raw export file to be provided locally

Practical recommendation

• use eu_energy_partner_official_api_annual.csv for reproducible official annual quantity analysis
• use eu_energy_partner_shares.csv for the published 2025 workbook figures already integrated in the app
• use the raw Comext route when publication-grade replication is required