A short Python script that mines two open data sources, the World Bank API and India's Agmarknet mandi prices on data.gov.in, into one local DuckDB file with a table per source.
Open-Source Tools · How-To · Dev & ops tools
How to use data-source-mining: pull World Bank indicators and mandi prices into DuckDB
Revised
· v1.0.0 · what changed
Code: github.com/herrrickshaw/data-source-mining
Watch the walkthrough
Watch on YouTube: youtu.be/gQ6XYU7O9VE
Why it is useful
- Who benefits: analysts who want a macro panel for India and six comparison economies, or daily vegetable and grain market prices, without clicking through portals.
- What it replaces: manual downloads from the World Bank site and data.gov.in.
- Evidence from the run: a live call to the World Bank for India and Germany returned 326 rows for GDP growth, CPI inflation and renewable energy share in about a minute; one mandi API call returned records.
- Result is a single DuckDB file you can query with SQL; the repo's committed copy (pulled 21 July 2026) has 4,532 World Bank rows and 4 mandi rows.
- Small enough to read in five minutes, so you can add your own indicators.
What it is for
data-source-mining was written after an audit of a Claude usage log showed two open sources that were under-used: the World Bank API and Agmarknet daily mandi prices. The script mine_underused_sources.py pulls both and stores them in underused_sources.duckdb.
World Bank: 12 indicators (GDP, growth, inflation, agriculture share, fertiliser use, trade, energy mix, FDI, real interest rate, INR/USD) for IND, USA, CHN, BRA, DEU, JPN and KOR, paged through the public API.
Mandi: the data.gov.in Agmarknet resource, using the public sample key that data.gov.in documents, appended with a pulled_on date so a re-run on the same day replaces that day's rows.
Note on scope: the script mines a fixed pair of sources. It is not a general source-discovery search tool.
Who it is for
- Economists and students who want a quick cross-country panel
- Agri-trade and food-price analysts
- Anyone learning DuckDB with real open data
- Investment decisions: this is raw open data, not investment advice
- Real-time prices: mandi data is whatever the API returns on the pull day and the sample key is rate-limited
- Running it inside the cloned folder if you want to keep the committed database: the script rebuilds worldbank_indicators with CREATE OR REPLACE in that file
What you need
| Requirement | Why | How to get it |
|---|---|---|
| Python 3 and the duckdb package | stores the results | python3 -m venv .venv && .venv/bin/pip install duckdb |
| Internet access | calls api.worldbank.org and api.data.gov.in | none |
| No account or key | World Bank is open; the mandi call uses data.gov.in's public sample key built into the script | nothing to do |
Get the code and an AI assistant
You do not have to be a programmer. Get the repository, give it to an AI assistant, and ask in plain language. Any of these routes works:
| Tool | What you need | Best for |
|---|---|---|
| GitHub | Free account to fork or star; git to clone, or the green Code button, then Download ZIP, or open in Codespaces | Getting the code and keeping up with updates |
| Claude Code | Claude subscription or API key; run claude inside the cloned folder | Letting the AI read and run the whole repo |
| Gemini CLI | Google account; run gemini inside the cloned folder | The same hands-on workflow with Gemini |
| Claude.ai, Gemini or ChatGPT in the browser | A chat account; upload README.md and the files you care about (or paste the output of git ls-files) | Understanding the project and asking questions without installing anything |
| Other tools (Copilot Chat, Cursor, Codex CLI) | Open the cloned folder in the tool and point it at README.md first | Editing and extending the code |
git clone https://github.com/herrrickshaw/data-source-mining.git
cd data-source-mining
claude # or: geminiInstall
- Clone the repo
git clone https://github.com/herrrickshaw/data-source-mining.git cd data-source-mining - Create a virtual environment and install DuckDB
python3 -m venv .venv .venv/bin/pip install duckdb
Quick start: commands and what you should see
World Bank only (about 84 API calls, a few minutes)
.venv/bin/python mine_underused_sources.py --wb-onlyOne line per country, then 'worldbank_indicators: N rows' and a table list. N was 4,532 in the committed 21 July 2026 pull.
Mandi prices only
.venv/bin/python mine_underused_sources.py --mandi-only'mandi_prices: N rows pulled today'. The data.gov.in sample key returns a small sample, not the full market.
Query the result
.venv/bin/python -c "import duckdb; c=duckdb.connect('underused_sources.duckdb', read_only=True); print(c.sql(\"select country, round(value,2) as v from worldbank_indicators where indicator='gdp_growth_pct' and year=2024\"))"GDP growth by country for 2024.
Sample outputs
Real output from a run on the date shown in the sources line; trimmed to fit.
Live World Bank pull, India and Germany, 3 indicators (run 19 Oct 2026)
(script functions called with WB_COUNTRIES=[IND,DEU] and 3 indicators)
worldbank: IND done (163 rows cumulative)
worldbank: DEU done (326 rows cumulative)
rows 326
country indicator latest_year latest_value
DEU cpi_inflation_pct 2025 2.17
IND cpi_inflation_pct 2025 2.4
DEU gdp_growth_pct 2025 0.24
IND gdp_growth_pct 2025 7.57
DEU renewable_energy_pct 2021 17.6
IND renewable_energy_pct 2021 34.9Written to a throwaway database, not to the repo's file. Values are as returned by the World Bank API that day.
GDP growth 2024 from the committed database (pulled 21 July 2026)
select country, round(value,2) as v, year from worldbank_indicators where indicator='gdp_growth_pct' and year=2024
BRA 3.42
CHN 4.96
DEU -0.5
IND 7.1
JPN -0.24
KOR 2.0
USA 2.79Percent. 4,532 rows, 7 countries, years 1960 to 2025 in that file.
Mandi prices, committed sample (pulled 21 July 2026)
select state,market,commodity,modal_price,arrival_date from mandi_prices
Bihar Gaya APMC Bitter gourd 1840.0 21/07/2026
Bihar Gaya APMC Tomato 4700.0 21/07/2026
Bihar Gaya APMC Capsicum 5400.0 21/07/2026
Bihar Gaya APMC French Beans(Frasbean) 8300.0 21/07/2026Modal price is Rs per quintal as published on Agmarknet. The committed table has only these 4 rows.
One live mandi API call (limit 2)
get_json(.../resource/9ef84268-...?limit=2)
mandi ok 2The endpoint answered with 2 records on 19 Oct 2026; record contents not shown.
Ways to use it
| Method | How it works |
|---|---|
| Command line | mine_underused_sources.py with --wb-only or --mandi-only. |
| Extend it | Add World Bank codes to WB_INDICATORS or countries to WB_COUNTRIES; the loop and table rebuild adapt. |
| Query with SQL | Open underused_sources.duckdb in the DuckDB CLI, Python or any DuckDB-aware notebook. |
| Via an AI assistant | Ask Claude Code or Gemini CLI to add an indicator, run the pull and chart India against Brazil. |
Sample prompts for your AI assistant
Copy, paste and edit the parts in [brackets].
- Explain the data-source-mining repository in plain English: what problem it solves, what the main files do, and what I need installed.
- Add World Bank indicators for life expectancy and population to mine_underused_sources.py, run it for IND and BRA, and show me the latest values.
- Using underused_sources.duckdb, compare India's CPI inflation with the other six countries since 2015 and describe the pattern in plain words.
- Explain the mandi_prices table: what do min, max and modal price mean, and what are the units?
- Change the script so it writes to a new file name each run instead of replacing worldbank_indicators.
- Find other open APIs like these two that need no registration, and list how I would add each to the script.
Limits and honest caveats
- Not investment advice. These are raw open statistics.
- Only the World Bank leg was run end to end here, on a reduced set (2 countries, 3 indicators) in a throwaway database; the full 7 x 12 pull and the full mandi pull were not re-run. Table sizes quoted come from the committed 21 July 2026 file.
- Running the script in the repo folder overwrites worldbank_indicators in the committed underused_sources.duckdb. Copy the folder first if you want to keep it.
- The README in the repo is only a placeholder pointing to a how-to-use page; the docstring in the script is the real documentation.
- Mandi data from the sample key is small and the committed table has 4 rows; do not draw market conclusions from it. The World Bank latest year varies by indicator (renewables was 2021).
Sources: data-source-mining mine_underused_sources.py and committed underused_sources.duckdb; World Bank pull for IND and DEU and a single data.gov.in call run on 19 October 2026 on a fresh clone in a throwaway virtual environment; committed-database figures are from the 21 July 2026 pull.
- v1.0.0 — 26 October 2026 — first published.
About this article: Researched, written and edited by Umashankar Triplicane Dwarakanathan, with AI research assistance; every figure is meant to trace to the primary source cited. See the Editorial Policy for how sourcing, AI use and corrections work.
