Jinsi ya kujenga hifadhi ya data ya A-share kwa AI

Hifadhi ya data ya ndani ya A-share

Hifadhi ya data ya ndani ya A-share ni historia ya soko la hisa la China bara iliyohifadhiwa kwenye diski yako kama faili za safu wima za Parquet. Faili hizi zinaweza kusomwa na DuckDB au Polars, badala ya kutumia mfumo wa mtoa huduma unaohitaji kuita ukurasa mmoja kwa wakati mmoja. ashare-lake ni mradi wa chanzo huria unaojenga hifadhi hiyo, pamoja na kazi ya kila siku inayoiweka kisasa na seva ya Model Context Protocol inayoruhusu wakala wa AI kuhoji data hiyo. Chaguzi mbili za usanifu zinaifanya iandikiwe hapa: majina ya kampuni zilizofutwa kwenye soko (delisted) yanahifadhiwa, na misingi ya kifedha inaweza kusomwa kulingana na tarehe ya nyuma.

Kwa nini wakala wa AI anahitaji hifadhi ya data ya ndani ya A-share

Wakala anayefanya utafiti wa hisa za China bila kuwa na nakala ya ndani ana njia mbili. Anaweza kuchunguza kurasa za kifedha, akitumia nafasi yake ya muktadha (context window) kwa HTML na kutoa namba ambazo hakuna mtu anayeweza kuzizalisha upya mwezi ujao. Au anaweza kupiga simu kwa muuzaji anayehitaji usajili, ambaye huweka kikomo cha safu na kufunga kila matokeo kwenye akaunti.

Kiwango (scale) ndicho kipengele kinachodharauliwa. Ghala letu lenyewe hubeba data ya soko la US kwa mwonekano wa dakika, na wiki moja ya kawaida ya data hiyo inaonekana hivi:

SELECT toDate(toTimeZone(window_start, 'America/New_York')) AS session,
       formatDateTime(toDate(toTimeZone(window_start, 'America/New_York')), '%b %e') AS session_label,
       uniqExact(ticker) AS tickers_count,
       round(count() / 1000000, 2) AS minute_bars_millions
FROM global_markets.delayed_stocks_minute_aggs
WHERE toDate(toTimeZone(window_start, 'America/New_York')) >= toDate('2026-07-20')
  AND toDate(toTimeZone(window_start, 'America/New_York')) <= toDate('2026-07-24')
GROUP BY session, session_label
ORDER BY session

Mnamo Jul 20 mkanda wa data ulizalisha 1.79 milioni ya pau za dakika kwa alama 11737, na vipindi vingine vinne kwenye paneli hurudia hali hiyo. Soko moja la kitaifa, wiki moja, mwonekano mmoja. Muongo mmoja wa pau za kila siku, misingi ya kifedha, uanachama wa fahirisi, na rekodi za mtiririko wa fedha kwa soko lingine huwa na umbo sawa, na kuipata kupitia HTTP kunamaliza uwezo wa utendaji wa wakala.

Hifadhi ya ndani hubadilisha mambo mawili kwa wakati mmoja. Usomaji huwa ni uchanganuzi wa faili badala ya kufuata kiasi (quota), na hoja (query) iliyoandikwa leo inarejesha safu zilezile baada ya miezi sita, jambo ambalo ni muhimu kwa backtest ili iweze kuhakikiwa. Vidokezo vyetu kuhusu ujuzi wa data ya soko kwa mawakala wa AI na kuhusu SQL API juu ya data ya soko vinatoa hoja hiyo hiyo kwa data ya US.

Isakinisha, na uifunge kwenye toleo moja

Python 3.10 au mpya zaidi. Funga toleo hilo: matoleo sita yalitoka kati ya Julai 27 na Agosti 2, 2026, na usakinishaji usiofungwa ndani ya hati ya usanidi ya wakala ni kama shabaha inayohamahama. Msimbo huu unapatikana katika rootSunc/ashare-lake chini ya leseni ya Apache 2.0.

• pip install ashare-lake==0.5.0 inasakinisha toleo la sasa kufikia mwanzoni mwa Agosti 2026.
• asl --version inachapisha build iliyosakinishwa.
• asl config init --data-root /path/to/ashare-lake inaandika mfano wa TOML uliopakiwa huku mzizi wa data yako ukiwa umejazwa, hakuna haja ya kufanya repository checkout. --config inaweka njia ya pato, --force inaandika juu ya faili iliyopo.
• asl doctor inaendesha ukaguzi wake nje ya mtandao, kabla ya data yoyote kusonga.
• asl servers test inachunguza seva za bei za juu (upstream quote hosts). asl sources inachunguza kila chanzo, ikiwa na --vantage ya cn, overseas, au local.

Simama hapo. Amri inayofuata ni ya backfill, na si kitu cha kuanzisha wakati unasoma.

Kazi ya backfill

asl init hutengeneza muundo wa saraka na kujaza historia. Nyaraka za mradi huu zinaeleza kuwa mchakato kamili huchukua saa kadhaa za muda halisi na nafasi ya GB kadhaa kwenye diski, na unahitaji muunganisho wa moja kwa moja na seva ya bei ya Tongdaxin, jambo ambalo asl servers test huthibitisha. asl init --profile quick badala yake huchakata miaka mitatu ya hivi karibuni ndani ya dakika chache, na bado huhifadhi kila jina lililofanya biashara ndani ya kipindi hicho, ikiwemo zile kampuni ambazo zimeondoka sokoni tangu wakati huo. asl run daily ni kazi ya nyongeza inayofuata, asl status --datasets huripoti ufikiaji na usasishaji kwa kila mkusanyiko wa data, na asl serve huweka dashibodi ya kusoma pekee kwenye 127.0.0.1:8787.

Hifadhi hii haisafirishi data yoyote. Kila faili ya Parquet hujengwa kwenye mashine yako, huku ikiwa na rekodi ya asili ya kila mstari (row-level lineage) inayoonyesha ni chanzo kipi kilichozalisha data hiyo na wakati ilipochotwa.

Dataset thelathini na tisa ni zipi?

Zimepangwa kwa matabaka, kuanzia data za marejeleo kuelekea nje: majedwali thelathini na sita yaliyochaguliwa na matatu yaliyotokana na hayo.

• Marejeleo: vyombo vya kifedha (instruments), kalenda ya biashara inayohusu mwaka elfu mbili na kumi na sita hadi elfu mbili na ishirini na saba, na hali ya biashara.
• Data za soko: bei za kila siku (daily bars), bei za fahirisi (index bars), bei za dakika moja na dakika tano, prints za biashara (trade ticks), bei za bidhaa (commodity bars), viashiria vya marekebisho (adjustment factors), na matukio ya kuondolewa kwenye soko (delisting events).
• Matukio ya kampuni: hatua za kampuni (corporate actions), fahirisi ya matangazo, na ratiba ya utoaji wa taarifa za mapato (earnings disclosure).
• Misingi na tathmini: vipengele vya taarifa za kifedha, vipimo vya tathmini (valuation metrics), na makubaliano ya wachambuzi (analyst consensus).
• Mtiririko wa mtaji: mtiririko wa fedha za mfuko (fund flow), biashara ya margin (margin trading), mtiririko na umiliki wa northbound, ufichuzi wa oda kubwa za dragon-tiger board, biashara za block (block trades), na umiliki wa taasisi.
• Muundo na sekta: wanachama wa sekta, washiriki wa fahirisi (index constituents), wanachama wa viwanda, na fahirisi ya viwanda.
• Macro: viashiria vya macro, upana wa soko (market breadth), na kalenda ya kiuchumi.
• Hisia na mzunguko: alama za hisia (sentiment scores), viwango vya umaarufu (hot rank), bei za sekta, mtiririko wa fedha wa sekta, vichwa vya habari, na mtandao wa habari za papo hapo (flash news wire).
• Hatari na utiifu: ratiba ya kufunguliwa kwa hisa (share unlock schedule), na matukio ya udhibiti.

Mahali ambapo dataset inakaa hukuonyesha kile ambacho mwandishi anakipa kipaumbele. Viashiria vya marekebisho na matukio ya kuondolewa kwenye soko viko kwenye tabaka la data za soko kando ya bei za kila siku, havijawekwa kwenye kiambatisho, na mpangilio huo ndio nusu ya mradi yenye thamani zaidi kwa yeyote anayejaribu mawazo yake kwenye historia.

How a local lake handles survivorship bias

Survivorship bias is what you get when the universe of a study is drawn from the names that are still listed today. Every company that merged, went private, or was delisted is silently absent, and the names that vanish are rarely the winners.

Our warehouse can size that hole for the US market, the same arithmetic in a different alphabet. For each year, take every symbol that printed a minute bar in the second week of March, then check which of them were still printing in the last two weeks of July 2026:

WITH on_tape_now AS (
    SELECT ticker
    FROM global_markets.delayed_stocks_minute_aggs
    WHERE toDate(toTimeZone(window_start, 'America/New_York')) >= toDate('2026-07-20')
      AND toDate(toTimeZone(window_start, 'America/New_York')) <= toDate('2026-07-31')
    GROUP BY ticker
),
cohort AS (
    SELECT toYear(toTimeZone(window_start, 'America/New_York')) AS cohort_year,
           ticker
    FROM global_markets.delayed_stocks_minute_aggs
    WHERE toYear(toTimeZone(window_start, 'America/New_York')) BETWEEN 2016 AND 2025
      AND toMonth(toTimeZone(window_start, 'America/New_York')) = 3
      AND toDayOfMonth(toTimeZone(window_start, 'America/New_York')) BETWEEN 10 AND 14
    GROUP BY cohort_year, ticker
)
SELECT c.cohort_year AS year,
       count() AS names_on_tape_count,
       countIf(n.ticker != '') AS still_trading_count,
       count() - countIf(n.ticker != '') AS gone_count,
       round(100 * countIf(n.ticker != '') / count(), 1) AS still_trading_pct
FROM cohort AS c
LEFT JOIN on_tape_now AS n ON c.ticker = n.ticker
GROUP BY year
ORDER BY year

Of the 8101 symbols trading that week in 2016, 50.1% were still on the tape in late July 2026, and 4040 were not. The 2025 cohort reads 86.7%. Run a screen built from today's listings backwards across those 10 years and it drops a widening share of the market as it goes.

The comfortable assumption is that the missing names were all penny stocks. Sorting the March 2021 cohort into tiers by its average daily dollar volume that month says otherwise:

WITH on_tape_now AS (
    SELECT ticker
    FROM global_markets.delayed_stocks_minute_aggs
    WHERE toDate(toTimeZone(window_start, 'America/New_York')) >= toDate('2026-07-20')
      AND toDate(toTimeZone(window_start, 'America/New_York')) <= toDate('2026-07-31')
    GROUP BY ticker
),
march_2021 AS (
    SELECT ticker,
           sum(toFloat64(close) * toFloat64(volume))
             / uniqExact(toDate(toTimeZone(window_start, 'America/New_York'))) AS avg_daily_dollar_volume
    FROM global_markets.delayed_stocks_minute_aggs
    WHERE toDate(toTimeZone(window_start, 'America/New_York')) >= toDate('2021-03-01')
      AND toDate(toTimeZone(window_start, 'America/New_York')) <= toDate('2021-03-31')
    GROUP BY ticker
)
SELECT multiIf(d.avg_daily_dollar_volume >= 1000000000, '$1B or more',
               d.avg_daily_dollar_volume >= 100000000, '$100M to $1B',
               d.avg_daily_dollar_volume >= 10000000, '$10M to $100M',
               d.avg_daily_dollar_volume >= 1000000, '$1M to $10M',
               'under $1M') AS liquidity_bucket,
       count() AS names_count,
       countIf(n.ticker = '') AS gone_count,
       round(100 * countIf(n.ticker = '') / count(), 1) AS gone_pct
FROM march_2021 AS d
LEFT JOIN on_tape_now AS n ON d.ticker = n.ticker
GROUP BY liquidity_bucket
ORDER BY min(d.avg_daily_dollar_volume)

The under $1M tier lost the largest share, 57.5% of 3968 names. The busiest tier was not exempt: 3.4% of the 89 symbols trading $1B or more a day in March 2021 had left by late July 2026. Mergers, take-privates, bankruptcies, and index exits all end the same way in a price table: the rows stop.

ashare-lake treats that as a first-class problem. The instruments dataset retains delisted symbols rather than filtering down to live ones, a delisting_events table records how each departed name ended, and universe="all_a" in the Python API resolves a historical snapshot that includes them. The project's own docs report roughly a two-fold gap between a survivor-only backtest and a delisting-inclusive one over 2016 to 2021. That is the mirror image of the trap in our look-ahead bias in backtesting notes: a universe that quietly knows the future.

Point-in-time reads

load("financial_statement_items", as_of="2018-04-30") returns the latest version of each line item announced on or before that date, not the number as later restated. Bars carry an adjust argument: hfq for backward adjustment, qfq normalized inside the query window, or raw prices. A factor fitted on numbers the market had not published yet measures nothing, which is the first thing to check in any LLM generated alpha factor pipeline.

Kuisajili kama seva ya MCP

Model Context Protocol ndiyo njia ambayo wakala hutumia kuchagua zana. asl mcp huzungumza itifaki hiyo kupitia stdio, na mteja huzindua mchakato huo:

• claude mcp add ashare-lake -- asl mcp --config /abs/path/to/ashare-lake.toml
• Mteja yeyote mwingine wa MCP huhitaji vipengele vilevile viwili: asl kama amri, na mcp --config pamoja na njia kamili (absolute path) kama hoja (arguments). Njia hiyo lazima iwe kamili, kwa sababu mteja anaweza kuanzisha mchakato huo kutoka saraka yoyote.

Zana sita hujitokeza, zilizopangwa kulingana na swali badala ya seti ya data: describe_lake kwa ajili ya kile kilichopo na jinsi ya kukisoma, resolve_symbol kwa ajili ya kubadilisha jina kuwa msimbo (ikijumuisha majina yaliyofutwa kwenye orodha), query_bars, query_fundamentals pamoja na hoja yake ya as_of, query_dataset kwa kila kitu kingine, na run_sql kwa ajili ya SELECT moja ya DuckDB ya kusoma pekee (read-only) katika seti za data. Bendera ya --live inaruhusu seva kujibu utafutaji wa symbol na bei za kila siku ambazo hazijarekebishwa (unadjusted daily bars) kutoka chanzo kikuu bila kuandika kwenye ziwa la data (lake).

Vikwazo vinavyopaswa kufahamika kwanza

• A-shares pekee. Hakuna orodha za Hong Kong, Marekani, wala chochote nje ya China bara.
• Mradi wa kibinafsi. Masuala na maombi ya pull requests hupata uangalizi kwa kadiri ya uwezo, na nyaraka zinaeleza wazi kuwa hakuna hakikisho la upatikanaji: tovuti za chanzo hubadilika, na IP inaweza kuzuiwa, jambo ambalo husimamisha uingizaji wa data hadi mtu atakapofanya marekebisho.
• Leseni ya Apache 2.0 inashughulikia msimbo, si data. Kila chanzo cha awali kinabaki na masharti yake, na msimamizi hatoi haki ya kusambaza upya au kuuza faili za Parquet unazotengeneza. Soma masharti ya chanzo kabla ya matumizi yoyote ya kibiashara.
• Hakuna akaunti au token inayohitajika kwa vyanzo vinavyosomwa, jambo ambalo ndilo mvuto wake na pia udhaifu wake.
• Usaidizi wa Windows ulianza katika toleo la 0.3.0, na kiwango cha chini cha Python kilipandishwa hadi 3.10 katika toleo la 0.3.1.

Maswali Yanayoulizwa Mara kwa Mara

Je, unahitaji API key ili kujenga data lake ya A-share ya ndani?

Huna haja nayo kwa mfumo huu. Vyanzo vya juu vinavyosomwa havihitaji usajili au token, na data lake yenyewe hukaa kwenye mashine yako. Kila chanzo kina masharti yake ya matumizi, ambayo ni muhimu kuzingatiwa kabla ya kuanza kazi yoyote ya kibiashara.

Inachukua muda gani kukamilisha backfill ya ashare-lake?

Nyaraka zinaeleza kuwa backfill ya historia nzima inachukua saa kadhaa za muda halisi na GB kadhaa za nafasi kwenye diski, huku muunganisho thabiti wa intaneti na mwenyeji wa data za soko ukihitajika muda wote. asl init --profile quick inashughulikia miaka mitatu ya hivi karibuni ndani ya dakika chache na bado inajumuisha majina ya hisa ambazo zimeondolewa kwenye soko (delisted).

Je, data lake ya A-share ya ndani inajumuisha hisa zilizofutwa?

Ndiyo, inajumuisha. Alama za hisa zilizofutwa hubaki kwenye dataset ya instruments, jedwali la delisting_events hurekodi jinsi kila jina lilivyomaliza muda wake, na universe="all_a" hujenga snapshot ya kihistoria inayozijumuisha. Kipimo chetu cha soko la Marekani hapo juu kinaonyesha ukubwa wa data inayopotea ikiwa utazingatia hisa zilizosalia pekee.

Je, AI agent inaweza kuuliza maswali (query) kwenye ashare-lake moja kwa moja?

Ndiyo. asl mcp inatoa zana sita kupitia Model Context Protocol, mojawapo ikiwa ni zana ya SQL ya kusoma pekee (read-only), hivyo agent huuliza maswali kwenye faili za Parquet za ndani badala ya kufanya scraping. Ijisajili kupitia claude mcp add ashare-lake -- asl mcp --config /abs/path/to/ashare-lake.toml.

Je, ashare-lake ni mbadala wa AkShare au Baostock?

Hapana. Inakaa juu yao. Maktaba hizo huchota data kutoka vyanzo vya juu; mradi huu huhifadhi, huweka matoleo, na kusawazisha data hizo kuwa Parquet iliyopangwa vizuri, ikiwa na lineage ya kiwango cha safu (row-level) na mkataba mmoja kwa kila dataset.

Kila takwimu hapo juu ni matokeo ya query iliyohifadhiwa na kuwekewa toleo juu ya data halisi ya soko. Panua paneli yoyote ili kusoma SQL, au endesha ukaguzi uleule wa survivorship kwenye ulimwengu wako wa data kupitia Strasmore terminal.