Ájd5 dZddlmZddlZddlZddlZddlZddlZddl ZdZ dZ dddd d ddd d dd d d did diddd ddd dgdddddddgddZ dZ d!d"dZd#dZd$dZd%d&dZd'd(dZdZd)dZ d* d+dZ d, d-d Zy).a,Backfill sports_teams and match_level from Wikipedia. The two factors that decide whether a broadcast is a blockbuster -- who played and what stage it was -- are missing from 60%/70% of the real-match rows in global_sports, and absent entirely from a raw EPG. They are, however, public knowledge: a tournament's Wikipedia page lists every fixture with its date, teams and stage. Flow, mirroring the one proven in crystalcast/index.php: suggest_url(event, season) -> a best-guess Wikipedia page, pre-filled for the analyst rather than asked for cold fetch_wikitext(url) -> ?action=raw, not rendered HTML compact_wikitext(text) -> keep section headers + match-template lines parse_fixtures(client, ...) -> Claude turns that into structured fixtures Nothing is committed without a human looking at it: the caller previews the fixtures, edits them, and only then confirms. Results are written to the `crystal` application database -- NEVER to tvviewers, which stays read-only. The same fixtures serve both directions: historical rows (so the engine can learn what a marquee fixture is worth) and future schedule rows (so it can recognise one). Enriching only the forecast side would give the model the question without ever having taught it the answer. ) annotationsNz"https://en.wikipedia.org/w/api.phpzRGSIQ-Crystal2/1.0 (broadcast audience estimation; contact denson.joseph@gmail.com)objectstringzHThe tournament as the page names it, e.g. '2026 ICC Men's T20 World Cup')type descriptionarrayz(ISO date YYYY-MM-DD, or empty if unknownz8Local start time HH:MM if the page gives one, else emptyrzOne of GROUP, LEAGUE, QUALIFIER, ROUND_OF_32, ROUND_OF_16, QUARTER_FINAL, SEMI_FINAL, FINAL, THIRD_PLACE, SUPER_8, SUPER_4, PLAYOFF, ELIMINATOR, REGULAR, WARMUP, or UNKNOWNz!Match number if given, else empty) match_date match_timeteam1team2stagematch_noF)r propertiesrequiredadditionalProperties)ritems) event_labelfixturesrra[Below is the compacted wikitext of a Wikipedia page for a sports tournament. Extract every individual match/fixture you can find. For each one give: - match_date : the calendar date it was played (YYYY-MM-DD). Use the season year {season} if the wikitext gives only a day and month. Empty string if genuinely absent. - match_time : the local start time as HH:MM if the page states one, else empty string. - team1, team2: the two competing teams, as full names (not abbreviations) where possible. - stage : which round of the tournament, using the controlled vocabulary in the schema. - match_no : the match number if the page numbers them, else empty string. Also give event_label: the tournament's own name as this page states it. Rules: - Only real fixtures between two teams. Skip standings tables, squad lists, venue lists, broadcast-rights tables, and prose. - Do not invent fixtures. If the page has none, return an empty list. - A match listed with a date range or "TBD" gets an empty match_date rather than a guess. Event as our EPG names it: {event} Season: {season} WIKITEXT: {wikitext} ctjj|dti}tjj ||5}|j j ddcdddS#1swYyxYw)Nz User-Agent)headers)timeoutzutf-8replace)errors)urllibrequestRequestUAurlopenreaddecode)urlrreqrs 4/var/www/html/crystal/crystal2/engine/wiki_enrich.py_getr%dsd .. |R.@ AC   W  5:vvxwy9:::s !A44A=c\|r|d|n|}tdtjj|d} t j t |jdijdg}|sy|r|D]}t||dvs|dcS|ddS#t$rYywxYw) z1Search Wikipedia for the tournament's page title. z#?action=query&list=search&srsearch=z&srlimit=5&format=jsonquerysearchNtitler) WIKI_APIrparsequotejsonloadsr%get Exceptionstr)eventseasonqr!hitshs r$ find_pager8js%6(!E75AZ:ll  # $$: B B+*B+ct||}|s=|r|d|n|}dtjj|j ddzSdtjj|j ddzS)zA pre-filled guess for the analyst to accept or correct. Always returns something: an empty box makes the analyst do the search, and a wrong guess is cheaper to fix than a blank one is to fill. r'zhttps://en.wikipedia.org/wiki/_)r8rr,r-r)r3r4r*r5s r$ suggest_urlr;~ss eV $E #)vhaw u/&,,2D2DQYYsTWEX2YYY +fll.@.@sTWAX.Y YYcDtjd|tj}|rF|jdtj j |jd}}d|d}n(|jdr t|d|St|}}|dtj j|d } tjt|}|jd ijd }|r|d|SdS#t$rYywxYw#t$rYywxYw) zCRaw wikitext is far smaller and more structured than rendered HTML.z8^https?://([a-z]+)\.(?:m\.)?wikipedia\.org/wiki/([^?#]+)zhttps://z.wikipedia.org/w/api.phphttpNz?action=parse&page=z6&prop=wikitext&formatversion=2&format=json&redirects=1r,wikitext)rematchIgrouprr,unquote startswithr%r1r+r-r.r/r0) url_or_title max_charsmlangr*apir!datatexts r$fetch_wikitextrOs Ll\^\`\`aAggaj&,,"6"6qwwqz"Be67   (  %jy1 1|UU%fll&8&8&?%@ C ECzz$s)$ 88GR $ $Z 0D#4  --   s$ D<D DD DDc.|sytjdtj}g}|jdD]}|j }tj d|r2|j d|j dj z\|j |sntj d|tjs%tj d|tjr|j d tjd d|tj }|j |d d t||k\sndj|}d |vr|d |Sg}|jdD]}tjdd|j }dt|cxkrdksn?ttjd|tjxstjd|}ttjd|tjxstjd|} |r| r|j |t|dk\sndj|}t|dkDr|d dSdS)aThrow away everything that is not a fixture before the model sees it. Keeps section headers (they carry the stage: "Semi-finals", "Final"), any line declaring a match template, and any line carrying team/date/time/score parameters. Falls back to a looser date+versus filter when the page does not use match templates at all, so non-template pages still yield something. ze^\|\s*(team1|team2|date|time|round|stage|score1|score2|home ?team|away ?team|venue|aet|penalties)\s*= z^=+\s*(.+?)\s*=+$z SECTION: z= z^\|\s*team1\s*=z^\|\s*home ?team\s*=z --- match ---z]*>.*?|]*/>)flagsNz\s+r',zb\b(\d{1,2}\s+)?(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]*\.?\s*\d{0,2}(,?\s*\d{4})?\bz/\d{4}-\d{2}-\d{2}|\d{1,2}[/.]\d{1,2}[/.]\d{2,4}u\bvs?\.?\b|\bv\b|–|—z\d{1,2}[:.]\d{2}i(>) rBcompilerDsplitstriprCappendsubSlenjoinboolr)) rN max_linesrIWANTEDkeeprawsouthas_date has_matchs r$compact_wikitextrjs.  ZZJKM44QF Dzz$  IIK 88(! , KK aggdm&9&9&;; < ||A  88&244 0BHH=TVWY[Y]Y]4^ KK ( FF5r1BDD I AdsG t9 !   ))D/C#:I Dzz$  FF63 ,SV"s"  #HIJBDDR\99%WYZ[]#>244H>II&91=?  KKN t9    ))D/Cs8b=3w<0b0r<cddl}tjjdxsd}|sddlm}|t jjjdz dz }|jr6tj|jjdxsd}|s td |j| S) a-Anthropic client. Key resolution order: the environment first (what a developer shell has), then worker/config.json. The config route matters because the worker and the enrichment runner are started by Apache/systemd, neither of which inherits an interactive shell's environment. rNANTHROPIC_API_KEYrQ)Pathworkerz config.jsonanthropic_api_keyz^No Anthropic API key. Set ANTHROPIC_API_KEY, or add "anthropic_api_key" to worker/config.json.)api_key) anthropicosenvironr0pathlibrm__file__resolveparentexistsr.r/ read_text RuntimeError Anthropic)rqkeyrmcfg_paths r$_clientr~s **.., - 3C  >))+2299HD}T ?? **X//12667JKQrC  9: :   s  ++r<c |jjddddiddtddd tj ||xsd | d g }|j dk(rgdfSt d|jDd} tj|}|jdg|jddfS#tj$rgdfcYSwxYw)z=Have Claude turn compacted wikitext into structured fixtures.z claude-opus-5rXradaptivemedium json_schema)rschema)effortformatuserunknown)r3r4rA)rolecontent)model max_tokensthinking output_configmessagesrefusalrQc3TK|] }|jdk(s|j"yw)rNN)rrN).0bs r$ z!parse_fixtures..sAA&0@As((rr) rcreateFIXTURE_SCHEMAPROMPTr stop_reasonnextrr.r/JSONDecodeErrorr0)clientr3r4rAmsgrNrMs r$parse_fixturesrs // *%!)*7>!RT!fmm 3)h/</HIJ ! C )#2v AA2 FDzz$ 88J #TXXmR%@ @@   2v s<B55CCcZ|sddgdddSt|}|sd|gdddStj|t|}|sd|gdddS t ||||\}}|rd nd|||t|ddS#t $r-} d |gdt | jd | d d dcYd } ~ Sd } ~ wwxYw)zOne event, one page. Network-and-AI only; no DB writes. Returns {status, url, title, fixtures, label, note} so the caller can show the analyst exactly what happened for each event it looked up. no_urlrQz no page given)statusr!rlabelnoteno_pagezcould not read that page no_fixturesz"page has no fixture-shaped contenterrorz: NrVokz fixtures read) rOtimesleeprjrr1r__name__r_) rr3r4r!rwtcompactfxrexcs r$enrich_from_urlr s "22Tcdd  B #CR"24 4JJur"G 'b<> >>"65&'B E!dmCRs2wi~$> @@ >!#29--.b6t<> >>s A44 B*="B%B*%B*ct||}|sgddfStj|t|}|sgd|fSt |||t |\}}||rd|fSd|fS)z:Search-then-parse, used by the historical backfill runner.rzno Wikipedia search hit no_wikitextrr)r8rrrOrrj)rr3r4rr*rrr:s r$enrich_event_seasonr&sz eV $E 9777JJu  B =%'' 65&2B22F GEB  55  55r<))r!r2rintreturnr2)r3r2r4 int | Noner str | None)r3r2r4rrr2)i)rHr2rIrrr)i i`)rNr2rbrrIrrr2)r3r2r4rrAr2rztuple[list[dict], str])g333333?) r3r2r4rr!r2rfloatrdict)g?)r3r2r4rrrrztuple[list[dict], str, str])__doc__ __future__rr.rrrBr urllib.parserurllib.requestr+rrrr%r8r;rOrjr~rrrr<r$rs2#  /Y  ((HI +32\#^+32l#n$h/$h/&..WX *2Be f `(-# 6 +!=B 6: ( Z.271v,,A,$'@ @+/@:(+ 6$ 6/J 6r<