j!fdZddlmZddlZddlZdZdZd d dZ d dZ d dZ dd Z d dd Z y)a Measure what the estimator currently assumes. Three components of the model were constants someone typed rather than quantities estimated from evidence: * the hour-of-day curve and the weekday curve (engine/world.py) * the exponents on TV universe, channel share and sport affinity, all fixed at 1.0 -- i.e. the assumption that audience scales proportionally with each * the per-tier residual sigma behind every 80% interval Every one of them is measurable from the 5.9M rows already in global_sports. This module estimates them; worker/fit_model.py runs it and writes the result to calibration.json, which ReferenceData loads in preference to the constants. Method, so it can be checked: Curves within-stratum ratio estimation. Inside each (country, channel, event) group the median audience at each hour is expressed as a ratio to that group's own overall median, then the median ratio across groups is taken. Comparing only inside a group holds channel size and event pull fixed, so what survives is the hour effect. This is the same estimator already used for telecast multipliers, and the same logic as a ratio-to-moving-average seasonal index. Exponents weighted least squares on log audience, with event fixed effects absorbed by within-event demeaning. Identification therefore comes from cross-market variation WITHIN an event, which is exactly the comparison a cross-market projection makes. Weights are sqrt(n) on the cell. Sigma standard deviation of log residuals per tier, from a hold-one-out backtest against known audiences. ) annotationsN c 4||ddkD||jzj}t|}|j||gzdj j }|j|dj j j ddi}|j||d}||ddkD}|jrtj|ddgS|d|dz |d <|j|d jd d g}||d tk\}tj||jd|d jtj d|d jt"j iS) zMedian audience at each level of `key`, as a ratio to its group's median. Returns columns [key, weight, n_groups]. Levels seen in fewer than MIN_GROUPS groups are dropped rather than reported on thin evidence. ama_000rgrp_medcolumnsinner)onhowweightn_groupsratiomediansize)notnacopylistgroupbyr reset_indexrenamemergeemptypd DataFrameagg MIN_GROUPSindexastypefloatvaluesint)gskeykeysdmedbasejrs 2/var/www/html/crystal/crystal2/engine/calibrate.py_within_group_ratior,-s ByMA C 01668A :D ))DC5L !) , 3 3 5 A A CC 99T?9 % , , . : : < C CI& D (D $4W -A !I, Aww||S(J$?@@9) ,AgJ ))C. ! % %x&8 9C c&kZ' (C <<cii!3x=#7#7#>#E#E#S[%7%7%<%C%CE FFct|d}|jr|S|d|djz |d<|jddij dj dS)z8Hour-of-day weights, normalised so the peak hour is 1.0.hourrr Tdrop)r,rmaxr sort_valuesrr$outs r+measure_hour_curver6Esi b& )C yy MCM$5$5$77CM ::vv.: / ; ;F C O OUY O ZZr-ct|d}|jr|S|d|djz |d<|jdj dS)z7Day-of-week weights, normalised so the peak day is 1.0.weekdayrTr0)r,rr2r3rr4s r+measure_weekday_curver9NsV b) ,C yy MCM$5$5$77CM ??9 % 1 1t 1 <r?appendnplogr!sqrtrlenarrayuniquesumwhereaveragevstack concatenatelinalglstsqshapepinvTdiagr2r#)!r$rJrKflagship affinity_ofXYWmetacellsrtsflamevXdYdWdeiwbeta_residrPss_totr2nkxtx_invses! r+ fit_exponentsrXs&BNMAq!T 9 ! "g:;IGc66"#KKM  %-=0 1E   E  * / GGAII  IIqyy!)), - \\!)) $  1<< 0aB1!  "&&)RVVAF^RVVAY78 aff &' qvv'( Q\\199-. / 1v{s1vhm'DEE  A!Q"((1+a &A1Q4& 'BRBB YYr] !G 557Q;#288A;q>BatAwqzBCaG  aD !A$AaDq!<<= !A$AaD!445 !  '@AA 2BR^^B/bnnR6HyyrBq$wK/bEHD1 dNE "''"**UaZ<= >E 266"rQw,' (F8> U266"uz/* +f 4 4e B 88DAqiinnb2ag;.11R"QW+5EFGG !RVVB!O%E$q'NdSTgBqElbe USUVWSX\r1v3s299R=/A+B  DD7' Cs  VV c tj|tj|z|dkDz|dkDz}tj||||z }||}t|rt tj |nd}ii} }tj |D]} ||| k(} t| dkrt| t| |zz } t tj| t tj | zd| z |zz|t| <t tjtj| ddd| t| <|| fS)aPer-tier log-residual sigma and a shrunk multiplicative correction. correction_tier = exp( w * mean(log truth/pred)_tier + (1-w) * global mean ), w = n / (n + shrink_n). That is a James-Stein / empirical-Bayes estimator: a tier with few backtest rows is pulled toward the global correction rather than trusting its own noisy mean. Without it a tier seen ten times would be calibrated as confidently as one seen ten thousand times. rgrrD)ddofg?g@) rXisfiniterYr[r!r?r]expr#clipstd) predtruthtiershrink_nrAlrtrmu_gcorrsigrqvr|s r+residual_statsrs6 T R[[/ /4!8 < JB b DH$ %B bB!$R5 cDB#D YYr] D rQwK q6A:  Fc!fx' (RVVAbggaj(9$9QUdN$JKLSV BGGBFF11$5tTBCCF  D 9r-)r;)r$ pd.DataFramer%strreturnr)r$rrr) r$rrJdictrKrrirrr)g @) r np.ndarrayrrrrrr!rztuple[dict, dict])__doc__ __future__rnumpyrXpandasrrrTr,r6r9rrr-r+rsf!D#   F)F0[=FD"&FDV&)"->r-