青青草原综合久久大伊人导航_色综合久久天天综合_日日噜噜夜夜狠狠久久丁香五月_热久久这里只有精品

O(1) 的小樂

Job Hunting

公告

記錄我的生活和工作。。。
<2025年9月>
31123456
78910111213
14151617181920
21222324252627
2829301234
567891011

統(tǒng)計(jì)

  • 隨筆 - 182
  • 文章 - 1
  • 評(píng)論 - 41
  • 引用 - 0

留言簿(10)

隨筆分類(70)

隨筆檔案(182)

文章檔案(1)

如影隨形

搜索

  •  

最新隨筆

最新評(píng)論

閱讀排行榜

評(píng)論排行榜

Kullback–Leibler divergence KL散度

In probability theory and information theory, the Kullback–Leibler divergence[1][2][3] (also information divergence,information gain, relative entropy, or KLIC) is a non-symmetric measure of the difference between two probability distributions P and Q. KL measures the expected number of extra bits required to code samples from P when using a code based on Q, rather than using a code based on P. Typically P represents the "true" distribution of data, observations, or a precise calculated theoretical distribution. The measure Q typically represents a theory, model, description, or approximation of P.

Although it is often intuited as a distance metric, the KL divergence is not a true metric – for example, the KL from P to Q is not necessarily the same as the KL from Q to P.

KL divergence is a special case of a broader class of divergences called f-divergences. Originally introduced by Solomon Kullbackand Richard Leibler in 1951 as the directed divergence between two distributions, it is not the same as a divergence incalculus. However, the KL divergence can be derived from the Bregman divergence.

 

 

注意P通常指數(shù)據(jù)集,我們已有的數(shù)據(jù)集,Q表示理論結(jié)果,所以KL divergence 的物理含義就是當(dāng)用Q來編碼P中的采樣時(shí),比用P來編碼P中的采用需要多用的位數(shù)!

 

KL散度,也有人稱為KL距離,但是它并不是嚴(yán)格的距離概念,其不滿足三角不等式

 

KL散度是不對(duì)稱的,當(dāng)然,如果希望把它變對(duì)稱,

Ds(p1, p2) = [D(p1, p2) + D(p2, p1)] / 2

 

下面是KL散度的離散和連續(xù)定義!

D_{\mathrm{KL}}(P\|Q) = \sum_i P(i) \log \frac{P(i)}{Q(i)}. \!

D_{\mathrm{KL}}(P\|Q) = \int_{-\infty}^\infty p(x) \log \frac{p(x)}{q(x)} \; dx, \!

注意的一點(diǎn)是p(x) 和q(x)分別是pq兩個(gè)隨機(jī)變量的PDF,D(P||Q)是一個(gè)數(shù)值,而不是一個(gè)函數(shù),看下圖!

 

注意:KL Area to be Integrated!

 

File:KL-Gauss-Example.png

 

KL 散度一個(gè)很強(qiáng)大的性質(zhì):

The Kullback–Leibler divergence is always non-negative,

D_{\mathrm{KL}}(P\|Q) \geq 0, \,

a result known as , with DKL(P||Q) zero if and only if P = Q.

 

計(jì)算KL散度的時(shí)候,注意問題是在稀疏數(shù)據(jù)集上KL散度計(jì)算通常會(huì)出現(xiàn)分母為零的情況!

 

 

Matlab中的函數(shù):KLDIV給出了兩個(gè)分布的KL散度

Description

KLDIV Kullback-Leibler or Jensen-Shannon divergence between two distributions.

KLDIV(X,P1,P2) returns the Kullback-Leibler divergence between two distributions specified over the M variable values in vector X. P1 is a length-M vector of probabilities representing distribution 1, and P2 is a length-M vector of probabilities representing distribution 2. Thus, the probability of value X(i) is P1(i) for distribution 1 and P2(i) for distribution 2. The Kullback-Leibler divergence is given by:

   KL(P1(x),P2(x)) = sum[P1(x).log(P1(x)/P2(x))]

If X contains duplicate values, there will be an warning message, and these values will be treated as distinct values. (I.e., the actual values do not enter into the computation, but the probabilities for the two duplicate values will be considered as probabilities corresponding to two unique values.) The elements of probability vectors P1 and P2 must each sum to 1 +/- .00001.

A "log of zero" warning will be thrown for zero-valued probabilities. Handle this however you wish. Adding 'eps' or some other small value to all probabilities seems reasonable. (Renormalize if necessary.)

KLDIV(X,P1,P2,'sym') returns a symmetric variant of the Kullback-Leibler divergence, given by [KL(P1,P2)+KL(P2,P1)]/2. See Johnson and Sinanovic (2001).

KLDIV(X,P1,P2,'js') returns the Jensen-Shannon divergence, given by [KL(P1,Q)+KL(P2,Q)]/2, where Q = (P1+P2)/2. See the Wikipedia article for "Kullback–Leibler divergence". This is equal to 1/2 the so-called "Jeffrey divergence." See Rubner et al. (2000).

EXAMPLE: Let the event set and probability sets be as follow:
   X = [1 2 3 3 4]';
   P1 = ones(5,1)/5;
   P2 = [0 0 .5 .2 .3]' + eps;
Note that the event set here has duplicate values (two 3's). These will be treated as DISTINCT events by KLDIV. If you want these to be treated as the SAME event, you will need to collapse their probabilities together before running KLDIV. One way to do this is to use UNIQUE to find the set of unique events, and then iterate over that set, summing probabilities for each instance of each unique event. Here, we just leave the duplicate values to be treated independently (the default):
   KL = kldiv(X,P1,P2);
   KL =
        19.4899

Note also that we avoided the log-of-zero warning by adding 'eps' to all probability values in P2. We didn't need to renormalize because we're still within the sum-to-one tolerance.

REFERENCES:
1) Cover, T.M. and J.A. Thomas. "Elements of Information Theory," Wiley, 1991.
2) Johnson, D.H. and S. Sinanovic. "Symmetrizing the Kullback-Leibler distance." IEEE Transactions on Information Theory (Submitted).
3) Rubner, Y., Tomasi, C., and Guibas, L. J., 2000. "The Earth Mover's distance as a metric for image retrieval." International Journal of Computer Vision, 40(2): 99-121.
4) <a href="
http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence"&gt;Kullback–Leibler divergence</a>. Wikipedia, The Free Encyclopedia.

posted on 2010-10-16 15:04 Sosi 閱讀(10034) 評(píng)論(2)  編輯 收藏 引用 所屬分類: Taps in Research

評(píng)論

# re: Kullback&ndash;Leibler divergence KL散度 2010-11-30 16:17 tintin0324

博主,本人的研究方向需要了解kl距離,有些問題想請教下,怎么聯(lián)系呢?

# re: Kullback&ndash;Leibler divergence KL散度 2010-12-05 22:37 Sosi

@tintin0324
KL 距離本身很簡單,如果就是那樣子定義的,意義也如上面所說。。如果你想深入了解的話,可以讀以下相關(guān)文獻(xiàn)
統(tǒng)計(jì)系統(tǒng)
青青草原综合久久大伊人导航_色综合久久天天综合_日日噜噜夜夜狠狠久久丁香五月_热久久这里只有精品
  • <ins id="pjuwb"></ins>
    <blockquote id="pjuwb"><pre id="pjuwb"></pre></blockquote>
    <noscript id="pjuwb"></noscript>
          <sup id="pjuwb"><pre id="pjuwb"></pre></sup>
            <dd id="pjuwb"></dd>
            <abbr id="pjuwb"></abbr>
            久久米奇亚洲| 久久精品一区二区三区不卡| 尤物九九久久国产精品的特点| 亚洲伦理一区| 亚洲欧洲一区二区三区| 久久精品九九| 久久国内精品自在自线400部| 欧美性事在线| 亚洲伦伦在线| 一区二区三区高清| 欧美激情一区二区久久久| 欧美成人一区二区三区在线观看| 国产一区二区| 新狼窝色av性久久久久久| 羞羞色国产精品| 国产精品一二三四| 野花国产精品入口| 亚洲视频第一页| 欧美理论视频| 亚洲免费成人| 亚洲一区日韩在线| 国产精品分类| 亚洲欧美中文另类| 久久久福利视频| 黄网动漫久久久| 久久综合五月天婷婷伊人| 麻豆精品精华液| 亚洲精品国产品国语在线app | 欧美另类久久久品| 亚洲第一天堂av| 亚洲人午夜精品| 欧美日韩国产电影| 一区二区三区视频在线观看| 亚洲一区二区日本| 国产女人精品视频| 久久久精品网| 亚洲精品国产精品国自产观看浪潮| 日韩小视频在线观看| 欧美日韩综合不卡| 欧美一级久久久| 欧美成人亚洲成人| 亚洲一区二区精品| 国产午夜精品一区二区三区视频| 久久精品视频免费| 亚洲韩日在线| 欧美在线短视频| 在线看欧美日韩| 欧美日韩国产综合网| 亚洲人成亚洲人成在线观看图片 | 欧美肥婆bbw| 99re热精品| 国产乱码精品| 裸体歌舞表演一区二区 | 亚洲激情不卡| 午夜久久久久久久久久一区二区| 狠狠色狠狠色综合日日91app| 欧美jizzhd精品欧美巨大免费| 中文国产成人精品| 免费久久99精品国产| 亚洲伊人伊色伊影伊综合网| 激情自拍一区| 欧美小视频在线| 麻豆成人在线播放| 亚洲欧美另类国产| 亚洲日韩成人| 美女日韩欧美| 亚洲欧美日韩在线一区| 亚洲国产日韩一区| 国产欧美精品一区| 欧美日韩少妇| 男女视频一区二区| 欧美一区二区三区男人的天堂 | 欧美精品大片| 久久免费黄色| 欧美一级淫片aaaaaaa视频| 亚洲蜜桃精久久久久久久| 农夫在线精品视频免费观看| 久久在线免费观看视频| 国产精品99久久久久久白浆小说| 亚洲第一区在线观看| 久久久99国产精品免费| 亚洲综合久久久久| 日韩一区二区精品葵司在线| 一区二区在线免费观看| 国产欧美日韩精品丝袜高跟鞋 | 老牛影视一区二区三区| 欧美一级大片在线观看| 亚洲一区二区视频在线观看| 亚洲人体一区| 亚洲国产一区二区a毛片| 极品尤物一区二区三区| 国产一级精品aaaaa看| 国产欧美日韩三级| 国产精品一级久久久| 国产精品男女猛烈高潮激情| 欧美日韩国产精品自在自线| 欧美国产亚洲精品久久久8v| 久久久久久久999| 久久国产婷婷国产香蕉| 午夜久久一区| 性欧美video另类hd性玩具| 亚洲一区国产| 亚洲欧美日韩在线一区| 欧美一区二区三区四区高清| 亚洲欧美乱综合| 欧美一区二区三区啪啪| 久久精品视频导航| 久久综合九色综合久99| 久久夜精品va视频免费观看| 久久先锋影音av| 久久综合伊人77777麻豆| 久久亚洲捆绑美女| 欧美成人在线影院| 欧美精品 国产精品| 欧美另类高清视频在线| 欧美视频一区二区三区| 国产精品久久久一区麻豆最新章节| 国产精品久久久久9999吃药| 国产欧美精品一区aⅴ影院| 国产亚洲欧洲| 亚洲国产一区二区视频 | 91久久精品国产91久久| 99精品国产热久久91蜜凸| 亚洲午夜精品视频| 欧美一区二区三区在| 老司机精品视频网站| 欧美激情一区三区| 一本色道久久加勒比精品 | 亚洲小视频在线观看| 欧美一区二区在线免费观看| 久久永久免费| 欧美午夜视频在线观看| 国产一区成人| 亚洲伦理在线| 久久国产精品亚洲va麻豆| 免费一级欧美片在线观看| 亚洲精品国产精品久久清纯直播| 一区二区三区四区国产精品| 久久精品国产99国产精品| 欧美激情第10页| 国产麻豆精品在线观看| 亚洲高清成人| 性欧美videos另类喷潮| 欧美国产日韩一区二区| 国产精品99久久久久久久vr | 欧美国产亚洲视频| 欧美在线一二三区| 欧美黄色视屏| 欧美一级淫片播放口| 欧美日韩国产免费| 韩日精品在线| 亚洲欧美日韩国产综合| 欧美高清在线一区| 欧美一区二区三区男人的天堂 | 99国内精品久久| 久久免费精品视频| 正在播放亚洲一区| 欧美成人黑人xx视频免费观看| 国产精品一区二区久久精品| 亚洲久久在线| 欧美福利一区二区三区| 性色av一区二区三区红粉影视| 欧美日韩二区三区| 亚洲激情午夜| 久久天天狠狠| 亚洲一区二区三区四区在线观看| 欧美gay视频| 尤物视频一区二区| 久久精品国产精品亚洲精品| 一区二区三区欧美在线| 欧美久久久久久| 亚洲国产精品久久91精品| 久久久人人人| 亚洲欧美国产日韩中文字幕| 欧美午夜在线一二页| 日韩午夜精品| 亚洲激情在线播放| 久久影院午夜论| 一区二区视频免费完整版观看| 久久大逼视频| 亚洲欧美一区二区在线观看| 国产精品久久久久高潮| 亚洲免费视频一区二区| 99在线精品视频| 欧美日韩一区二区三区在线 | 国产精品无码永久免费888| 正在播放亚洲一区| 日韩视频一区二区三区在线播放| 欧美电影免费观看高清完整版| 亚洲第一级黄色片| 欧美国产日本高清在线| 久久夜色精品国产| 亚洲欧洲精品成人久久奇米网| 欧美凹凸一区二区三区视频| 久久久久久网站| 在线日韩欧美| 亚洲成在线观看| 在线性视频日韩欧美| 欧美日一区二区在线观看| 亚洲一区二区av电影|