锘??xml version="1.0" encoding="utf-8" standalone="yes"?>国产精品久久自在自线观看,欧美精品一区二区久久,午夜精品久久久久9999高清http://m.shnenglu.com/whspecial/category/19559.htmlzh-cnSun, 27 Oct 2013 15:52:56 GMTSun, 27 Oct 2013 15:52:56 GMT60璺ㄦ満鎴跨殑hadoop闆嗙兢http://m.shnenglu.com/whspecial/archive/2013/10/27/203940.htmlwhspecialwhspecialSun, 27 Oct 2013 15:28:00 GMThttp://m.shnenglu.com/whspecial/archive/2013/10/27/203940.htmlhttp://m.shnenglu.com/whspecial/comments/203940.htmlhttp://m.shnenglu.com/whspecial/archive/2013/10/27/203940.html#Feedback0http://m.shnenglu.com/whspecial/comments/commentRss/203940.htmlhttp://m.shnenglu.com/whspecial/services/trackbacks/203940.html

榪欐槸鏉ヨ嚜浜庨樋閲屾妧鏈槈騫村崕鐨勪竴涓垎浜紝鍥犱負(fù)鍦ㄧ櫨搴︿篃鑰冭檻榪囩被浼肩殑浜嬫儏錛屾墍浠ュ惉寰楁瘮杈冩湁鎰熸?zhèn)熷Q岃繖閲屾妸鐩稿叧鍐呭鏁寸悊涓涓嬨?/span>

棣栧厛灝婇噸鐗堟潈錛岃繕鏄妸鍘熼摼鎺ュ拰浣滆呰創(chuàng)涓婏細(xì)

http://adc.alibabatech.org/carnival/history/schedule/2013/detail/main/286?video=0

鏉ヨ嚜浜庨樋閲屽惔濞佸伐紼嬪笀鐨勫垎浜?/span>

 

棣栧厛闇瑕佽鏄庝竴鐐癸紝璺ㄦ満鎴?/span>hadoop鍙兘搴旂敤鍦烘櫙騫朵笉鏄緢澶氾紝鍥藉唴鍍?/span>BAT榪欑宸ㄥご涔熻闇瑕侊紝浣嗘槸澶ч儴鍒嗙殑涓皬鍏徃涔熻騫朵笉闇瑕佽繖涓紝涔熻榪欐槸涓睜榫欎箣鎶錛屽懙鍛點(diǎn)?/span>

鎶婅繖涓棶棰樺垎涓夋鏉ヨ錛岀涓孌墊槸闂鍑虹幇鐨勮儗鏅紝絎簩孌墊槸瑙e喅璇ラ棶棰樼殑闅劇偣錛岀涓夋鏄渶緇堢殑瑙e喅鏂規(guī)銆?/span>

錛堜竴錛?nbsp;鑳屾櫙錛?/span>

鍏堣鐪嬩笅涓轟粈涔堥渶瑕佸仛涓涓法鏈烘埧鐨勫ぇ闆嗙兢錛?/span>

澶ч泦緹ょ殑浼樼偣鍦ㄤ簬鏁版嵁綆$悊鍜屾巿鏉冨鏄擄紙榪欎釜闂鍦ㄤ竴涓閮ㄩ棬鐨勫ぇ鍏徃榪樻槸寰堥噸瑕佺殑錛夛紱璺ㄩ儴闂ㄧ殑浣跨敤鏁版嵁瀹規(guī)槗錛屾棤闇閲嶅鎷夊彇鏁版嵁銆?/span>

鍦ㄩ泦緹よ揪鍒頒竴瀹氳妯℃椂錛屽崟鏈烘埧錛堟満鎴垮唴鐨勫閲忔槸鏈夐檺鐨勶級(jí)宸茬粡鏃犳硶婊¤凍闆嗙兢鐨勯渶姹備簡(jiǎn)錛岃鎯充竴鍔蟲案閫哥殑瑙e喅闂錛岄渶瑕佸緩璁句竴涓法鏈烘埧鐨?/span>hadoop闆嗙兢銆?/span>

錛堜簩錛夋妧鏈寫鎴橈細(xì)

2.1 NameNode鐨勬ц兘闂錛?/span>

         鍦ㄧ鐞嗕竴涓法澶х殑hadoop闆嗙兢鏃訛紝鐢變簬鍘熷鐨?/span>Namenode鏄崟鑺傜偣錛屽洜姝や細(xì)鎴愪負(fù)涓涓ц兘鐡墮錛岄亣鍒扮殑鎬ц兘闂涓昏鍖呮嫭涓ゆ柟闈細(xì)瀛樺偍瀹歸噺闂錛堝瓨鍌ㄥ厓鏁版嵁錛夊拰璁$畻鍘嬪姏錛堝鐞?/span>rpc璇鋒眰錛屼慨鏀瑰唴瀛樻爲(wèi)鏃跺欓渶瑕佸叏灞閿侊級(jí)闂銆?/span>

         鍏朵腑瀛樺偍瀹歸噺闂鍙互渚濊禆鍐呭瓨鐨勫瀭鐩存墿灞曟潵瑙e喅錛屼絾鏄綆楀帇鍔涘嵈寰堥毦閫氳繃鎻愬崌紜歡鏉ヨВ鍐籌紙鍥犱負(fù)鐩墠鍘傚晢鐨勪富瑕佸彂灞曟柟鍚戞槸澶氭牳錛岃岄潪鎻愰珮涓婚錛?/span>

2.2鏈烘埧涔嬮棿鐨勭綉緇滈檺鍒訛細(xì)

         鏈烘埧涔嬮棿鐨勭綉緇滄案榪滄槸涓‖浠舵潯浠剁殑闄愬埗錛岃法鏈烘埧鐨勭綉緇滀紶杈撳甫鏉ヤ簡(jiǎn)鏁版嵁寤舵椂鍜屽甫瀹介檺鍒訛細(xì)

1錛?nbsp;寤舵椂涓鑸槸鍦?/span>10ms涔嬪唴錛岃?/span>hadoop涓婂ぇ閮ㄥ垎榪愯鐨勬槸紱葷嚎浣滀笟錛屽熀鏈彲鎺ュ彈

2錛?nbsp;甯﹀闄愬埗鐨勯棶棰樻瘮杈冨ぇ錛屽洜涓哄崟鏈烘埧鍐呯殑鐐瑰鐐瑰甫瀹戒竴鑸槸鍦?/span>1Gbps錛岃屾満鎴夸箣闂寸殑甯﹀紜湪20Mbps宸﹀彸錛岄潪甯告湁闄愩?/span>

2.3璧勬簮緇勪箣闂寸殑綆$悊

         姣忎釜閮ㄩ棬鍙互鐪嬪仛涓涓祫婧愮粍錛屽畠浠彲鑳戒細(xì)浜掔浉浣跨敤瀵規(guī)柟鐨勬暟鎹紝鍥犳濡備綍瑙勫垝璁$畻鍜屽瓨鍌ㄧ殑浣嶇疆灝卞緢閲嶈錛屽惁鍒欎細(xì)鍦ㄥ涓満鎴夸箣闂村嚭鐜板ぇ閲忕殑鏁版嵁鎷瘋礉銆?/span>

錛堜笁錛夎В鍐蟲柟妗堬細(xì)

鍏堢湅涓嬫暣涓法闆嗙兢hadoop鐨勬灦鏋勫浘錛?/span>


 

閲嶇偣浠嬬粛閲岄潰涓夌偣錛屼篃灝辨槸鍜屼笂闈笁涓棶棰樼浉瀵瑰簲鐨勶細(xì)

1錛?nbsp;鍙互鐪嬪埌榪欓噷鐢誨嚭浜?jiǎn)涓や?/span>NN錛?/span>namenode錛夛紝瀹冧滑瀹為檯涓婅繕鏄睘浜庝竴涓?/span>hadoop闆嗙兢錛岃繖鏄笟鐣岄噷鐨勪竴涓В鍐蟲柟妗堬細(xì)HDFS Fedaration錛屽畠涓轟簡(jiǎn)瑙e喅鍏冩暟鎹妭鐐規(guī)ц兘闂錛?/span>

2錛?nbsp;鍙互鐪嬪埌榪欓噷鏈変竴涓?/span>cross node鑺傜偣錛屽畠鏄敤鏉ュ湪涓や釜鏈烘埧涔嬮棿鍚屾鏁版嵁鐨勶紝瀹冪殑璁捐鑰冭檻鍒頒簡(jiǎn)鏈烘埧闂寸殑緗戠粶闄愬埗錛?/span>

3錛?nbsp;鏈鍚庢槸groupA銆?/span>groupB錛岃繖鏄負(fù)浜?jiǎn)瑙e喯x暟鎹駭鍑烘柟鍜屼嬌鐢ㄦ柟鍏崇郴鏉ョ敤鐨勩?/span>

3.1 Federation

Federation鐩稿叧璧勬枡瑙侊細(xì)

http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#HDFS_Federation


涓轟簡(jiǎn)姘村鉤鎵╁睍Namenode錛?/span>federation浣跨敤浜?jiǎn)澶氫釜浜掔浉鐙珛鐨?/span>namenode銆傚畠浠箣闂翠簰鐩鎬笉闇瑕侀氫俊錛屾瘡涓?/span>datenode闇瑕佸悜鍏ㄩ儴namenode娉ㄥ唽騫跺彂閫佷俊鎭?/span>

BlockPool鏄睘浜庝竴涓?/span>namenode鐨?/span>block闆嗗悎錛屾瘡涓?/span>blockpool涔嬮棿涔熸槸浜掔浉鐙珛鐨勩?/span>

         鍦?/span>federation閲岋紝鏈変竴涓渶瑕佸叧娉ㄧ殑闂錛屽氨鏄涓?/span>namenode鐨勫湴鍧濡備綍瀵圭敤鎴瘋繘琛岄忔槑錛熷畠閲囩敤鐨勮В鍐蟲柟妗堟槸鐩綍鏍?wèi)鎸傝浇鐨勬栆?guī)錛堢ぞ鍖烘湁涓?/span>viewFS錛屽簲璇ュ氨鏄負(fù)浜?jiǎn)瑙e唽櫩欎釜闂锛?jí)錛氱啛鎮(zhèn)?/span>linux鎴栬?/span>nfs鐨勬湅鍙嬪簲璇ラ兘鐭ラ亾mount榪欎釜姒傚康錛岀洰褰曟爲(wèi)鎸傝澆灝辨槸榪欎釜鎰忔濄?/span>

涓嶈繃浣跨敤鐩綍鏍?wèi)鎸傝浇涔熷瓨鍦ㄧ潃涓涓棶棰橈紝灝辨槸鍚勪釜瀛愮洰褰曚笅鐨勫瓨鍌ㄨ祫婧愰渶瑕佷漢涓虹殑浠嬪叆綆$悊錛屼笉鑳藉嚭鐜頒弗閲嶇殑涓嶅潎銆?/span>

3.2 crossNode

         鏈烘埧闂寸殑緗戠粶闄愬埗瑕佹眰涓嶈兘鍑虹幇澶ц妯°侀暱鏃墮棿鐨勬暟鎹嫹璐濓紝闇瑕佷竴涓笓闂ㄧ鐞嗘満鎴塊棿鏁版嵁鎷瘋礉鐨勮繘紼嬶紝鍙仛crossNode銆傚畠鏄嫭绔嬮儴緗茬殑涓涓妭鐐癸紝鍜屽厓鏁版嵁鑺傜偣鏄垎紱葷殑銆?/span>

         瀹冭兘鎻愪緵鐨勫姛鑳芥鎷潵璇翠富瑕佸寘鎷互涓嬩笁鐐癸細(xì)

a錛?nbsp;鏍規(guī)嵁棰勭疆鐨勮法鏈烘埧鏂囦歡錛岃繘琛屾暟鎹嫹璐?/span>

b錛?nbsp;澶勭悊瀹炴椂鐨勬暟鎹嫹璐濊姹?/span>

c錛?nbsp;榪涜璺ㄦ満鎴跨殑鏁版嵁嫻侀噺鎺у埗

濡備綍寰楃煡璺ㄦ満鎴挎枃浠跺垪琛紵

         鐢變簬紱葷嚎浠誨姟鍩烘湰閮芥槸瀹氭椂瑙﹀彂鐨勶紝鍙互鏍規(guī)嵁瀵瑰巻鍙蹭綔涓氱殑鍒嗘瀽鏉ュ艦鎴愪竴涓法鏈烘埧鏂囦歡鍒楄〃

3.3   璧勬簮緇勪箣闂寸殑綆$悊

鍚勪釜璧勬簮緇勪箣闂村瓨鍦ㄦ暟鎹殑渚濊禆錛屾垜浠笇鏈涢氳繃璧勬簮緇勭鐞嗭紝鑳藉疄鐜板ぇ閮ㄥ垎浠誨姟鍦ㄦ湰鏈烘埧鍐呬駭鍑烘暟鎹紝鍙湁灝戦噺璺ㄦ満鎴夸駭鍑烘暟鎹紱澶ч儴鍒嗕換鍔¤鍙栨湰鏈烘埧鐨勬暟鎹壇鏈紝鍙湁灝戦噺璺ㄦ満鎴胯鍙栨暟鎹?/span>

涓轟簡(jiǎn)鏍囪瘑璧勬簮緇勪箣闂寸殑鏁版嵁渚濊禆鎬э紝瀹氫箟涓涓祫婧愮粍涔嬮棿鐨勮窛紱繪蹇碉細(xì)涓涓祫婧愮粍璁塊棶鍙︿竴涓祫婧愮粍鐨勬暟鎹噺瓚婂錛屽垯涓よ呯殑璺濈瓚婅繎錛屽簲璇ュ皢璺濈鎺ヨ繎鐨勮祫婧愮粍鏀懼湪鍚屼竴涓満鎴垮唴銆?/span>

涓轟簡(jiǎn)璁╄綆楀拰浜у嚭灝藉彲鑳藉湴闈犺繎錛屼嬌鐢ㄤ竴涓?/span>MRProxy錛屽浜庝笉鍚岀被鍨嬬殑浠誨姟鍋氫笉鍚屽鐞嗭細(xì)

a錛?span style="font-size: 7pt; line-height: normal; font-family: 'Times New Roman';">            紱葷嚎璁$畻錛氳法鏈烘埧鍒楄〃涓殑鏁版嵁姝e湪浼犺緭涓紙DC1->DC2錛夛紝DC2涓婄殑 Job 琚殏鍋滆皟搴︼紝絳夊緟浼犺緭瀹屾瘯

b錛?span style="font-size: 7pt; line-height: normal; font-family: 'Times New Roman';">            Ad-hoc鏌ヨ錛?/span>DC2涓婄殑 Job 闇瑕佽DC1涓婄殑鏁版嵁錛?/span>Job鏆傚仠璋冨害錛岄氱煡 CrossNode錛屾暟鎹紶杈撳畬姣曞悗緇х畫璋冨害

c錛?span style="font-size: 7pt; line-height: normal; font-family: 'Times New Roman';">             鐗規(guī)畩鎯呭喌錛氳法鏈烘埧鏁版嵁 Join錛?/span>DC1澶ц〃錛?/span>DC2灝忚〃錛?/span>Job 璋冨害鍒?/span>DC1涓婏紝璺ㄦ満鎴跨洿鎺ヨ鍙?/span>DC2鏁版嵁錛屾棤闇絳夊緟

 

鐢變簬鏄牴鎹棰戝拰ppt鏁寸悊錛屽茍娌℃湁浠g爜鎴栬呮枃妗o紝鎵浠ュ彲鑳芥湁浜涘湴鏂圭殑鐞嗚В鏈夊亸宸紝嬈㈣繋鏉ユ彁鎰忚~



whspecial 2013-10-27 23:28 鍙戣〃璇勮
]]>
Dremel瀛樺偍鏍煎紡瑙f瀽http://m.shnenglu.com/whspecial/archive/2013/08/14/202546.htmlwhspecialwhspecialWed, 14 Aug 2013 15:17:00 GMThttp://m.shnenglu.com/whspecial/archive/2013/08/14/202546.htmlhttp://m.shnenglu.com/whspecial/comments/202546.htmlhttp://m.shnenglu.com/whspecial/archive/2013/08/14/202546.html#Feedback1http://m.shnenglu.com/whspecial/comments/commentRss/202546.htmlhttp://m.shnenglu.com/whspecial/services/trackbacks/202546.html    Dremel鏄痝oogle鎺ㄥ嚭鐨勫張涓紲炲櫒錛宲aper涓縐拌兘澶熷湪3s鍐呭垎鏋?PB鐨勬暟鎹紝涓昏鏄潰鍚戜氦浜掑紡鏌ヨ銆傝繖綃噋aper瀵瑰祵濂楃被鍨嬬殑瀛樺偍鏂瑰紡鏂歸潰錛屾濈淮紜疄鏈変簺璺寵穬錛岃繖綃囨枃绔犱富瑕佽璁茶繖涓紝涓鏂歸潰鏄柟渚垮悗鏉ヨ呯悊瑙o紝鍙︿竴鏂歸潰鏄鑷繁涔熸暣鐞嗕笅鎬濊礬銆?br />

    棣栧厛Dremel浣跨敤鐨勬槸鍒楀瓨妯″瀷錛屽浜庡熀鏈被鍨嬪垪瀛樿緝瀹規(guī)槗鍋氬埌錛涗絾鏄浜庡祵濂楃被鍨嬶紝Dremel涔熻兘鍋氬埌灝嗗叾鎷嗚В鎴愬熀鏈被鍨嬪茍榪涜鍒楀瓨錛岃繖鏄煎緱鎴戜滑鐮旂┒鐨勩?br />

    鐩磋鐪嬩笅宓屽綾誨瀷鎸夎瀛樺偍鍜屾媶瑙e悗鎸夊垪瀛樺偍鐨勫姣旀晥鏋滐細(xì)

    鐒跺悗瀵逛簬宓屽鏁版嵁綾誨瀷錛孌remel閲岄潰瀹氫箟浜?jiǎn)閲岄潰涓壙U嶇被鍨嬬殑瀛楁

    1錛屽繀欏誨嚭鐜?嬈¤屼笖浠呭嚭鐜?嬈$殑瀛楁錛歳equired

    2錛屽彲鑳藉嚭鐜?嬈℃垨鑰?嬈$殑瀛楁錛歰ptional

    3錛屽彲鑳藉嚭鐜?嬈℃垨鑰匩嬈″瓧孌碉細(xì)repeated

    涓嬮潰浠aper鐨勪緥瀛愭潵璁茶堪鍚э細(xì)

    鍏朵腑DocId鏄痳equired瀛楁錛屽洜姝ゅ湪r1,r2涓繀欏誨嚭鐜?嬈★紱url瀛楁鏄痮ptional瀛楁錛屽洜姝ゅ湪r1鐨勭涓変釜Name閲屾湭鍑虹幇錛屽湪r1鐨勫墠涓や釜Name閲屽嚭鐜頒簡(jiǎn)1嬈★紱Backward瀛楁鏄痳epeated瀛楁錛屽洜姝ゅ湪r1鐨凩inks閲屾湭鍑虹幇錛屽湪r2鐨凩inks閲屽嚭鐜頒簡(jiǎn)2嬈°?br />

    鐞嗚В浜?jiǎn)涓婇潰杩欎簺锛岀洿鎺ユ潵鐪嬩笅Dremel鏄庝箞鏉ュ瓨瀹冪殑鍚э細(xì)

    涓婅〃涓殑姣忔潯璁板綍閮芥湁涓や釜灞炴э紝"r"浠h〃repetition level錛?d"浠h〃definition level錛屽畾涔夊涓嬶細(xì)

    repetition level:what repeated field in the field’s path the value has repeated錛岃褰曡瀛楁鏄湪鍝釜repeated綰у埆涓婇噸澶嶇殑

    definition level:how many fields inpthat could be undefined (because they are optional or repeated) are actually present錛岃褰曡瀛楁涔嬩笂鏈夊灝戜釜optional鎴栬卹epeated瀛楁瀹為檯鏄湁鍊肩殑錛堟湰鏉ュ彲浠ヤ負(fù)null鐨勶級(jí)

    鐪嬪埌榪欓噷錛屽悇浣嶅彲鑳藉凡緇忓湪蹇?jī)閲岄粯蹇典簡(jiǎn)锛?xì)WTF錛佸埆鎬ワ紝鍙互緇撳悎涓涓緥瀛愭潵鐪嬶細(xì)

鍏堢湅repetition level錛堜笅闈互r鏇夸唬錛夛紝浠ame.Language.Code涓轟緥錛?/p>

    1)瀵圭1涓嚭鐜扮殑鍊鹼紝鍏秗濮嬬粓涓?錛屽洜姝?en-us'鐨剅涓?

    2)瀵逛簬絎?涓?en'錛屽叾涓婁竴涓兼槸'en-us'錛屽畠浠槸鍦↙anguage綰у埆鍙戠敓鐨勯噸澶嶏紝Name.Language鏄袱綰х殑repeated瀛楁錛屽洜姝涓?

    3)瀵逛簬絎?涓糿ull錛屾槸涓轟簡(jiǎn)璁板綍'en-gb'鏄嚭鐜板湪絎笁涓狽ame鑰岄潪絎簩涓狽ame閲岋紝鐗規(guī)剰鍗犱綅鐢ㄧ殑銆俷ull鐨勪笂涓涓兼槸'en'錛屽畠浠槸鍦∟ame綰у埆鍙戠敓鐨勯噸澶嶏紝鍥犳r鏄?

    4)瀵逛簬絎?涓?en-gb'錛屽叾涓婁竴涓兼槸null錛屽畠浠篃鏄湪Name綰у埆鍙戠敓鐨勯噸澶嶏紝鍥犳r鏄?

    5)瀵逛簬絎?涓糿ull錛屽叾涓婁竴涓兼槸'en-gb'錛屽畠浠嚭鐜板湪涓や釜涓嶅悓Document閲岋紝鍥犳r鏄?

    鎬葷粨涓嬶紝鐪媟epetition level娉ㄦ剰涓ょ偣錛?,鍙瘮杈冭鍊煎拰涓婁竴涓鹼紱2,鍙渶瑕佺湅榪欎袱涓肩殑閲嶅浣嶇疆涓婃湁鍑犱釜repeated瀛楁

鍐嶇湅definition level錛堜笅闈互d鏇夸唬錛夛紝涔熶互Name.Language.Code涓轟緥錛?/p>

    1)瀵逛簬'en-us'錛屽叾涓婄殑Name錛孡anguage閮藉嚭鐜頒簡(jiǎn)錛屽洜姝涓?錛堝叾瀹炲浜庨潪null鍊肩殑瀛楁錛屽叾涓婄殑optional鎴栬卹epeated瀛楁鑲畾鏄嚭鐜頒簡(jiǎn)錛屾墍浠ラ兘鏄浉鍚岀殑錛屽彧鏄痭ull瀛楁鐨刣鍊兼湁宸埆錛?br />

    2)瀵逛簬'en'錛屽悓鐞哾涔熶負(fù)2

    3)瀵逛簬null錛屽叾涓婂彧鍑虹幇浜?jiǎn)Name錛屾病鏈夊嚭鐜癓anguage錛屽洜姝涓?

    4)瀵逛簬'en-gb',d涔熶負(fù)2

    5)瀵逛簬鏈鍚庝竴涓猲ull錛屽叾涓婁篃鍙嚭鐜頒簡(jiǎn)Name錛屾病鏈夊嚭鐜癓anguage錛屽洜姝涓?


    浠ヤ笂鍙槸璁蹭簡(jiǎn)dremel鎬庝箞鍘誨瓨宓屽綾誨瀷錛岃嚦浜庤繖縐嶅瓨娉曟槸鎬庝箞鎯沖嚭鏉ョ殑錛岀湡闈炴垜杈堣兘鐞嗚В鐨勪簡(jiǎn)銆傘傘傛洿澶氬唴瀹癸紝璇峰弬鑰冨師钁梡aper鍙?qiáng)缃戜笂瑙f瀽銆?br />



whspecial 2013-08-14 23:17 鍙戣〃璇勮
]]>
Orcfile鏂囦歡鏍煎紡瑙f瀽錛?錛?/title><link>http://m.shnenglu.com/whspecial/archive/2013/08/14/202545.html</link><dc:creator>whspecial</dc:creator><author>whspecial</author><pubDate>Wed, 14 Aug 2013 15:13:00 GMT</pubDate><guid>http://m.shnenglu.com/whspecial/archive/2013/08/14/202545.html</guid><wfw:comment>http://m.shnenglu.com/whspecial/comments/202545.html</wfw:comment><comments>http://m.shnenglu.com/whspecial/archive/2013/08/14/202545.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://m.shnenglu.com/whspecial/comments/commentRss/202545.html</wfw:commentRss><trackback:ping>http://m.shnenglu.com/whspecial/services/trackbacks/202545.html</trackback:ping><description><![CDATA[<p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    涓婄瘒鏂囩珷浠庢暣浣撲粙緇嶄簡(jiǎn)Orcfile鐨勫瓨鍌ㄦ牸寮忥紝鎺ヤ笅鏉ラ噸鐐逛粙緇嶄笅Orc閲岀敤鍒扮殑鍑犵緙栫爜鏍煎紡錛?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    瀛楀吀緙栫爜錛氱敤浜嶴tring綾誨瀷鐨勫瓧孌?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Run-Length緙栫爜錛氱敤浜巌nt錛宭ong錛宻hort絳夌被鍨嬬殑緙栫爜</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Bit緙栫爜錛氬彲浠ョ敤浜庡悇縐嶆暟鎹被鍨?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><strong>1錛屽瓧鍏哥紪鐮侊細(xì)</strong></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    瀵逛簬String綾誨瀷鐨勬瘡涓瓧孌靛垎鍒繚瀛樹竴涓瓧鍏革紝璁板綍姣忎釜鍊煎湪瀛楀吀涓殑浣嶇疆錛屼繚瀛樺瓧鍏哥殑鏁版嵁緇撴瀯閲囩敤涓媯電孩榛戞爲(wèi)銆傚浜庢瘡涓猄tring瀛楁錛屾渶緇堜細(xì)鏈変笁涓緭鍑篠tream錛屽垎鍒槸StringOuptut(璁板綍瀛楀吀涓殑鍊?錛孡engthOutput(璁板綍姣忎釜瀛楀吀鍊肩殑闀垮害)錛孯owOutput(璁板綍瀛楁鍦ㄥ瓧鍏鎬腑鐨勪綅緗?銆?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鎬濊?錛氫負(fù)浠涔堣鐢ㄧ孩榛戞爲(wèi)錛?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鍥犱負(fù)綰㈤粦鏍?wèi)鏃犺鏄彃鍏ュQ屽垹闄わ紝鏌ユ壘鐨勬ц兘閮芥瘮杈冨鉤鍧囷紝閮芥槸O(logN)錛岃屼笖鏄鉤琛℃煡鎵炬爲(wèi)錛屾渶鍧忔儏鍐典篃涓嶄細(xì)閫鍖栨垚O(N)<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鎬濊?錛氬叾瀹炰竴鑸瓨鍌ㄦ椂榪樹細(xì)浣跨敤LZO涔嬬被鐨勫帇緙╋紝瀹冧滑鏈韓灝辨槸涓縐嶅瓧鍏稿帇緙╋紝涓轟粈涔圤rc閲岄潰瑕佽嚜宸卞仛瀛楀吀鍘嬬緝錛?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鍥犱負(fù)LZO涔嬬被鐨勫帇緙╃獥鍙d竴鑸瘮杈冨皬錛圠ZO榛樿鏄?4KB錛夛紝鑰孫rc鐨勫瓧鍏稿帇緙╂槸浠ユ暣涓瓧孌典負(fù)鑼冨洿鏉ュ帇緙╃殑錛屽帇緙╃巼浼?xì)鏇村ソ銆?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><strong>2錛孯un-Length緙栫爜錛?/strong><br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    瀵逛簬int,long,short綾誨瀷鐨勫瓧孌碉紝浣跨敤Run-Length緙栫爜銆傝Run-Length鑳藉瀵圭瓑宸暟鍒楋紙瀹屽叏鐩哥瓑涔熷睘浜庣瓑宸暟鍒楋級(jí)榪涜鍘嬬緝錛岃絳夊樊鏁板垪闇瑕佹弧瓚充互涓嬩袱涓潯浠訛細(xì)</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    1錛岃嚦灝戝寘鍚?涓厓绱?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    2錛屽樊鍊煎湪-128~127涔嬮棿錛堝洜涓哄樊鍊肩敤1Byte鏉ヨ〃紺猴級(jí)</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    瀵逛簬涓嶆弧瓚崇瓑宸暟鍒楃殑鏁板瓧錛孯un-Length緙栫爜涔熻兘瀛樺偍錛屼絾鏄病鏈夊帇緙╂晥鏋滐紝Run-Length鐨勫叿浣撳瓨鍌ㄥ涓嬶細(xì)</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    絎竴涓狟yte鏄疌ontrol Byte錛屽彇鍊煎湪-128~127涔嬮棿錛屽叾涓?1~-128浠h〃鍚庨潰瀛樺偍鐫1~128涓笉婊¤凍絳夊樊鏁板垪鐨勬暟瀛楋紝0~127浠h〃鍚庨潰瀛樺偍鐫3~130涓瓑宸暟鍒楃殑鏁板瓧錛?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    濡傛灉Control Byte>=0錛屽垯鍚庨潰璺熺潃涓涓狟yte瀛樺偍宸鹼紝鍚﹀垯涓嶅瓨鍌ㄨByte錛?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    濡傛灉Control Byte>=0錛屽垯鍚庨潰璺熺潃絳夊樊鏁板垪鐨勭涓涓暟錛屽惁鍒欒窡鐫-Control Byte涓暟瀛椼?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    渚嬪瓙錛?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鍘熷鏁板瓧錛?2,12,12,12,12,10,7,13</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    緇忚繃Run-Length鐨勬暟瀛楋細(xì)<span style="color: #ff0000;">2</span>,<span style="color: #ffc000;">0</span>,12,<span style="color: #ff0000;">-3</span>,10,7,13</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    綰㈣壊浠h〃Control Byte錛岄粍鑹蹭唬琛ㄥ樊鍊鹼紝榛戣壊浠h〃鍏蜂綋鐨勬暟瀛椼?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><strong>3錛孊it緙栫爜錛?/strong></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">瀵規(guī)墍鏈夌被鍨嬬殑瀛楁閮藉彲浠ラ噰鐢˙it緙栫爜鏉ヨ〃紺鴻鍊兼槸鍚︿負(fù)null銆傚湪鍐欎換浣曠被鍨嬪瓧孌典箣鍓嶏紝鍏堝垽鏂瀛楁鍊兼槸澶熶負(fù)null錛屽鏋滀負(fù)null鍒檅it鍊煎瓨涓?錛屽惁鍒欏瓨涓?錛屽浜庝負(fù)null鐨勫瓧孌靛湪瀹為檯緙栫爜鏃朵笉闇瑕佸瓨鍌ㄤ簡(jiǎn)銆傜粡榪嘊it緙栫爜涔嬪悗錛屽彲浠ュ浜?涓猙it緇勬垚涓涓狟yte錛屽啀瀵瑰叾榪涜Run-Length緙栫爜銆?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鍏跺疄闄や簡(jiǎn)榪欎笁縐嶇紪鐮佹牸寮忎箣澶栵紝Orc瀵逛簬hive鐨勫鏉傜被鍨媋rray,map,list絳夛紝灝嗗叾闄嶇淮鎴愬熀鏈被鍨嬫潵瀛樺偍錛岃繖涓篃鏄煎緱鍊熼壌鐨勶紝濡傛灉鏈夌┖涔嬪悗浼?xì)杩涜鍒嗘瀽銆?/p><img src ="http://m.shnenglu.com/whspecial/aggbug/202545.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://m.shnenglu.com/whspecial/" target="_blank">whspecial</a> 2013-08-14 23:13 <a href="http://m.shnenglu.com/whspecial/archive/2013/08/14/202545.html#Feedback" target="_blank" style="text-decoration:none;">鍙戣〃璇勮</a></div>]]></description></item><item><title>Orcfile鏂囦歡鏍煎紡瑙f瀽錛?錛?/title><link>http://m.shnenglu.com/whspecial/archive/2013/08/14/202544.html</link><dc:creator>whspecial</dc:creator><author>whspecial</author><pubDate>Wed, 14 Aug 2013 15:12:00 GMT</pubDate><guid>http://m.shnenglu.com/whspecial/archive/2013/08/14/202544.html</guid><wfw:comment>http://m.shnenglu.com/whspecial/comments/202544.html</wfw:comment><comments>http://m.shnenglu.com/whspecial/archive/2013/08/14/202544.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://m.shnenglu.com/whspecial/comments/commentRss/202544.html</wfw:commentRss><trackback:ping>http://m.shnenglu.com/whspecial/services/trackbacks/202544.html</trackback:ping><description><![CDATA[<p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Orcfile(<em>Optimized Row Columnar</em>)鏄痟ive 0.11鐗堥噷寮曞叆鐨勬柊鐨勫瓨鍌ㄦ牸寮忥紝鏄涔嬪墠鐨凴CFile瀛樺偍鏍煎紡鐨勪紭鍖栥傚啓榪欎釜鐨勫摜浠潵鑷簬HortonWorks錛屼唬鐮佸啓鐨勫緢涓嶉敊錛屾瘮涔嬪墠鐨剅cfile寮哄浜?jiǎn)锛堟嵁璇磖cfile鏄釜涓闄㈢殑绔ラ瀷璺戝幓facebook鍐欑殑錛岀湅鏉ヤ腑鍥界殑璁$畻鏈烘暀鑲叉按騫寵繕鏄湁闄愬晩銆傘傘傚洤錛岃窇棰樹簡(jiǎn)錛?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鍏堜粙緇嶄笅Orc鐨勬枃浠舵牸寮忥紝鎴竴寮犲畼鏂圭殑鍥撅細(xì)<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><a target="_blank" data_ue_src="http://blog.51cto.com/attachment/201306/234709762.png"><img src="http://blog.51cto.com/attachment/201306/234709762.png" title="OrcFileLayout.png" data_ue_src="http://blog.51cto.com/attachment/201306/234709762.png" alt="" /></a></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鍙互鐪嬪埌姣忎釜Orc鏂囦歡鐢?涓垨澶氫釜stripe緇勬垚錛屾瘡涓猻tripe250MB澶у皬錛岃繖涓猄tripe瀹為檯鐩稿綋浜庝箣鍓嶇殑rcfile閲岀殑RowGroup姒傚康錛屼笉榪囧ぇ灝忕敱4MB->250MB錛岃繖鏍峰簲璇ヨ兘鎻愬崌欏哄簭璇葷殑鍚炲悙鐜囥傛瘡涓猄tripe閲屾湁涓夐儴鍒嗙粍鎴愶紝鍒嗗埆鏄疘ndex Data,Row Data,Stripe Footer錛?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    1錛孖ndex Data錛氫竴涓交閲忕駭鐨刬ndex錛岄粯璁ゆ槸姣忛殧1W琛屽仛涓涓儲(chǔ)寮曘傝繖閲屽仛鐨勭儲(chǔ)寮曞簲璇ュ彧鏄褰曟煇琛岀殑鍚勫瓧孌靛湪Row Data涓殑offset錛屾嵁璇磋繕鍖呮嫭姣忎釜Column鐨刴ax鍜宮in鍊鹼紝鍏蜂綋娌$粏鐪嬩唬鐮併?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    2錛孯ow Data錛氬瓨鐨勬槸鍏蜂綋鐨勬暟鎹紝鍜孯Cfile涓鏍鳳紝鍏堝彇閮ㄥ垎琛岋紝鐒跺悗瀵硅繖浜涜鎸夊垪榪涜瀛樺偍銆備笌RCfile涓嶅悓鐨勫湴鏂瑰湪浜庢瘡涓垪榪涜浜?jiǎn)缂栫爜锛屽垎鎴愬涓猄tream鏉ュ瓨鍌紝鍏蜂綋濡備綍緙栫爜鍦ㄤ笅涓綃囪В鏋愰噷浼?xì)璁层?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    3錛孲tripe Footer錛氬瓨鐨勬槸鍚勪釜Stream鐨勭被鍨嬶紝闀垮害絳変俊鎭?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    姣忎釜鏂囦歡鏈変竴涓狥ile Footer錛岃繖閲岄潰瀛樼殑鏄瘡涓猄tripe鐨勮鏁幫紝姣忎釜Column鐨勬暟鎹被鍨嬩俊鎭瓑錛涙瘡涓枃浠剁殑灝鵑儴鏄竴涓狿ostScript錛岃繖閲岄潰璁板綍浜?jiǎn)鏁翠釜鏂囦欢鐨勫帇缇兙c誨瀷浠ュ強(qiáng)FileFooter鐨勯暱搴︿俊鎭瓑銆傚湪璇誨彇鏂囦歡鏃訛紝浼?xì)seek鍒版枃浠跺熬閮ㄨPostScript錛屼粠閲岄潰瑙f瀽鍒癋ile Footer闀垮害錛屽啀璇籉ileFooter錛屼粠閲岄潰瑙f瀽鍒板悇涓猄tripe淇℃伅錛屽啀璇誨悇涓猄tripe錛屽嵆浠庡悗寰鍓嶈銆?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鎺ヤ笅鏉ョ湅涓婳Rcfile鐩稿浜嶳Cfile鍋氫簡(jiǎn)鍝簺鏀硅繘錛屼粠Orc浣滆呯殑ppt閲屾埅浜?jiǎn)寮犲浘锛屽垎鍒В閲婁笅鍚勮锛?xì)<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><a target="_blank" data_ue_src="http://blog.51cto.com/attachment/201306/003840639.png"><img src="http://blog.51cto.com/attachment/201306/003840639.png" title="鍥劇墖1.png" width="500" height="345" border="0" hspace="0" vspace="0" data_ue_src="http://blog.51cto.com/attachment/201306/003840639.png" style="width: 500px; height: 345px;" alt="" /></a></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Hive type model:RCfile鍦ㄥ簳灞傚瓨鍌ㄦ椂涓嶄繚瀛樼被鍨嬶紝閮藉綋鍋欱yte嫻佹潵瀛樺偍</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Separtor complex columns:Orc灝嗗鏉傜被鍨嬫媶寮瀛樺偍</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Splits Found Quickly錛氫笉寰堢悊瑙?br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Default Column group size錛氫笉鐢ㄨВ閲婁簡(jiǎn)</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Files per a bucket錛氫笉寰堢悊瑙?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Store min錛宮ax錛宑ount錛宻um錛氬瓨浜?jiǎn)杩欎簺渚夸簬蹇熷湴skip鎺変竴涓猻tripe</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Versioned metadata:涓嶅緢鐞嗚В<br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Run-Length Data-coding錛氭暣鏁扮被鍨嬪仛Run-Length鍙橀暱緙栫爜</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Store Strings in dictionary錛歋tring綾誨瀷鍋氬瓧鍏哥紪鐮?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Store Row Count錛氭瘡涓猄tripe浼?xì)瀛樺偍琛屾?/p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Skip Compressed blocks:鍙互鐩存帴skip鎺夊帇緙╄繃鐨刡lock</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    Store internal indexes:瀛樺偍浜?jiǎn)涓涓交閲忕駭鐨刬ndex</p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;"><br /></p><p style="margin-right: 0px; margin-left: 0px; font-family: sans-serif; font-size: 16px; line-height: normal;">    鏁翠釜Orc鐪嬩笅鏉ワ紝浠g爜鍐欑殑榪樻槸姣旇緝娓呮櫚鏄庝簡(jiǎn)鐨勶紝鑰屼笖鎴戜滑涔熻繘琛屼簡(jiǎn)嫻嬭瘯錛屽帇緙╂晥鏋滄瘮RCfile鎻愬崌浜?jiǎn)涓崬畱锛屾湁鍏喘懀鐨勬湅鍙嬪彲浠ユ潵鐪嬩笅锛屼箣鍚庝細(xì)鍐櫧W簩綃囪В鏋愶紝涓昏鏄Orc鐢ㄥ埌鐨勫嚑縐嶇紪鐮佹牸寮忋?br /></p><div></div><img src ="http://m.shnenglu.com/whspecial/aggbug/202544.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://m.shnenglu.com/whspecial/" target="_blank">whspecial</a> 2013-08-14 23:12 <a href="http://m.shnenglu.com/whspecial/archive/2013/08/14/202544.html#Feedback" target="_blank" style="text-decoration:none;">鍙戣〃璇勮</a></div>]]></description></item><item><title>鍏充簬map/reduce鐨刢ombiner榪愯鏃舵満鐨勯棶棰?/title><link>http://m.shnenglu.com/whspecial/archive/2012/11/06/194819.html</link><dc:creator>whspecial</dc:creator><author>whspecial</author><pubDate>Tue, 06 Nov 2012 15:52:00 GMT</pubDate><guid>http://m.shnenglu.com/whspecial/archive/2012/11/06/194819.html</guid><wfw:comment>http://m.shnenglu.com/whspecial/comments/194819.html</wfw:comment><comments>http://m.shnenglu.com/whspecial/archive/2012/11/06/194819.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://m.shnenglu.com/whspecial/comments/commentRss/194819.html</wfw:commentRss><trackback:ping>http://m.shnenglu.com/whspecial/services/trackbacks/194819.html</trackback:ping><description><![CDATA[<blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"></blockquote><span style="font-size: 12pt;">   </span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">map/reduce</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鐨?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combiner</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鍒板簳鍦ㄤ粈涔堟椂鍊欒繍琛岋紵</span> <p align="left" style="text-indent: 21pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鍦ㄧ綉涓婂ぇ澶氭暟璧勬枡涓紝閮芥槸璇?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combiner</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鍦?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">map</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">绔繍琛岋紝鍙戠敓鍦?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">map</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">杈撳嚭鏁版嵁涔嬪悗錛岀粡榪?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combiner</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鍐嶄紶閫掔粰</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">reducer</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">銆備絾鏄箣鍓嶅湪宸ヤ綔涓嚭鐜扮殑涓涓棶棰樺鑷存垜鍙戠幇鍘熸潵</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combiner</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">灞呯劧涔熶細(xì)鍦?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">reducer</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">绔繍琛岋紝騫朵笖浼?xì)澶殝啤杩愯銆?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;"><br /> </span><span style="font-size: 8pt; line-height: 150%; font-family: Verdana, sans-serif;">   </span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鍦ㄧ綉涓婃煡浜?jiǎn)涔嬪悗鍙戠帋图寴q欐槸</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">hadoop-0.18</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鐗堟湰寮曞叆鐨勬柊</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">feature</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">錛?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;"><br /> </span><span style="font-size: 8pt; line-height: 150%; font-family: Verdana, sans-serif;">   </span><span style="font-size:12.0pt;line-height:150%;font-family:"Arial","sans-serif";color:#333333;background:white;">Changed policy for running combiner. The combiner may be run multiple times as the map's output is sorted and merged. Additionally, it may be run on the reduce side as data is merged. The old semantics are available in Hadoop 0.18 if the user calls: job.setCombineOnlyOnce(true)</span><span style="font-size:12.0pt;line-height:150%;font-family:瀹嬩綋;color:#333333; background:white;">銆?/span><span style="font-size: 12pt; line-height: 150%; font-family: Verdana, sans-serif;"><br /> </span><span style="font-size: 8pt; line-height: 150%; font-family: Verdana, sans-serif;">   </span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">瀹為檯涓?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combiner</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">浼?xì)鍦?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">mapper</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">绔拰</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">reducer</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">绔垎鍒繍榪愯錛岀湅浜?jiǎn)涓嬩唬鐮佸Q屽彂鐢?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combine</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鐨勬椂鏈哄湪浠ヤ笅錛?/span><span style="font-size: 8pt; line-height: 150%; font-family: Verdana, sans-serif;"><br /> </span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">   1</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">錛?/span> <span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鍦?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">mapper</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">绔殑</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">spill</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">闃舵錛屽湪緙撳瓨涓殑璁板綍瓚呰繃闃堝兼椂浼?xì)杩涜?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combine</span></p> <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">if (spstart != spindex) {</span></p> <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">         …</span></p> <p align="left" style="margin: 2.9pt 0cm; text-indent: 18pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combineAndSpill(kvIter, combineInputCounter);</span></p> <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">}</span></p> <p align="left" style="margin: 2.9pt 0cm 2.9pt 18pt; text-indent: -18pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">2</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">錛?/span> <span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鍦?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">mapper</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">绔殑</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">merge</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">闃舵錛岃繘琛?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">merge</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鐨?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">spill</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鏂囦歡鏁扮洰</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">>=3</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鏃朵細(xì)榪涜</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combine</span></p> <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">if (null == combinerClass || numSpills < minSpillsForCombine) {</span></p> <p align="left" style="margin: 2.9pt 0cm; text-indent: 21pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">Merger.writeFile(kvIter, writer, reporter);</span></p> <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">} else {</span></p> <p align="left" style="margin: 2.9pt 0cm; text-indent: 21pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combineCollector.setWriter(writer);</span></p> <p align="left" style="margin: 2.9pt 0cm; text-indent: 21pt; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combineAndSpill(kvIter, combineInputCounter);</span></p> <p align="left" style="margin: 2.9pt 0cm; line-height: 150%;"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">}</span></p> <p style="line-height:150%"><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">3</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">錛?/span> <span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">鍦?/span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">reducer</span><span style="font-size: 12pt; line-height: 150%; font-family: 瀹嬩綋;">绔紝涓瀹氫細(xì)榪涜</span><span style="font-size: 12pt; line-height: 150%; font-family: Arial, sans-serif;">combine</span></p><span style="font-size: 12pt; font-family: Arial; line-height: 17px; background-color: #ffffff; "><br /></span><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"></blockquote><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"></blockquote><div class="uea8a6c" id="isChromeWebToolbarDiv" style="display:none"></div> <div class="w0mw8ya" id="isChromeWebToolbarDiv" style="display:none"></div> <div class="0aikg0o" id="isChromeWebToolbarDiv" style="display:none"></div><img src ="http://m.shnenglu.com/whspecial/aggbug/194819.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://m.shnenglu.com/whspecial/" target="_blank">whspecial</a> 2012-11-06 23:52 <a href="http://m.shnenglu.com/whspecial/archive/2012/11/06/194819.html#Feedback" target="_blank" style="text-decoration:none;">鍙戣〃璇勮</a></div>]]></description></item></channel></rss> <footer> <div class="friendship-link"> <p>感谢您访问我们的网站,您可能还对以下资源感兴趣:</p> <a href="http://m.shnenglu.com/" title="精品视频久久久久">精品视频久久久久</a> <div class="friend-links"> </div> </div> </footer> <a href="http://www.cq81.cn" target="_blank">久久久噜噜噜久久中文字幕色伊伊 </a>| <a href="http://www.tjywmc.cn" target="_blank">77777亚洲午夜久久多人</a>| <a href="http://www.gaohuirong.cn" target="_blank">久久精品无码一区二区三区日韩 </a>| <a href="http://www.rhtyyls.cn" target="_blank">少妇人妻88久久中文字幕</a>| <a href="http://www.qian-mi.cn" target="_blank">欧美大香线蕉线伊人久久</a>| <a href="http://www.x6844.cn" target="_blank">99久久精品国内</a>| <a href="http://www.fmlw.net.cn" target="_blank">四虎国产精品成人免费久久</a>| <a href="http://www.batlive.cn" target="_blank">一本色道久久88精品综合</a>| <a href="http://www.ssc695.cn" target="_blank">国产成人精品久久亚洲高清不卡 </a>| <a href="http://www.vueq.cn" target="_blank">久久伊人五月丁香狠狠色</a>| <a href="http://www.xxupng.cn" target="_blank">久久久久99精品成人片试看</a>| <a href="http://www.ru87.cn" target="_blank">国产A级毛片久久久精品毛片</a>| <a href="http://www.mmmbbb.cn" target="_blank">国产精品久久久久久久久软件</a>| <a href="http://www.phatmedia.cn" target="_blank">国产麻豆精品久久一二三</a>| <a href="http://www.txt115.cn" target="_blank">久久综合九色综合欧美就去吻</a>| <a href="http://www.jxsrgh.com.cn" target="_blank">97精品伊人久久久大香线蕉</a>| <a href="http://www.ptrjmfv.cn" target="_blank">久久国产乱子伦精品免费午夜</a>| <a href="http://www.shqidao.cn" target="_blank">东方aⅴ免费观看久久av</a>| <a href="http://www.linan521.cn" target="_blank">国产成人精品久久</a>| <a href="http://www.yc-shop.cn" target="_blank">久久777国产线看观看精品</a>| <a href="http://www.gameforpeace.cn" target="_blank">亚洲欧美一区二区三区久久</a>| <a href="http://www.blog91.cn" target="_blank">国内精品久久久久久久影视麻豆</a>| <a href="http://www.shdad.cn" target="_blank">午夜精品久久久久久99热</a>| <a href="http://www.vstmall.com.cn" target="_blank">亚洲精品无码久久久</a>| <a href="http://www.manini.cn" target="_blank">7国产欧美日韩综合天堂中文久久久久 </a>| <a href="http://www.songyufen.cn" target="_blank">avtt天堂网久久精品</a>| <a href="http://www.allywll-800.cn" target="_blank">偷偷做久久久久网站</a>| <a href="http://www.reeway.cn" target="_blank">欧美一级久久久久久久大片</a>| <a href="http://www.hch1989.cn" target="_blank">国产精品熟女福利久久AV</a>| <a href="http://www.veixzew.cn" target="_blank">国产精品青草久久久久婷婷 </a>| <a href="http://www.7708.com.cn" target="_blank">久久婷婷午色综合夜啪</a>| <a href="http://www.9503miwang.cn" target="_blank">久久精品国产99久久丝袜</a>| <a href="http://www.ftfygs.cn" target="_blank">中文字幕久久欲求不满</a>| <a href="http://www.a0037.cn" target="_blank">一级做a爱片久久毛片</a>| <a href="http://www.sgcam.com.cn" target="_blank">国产精品久久久久久福利漫画</a>| <a href="http://www.22980000.cn" target="_blank">久久精品亚洲一区二区三区浴池 </a>| <a href="http://www.9258bbs.cn" target="_blank">欧美精品一区二区久久</a>| <a href="http://www.fangongdi.cn" target="_blank">国产AⅤ精品一区二区三区久久 </a>| <a href="http://www.wirja9.cn" target="_blank">精品久久久久一区二区三区</a>| <a href="http://www.suzhougarden.cn" target="_blank">好属妞这里只有精品久久</a>| <a href="http://www.fkmlt.cn" target="_blank">99久久精品日本一区二区免费 </a>| <script> (function(){ var bp = document.createElement('script'); var curProtocol = window.location.protocol.split(':')[0]; if (curProtocol === 'https') { bp.src = 'https://zz.bdstatic.com/linksubmit/push.js'; } else { bp.src = 'http://push.zhanzhang.baidu.com/push.js'; } var s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(bp, s); })(); </script> </body>