信息技术-人工智能系列之一从RNN到ChatGPT：大模型的发展与应用-东北证券[]-20231114【58页】正式版.docx

Ez6-÷y2023-ll-14“ 300 淋-20231103yri Transformer yx÷ -y k GPT-20231030:2023 Ot> ugt>yRce子;-20231027:p÷i6BND ;-20231026tzvcyGvgEn ÷ySO55O521100001 021-61002390 wangqi_5636Ez/NyIRNNrChatGPTykNP-yyyWONFBTWybnaoyjyyDUWDnyyyyewiv×IY2ik_Iao6o_ysoINnOyvwyoNoW÷luotw×UUyt>g2oV)n0VyyuV÷ikoao×l×yUPyi0iW÷IoP-o2IiuWkrgyDVCVoCNuIyt1:NOtkIryIoe6)u_；kaoUyO_iyf)newivoyotuewiU3veUry2Iyiyyy|?oyt)ik：zyPwt>yYg)yY0;UBERTyIeioOvrw×tiyynyoo×lUIgogyygbU-UiyoChatGPT2oIuIytkyyrlyl1×1VIeNUiOw×PoyV÷ikU-yoOUniuUYy-IyIkakaoeyiUiGPT-4Vkak-ie-kak662gyoV÷IikW÷toPyU2a<<IikBO_RW÷l5<iu9uvW“人aV÷lbyol÷Tur-WyiONaoO_÷iugm÷6W2ikNPkt>rNnY6|，U2IyDnUewivaoOioykOIy-veUrgewiyyW÷I-g-2IaoOryyYr×t>yk6i0iNyoyU2oW÷loyikPi6UtuovOV÷m÷yy-PyoU÷Iu6Uo2ogywaoNoU_WysiyiikNgYyi3×0Umoc2t>yykymPyyPoyi×6uoy0ieoge62东北证券股份有限公司.NORTHEASTSECURITIESCO.,LTN91. ikvUao641.1. _44IRNNrChaiGPT41.2. eP_ik41.2.1. yyy)-RNN61.2.2. W>>b-LSTMNyyyWS-GRU81.2.3. y-ey-÷H1.3. ioOv121.3.1. ioOv121.3.2. Transformer÷14'y!k201.4.L'yluk201.3.3. N0os'yiuk201.3.4. 'ykyGPT211.3.5. ykryBERT221.4. iky-yio241.4.1. Oy'yik2415.2.n-Prompt251.5.3. ik8o-yio44GPT-3.5UChaiGPT281.5.4. k-yioyUgb331.6. _yUO352. ikPN372.5. ikf>372.6. kY432.7. kak443. ikW÷IPN-o493.5. ioo×NyyU493.1.1. kuu×493.1.2. ÷Iuu×503.1.3. eIIouv5132.oV÷NVyU523.2.1.oNuo523.22ouoV÷v523.3. iyrNiyU533.3.1. iyr533.3.2. iyiU543.3.3. iy|)Ndebug543.4. og6yU554. f>yo56I)IWDkB5D2yUBeD*ea5D3，yyyDoD6D4vjjM0yyyDoj“*x88*8“*7Dyyyb°yc9东北证寿R9份有限公司I-5Northcqstsccunesco.ltoN旷|一t>6yyyyWSy10p7yy-ey÷11|)8yo0vot>13p9yeioUvop14DIOytransforiner÷15plly)ioUv<17。12yeyeyot>18b13yELMot)ycUy_io|)21b14yGPT"yky0j22pISyELMolGPTlBERT0uy>23D16yGPT-2TNLPyooeoj25D17VoyAIn-ContentLearningNFine-Tuninguy)26b18yOtkOIuSukBedoVb27卜19yGPT-3rChatGPTaoi)2920yyibU-yi)31b21yO-yy-QGPT-3kVt)34)22yikyo)3823yChatGPTyy»3924yChatGPTyy件39b25yChatGPT6×yUp40b26yChatGPTVy件41p27yChatGPTUyU件43f)28ykaenyJiyD45t>29ykayOfo)45b30ykauob46)31yka<46()32yka_n<47t>33yChatGPTuPDFkijJ)49D34yChatGPTuPDFkIjbyyy50t>35yChatGPT6×Iioy-j)51D36yChatGPTV÷eptjp51卜37yChatGPTiyrijb53D38yChatGPTiyiUijb54D39yChatGPTwvGPTsBVjD55东北证方RS份有限公司I-3xNORTHeQSTMCURITiesCO.LTONyI1. ikvao61.1. _441RNNrChatGPTewiVyNaturalLanguageProcessingyNLPyo<<yuljy-yiSoNniOuy|-U_««yy_velvlrUkyDiyyt6NbiewuyU2IewivaoyyioyUlyyylV÷loyP2wyf>nao6yjyyDijWDnyyyyewiv×iY2ewivyyir20z50-iy-÷<<yuy-U1«ygouewiver2isilUyI÷IyyyUiyfUIygi÷V÷220p80iyw<<yyU6UUiyuoyy<<yyewivsoywO2N÷y>>yly<<yUlVIligoiyy)yu62u21f>ywoShyyf)nao6weulewivo2I2010-yTomasM汰Ok)VOw_ulytyyyjyRNNyik-0yewivuIU6÷22015DzmitryBahdanauf>»:Neuralmachinetranslationbyjointlylearningtoalignandtranslate;ouioOvyikyinriuiyksyylwveUri÷2×Transformeryy×IioOvy_Iyewivao2i<o2018-uBERT-ie-yikyYkewiv6erINn3y2-yika÷wnIkiyNi'yyrNnikyw0y6NiyyuolyoyWlkgyy÷yo-×W2OpenAI6_GPT-3ky6uer1750yywyyOowjyykewivyUuION115y2g-GPT-4.0zoyulkiriy!NyuWyoikNoyivgeOkayjli÷loyiokak2ouk÷uiyikT5yyiUT5ywoiegaov21.2. eP_ikikNny×yooWo2UlououoyuuoyW÷WioywoouoOsiyu2ooioku6WyUyu<<syuo-Uy5yve2oyvSWuoykSco)IOEoIo92WyoNnuoRW*.4,lv*-.l*sygol8W0Nn÷QWt½-yM-2Kr*)UyP-kgo11oyNnyy,W!)u_2_oyyuuou-w×ruoPl>yy<<y6uw÷0iPy26oNnyyg11<<yuwv2CiiBnyyNonyu2Nyuj?J)U+y6URWyO×÷jISW×6iopsEo8wg%”,峰$&，)2Byuo->M÷yduuN w东北证券股份有限公司Northcqstmcuriticsco.ltdoOyyi-yNnuWye6y2iu_oyUi-t÷y60-uuoyyo×r÷y0-×6ioyomyO2yk<<ueP-kyautoregressivemodelsy2HH)t-1时樽步tfl神及用络入序列歙据序列全速M同塔FNNUOgbyAWFZUG gWoyYytyu(XwoI(O1WW>IOsg0ftW0Nn÷i>降S2yUyi降OS9(Ws)«$外_/$K%/s&DbdU/$2Uer>.kCylatentautoregressivemodelsy2D2yel>-kot)象据序列时同步t-1时同步t时同步t÷lUOgDyAWFZN w东北证券股份有限公司Northcqstmcuriticsco.ltdet>-ky-e6WyoyyyyyyrecurrentneuralnetworkyRNNyi0wO6y-yyyWSygatedrecurrentUnitsyGRUylyy>>f)ylongshort-termmemoryyLSTMyyoyeP-k2ik÷NnYyooWicyyUoWickNnyywuUuyoyt)yOW2IeOyOWrWysequencetosequenceyseq2seqynyoyj6uiy-爸yycncoder-decodery÷2oIyyeP-ikyyRNN1LSTM1GRUiIWyWIWyy-ey21.2.1. yyyD-RNN12.1.1.yyyDyyybyRNNyl2010kPiikyywyoys-yo0et>-k2RNNkiNn÷oyiO<<yyyl<<yrOuo÷oio2uiiNn÷oyRNNUNNNn÷ob0yy-÷ouioyyw<<yU2Ioyiyy<<ydayb<<-yyy时同步制新>3yyyyD<输出输出从惊蠹屡输入UOgDyAWEZyyy×>ygtOiuyaAiIAyIOR«yoyyuiua2u!o阵gM*N口嗝；6«y_0t>y*-V*iW1÷*)东北证券股份有限公司Northcqstmcuriticsco.ltdN wwoyW_÷oyW*Wo÷NnoyWsU*wV-k_÷oi-Nn÷o21G2iAoygV6u9(>O9»VIiy_*oU$ul_u,幺N_Nn*oOB“！io6yt)-0u2uAoygY(5uWiy-IOft$oOyy需6u20tRNNgyykOuyUUmMbwUdI)MUOeMd。冲UR26UynWNWyjs2oyRNNo6u_0_w÷oPP21.2.1.2. y÷oyyRNNk×<y÷o=yBack-PropagationThroughTimeyBPTTyyy8y2-y+yin+OYi-0+(U*_U«ydgA*>+9$yVI+oiyyy<<yloiNn+OByf+,yuyW+,9,9$y2oyRNNy+sOO-y+o_w8昉时MOiD2×r>IoyN-IjioyyyuWyiv2NNo0eP-kjiOOIyRNNk的歹÷ydOJSBNX必6g6Oey*,G,g02"GeWgtjWJciORO6f"gwo2P4y)÷-yyyMOD出出区HWft时同多t时同步tlUOgbyAWEZiyRNNk"yi÷y.iW心2U$1uHsyOoe-e4oyWjUV(etXaUuAogY2,NiNn*oauyiO*-ey人望溜畀)WO<v(oiAUSi¼yyu2RNN刚“网JtoT11÷obaRRSyWE>/»%旧/加以以E。/%>6Uo÷yN w东北证券股份有限公司Northcqstmcuriticsco.ltd-$%&,%*，*)=不C啕g,丽oi÷y«yyyustgYui3yYIi3yygY2iyygY*iy1C-事。-1c-啕Ek电$)IMl(/、¢)WS.嘿.唠-.yiryN_oNniwyDUu6EtyyfRNniWyt>guoguu2_SWyt)0iPdyiuPyv_ob_Oioe，吵_ot)_0idy2_RWyt)÷yyuuIly_oripv0yUyiyuoCu_tly_oripiy2ipv0_yoyf)yokgY-f)yykUy-yywyigyJ6yb-y÷kgYkyykywr0g_e2v>>NyOyy-ukgYiv-UWyyiyit3v0Uit>iyyy×-yioyrNf)-UW6u-X2×Poy-_x)jiy3ki!?v0Ni|)iy62yl<<yyygugy×RNNkyoy-I!)kWIyy-y÷uit)iyitjv0yyi-y|d_s0Iyy-Wgyt)CI0v2_SWyt)IIUyt)OAyuWIj2oyRNNky-×yylong-termJependenciesyyysy>>Yyy>>YwouPOiB_06Byoy21.2.2. yy>>f>-LSTMNyyyWS-GRUIeoyyyDkByyy|)YA6uiyy»DWongshort-termmemoryyLSTMyiUwuWO_44yyyWSygatedrecurrentUnityGRUy2pt-blyyyyDVLSTMNGRUQiyBWyvmyy-ewivli÷Il÷WorfVWDi2yy>>D-LSTM1.STMoNyRNNyiomwvUwkysyWuoyU6o-uNyeO0avyiwvyWyyiDIyy2LSTM_ui»CWSymemorycellyguvoao»_oioyInnyyuwivyOyyoutputgateygIWSoQgDyuyyinputgateyivuouuy×yyforgetgateyIY>>WSu20yyypYovyi6tN÷Yiuauoio_»N w东北证券股份有限公司Northcqstmcuriticsco.ltdyIN÷Yyu2-Iuy>>WSItt>0io)三yoa_oruio-÷ouyy2D5yyy»BpbyUOgDyAWFZ1.STMkRNNayy<<yi-U)5boi2AoooU嗝0-Nn÷flo&a<k!uUy吁UyRC小卯y00nI()Y»BWS4|21.STMkeoO)W-Iy,tyiAUdUOij)o,)oylyao也行IMgyUl泮。，y2YaW-(-tCWldN*NS$XUba嗝>y)tanhyu_y«cAbiy2y6u<*yyj>2woyWuy三(y三cy三(4y<lj!yW>3y三ry三>4-gYduyW两"用y叫o_u2nftWSuuyi*8y*BkVyMauyv÷ouou11ftWSyl)w«AWodyoyftycvco*WSouoyiywWoywoy+<a瓦®®9annt6ykS)*Br68g2N w东北证券股份有限公司northcqstmcuriticsco.ltdGy*(vwRS吗ooa$SyIy0or6a噜$DpiRNNo6ayPtyn×6uoiogY2LSTMObiy_oRNN÷yo0ey"y÷(oyiIeuygyi)i(oguugi>iyUit>v0y21.2.2.1. yyyWS-GRUt>ylLSTMyyyyWSyGRUyoNnyuW-2-yGRUy_6NLSTMmyywMV2D 6yyyyWSyGRUuonnyyYyyresetgateyUyyupdategateyyw<<yy_NLSTMys,sWr+l>l六+三)*.*JWOy$河M两)5洌>6»删UMS网6O_加2<<yu2YaWsu_+ouUNNn÷odas!YyCVYauouivwoyioj?4Ws-tui(9,+(WS>边吗+9)y 彝iv da $ oyNNn÷o da R|$&! Ya *Ws bsyIycvWoy ioyN÷东北证券股份有限公司northcqstmcuriticscoxo-三s=叨s»+(12N wGRUt)ylLSTMuWy-U-tLSTMuy×yyt6uuy÷Owo->>WSayo>>yuaoyowwGRUyuWy<<ywoy20÷gWuoNyGRUeoyyOLSTMu2oynyRNNyey×6IyoniUyo«yft)2wyuiWyYugy-yLSTMUGRUWkroyof)-U2o!)yWkuiyyUyUvyYuI-yyoWoNYu21.2.3.y7y÷quWicruWWickysequenceIransduclionyT6ipyyPogwsYo_2yoWickNnyy_oikg0gry×o2y6uoGOUbi(oioWuf)rywuuyoyt)y0W2y-eyyencoder-decodery÷koIvyuu<<<yorWicky×yuiky6-yriYo_2D7yy-苍y÷c输入序列马向量弊码器输入UOgbyAWEZy-ey u yyencoderyUeyydecoderynnp|Dt)r2uWicwojOyt>oay <<uNO CUBV yly2 uWoinSyi y6ow>>ioo r 6aoygt)t3uN0U2 y Ioa « _eyo_Uoa2yAVDyO6 y t)lvyey× y rNOUiU)WyUo>>yk÷o yuey r_ ÷o uSy_wONn÷o uoo v rt>oWSyorey-oyiyy r WONnS)2y-eyt>Wi -y01o lu rUi÷I2 U6o0UyuWfrt)Uy)NOUyYINO rOy uWy IyWrWy2N÷ uieuWry6Uewi veyoi ty Y2输出序列东北证券股份有限公司Northcqstmcuriticsco.ltdN W1.3. ioOvywLSTM1GRUyikyII>>WSwkW÷ioyOwNy2uWboioyrNrCVb6aoy-°rioyy_u÷yf)y÷ykgyew2÷_yQyrf)oWioyyuw_OWoio_vrl<<<×=2oIeooy6uIPN0ioRNNBWky0y6iNn÷6oyy0un÷ioNey_tokenNyuey-oy!)k)yNOioWDOBW6Dy02OoNnyyWrWkeyyO÷otokent>irN0ioWyhwo02iiy0iyt>oi÷yUIoy_ISNM÷00NoISot)st)nOyuy0rNOioyoio2ioOyio0vt>122015DzmitryBahdanau件>>:Neuralmachinetranslationbyjointlylearningtoalignandtranslate;ouioOVyAttentionMechanismyyIuuioO0VltgYyikyinriuiyksyytwveUri÷2_uioUvyikgyrl_t)o6W22017Vaswanif56eI:Attentionisallyouneed;y6ulyIeioOvyself-attentionInechanismyoyiloyRNNyyyTransformery2TransformeryUIkIgygi×gyW2÷yWnyyTransformerIkoiyNi-y-u0-ylyy_IyBERT1GPTykik2OyTransformerkuoowewivNkyyrkri«yIEWDykay66lyyg21.3.1. iovQibgyyITyOyywrioIIy_Cyv2<ioO=ieI_vioUgy÷Uit)iv÷N0Vio2ewyUo-ybyOgioy0t)2eyt>i*-io0owouugUiooyyO÷wouu16uz3vi_o/iwo_2uf)oregioU60yu3gy0NUoy12ir_yonyt3i_Iegoyio0oiNy6t)sf)oNyvviolt3÷uiwry62t)yyoioUvio-Ng>>yNt>egioUVDyoIyIyooyioOOoi_0VYioyyu0Vrio2yf)noioUvo-NoNneIyWineyyeyqueryyl(ykeyyUyvalueyykeyUvalueoru2ye_*-IeUUi-y3Nbokey-valueut>_luntyykeyD_IiyIyos<_U|yvaluet)_iiylyow_0uyquery|j_ieuoolyiy2e÷yqueryNbokeyiYUyyrf)sgUkeyuPvalue-eyy2×NyioOvoqueryIkeyUvalueeyyiygt)GyyoikeyNquerysygYynkeybuPvaluegU2东北证券股份有限公司Northcqstmcuriticsco.ltdN火|)8yioOvot>诧的集合query注重力打分F(QK)IJiH11vT：注金力分做九一化法京力收支111UogDyAWUZOD8byioOvo«yV0niyylyioOiVy2y_NWgYy3ygn(jioUvoyIVyuI<<yqueryyoey_avuyNkeysyoyuvIay0JSgVu2VuiU_÷oNkPisit)Wot)0Vio2ioUIVyuo>>yyyUgioOiVyuluyioOlVyuluyioUIVyuy0IVyuyiI0PUy2OCc)QTyiVyUZwjvylii人cty6Transformerkb)UyioUyscaleddot-productattentionyIVyuIIyioOlVyu-Uyi2uyyir<<ym÷UIVyuyOouyi-ne<wo)yt)2uyioUIVyuOy(S东H证券股份有隰公司Vf¾FNORTHCQSTSCCURITIESCO.LTDN火|-War¾.woye窖WQBWj瑚6UB2Auyoi:嚼y_AyyU-IyIVyuV_<÷y=t>NUyj嘲Ssyt'yioij)c02Byo->t(<<ygm÷ylyi<ncen<y*yioOywoeU<>y)-iyyj><2yf>>V.暇：X*X啕暇：嘲x2yXgY_NW0dgnUAo.uyioU«y_-y咽降哗嘲嘲哗嘲嘲嘲(暇，«)=障嘲嘲阵嘲"5(Jl)S1.3.2. Transformer÷13.2.1.eioOvioUviNnSyqueryyNwPSykey-valueyiYyirNngoio2iyyIy_uioUvyieyiNn÷oou_queryyNyiNn÷ooaiYyNniNqueryJSg-gYg6a_N0qioV_y6iUuIONn÷ouoy6UyUyo×j2owweyiNn÷oyuuWboOiygY<<yyyWN_iU<<ygf)yykyUevm÷02oy×f)0y6vyygvinONWDOt)sgyyoINoyikyYuyyk%×-wkrYuOOsy2b9yeioUv<uogt>yWEzeioOvyself-attentionmechanismyuWoinSIquerylkeyUvaluey<<yuWoinSNwr>St>sgyyiomwwkWO0OSOsyU3Sg2<<yioUgY÷yinOioN w东北证券股份有限公司Northcqstmcuriticsco.ltdUyobr>t>uyeiovy-UV6if)yOGPUvTPUy-I<<yyU2y-i<<y-yvyW÷-Yy-UwUI<<ym÷yky-ywl-yev21.3.2.2.Transformeru-÷2017VaswaniJeI:Attentionisallyouneed;y6ulyleio0vyself-attentionInechanismyoyioioyRNNyyyTransformeryy(3w6Uik<<yigy-gi×gy6W20÷-uTransformerkyluly2DIOytransfbrmer÷OutputProbaMtiesPositionai-/×EncodingVInputEmbedOngInputsOutputs(shiftedright)UOgby：Attentionisal!youneed;Transformeru_×Iyoyyey÷ywy0ey××IyleioOvkW6y2yUeyWbVuy东北证寿R9份有限公司I-5Northcqstsccunesco.ltoN旷|一WU|)oSWoueAdNOy2yOeyO0kWWiteioOOylo_Dnn2wWOD10b2u_NyTransfbrmery0eyyoun)Wyblocky6ryinWyonn2NnoeioUymuIti-headself-attentionyyInoyio_DyPoSition-WiSefeedforwardnetworky2«yyeioU÷yquerylkeyUvalueyge_Nny2-it>uyt3k-yyyyif)iyyU6U'yUgyinAy×tyyreMdualcannedi11y2*t1yuOSngTnmdanner118W-0uN(auy三7y2BteyBcodywyOyxYwt-NWylayernoniilizationy2oyuWQt>inOyTransformcry-yIjNn嘲|j6e(Ug2Biey|>cU_NWAyiNAou_yImt<m+AMlmM5m6Transformerey-ounf)WryuWOiNnA_ileyU-NW2iy-oiynnO-yy,vnn0ouiOnyu-yyeyioUyencoder-decoderattentiony2yyeyioUoyqueryge_NneyWduykeyUvaluegeuny-602eyeioUoyquerylkeyOvalueygeNNneyWu20oyeyoin0uy×i0_obo02yeXymaskedyio0IeP-gyo÷06uoiysy-eyu-NeyoboyyrISOiioU«y21.3.2.3.ioOvt-÷iyeioUVyVaswani(f)6ulioUymuli-headattentionyvwDTransformerWo2ioUv«kSy_iuf)querylkeyUvalueyytDioOvnrlyyoOIt>wkBWOTysy2w-oyyoyIuuuolt>unyquerylkeyOvaluet>yg÷yli11earprojectionsyyYiND÷querylkeyUvalueuTet>uioUo2gAAboioUAouyINk-yg÷yig)uyy2iNt>÷UcUioUyoP)3uOiyiNnioUy«u_Nnyheadyyoyyy«u_ioOv2东北证券股份有限公司Northcqstmcuriticsco.ltdN火|。llyioOvo)uogjjyWEzeioUu0uoIOyg÷yyyUcyIuyunqueryIkeylvalue60z64ympresentationsubspacesyyRiiocl6g人nioOR:WRiNkyg÷Ucygerw6×W0TS0osgiomy23NNnonnioOuuO_«yy啕理嘲H?喝嘲唠嘲(喝嗝唠)=唱嘲嘲嘲唠降(/嘲唠嘲i,&,胆为嚼J暇9> <><<*o- Ieuyio0yu29«,IjhlZ'6uuowj<EioOOwyy及ofNnioU)0Puuoj÷UW*-cl<ylyogYuy女,OAPCeiOOOgUWy(f网gV6uygVdUyOgn2PioOvyy_<<k÷wkyWsyUyoUke23uioyio0vyioUy-wkryWOoOSsyyotioOvvyW÷v2÷-yIuioUoO)V÷yyZNt>Nwvekoiy6Utkyeg2IuOV÷y-yiuOYgINo×OoyX_N0VYgyiwek-1of)ie6213.2.4.eyeyyResidualConneCtiOnVOCDyDyf)Yywyuyf)yt>"yUgywouy÷yweyg-ge-znt>6uey东北证券股份有限公司Northcqstmcuriticsco.ltdN w3yResNety2ResNet2015ImageNetIkIEIsznyywYIgyt3|jo<«2OirgirynyJ>WNyW(均如)6UuyDnrv"TKOynwIgYnoiy