谷歌-教育生成式人工智能开发技术报告 2024.docx

TowardsResponsibleDevelopmentofGenerativeAlforEducation:AnEvaluation-DrivenApproachIrinaJurenkavt1,FAarkusKunesch-t.,KevinMcKeeyDanielGillickgi1ShaojianZhut1,SaraWiltberge省ShubhamMilindPhal1,KatherineHermann1,DanielKasenborgslAvishkarBhoopchand1,AnkitAnand1,MirunaPislarilStephanieChan,t1.isaWang.JenniferShe'.ParsaMahmoudieh1lAJiyaRysbek1tWei-JenK3,AndreaHuber'.BrettWiltshire1,GalElklant2,RoniRabln2,JasminRublnovltzt-4.AmitPitaru4,MacMcA)ltster3.JuliaV/llkowskP,DavidChol,RoeeEngelberg2,1.ldanHackmon2,Adva1.evln2tRachelGrlftin5,MichaelSears5,FilipBaEMIaMesar±ManaJabbour3fArslanChaudhry1,JamesCohan3.SrldharThiagaraja11,Nir1.evine,.BenBrowm.DilanGorur§.SvetlanaGrant1,RachelHashimoshoni3.1.auraWeidinger1,JieruHu1,DawnChen3,KubaDoleckt3,CanferAkbulut19MaxwellBileschi1y1.auraCulp',Wen-XinDong3.NahemaMarchal1.KelsieVanDema114,HemaBaiajMisra3.MichaelDuahslMoranAmbar2.AviCaciularu?,Sandra1.efda,ChrisSummerBeIdTyJamesAnPierre-AlexandreKamienny1tAbhinitMohdi3,TbeofilosStrinopoulous3.AnnieHaleWayneAnderson5.1.uisC.CoboilNivEfront2.MukthaAnanda3.ShakirMohameda,MaureenHeymam3,ZoubinGhahramani1,Yo$lMatias2,BenGomes3and1.ilaIbrahim1'EqualOonMbutone,tTechnicala<J,:Researchlead.5Worketreamlead.,6cogDp<.lca2GcoqResearch.3Gocq,4GooflJeCreative1.ab,5AnzonaStateUwversity.61.urjUniversity,7U11iversdyofOxford.sAtropc.workcarriedOUlwhileOrnPlOy9。alGsg匕DeepMindAmajorchallengefacingtheworldistheprovisionofequitableanduniversalaccesstoqualityeducation.RecentadvancesingenerativeAl(genAl)havecreatedexcitementaboutthepotentialofnewtechnologiestoofferapersonaltutorforeverylearnerandateachingassistantforeveryteacher.Thefullextentofthisdream,however,hasnotyetmaterialised.WearguethatthisisprimarilyduetothedifficultieswithverbalisingpedagogicalintuitionsintogenAlpromptsandthelackofgoodevaluationpractices,reinforcedbythechallengesindefiningexcellentpedagogy.Herewepresentourworkcollaboratingwithlearnersandeducatorstotranslatehighlevelprinciplesfromlearningscienceintoapragmaticsetofsevendiverseeducationalbenchmarks,spanningquantitative,qualitative,automaticandhumanevaluations;andtodevelopanewsetoffine-tuningdatasetstoimprovethepedagogicalcapabilitiesofGemini,introducing1.earn1.M-Tutor.Ourevaluationsshowthat1.earn1.MTutorisconsistentlypreferredoveraprompttunedGeminibyeducatorsandlearnersonanumberofpedagogicaldimensions.Wehopethatthisworkcanserveasafirststeptowardsdevelopingacomprehensiveeducationalevaluationframework,andthatthiscanenablerapidprogresswithintheAlandEdTechcommunitiestowardsmaximisingthepositiveimpactofgenAlIneducation.1. IntroductionTheroughly70yearhistoryofArtificialIntelligence(AI)hasbeenoneofparadigmshifts:fromsymbolicsystems,toBayesianapproaches,todeeplearning,andinthelastfewyears,generativeAl(genAIJ-Iargefoundationalmodelstrainedonhugeswathsofmediaavailableontheinternettogainanimpressivesetofgeneralcapabilities,wherebytheyare(mostofthetime)abletoprovideausefulresponsetoanyuserpromptorenquiry.Eachparadigmshiftbroughtwithitauniquesetofhopes,opportunities,andchallenges.YetthecurrentgenAleraisunprecedented:Alismoreaccessiblethanever(becauseitonlyrequirespromptingthroughnaturallanguage),morecapablethanever,andappearstobeimprovingfasterthanever.Questionsnaturallyariseabouthowtoharnessthistechnologyformaximalsocialbenefit.POfXan松贝3)2WMr”图小YeOa密室Q券.8m®2024G009teDecpMndAJIrightsreservedDeployment:ASUStudyHall*>MM*IlM(M<w*aMtf4ltf<CM>*aU.<(>l<ar>4*t4rMC4*<M'rwt*r*44Ke，n<,山*r,nxX，y4r*g-，AfMlSl«4<ll»Ml»Tt«f27<4*"<4i«!«.,*“，*g”»一u*>f.<UTAC(C*O)<VMrTnx.<tv*(C*J11.><<<*<r>:l*t4.>.rMM'<bcm公“)；lOf.Tv"Gtr,C*,，«联，G»,*<CharAtO-，x*Participation:1.earnerfeedbackIwoulddescbeitasahelpfulfriendthatknowsalotaboutonesubjectthatcanhefpyouIeamthedass.一1.eart1.MTOStudyHallUser«Ug,x<*f*<49kM»1.>«e*<a<，z«T-(fctlrlo<*nt<“<41*1,a.*，fc<wUI2”T><r*M*rM0'T>r«，(“Gnem4rv*vi*«<M*ttkfrw«ltt<»*%<<.O»y>«*m1<<*v>>rv4*，“<<.«(IM>aUUltammIkMd»，I)mmkt，CWv-m>'lMr1.Mt4M*H?IeUJ-Wl«MM41k4(，«!(QMMbMAFigure11.earn1.M-TutorDevelopmentoverviewotourapproachtoresponsibledevelopmento1geAltoreducation.Bofdaw心showthedevelopmenttlow,do*Ma”gstheSIOfmatlonnow.OrapproachSlanSandendswith西mCgat6.Westartbyansweringtheqestkx毋of'whoarewtryingtohelp?*,whatdotycareabout9",whoarealltherelevantstakeholders?",andbringthemintoourdevelopmentprocess.Thisinformstheprioritisationofourmode?improvementsworkandthe<Jeelopmetofourmp<eensiyeevaluationbenchmarks.ThesefurtherWorrnmodelimprovements(andeachother)throughafastautofnaticev引匕献kvbasedandasloweru11wevQ*>nsMsediteration100PFinalMWeuseth©deploymentofour11xxilstorealuserstofurtherinformourresearchanddevelopmentwork,andtofeedbackintothePaeepaMnstageWBusthisapproachtodevelop1.earn1.MTutor,aconversationalAltutor.Evaluation(teacherpreferences):oneofsevenevaluationbecmarksncroicdinthisrepod.ItShowSthateducatorsprefer1.eam1.M-T!oroverPrOmPd1baseGemiN1.OonthemaotyotmeasuredPedagoglCaiattributes.Deployment(ASUStudyHall):exampleconversationbetween1.earn1.MTutorandanASUStudyHaestudente<vcdintheIntroductiontoProgrammingcourse.Participation(learnerfeedback):aninterviewquoteIromanASUStudyHallstudentwhohasused1.eam1.M-Tutorduringtheircourse.Weuseinterviewstogetqualitativefeedbackontheefficacyandsafetyofthetutor.Oneofthekeychallengesfacingtheworldisthelackofuniversalandequitableaccesstoqualityeducation2J.Educationisakeyeconomicdriver3andafacilitatorotupwardsocialmobility4:however,evenbeforetheCOVID-19pandemic,53%ofallten-year-oldchildreninIow-tomiddle-incomecountrieswereexperiencinglearningpoverty5,and40%ofUSschooldistrictleadsdescribedtheirteachershortagesas-severe*or"verysevere"6.Thelong-standingproblemswitheducationalattainmentandteacherretentionhavebeenfurtherexacerbatedbythepandemic,disproportionatelyaffectingthosefromlessprivilegedbackgrounds5.6.TheriseingenAlthatfollowedthepandemichasbeenmetwithmixedreactions.Ontheonehand,itappearstoholdsomepromisetodemocratiseaesstoknowledgeandeducation:studentsareearlyadoptersandtopusersofthetechnology7.andgenAlisdominatingtheEdTechlandscape8.Ontheotherhand,severalncershavebeenraisedaboutthemisuseofthistechnologyineducationalsettings7,9.Forexample,thegenAlmodelsthatpowermostofthelatestEdTechsystemsarenotexplicitlyoptimisedforpedagogy.Instead,modelsaretrainedtobe"helpful*10-14,butthisspecificdefinitionofhelpfulnessmayoftenbeatoddswithpedagogyandlearning.Forexample,studentscaneasilygetdirectanswerstohomeworkassignmentsinsteadofworkingthroughthemforthemselvestogettheintendedpractice.Theavailabilityofwhatappearstobe"expert"informationbypromptingagenAlmodelforanansweralsogivesstudentsanillusionofmasterybeforeithasbeenachieved,whichmayeventuallyleadtoproblemsintheworkplace(9,15.ThisreportdescribesourfirststepstowardsoptimisinggenAlforeducationalusecases.Inparticular,wefocuson1:1conversationaltutoring,andproposeacomprehensiveevaluationprotolforthisusecase.Wefocusonconversationaltutoringbecausewebelievethatitisoneofthemostimpactfulandgeneralusecases,andbecauseitrequirestheintegrationofmanyimportanteducationalcapabilitiesintoasinglesystem.AnexcellentconversationalAltutorhasthepotentialtoenhancetheeducationalexperienceofbothlearners(byprovidingthemwithinstantfeedbackandadaptingtotheirindividualneeds)andteachers(bymultiplyingtheirimpactandlighteningtheirworkload).Wefocusonevaluation,becauseitisclearthatasharedframeworkacross(andevenwithin)learningscience(seeSection3.1),EdTech(seeSection3.2),andAlforEducation(seeSection4.2)islacking,andsuchaframeworkwouldlikelyenableprogressmorethananysingleproduct.Furthermore,effectivemeasuresofpedagogicalsuccessareaprerequisiteforoptimisingAlsolutions,whichneedsuchsignalsfor-hill-climbing".Ourmaincontributionsarethefollowing:1. WedescribeourapproachtoresponsibledevelopmentofAlforeducation(Figure1).whichisinformedbyIheethicsandpolicyliterature16-26.Weemphasiseaparticipatory(Section2)andmultidisciplinaryapproachtoresearch,bringingtogetherexpertsinpedagogy,cognitivescience.Al.engineering,ethics,andpolicy,aswellastheultimateStakeholders-StudentsandteacherstotranslateinsightsfromlearningscienceintopragmaticandusefulpedagogicalimprovementsofGemini1.0(10)foreducation.2. Weintroduce1.eam1.M-Tutor,anewtext-basedgenAltutorbasedonGemini1.0,furtherfinetunedfor1:1conversationalIutoring(Section3),andshowthatwimproveitseducation-relatedcapabilitiesoveraprompttunedGemini1.0.3. Wedevelopacomprehensivesuiteofsevenpedagogicalbenchmarks(quantitativeandqualitative.andusingbothhumanandautomaticevaluations;Figure2)intendedforassessingtheperformanceofconversationalAltutorsfromvariousangles.Asacasestudy,weapptytheseevaluationstoaprompttuned1Gemini1.0and1.eam1.M-Tutor,providingaportfol100fevidenceforpedagogicalprogress.Wealsodiscussexamplesofmoretargetedevaluationsanddescribehowweusethemtodevelopspecificeducationalcapabilitiesfor1.earn1.M-Tutorllikeevaluativepractice(Section8.1)andfeedbackonproceduralhomeworkproblems(Section8.2).OurcomprehensiveapproachgoesbeyondaddressingthemorecommonquestionofDoesitwork?"(quantitativeresearch),toalsoinclude"Howandwhydoesitwork?,(qualitativeresearch)and"Willitworkforeveryone?"(participatoryresearch),inlinewiththerecommendationsinDataRatings5tbti*<l<3ftwC<<dKkFigure2OVefVieWoftheevaluationtaxonomyintroducedInSection4.3.2thatunderpinstheSeVenpe<3agcgicaJevaluationbenchmarksintroducedInthisreport.EachbenchmarkisuniquenitsPfaCewithintetaxonomyandcomeswith1sownbenefitsandchallengesTOoelhe1.thesed>iferentbenchmarksprovideam<ecomprehensiveviewonthepedagogicalCapabiStiesofgenAltutors.Numbersinbracketsrepresentsectionnumbersdescribingeachparticularbenchmark.Fosteretal.21.4. Finally.Wediscussthelimitations,aswellasthesafety,ethical,andpolicyimplicationsofourwork.OurapproachtoethicsandsafetygoesbeyondthecommongenAlguidelines,aswedevelopeducation-specificinterventions(Section9).Asacommunity.WearejustatthebeginningofalongjourneytowardsbuildinggenAltechnologycapableenoughtomeaningfullycontributetouniversalandequitableaccesstoqualityeducation2.Hence,wehopethatthisreportisseenasaninvitationtostakeholdersinresearch,EdTech1ethics,policy,andeducation,toprovidefeedbackonourearlywork,andtocometogethertoestablishcommonguidelines,benchmarks,andworkingprinciplestosteerourjointworkontheresponsibledevelopmentoftransformationalAlforeducation,whileweareWorkingonmak»x)oureducationalbenchmarksaccessibletoIhecommunity,pleasereachouttou$vema>ifyouhaveanyimmediatesuggeSlionSorfeedback,orviathisf<xmforamoreformalresearchCollatxxation.2. ParticipatoryapproachThissectiondetailstheparticipatoryelementsthathelpedshapethisproject,includingthedesignofourevaluativeapproach,andourgoalsindeveloping1.eam1.M-Tutor.WefirmlybelievethatresponsibledevelopmentofeducationalAlsystemsrequiresengaginglearners,educators,policymakers,andacademicresearchers(27,toensurethatIheresultingsystemsalignwiththeirneeds,values,andaspirations28.29.Weutilisediverseparticipatoryresearchmethods,includingworkshops,-designexercises,semi-structuredinterviews,anduserstudies,inacollaborativeanditerativedevelopmentprocess-Thsrep<xtdescribespreviously11pub)isedwork,seeTombazzietal.30forattree-partarticlesenesonAlandtheFutureot1.earnngbyTheRSAandGocgleDeepMind.Inthisreporteachparticipantisassignedanumericalidentifier(P1throughP116).ThisincludesPaniCipantSfromourworkshops(Pl-P94),initialinterviews(P95P97).co-designactivities(P98-P106),anduserstudiesdescribedinSection7(PI07-116).2.1. Participatoryworkshops:ImaginingandcritiquingthefutureofeducationandAlWeconductedtwoparticipatoryworkshopsintheUK:onewithlearners,primarilyuniversitystudentscomingfromdiverseacademicbackgrounds(n=60),andanotherwitheducators,mainlyhighschoolteachersspecialisinginSTEMsubjects(11=34).Thechoiceoftheparticipantdemographicswasdictatedbypracticalconsiderations.Werealisethatfutureworkisneededtoexpandourreachtobroadercommunities,sincelearnersintheUKandotherWEIRDWestern.Educated.Industfiafcsed,Rich,Democfalic(WEIRD)countries(31)areoftenover-representedinpsychologicalstudies,despilenotbeingrepresentativeofIheglobalpopulation.countrieslikelyencounterfewerbarrierstoaccessinggenAltools,andperspectivesonAlineducationlikelydiffersubstantiallyacrossculturalcontexts.Followingestablishedbestpracticesforparticipatoryworkshops32,weemployedstructuredactivitiestofosterinteraction,collaborativelearning,andgroupcohesion(seeSectionB.1formoredetails).Participantsweredividedintosmallgroupsoffivetoeightindividualsandengagedintwokeyexercises: Groundingexercise:Thisactivityexploredparticipants'educationalexperiences,revealingcurrentneeds,challenges,andpotentialareasforimprovementregardinggenAltools. Speculativedesign:Thisexerciseencouragedparticipantstoenvisionascenarioinvolvingalearnerfacingvariouschallenges.Throughcollaborativebrainstorming,theyexploredhowAlandsocialfactorscouldexacerbateormitigatethesechallenges.Theseworkshopshighlightedcunentchallengesineducation:learnersstrugglewithtimemanagement,cognitiveoverload,anddemotivationwhentheyperceivetheirlearningmaterialsasirrelevant：whileeducatorsstruggletoprovidepersonalisedattentionandfeedbackinclassroomsettings.Personalisedtutoring,byAlorhumans,wasvaluedbybothlearnersandeducators.Tutorsareespeciallyeffectivewhentheyhaveknowledgeofthelearnerandcanadapttheirapproachacc<×dingty.1.earnersfeltmorecomfortableseekingclarificationsfromAltutorsthanhumantutors,perceivingAltutorsaslessformalandlesslikelytoinducefearsofjudgement.AsharedlimitationofbothhumanandAltutorswastheirlackoffamiliaritywiththenuancesofparticularsyllabiorexamboardrequirements.1.earnersintheworkshopwereoftenstrongadoptersofgenAl.Whileawareofitslimitations,theytendedtobehappytoworkaroundthem.Educatorsweremoresceptical,citingworriesabouthallucinations,thepotentialforcheating,andthelackofadaptationtoth©learnerslevelandcognitiveloadingenAs'wall-of-text*responses.BothgroupssawimmediatebenefitsofgenAltools,suchasfromgeneratingpracticequestions,critiquingandgeneratingideas,andsummarisingcontent.Asharedvisionforthefutureofeducationemerged,emphasisingtheroleofpersonalisedAltutorsinenablingflexible,cross-disciplinary,andrelevantlearningopportunities.AcJditionaIIy,virtualandaugmentedrealitytechnologieswereseenasbeneficialthroughenhancedimmersion.Educatorsdesiredreal-timefeedbackandactionableinsightsfromAltoolstoimproveteaching.TheyalsocautionedagainstafuturewherelearnersbecomedependentonAlandlosetheirautonomy.WhenaskediftheyfeltthreatenedbyAl.educatorsexpressedconfidencethattherewouldalwaysbearoleforhumansintheprocessofteachingandviewedgenAlasapositivetooltoassistthem,freeingupmoretimeformeaningfulinteractionswiththeirstudents.2.2. Understandinglearningexperiences:InitialinterviewsandWizard-Of-OzsessionsToinitiateouriterativeparticipatorydesignprocessfor1.earn1.M-Tutor,v/eco