Satisficing and Learning Cooperation in the Prisoner’s Dilemma
Jeff L. Stimpson
&RPSXWHU6FLHQFH'HSDUWPHQW
%ULJKDP<RXQJ8QLYHUVLW\
3URYR87
MVWLP#FVE\XHGX
Michael A. Goodrich
$VVLVWDQW3URIHVVRURI&RPSXWHU6FLHQFH
%ULJKDP<RXQJ8QLYHUVLW\
3URYR87
PLNH#FVE\XHGX
Abstract
7KH SULVRQHU¶V GLOHPPD LV D XVHIXO PRGHO IRU
VWXG\LQJWKHEDODQFHEHWZHHQVHOILQWHUHVWDQGJURXS
LQWHUHVW LQ PXOWLDJHQW V\VWHPV $OWKRXJK PDQ\
VWUDWHJLHV KDYH EHHQ GHYHORSHG WKDW SHUIRUP ZHOO
PRVW RI WKHVH VWUDWHJLHV PDNH VWURQJ DVVXPSWLRQV
DERXWWKHLQIRUPDWLRQDYDLODEOHWRWKHDJHQW,WLVLQ
WKLV FRQWH[W WKDW ZH GHVFULEH D VDWLVILFLQJ OHDUQLQJ
VWUDWHJ\ IRU WKH SULVRQHU¶V GLOHPPD DQG SUHVHQW
HYLGHQFH WKDW VWDEOH RXWFRPHV RWKHU WKDQ WKH 1DVK
HTXLOLEULXP DUH SRVVLEOH ,Q DGGLWLRQ ZH RIIHU
HPSLULFDOHYLGHQFHWKDWXQGHUW\SLFDOFLUFXPVWDQFHV
PXWXDO FRRSHUDWLRQ LV WKH PRVW OLNHO\ RXWFRPH DQG
LGHQWLI\ FRQGLWLRQV XQGHU ZKLFK WZR VDWLVILFLQJ
DJHQWVZLOOOHDUQWRFRRSHUDWH
1
E\GHIHFWLQJUHJDUGOHVVRIZKDWKLVRUKHURSSRQHQWGRHV<HW
LIERWKSOD\HUVPDNHWKLV³UDWLRQDO´GHFLVLRQWRGHIHFWERWK
UHFHLYHOHVVWKDQLIWKH\KDGFRRSHUDWHG
,Q VHDUFKLQJ IRU DQ HIIHFWLYH VWUDWHJ\ LQ WKH SULVRQHU¶V
GLOHPPDZHORRNIRUDVWUDWHJ\H[KLELWLQJIOH[LEOHEHKDYLRU
,W VKRXOG FRRSHUDWH ZKHQHYHU PXWXDO FRRSHUDWLRQ LV
SRVVLEOHEXWLWPXVWEHDEOHWRGHIHFWZKHQLWLVDSSDUHQWWKDW
LWVRSSRQHQWLVXQZLOOLQJWRFRRSHUDWH0DQ\VXFKVWUDWHJLHV
KDYHEHHQGHYHORSHGDQGVWXGLHGEXWRIWHQWKHVHVWUDWHJLHV
LQYROYHDWOHDVWRQHRIWKHIROORZLQJDVVXPSWLRQV
•
•
Introduction
,QVLWXDWLRQVLQYROYLQJVHYHUDOLQWHUDFWLQJDJHQWVHDFKDJHQW
LV RIWHQ IRUFHG WR FKRRVH EHWZHHQ WZR W\SHV RI EHKDYLRU
WKRVHWKDWEHQHILWWKHJURXSDVDZKROHDQGWKRVHWKDWOHDGWR
UHZDUGVIRUWKHLQGLYLGXDODWWKHH[SHQVHRIWKHJURXS7KH
VLWXDWLRQ EHFRPHV LQWHUHVWLQJ ZKHQ LQ WKH ORQJ UXQ SRRU
RXWFRPHV IRU WKH JURXS OHDG WR QHJDWLYH FRQVHTXHQFHV IRU
HDFKLQGLYLGXDO
7KHLWHUDWHGSULVRQHU¶VGLOHPPDLVDQHOHJDQWDQGZHOO
NQRZQH[DPSOHRIVXFKFLUFXPVWDQFHVWKDWKDVEHHQVWXGLHG
LQDZLGHYDULHW\RIGLVFLSOLQHV$W\SLFDOSD\RIIPDWUL[IRU
WKHSULVRQHU¶VGLOHPPDLVJLYHQLQ)LJXUH7KHGLOHPPDLV
$¶VSD\RII%¶VSD\RII
$JHQW$¶V
&KRLFH
$JHQW%¶V&KRLFH
&RRSHUDWH
'HIHFW
&RRSHUDWH
'HIHFW
Figure 1: $W\SLFDOSD\RIIPDWUL[IRUWKHSULVRQHU¶VGLOHPPD
WKDW HYHU\ SDLU RI DFWLRQV LV HLWKHU XQVWDEOH RU VXERSWLPDO
)RUPDOO\ VWDWHG WKH XQLTXH 1DVK HTXLOLEULXP LV WKH RQO\
RXWFRPHWKDWLVQRW3DUHWRRSWLPDO0XWXDOGHIHFWLRQLVWKH
GRPLQDQWVWUDWHJ\LQWKHVHQVHWKDWDSOD\HUZLOOEHEHWWHURII
Lawrence C. Walters
$VVRFLDWH3URIHVVRURI3XEOLF3ROLF\
%ULJKDP<RXQJ8QLYHUVLW\
3URYR87
ODUU\BZDOWHUV#E\XHGX
•
•
SOD\HUVDUHDZDUHRIWKHVWUXFWXUHRIWKHJDPHVXFK
DVWKHRWKHUSOD\HUVWKHRWKHUSOD\HU¶VSRVVLEOH
DFWLRQVDQGWKHUHODWLRQVKLSEHWZHHQWKHDFWLRQV
DQGWKHSD\RIIV
SOD\HUVDUHLPPHGLDWHO\DZDUHRIRWKHUSOD\HU¶V
GHFLVLRQV
SOD\HUVDUHDZDUHRIWKHRWKHUSOD\HU¶VSD\RIIV
SOD\HUVDUHDZDUHWKDWWKH\DUHLQDJDPHVLWXDWLRQ
PHDQLQJWKDWWKH\DUHDZDUHWKDWWKHDFWLRQVRI
RWKHUDJHQWVDUHDIIHFWLQJWKHLURXWFRPHV
,Q FRPSXWHU VLPXODWLRQV WKHVH UHTXLUHPHQWV DUH HDVLO\
PHWEXWLQUHDOZRUOGVLWXDWLRQVWKH\PD\EHTXLWHOLPLWLQJ
)RU H[DPSOH WKH SULVRQHU¶V GLOHPPD FDQ EH H[WHQGHG WR
PXOWLSOH SOD\HUV ,I WKHUH DUH PDQ\ SOD\HUV FKRRVLQJ IURP
PDQ\ DFWLRQV NHHSLQJ WUDFN RI WKH JDPH VWUXFWXUH PD\ EH
XQUHDOLVWLF LQ WHUPV RI VWRUDJH UHTXLUHPHQWV DQG
FRPSXWDWLRQDO FDSDFLW\ ,Q RWKHU FDVHV LQIRUPDWLRQ DERXW
WKHVWUXFWXUH RI WKH JDPHPD\ QRW HYHQ EH DYDLODEOH WR WKH
GHFLVLRQPDNHU)LQDOO\DOWKRXJKVLWXDWLRQVDQDORJRXVWRD
SULVRQHU¶VGLOHPPDDUHFRPPRQRFFXUHQFHVWKH\DUHUDUHO\
WKRXJKW RI LQ WHUPV RI JDPH WKHRU\ ,QVWHDG ZH DUH PRUH
LQWHUHVWHGLQPHHWLQJVSHFLILFJRDOV
5HPRYLQJ WKHVH DVVXPSWLRQV IURP WKH SULVRQHU¶V
GLOHPPDWDNHVWKHSUREOHPRXWRIJDPHWKHRU\DQGLQWRDUHDV
RI PDFKLQH OHDUQLQJ ,W LV LQ FRQWH[W RI WKHVH W\SHV RI
VLWXDWLRQV WKDW ZH FRQVLGHU D VDWLVILFLQJ VWUDWHJ\ IRU WKH
SULVRQHU¶VGLOHPPD6SHFLILFDOO\WKHSXUSRVHRIWKLVSDSHULV
WR SUHVHQW WKH VWUDWHJ\ DQG WKHQ VKRZ WKDW VWDEOH
RXWFRPHVRWKHUWKDQWKH1DVKHTXLOLEULXPIUHTXHQWO\RFFXU
DQGGHVFULEHWKHFLUFXPVWDQFHVXQGHUZKLFKWZRDJHQWV
HPSOR\LQJDVDWLVILFLQJVWUDWHJ\ZLOOOHDUQWRFRRSHUDWH
2
Related Literature
7KH SULVRQHU¶V GLOHPPD ZDV FRQFHLYHG LQ WKH V WR
TXHVWLRQVRPHRIWKHEDVLFWHQHWVRIJDPHWKHRU\6WDQGDUG
UDWLRQDO GHFLVLRQ PHFKDQLVPV VXFK DV PLQLPD[ OHDG WR
PXWXDOGHIHFWLRQDQGSRRURXWFRPHVIRUERWKSOD\HUV6LQFH
WKHQ WKHUH KDYH EHHQ QXPHURXV DWWHPSWV WR ³VROYH´ WKH
SULVRQHU¶V GLOHPPD E\ VKRZLQJ WKDW PXWXDO FRRSHUDWLRQ LV
UDWLRQDO DIWHU DOO 7KH PRVW LQIOXHQWLDO RI WKHVH KDV EHHQ
$[HOURG¶V ZRUN LQ WKH UHSHDWHG SULVRQHU¶V GLOHPPD >@
+H VKRZV WKDW PXWXDO FRRSHUDWLRQ LV UDWLRQDO DQG VWDEOH
ZKHQ WKH IROORZLQJ FRQGLWLRQV KROG WKH IXWXUH LV
LPSRUWDQWWKHUHLVVXIILFLHQWGLIIHUHQFHEHWZHHQSD\RIIV
IRUPXWXDOFRRSHUDWLRQDQGPXWXDOGHIHFWLRQDQGRQHLV
IDFLQJ DQ DGDSWLYH RSSRQHQW ,Q VXPPDU\ $[HOURG VKRZV
WKDWUDWLRQDOLW\LQUHSHDWHGSOD\JDPHVLVQRWWDQWDPRXQWWR
1DVKHTXLOLEULXP
7KHLGHDRIDSSO\LQJJDPHWKHRU\WROHDUQLQJLQPXOWL
DJHQW V\VWHPV LV IDU IURP QHZ )RU H[DPSOH 0LQLPD[4
>/LWWPDQ @ LV D UHLQIRUFHPHQW OHDUQLQJ DOJRULWKP WKDW
OHDUQV WKH 1DVK HTXLOLEULXP LQ ]HURVXP RU SXUHO\
FRPSHWHWLYH VWRFKDVWLF JDPHV )XUWKHU ZRUN VXFK DV >+X
DQG:HOOPDQ@KDVDWWHPSWHGWRH[WHQGWKHVDPHLGHDWR
JHQHUDOVXP VWRFKDVWLF JDPHV 7\SLFDOO\ WKH IRFXV RI WKLV
OLWHUDWXUH KDV EHHQ WRZDUGV OHDUQLQJ WKH 1DVK HTXLOLEULXP
:KLOH WKLV PD\ EH D GHVLUDEOH SURSHUW\ LQ PDQ\
FLUFXPVWDQFHV WKLV DSSURDFK KDV GUDZEDFNV )LUVW WKHVH
DOJRULWKPV XVXDOO\ UHTXLUH VLJQLILFDQW DVVXPSWLRQV DQG
NQRZOHGJH DERXW WKH JDPH VWUXFWXUH WKDW FDQ EH TXLWH
OLPLWLQJ6HFRQGLQOLJKWRI$[HOURG¶VZRUNLQDUHSHDWHG
SOD\ VLWXDWLRQ WKH 1DVK HTXLOLEULXP PD\ QRW EH WKH RQO\
VWDEOHVROXWLRQZLWKGHVLUDEOHSURSHUWLHV
/LNHPXFKRIWKHZRUNGRQHLQWKHSULVRQHU¶VGLOHPPD
WKH FRQFHSW RI VDWLVILFLQJ FDPH DERXW DV D PRGLILFDWLRQ RI
UDWLRQDOLW\7UDGLWLRQDOUDWLRQDOFKRLFHWKHRU\KROGVWKDWDQ
DJHQWIDFHGZLWKDGHFLVLRQZLOOFKRRVHWKHDOWHUQDWLYHWKDW
PD[LPL]HVDXWLOLW\IXQFWLRQ+RZHYHUDVQRWHGLQ>&RQOLVN
@ DQG RWKHUV WKHUH LV OLWWOH HPSLULFDO HYLGHQFH WKDW
SHRSOH PDNH GHFLVLRQV LQ WKLV PDQQHU LQGHHG HYLGHQFH
VWURQJO\ VXJJHVWV RWKHUZLVH $V D UHSODFHPHQW +HUEHUW
6LPRQKDVSURSRVHGVDWLVILFLQJ+HH[SODLQVWKHGLIIHUHQFH
EHWZHHQRSWLPL]LQJDQGVDWLVILFLQJ³$GHFLVLRQPDNHUZKR
FKRRVHV WKH EHVW DYDLODEOH DOWHUQDWLYH DFFRUGLQJ WR VRPH
FULWHULDLV VDLGWRRSWLPL]H RQHZKR FKRRVHVDQDOWHUQDWLYH
WKDW PHHWV RU H[FHHGV VSHFLILHG FULWHULD EXW WKDW LV QRW
JXDUDQWHHG WR EH HLWKHU XQLTXH RU LQ DQ\ VHQVH WKH EHVW LV
VDLG WR VDWLVILFH´ >6LPRQ @ 5DWKHU WKDQ FDOFXODWLQJ
RSWLPDO DFWLRQV D VDWLVILFLQJ DJHQW VLPSO\ VHOHFWV DQ
DOWHUQDWLYHWKDWPHHWVDVHWRIDVSLUDWLRQOHYHOV$VORQJDV
WKHVHDVSLUDWLRQOHYHOVDUHEHLQJPHWWKHDJHQWFDQFRQWLQXH
WRDFWZLWKRXWH[SHQGLQJDQ\VHDUFKFRVWV:KHQDVSLUDWLRQ
OHYHOV DUH QRW PHW D VHDUFK LV H[HFXWHG XQWLO D VDWLVIDFWRU\
DOWHUQDWLYHLVIRXQG
,QRUGHUWRKDQGOHDYDULHW\RIHQYLURQPHQWVDVSLUDWLRQ
OHYHOVFDQEHDGDSWLYH$FFRUGLQJWR6LPRQ³LILWWXUQVRXW
WREHYHU\HDV\WRILQGDOWHUQDWLYHVWKDWPHHWWKHFULWHULDWKH
VWDQGDUGVDUHJUDGXDOO\UDLVHGLIVHDUFKFRQWLQXHVIRUDORQJ
ZKLOHZLWKRXWILQGLQJVDWLVIDFWRU\DOWHUQDWLYHVWKHVWDQGDUGV
DUHJUDGXDOO\ORZHUHG´>6LPRQ@
:H VHH VHYHUDO DGYDQWDJHV LQ DSSO\LQJ VDWLVILFLQJ WR
PXOWLDJHQWV\VWHPV)LUVWEHFDXVHVDWLVILFLQJLVVLPSOHDQG
IOH[LEOHLWFDQEHDSSOLHGZKHQLQIRUPDWLRQVWRUDJHVSDFH
DQGH[HFXWLRQWLPHDUH OLPLWHG7KLVPHDQVWKDW DJHQWV GR
QRWQHHGFRPSOH[PRGHOVRIRWKHUDJHQWV6DWLVILFLQJLVDOVR
UREXVW²HYHQ LI WKH HQYLURQPHQW FKDQJHV RU LQLWLDO
LQIRUPDWLRQ DERXW WKH HQYLURQPHQW LV ZURQJ D VDWLVILFLQJ
DOJRULWKPFDQW\SLFDOO\DGDSW
3
A Satisficing Strategy For the Prisoner’s
Dilemma
$SSO\LQJ 6LPRQ¶V VDWLVILFLQJ DOJRULWKP WR WKH SULVRQHU¶V
GLOHPPD LV VWUDLJKWIRUZDUG ,Q WKLV SDSHU ZH DGDSW WKH
DOJRULWKP DQG QRWDWLRQ SUHVHQWHG LQ >.DUDQGLNDU et al.
@7KHVWDWHDWWLPHtIRUDQDJHQWXVLQJWKLVVWUDWHJ\LV
JLYHQE\WKHSDLUAt,αtZKHUHAtLVDQDFWLRQLQ^&'`DQG
αt LV WKH FXUUHQW DVSLUDWLRQ OHYHO 7KH SOD\HUV¶ DFWLRQV
GHWHUPLQHWKHSD\RIIVπW$DQGπW%$IWHUUHFHLYLQJDSD\RII
πWDQDJHQWHPSOR\LQJDVDWLVILFLQJVWUDWHJ\XSGDWHVLWVVWDWH
LQ WZR VWHSV )LUVW LI π t ≥ α t WKHQ AW AW RWKHUZLVH
A t ≠ At 7KHQ DVSLUDWLRQV DUH XSGDWHG DV D ZHLJKWHG
DYHUDJHEHWZHHQWKHFXUUHQWDVSLUDWLRQOHYHODQGWKHUHFHLYHG
SD\RII 7KLV XSGDWH UXOH LV JLYHQ E\ HTXDWLRQ ZKHUH
≤λ≤
αt λα t ( ± λ )π t
,W LV ZRUWK SRLQWLQJ RXW WKDW WKH GHFLVLRQ DOJRULWKP
PDNHVQRXVHRIWKHSD\RIIPDWUL[RUWKHDFWLRQVRIWKHRWKHU
SOD\HUV 7KXV LW FDQ EH DSSOLHG WR VLWXDWLRQV ZKHUH WKLV
LQIRUPDWLRQ LV HLWKHU FRPSOH[ RU XQNQRZQ $OO WKDW LV
QHHGHGLVWKHDELOLW\WRDVVRFLDWHDSD\RIIZLWKDQDFWLRQ,Q
DGGLWLRQLWLVLPSRUWDQWWRQRWHWKDWWKLVDOJRULWKPUHTXLUHV
WKUHHSDUDPHWHUVIRUHDFKDJHQWWKHXSGDWHUDWHλDQLQLWLDO
DFWLRQA0DQGDQLQLWLDODVSLUDWLRQα
%HIRUH PRYLQJ LQWR DQ DQDO\VLV RI WKH DOJRULWKP D
VLPSOHLOOXVWUDWLRQLVZRUWKZKLOH*LYHQWKDWλ A0 &
DQGα FRQVLGHUWKHH[DPSOHLQ)LJXUH
W
7LWIRU7DW
$W
πW
αW
&
&
&
'
'
'
'
&
Figure 2: $EULHIH[DPSOHRIDVDWLVILFLQJVWUDWHJ\DJDLQVWDWLW
IRUWDWVWUDWHJ\
,Q WKLV H[DPSOH DVDWLVILFLQJDJHQW LV SOD\LQJDJDLQVW D WLW
IRUWDWVWUDWHJ\WKDWVLPSO\FRRSHUDWHVRQWKHILUVWPRYHDQG
WKHQ UHSHDWV LWV RSSRQHQW¶V ODVW PRYH RQ VXEVHTXHQW
LWHUDWLRQV ,QLWLDOO\ ERWK SOD\HUV FRRSHUDWH UHFHLYLQJ D
SD\RII RI +RZHYHU EHFDXVHWKLV SD\RII LV OHVV WKDQ WKH
VDWLVILFLQJDJHQW¶VDVSLUDWLRQRIA 'DQGWKHDVSLUDWLRQV
DUHXSGDWHGDVDQDYHUDJHRIWKHROGDVSLUDWLRQDQGWKHQHZ
SD\RII
4
Cooperation Among Satisficing Agents
4.1
%HIRUH GHVFULELQJ RXU UHVXOWV LQ GHWDLO ZH PDNH WZR
REVHUYDWLRQV)LUVWUHLQIRUFHPHQWOHDUQLQJKDVEHHQDSSOLHG
WRWKHSULVRQHU¶VGLOHPPDZLWKPL[HGUHVXOWV,Q>6DQGKROP
DQG&ULWHV@VHYHUDOW\SHVRI4OHDUQHUVZHUHVKRZQWR
SOD\RSWLPDOO\DJDLQVWDIL[HGWLWIRUWDWVWUDWHJ\+RZHYHU
GXHWRWKHLQWHUDFWLRQRIWKHLUOHDUQLQJWKHVH4OHDUQHUVKDG
GLIILFXOW\ SOD\LQJ RSWLPDOO\ DJDLQVW HDFK RWKHU 6HFRQG
DOWKRXJK WKH VDWLVILFLQJ DOJRULWKP GHVFULEHG LQ WKH ODVW
VHFWLRQ LV VLPSOH WKH G\QDPLF LQWHUDFWLRQ EHWZHHQ WZR
DJHQWVLVGLIILFXOWWRWKHRUHWLFDOO\FKDUDFWHUL]H7KXVLQWKLV
SDSHU ZH UHVWULFW RXU DQDO\VLV WR WZR VDWLVILFLQJ DJHQWV
SOD\LQJ DJDLQVW HDFK RWKHU ,Q DGGLWLRQ ZH IRFXV RQ
SUHVHQWLQJHPSLULFDOHYLGHQFHRIFLUFXPVWDQFHVXQGHUZKLFK
WKHVHWZRDJHQWVZLOOOHDUQWRFRRSHUDWH
,QRUGHUWRH[WHQGWKHQRWDWLRQWRDWZRSOD\HUJDPHZH
LQWURGXFH Bt DQG βW DV WKH VHFRQG SOD\HU¶V DFWLRQ DQG
DVSLUDWLRQOHYHO UHVSHFWLYHO\ )RUVLPSOLFLW\ λ LVVHWWRWKH
VDPHYDOXHIRUERWKSOD\HUV:HDOVRJHQHUDOL]HWKHSD\RII
PDWUL[E\VHWWLQJWKHRIIGLDJRQDOSD\RIIVWRDQG
DQGWKHQXVHσDVWKHUHZDUGIRUPXWXDOFRRSHUDWLRQDQGδDV
WKHUHZDUGIRUPXWXDOGHIHFWLRQZLWKWKHFRQVWUDLQWVWKDW
δσDQGσ!7KLVPRGLILHGSD\RIIPDWUL[LVVKRZQ
LQ)LJXUH
$¶VSD\RII%¶VSD\RII
$JHQW$¶V
&KRLFH
$JHQW%¶V&KRLFH
&RRSHUDWH
'HIHFW
&RRSHUDWH
σσ
'HIHFW
δδ
Convergence and Stability
%HIRUH SUHVHQWLQJ RXU UHVXOWV ZH GLVFXVV WKH SRVVLEOH
RXWFRPHV RI D UHSHDWHG SULVRQHU¶V GLOHPPD SOD\HG E\
VDWLVILFLQJDJHQWV7KHVLPSOHVWRXWFRPHLVFRQYHUJHQFHWRD
A
B
SDLURIDFWLRQVAB7KLVRFFXUVZKHQ α t ≤ π t DQG β t ≤ π t
PHDQLQJ WKDW ERWK SOD\HUV DUH VDWLVILHG ZLWK WKHLU FXUUHQW
SD\RIIV DQG WKXV ERWK SOD\HUV ZLOO UHSHDW WKHLU DFWLRQV
LQGHILQLWHO\$WVXEVHTXHQWLWHUDWLRQVαZLOODV\PSWRWLFDOO\
DSSURDFK π$ DQG β ZLOO DV\PSWRWLFDOO\ DSSURDFK π% 7KLV
FDQEHVHHQDVDQHTXLOLEULXPLQWKHVHQVHWKDWQHLWKHUSOD\HU
KDVDQLQFHQWLYHWRFKDQJHJLYHQWKHLUJRDOVDQGZKDWWKH\
KDYHOHDUQHGDERXWWKHLUHQYLURQPHQW
$ VHFRQG SRVVLEOH RXWFRPH LV FRQYHUJHQFH WR VRPH
DFWLRQF\FOHPHDQLQJWKDWERWKSOD\HUVUHSHDWDVHTXHQFHRI
DFWLRQSDLUVLQGHILQLWHO\$VDIRUPDOGHILQLWLRQZHVD\WKDW
WKHSOD\HUVKDYHFRQYHUJHGWRDF\FOHRIGXUDWLRQNDWWLPHτ
LIIRUDOOt!τDQGDOOkVXFKWKDW ≤ k ≤ N ± AWN AWN1
DQGBWN BWN1
$ WKLUG DQG ILQDO SRVVLELOLW\ WR FRQVLGHU LV WKDW WKH
LQWHUDFWLRQEHWZHHQWZRDJHQWVLVHQWLUHO\FKDRWLF7KLVLVDW
OHDVW YHU\ XQOLNHO\ DV WKURXJKRXW RXU UHVHDUFK WKH SURFHVV
KDVDOZD\VFRQYHUJHGWRVRPHVWDEOHRXWFRPHUHJDUGOHVVRI
WKH SD\RII PDWUL[ RU LQLWLDO FRQGLWLRQV +RZHYHU WKLV
UHPDLQVWREHVKRZQWKHRUHWLFDOO\
)LJXUHLVDEULHILOOXVWUDWLRQRIWKHFRPSOH[LW\RIWKH
SURFHVV ,W GHSLFWV WKH RXWFRPH DV D IXQFWLRQRI WKH LQLWLDO
DVSLUDWLRQV IRU WKUHH SRVVLEOH JDPH VWUXFWXUHV DQG LQLWLDO
DFWLRQV &OHDUO\ WKHUH LV QR VLPSOH PDWKHPDWLFDO
FKDUDFWHUL]DWLRQ RI WKH UHODWLRQVKLS EHWZHHQ JDPH VWUXFWXUH
DQG LQLWLDO SDUDPHWHUV DQG FRQYHUJHQFH WR FRRSHUDWLRQ
+RZHYHUHPSLULFDOUHVXOWVSUHVHQWHGLQWKHQH[WVHFWLRQGR
DOORZXVWRLGHQWLI\FRQGLWLRQVXQGHUZKLFKWKHVHDJHQWVZLOO
OHDUQWRFRRSHUDWH
Figure 3: *HQHUDOL]HGSD\RIIPDWUL[IRUWKHSULVRQHU¶VGLOHPPD
(a)
(b)
2.0
β0
(c)
2.0
β0
1.0
0.5
1.0
α0
1.5
2.0
2.0
β0
1.0
1.0
0.5
1.0
α0
1.5
2.0
0.5
1.0
α0
1.5
2.0
Figure 4:7KHVHWKUHHJUDSKVVKRZWKHUHODWLRQVKLSEHWZHHQLQLWLDODVSLUDWLRQVDQGWKHILQDORXWFRPHIRUWKUHHGLIIHUHQWJDPHVWUXFWXUHV)RU
HDFKSDLURILQLWLDODVSLUDWLRQVLQWKHJUDSKWKHRXWFRPHRIWKHJDPHZDVUHFRUGHG:KLWHLQGLFDWHVFRQYHUJHQFHWRPXWXDOFRRSHUDWLRQEODFN
LQGLFDWHVFRQYHUJHQFHWRPXWXDOGHIHFWLRQDQGJUD\LQGLFDWHVFRQYHUJHQFHWRVRPHF\FOH,Q)LJXUHD$ '% 'σ δ DQGλ
,Q)LJXUHE$ &% &σ δ DQGλ ,Q)LJXUHF$ '% &σ δ DQGλ 4.2 General Results
:HVHWXSDVLPXODWLRQWKDWUDQGRPO\VHOHFWVWKHSDUDPHWHUV
IRUDJDPHIURPXQLIRUPGLVWULEXWLRQVDVGHVFULEHGLQ7DEOH
Parameter
Min. Value
Max. Value
αβ
λ
σ
σ
δ
A0, B0
& '
RI WKHVH SDUDPHWHUV DIIHFW FRQYHUJHQFH WR PXWXDO
FRRSHUDWLRQ
Initial Aspirations
)LJXUH VKRZV D FRQWRXU SORW RI WKH IUHTXHQF\ RI
PXWXDO FRRSHUDWLRQDVDIXQFWLRQRILQLWLDODVSLUDWLRQV,WLV
FOHDU WKDW KLJK DVSLUDWLRQV DUH PRUH OLNHO\ WR OHDG WR
FRRSHUDWLRQ $W ILUVW WKLV PD\ DSSHDU FRXQWHULQWXLWLYH²
SOD\HUVZLWKKLJKDVSLUDWLRQVPLJKWEHXQZLOOLQJWRVHWWOHIRU
FRRSHUDWLRQ+RZHYHULQPRVWFLUFXPVWDQFHVERWKSOD\HUV
DUH DEOH WR OHDUQ WKDW WKH\ FDQQRW H[SHFW PRUH WKDQ PXWXDO
FRRSHUDWLRQLQWKHORQJUXQ2QWKHRWKHUKDQGSOD\HUVZLWK
ORZ DVSLUDWLRQV WHQG WR UHPDLQ VDWLVILHG ZLWK PXWXDO
GHIHFWLRQRUVHWWOHLQWRF\FOHV
Table 1: 'LVWULEXWLRQRISDUDPHWHUVIRUVLPXODWLRQV
2
7KHVLPXODWLRQWKHQUXQVDUHSHDWHGSULVRQHU¶VGLOHPPDXQWLO
WKH SURFHVV FRQYHUJHV WR VRPH DFWLRQ SDLU RU VRPH DFWLRQ
F\FOH7KHILQDORXWFRPHVRIRIWKHVHVLPXODWLRQVDUH
GLVSOD\HGLQ)LJXUH
1.6
70- 80
1.2
Beta
0.8
60- 70
50- 60
40- 50
0.4
30- 40
20- 30
0
0
0.4
'''&'' ''&&'&
&'
0.8
1.2
1.6
2
Alpha
''
&&
Figure 5: )UHTXHQFLHVRIHDFKRIWKHSRVVLEOHRXWFRPHVIURP
WULDOV3DUDPHWHUVZHUHUDQGRPO\VHOHFWHGDVGHVFULEHGLQ7DEOH
Figure 6: $FRQWRXUSORWRIWKHSHUFHQWDJHRIWULDOVRXWRIWKDW
FRQYHUJHGWRPXWXDOFRRSHUDWLRQDVDIXQFWLRQRILQLWLDODVSLUDWLRQV
/LJKWFRORUVLQGLFDWHWKDWLQPRVWRIWKHWULDOVZLWKWKHJLYHQLQLWLDO
DVSLUDWLRQVWKHDJHQWVOHDUQHGWRFRRSHUDWH'DUNFRORUVLQGLFDWH
WKDWIHZRIWKHWULDOVOHGWRPXWXDOFRRSHUDWLRQ3DUDPHWHUVRWKHU
WKDQαDQGβZHUHVHOHFWHGUDQGRPO\DVGHVFULEHGLQ7DEOH
Structure of the Payoff Matrix
7KH VWUXFWXUH RI WKH SD\RII PDWUL[ FDQ DOVR KDYH
FRQVLGHUDEOH LQIOXHQFH RYHU WKH DELOLW\ RI WKH DJHQWV WR
FRQYHUJH WR OHDUQ WR FRRSHUDWH )LJXUH VKRZV WKH
IUHTXHQF\RIPXWXDOFRRSHUDWLRQDVDIXQFWLRQRIσDQGδ
,WLVLQWHUHVWLQJWRQRWHWKDWHYHU\JDPHFRQYHUJHGWRRQHRI
IRXU SRVVLELOLWLHV PXWXDO FRRSHUDWLRQ PXWXDO GHIHFWLRQ
VRPH YDULDWLRQ RQ '''&''&' RU VRPH YDULDWLRQ RQ
''&&'&
0.9
0.8
0.7
4.3 Factors Leading to Cooperation
$V VKRZQ SUHYLRXVO\ FRQYHUJHQFH WR PXWXDO
FRRSHUDWLRQ LV WKH PRVW IUHTXHQW RXWFRPH LQ D SULVRQHU¶V
GLOHPPD SOD\HG E\ WZR VDWLVILFLQJ DJHQWV 6HYHUDO IDFWRUV
LQIOXHQFH WKLV OHDUQLQJ SURFHVV EHWZHHQ LQWHUDFWLQJ DJHQWV
7KHVHDUH
•
•
•
•
LQLWLDODVSLUDWLRQV
VWUXFWXUHRIWKHSD\RIIPDWUL[
OHDUQLQJUDWHDQG
LQLWLDODFWLRQV
7KHUHPDLQGHURIWKLVVHFWLRQIRFXVHVRQDQDO\]LQJKRZHDFK
0
0.15
0.3
0.45
0.6
0.75
0.9
Sigm a
80- 100
60- 80
0.6
40- 60
0.5
20- 40
0- 20
Delta
Figure 7: $FRQWRXUSORWRIWKHSHUFHQWDJHRIWULDOVRXWRIWKDW
FRQYHUJHGWRPXWXDOFRRSHUDWLRQDVDIXQFWLRQRIHDFKδ, σSDLU
/LJKWFRORUVLQGLFDWHWKDWPRVWRIWKHWULDOVFRQYHUJHGWRPXWXDO
FRRSHUDWLRQZKLOHGDUNFRORUVLQGLFDWHWKDWIHZRIWKHWULDOV
FRQYHUJHGWRFRRSHUDWLRQ3DUDPHWHUVRWKHUWKDQδDQGσZHUH
FKRVHQUDQGRPO\DFFRUGLQJWR7DEOH
1RWHWKDWFRRSHUDWLRQLVPRVWOLNHO\ZKHQδLVVPDOODQGσLV
ODUJH 7KLV LV H[SHFWHG EHFDXVH WKH GLVWLQFWLRQ EHWZHHQ
FRRSHUDWLRQ DQG GHIHFWLRQ EOXUV ZKHQ σ DQG δ DUH FORVH
WRJHWKHU 7KLV W\SH RI EHKDYLRU VHHPV W\SLFDO RI QRQ
RSWLPL]LQJDOJRULWKPV,QGHVFULELQJKLVZRUNLQPRGHOLQJ
KXPDQEHKDYLRU$UWKXUZULWHVWKDWKXPDQEHKDYLRUDQGKLV
DOJRULWKP ³DSSHDU WR µGLVFRYHU¶ DQG H[SORLW WKH RSWLPDO
DFWLRQZLWKKLJKSUREDELOLW\as long as it is not difficult to
discriminate %XW EH\RQG D SHUFHSWXDO WKUHVKROG ZKHUH
GLIIHUHQFHV LQ DOWHUQDWLYHV EHFRPH OHVV SURQRXQFHG QRQ
RSWLPDORXWFRPHVEHFRPHPRUHOLNHO\´>$UWKXU@
Initial Actions
7RVWXG\WKHHIIHFWVRILQLWLDODFWLRQVRQFRRSHUDWLRQZH
UDQIRXUVHWVRIVLPXODWLRQVKROGLQJGLIIHUHQWLQLWLDODFWLRQV
FRQVWDQW HDFK WLPH 7KH SHUFHQWDJHV RI VDPSOHV WKDW
FRQYHUJHWRFRRSHUDWLRQIRUHDFKJURXSDUHVKRZQLQ7DEOH
Initial Actions
% of Cooperation
5DQGRP
&&
''
'&RU&'
Table 2: 3HUFHQWDJHRIFRRSHUDWLRQRXWRIWULDOVDVD
IXQFWLRQRILQLWLDODFWLRQV3DUDPHWHUVRWKHUWKDQ$DQG%
ZKHUHFKRVHQDFFRUGLQJWR7DEOH
:KLOH LQLWLDO DFWLRQV GR QRW DSSHDU WR EH DV VLJQLILFDQW DV
RWKHU IDFWRUV QRWH WKDW FRRSHUDWLRQ RFFXUV ZLWK WKH VDPH
SHUFHQWDJH UHJDUGOHVV RI ZKHWKHU WKH LQLWLDO DFWLRQV DUH
FRRSHUDWLRQRUGHIHFWLRQDVORQJDVERWKSOD\HUVFKRRVHWKH
VDPHDFWLRQ
Learning Rate
7KHUDWHDWZKLFKWKHDVSLUDWLRQVDUHXSGDWHGDOVRKDVD
FRQVLGHUDEOH HIIHFW RQ ZKHWKHU PXWXDO FRRSHUDWLRQ LV
OHDUQHG)LJXUHVKRZVWKHUHODWLRQVKLSEHWZHHQλDQGWKH
SHUFHQWDJH RI WULDOV WKDW FRQYHUJHG RQ PXWXDO FRRSHUDWLRQ
$V λ LQFUHDVHV WKH IUHTXHQF\ RI FRRSHUDWLRQ LQFUHDVHV DV
ZHOO7KHRQO\H[FHSWLRQLVZKHQλ DQGWKXVDVSLUDWLRQV
DUHQRWXSGDWHGDWDOOOHDGLQJWRYLUWXDOO\QRFRRSHUDWLRQ
% of Cooperation
100
80
60
40
20
0
0
0.2
0.4
0.6
0.8
1
Lam bda
Figure 8: 3HUFHQWDJHRIWULDOVRXWRIWKDWFRQYHUJHGWRPXWXDO
FRRSHUDWLRQDVDIXQFWLRQRIWKHXSGDWHUDWHλ3DUDPHWHUVRWKHU
WKDQλZHUHVHOHFWHGUDQGRPO\DVGHVFULEHGLQ7DEOH
5
Conclusions and Further Work
7RVXPPDUL]HWKHUHVXOWVRIWKHSUHYLRXVVHFWLRQZHUHVWDWH
ILYH LPSRUWDQW IDFWRUV WKDW LQFUHDVH WKH OLNHOLKRRG WKDW WZR
VDWLVILFLQJDJHQWVZLOOOHDUQWRFRRSHUDWH
•
•
•
•
$JHQWVVKRXOGOHDUQEXWVORZO\
7KHGLIIHUHQFHEHWZHHQSD\RIIVIRUPXWXDO
GHIHFWLRQDQGPXWXDOFRRSHUDWLRQVKRXOGEH
PD[LPL]HG
$JHQWVVKRXOGKDYHKLJKLQLWLDODVSLUDWLRQV
$JHQWVVKRXOGVWDUWRXWZLWKVLPLODUEHKDYLRU
$VDWHVWRIWKHVHSULQFLSOHVZHUDQDILQDOVHWRIVLPXODWLRQV
HQIRUFLQJWKHIROORZLQJFRQGLWLRQVA0 = B0σδ!!
λ ! α ! σ DQG β ! σ 8QGHU WKHVH FRQGLWLRQV WKH
DJHQWVOHDUQWRFRRSHUDWHLQRIWULDOV
7KHVH UHVXOWV PDNH D SURPLVLQJ FDVH IRU WKH XVH RI
VDWLVILFLQJLQPXOWLDJHQWV\VWHPVDVDZD\RIEDODQFLQJVHOI
LQWHUHVWDQGFRPPRQJRRGZKHQOLWWOHLQIRUPDWLRQDERXWWKH
HQYLURQPHQW LV DYDLODEOH %HFDXVH DJHQWV GR QRW GLUHFWO\
PRGHOHDFKRWKHUWKHDSSURDFKLVIDVWVLPSOHDQGVFDODEOH
WRPDQ\SOD\HUV
$V D ILQDO QRWH ZH UHFRJQL]H WKDW WKHUH DUH VHYHUDO
GLUHFWLRQV IRU IXUWKHU ZRUN WKDW VKRXOG SURYH XVHIXO DQG
LQWHUHVWLQJ WR UHVHDUFKHUV LQ PXOWLDJHQW V\VWHPV :H KDYH
OLPLWHG RXU GLVFXVVLRQ RI WKLV VDWLVILFLQJ DOJRULWKP WR WKH
SULVRQHU¶V GLOHPPD +RZHYHU EHFDXVH QR DVVXPSWLRQV
DERXWWKHUHODWLRQVKLSVEHWZHHQWKHSD\RIIVKDYHEHHQEXLOW
LQWRWKHDOJRULWKPLWVKRXOGH[WHQGHDVLO\WRRWKHUGRPDLQV
,Q DGGLWLRQ WKH DOJRULWKP ZH KDYH SUHVHQWHG LV OLPLWHG WR
WZRDFWLRQ GHFLVLRQ SUREOHPV ZLWK LPPHGLDWH IHHGEDFN
7KXV WKH DGGLWLRQ RI D VDWLVILFLQJ VHDUFK DOJRULWKP IRU
PXOWLSOHDFWLRQVLVQHFHVVDU\DQGDQH[WHQVLRQWRVHTXHQWLDO
GHFLVLRQ SUREOHPV ZRXOG SURYH XVHIXO IRU PDQ\
DSSOLFDWLRQV
Acknowledgements
7KH DXWKRUV JUDWHIXOO\ DFNQRZOHGJH WKH RI VXSSRUW RI WKH
1DWLRQDO6FLHQFH)RXQGDWLRQXQGHUJUDQW&06
References
>$UWKXU@:%ULDQ$UWKXU'HVLJQLQJHFRQRPLFDJHQWV
WR DFW OLNH KXPDQ DJHQWV $ EHKDYLRUDO DSSURDFK WR
ERXQGHG UDWLRQDOLW\ The American Economic Review
0D\
>$[HOURG @ 50 $[HOURG The Evolution of
Cooperation%DVLF%RRNV
>&RQOLVN @ -RKQ &RQOLVN :K\ ERXQGHG UDWLRQDOLW\"
Journal of Economic Literature
>+X DQG :HOOPDQ @ - +X DQG 0 3 :HOOPDQ
0XOWLDJHQW UHLQIRUFHPHQW OHDUQLQJ 7KHRUHWLFDO
IUDPHZRUN DQG DQ DOJRULWKP Proceedings of the
Fifteenth International Conference on Machine
Learning6DQ)UDQFLVFR0RUJDQ.DXIPDQ
>.DUDQGLNDUet [email protected]'0RRNKHUMHH'
5D\ DQG ) 9HJD5HGRQGR (YROYLQJ DVSLUDWLRQV DQG
FRRSHUDWLRQJournal of Economic Theory
>/LWWPDQ @ 0LFKDHO / /LWWPDQ 0DUNRY JDPHV DV D
IUDPHZRUN IRU PXOWLDJHQW UHLQIRUFHPHQW OHDUQLQJ
Proceedings of the Eleventh International Conference
on Machine Learning6DQ)UDQFLVFR0RUJDQ
.DXIPDQ
>6DQGKROP DQG &ULWHV @ 7XRPDV : 6DQGKROP DQG
5REHUW+&ULWHV0XOWLDJHQWUHLQIRUFHPHQWOHDUQLQJLQ
WKH,WHUDWHG3ULVRQHU¶V'LOHPPDBioSystems
>6HQet al. @66HQ06HNDUDQDQG-+DOH/HDUQLQJ
WRFRRUGLQDWHZLWKRXWVKDULQJLQIRUPDWLRQProceedings
of the Twelfth National Conference on Artificial
Intelligence$$$,6HDWWOH:$
>6LPRQ @ +HUEHUW $ 6LPRQ Models of bounded
rationality. 9RO Empirically grounded economic
reason&DPEULGJH0DVV0,73UHVV