US007051180B2
(12)
(54)
(75)
United States Patent
(10) Patent N0.:
Downer et a].
(45) Date of Patent:
US 7,051,180 B2
May 23, 2006
MASTERLESS BUILDING BLOCK BINDING
6,075,938 A
6/2000 Bugnion et 31.
TO PARTITIONS USING IDENTIFIERS AND
6,088,770 A
7/2000 Tarui et a1.
INDICATORS
6,345,317 B1*
2/2002
6,564,252 B1*
5/2003 Hickman et a1.
709/214
6,647,508 B1* 11/2003 Zalewski et a1.
711/173
Inventors: ‘7Va
6A- Downer’ Portland’ OR
S ;
Brut: M- Gilbert’ Beaverton’ ORG} )
(US); Thomas D. Lovett, Portland, OR
(Us)
Takeda ........................ .. 710/2
6,728,709 B1* 4/2004 Plasek et a1. ............. .. 711/152
OTHER PUBLICATIONS
Nanda et al., “Mapping Applications onto a Cache Coherent
(73) Assignee: International Business Machines
Corporation, Armonk, NY (US)
Multipreeessen” Proceedings on Supercomputing ,92’ PP'
368'377 (1992)'*
_
US 6,021,479, 02/2000, Stevens (Withdrawn)
*
( )
Notice:
Sub'ect to an
disclaimer, the term of this
pateJnt is extended or adjusted under 35
U.S.C. 154(b) by 363 days.
*
-
-
cued by exammer
Primary ExamineriPierre-Michel Bataille
(74) Attorney, Agent, or FirmiAbdy Raissinia
(21) Appl. No.: 10/045,796
(57)
(22)
Filed:
Jan. 9, 2002
(65)
_
Prior Publication Data
Us 2003/0131214 A1
ABSTRACT
_
_
_
A masterless approach brnds multrprocessor burldrng blocks
1111' 10’ 2003
to partitions of a computer system using identi?ers and
indicators. A number of building blocks communicate
among each other to determine a partition to Which each
building block is to be partitioned. For each unique partition
(51)
Int‘ C1‘
to Which one or more of the building blocks is to be
G06F 12/00
(52)
(58)
(200601)
partitioned, the building blocks communicate among each
US. Cl. ...................... .. 711/173; 709/215; 711/156
Field of Classi?cation Search .............. .. 709/214,
other to determine building block uniqueness’ and then each
Of the building hleeks joins the panitieh The building
709/215, 201; 707/8; 711/173, 170, 154,
711/147, 156
blocks share With one another their logical port identi?ers,
Which uniquely identify the building block Within a parti
See application ?le for Complete Search 11151013’(56)
References Cited
U.S. PATENT DOCUMENTS
tion. A commit indicator of each building block indicates
that the building block has committed itself to the partition
and that its identi?ers cannot be changed. A partition protect
indicator is set by one building block of a partition, pre
3,641,505 A
2/1972 ArtZ et a1.
venting changes to the commit indicators of other building
blocks Wishing to join the partition except by that one
5,887,138
5,893,144
5,905,998
5,926,829
3/1999
4/1999
5/1999
7/1999
Hagersten et a1.
Wood et a1.
Ebrahim et a1.
Hagersten et a1.
block protect indicators protect the building blocks them
6,035,378 A *
3/2000
James
6,049,853 A
4/2000 Kingsbury et a1.
A
A
A
A
building block, e?‘ectively protecting the partition. Building
selves.
...................... .. 711/147
27 Claims, 20 Drawing Sheets
U.S. Patent
May 23, 2006
Sheet 1 or 20
US 7,051,180 B2
FIG 1
102
104
106
COMMUNICATE AMONG BUILDING
BLOCKS TO DETERMINE
RESPECTIVE PARTITIONS FOR
BUILDING BLOCKS TO JOIN
FOR EACH UNIQUE PARTITION,
COMMUNICATE AMONG BUILDING
BLOCKS OF PARTITION TO
DETERMINE BOOT BUILDING
BLOCK FOR PARTITION
JOIN APPROPRIATE PARTITION BY
EACH BUILDING BLOCK, AND
PERFORM ACTIONS AS
APPROPRIATE TO PROTECT
PARTITION AND BUILDING BLOCK
100
U.S. Patent
May 23, 2006
Sheet 4 0f 20
US 7,051,180 B2
FIG 4
BUILDING BLOCK [US AND
400
INDICATORs
\I\I
PHYSICAL PORT ID
'42
LOGICAL PORT ID
M
PARTITION ID
"4i
PROTECT INDICATOR
w
COMMIT INDICATOR
"m
STATUS INDICATOR
£2
U.S. Patent
May 23, 2006
Sheet 5 0f 20
US 7,051,180 B2
FIG 5A
RESET BUILDING
502/‘,
BLOCK
I
SELECT ONE
504% PROCESSOR AS BSP
I
506
INITIALIZE MINIMUM
L/I/
HARDWARE AND
SOFTWARE
I
508W DETERMINE PHYSICAL
PORT ID
I
ENABLE AND UNPROTECT
510W
BUILDING BLOCK AND
REMOVE FROM PARTITION
I
512
SEND PHYSICAL PORT ID TO
L/‘/ OTHER BUILDING BLOCKS
I
514
INITIALIZE REMAINING
HARDWARE AND SOFTWARE
W AND RESET IF NECESSARY
516W
DETERMINE
PARTITION ID
518
SEND PARTITION ID
I/I/ TO OTHER BUILDING
BLOCKS
500a
U.S. Patent
May 23, 2006
WAIT TO RECEIVE PHYSICAL PORT
ID AND PARTITION ID FROM OTHER
BUILDING BLOCKS
520
US 7,051,180 B2
Sheet 6 0f 20
FIG 5B
ANY
522
BUILDING
BLOCKS TIMED
OUT?
REMOVE FROM
PART|TION(S)
NO
SEND ALL PHYSICAL PORT
526
ID'S OF BUILDING BLOCKS IN <_—
SAME PARTITION
I
527
RECEIVE ALL PHYSICAL PORT ID'S
OF BUILDING BLOCKS IN SAME
PARTITION FROM EACH BUILDING
BLOCK IN SAME PARTITION
528
530
I
I
DETERMINE
LOGICAL PORT ID
REINITIALIZE HARDWARE
AND SOFTWARE WITH
LOGICAL PORT ID
I
530
SEND LOGICAL PORT ID TO
OTHER BUILDING BLOCKS IN
SAME PARTITION
500D
U.S. Patent
May 23, 2006
Sheet 7 0f 20
WAIT TO RECEIVE LOGICAL PORT
534
ID'S FROM OTHER BUILDING
\I\I
BLOCKS IN PARTITION
US 7,051,180 B2
500C
I
VERIFY THAT OTHER BLOCKS
HAVE RECEIVED SAME PHYSICAL
PORT ID'S FOR PARTITION AND
THAT LOGICAL ID'S ARE UNIQUE
536
I
BOOT PARTITION IF HAVE LOWEST
LOGICAL PORT ID
“538
HAVE PARTITION ADD
BUILDING BLOCK TO
PARTITION
PROTECTED?
540
ADD SELF TO
PARTITION (COMMIT)
544
PARTITION (COMMIT) \f\>
546
l
WAIT FOR OTHER BUILDING
BLOCKS TO COMMIT TO
PARTITION
<—__
I
SEND BOOT ID TO OTHER
BUILDING BLOCKS IN PARTITION
N548
U.S. Patent
May 23, 2006
Sheet 8 0f 20
WAIT TO RECEIVE BOOT ID
550/»
FIG 5D
FROM OTHER BUILDING
BLOCKS IN PARTITION AND
VERIFY
I
I
I
552
WRITE PROTECT
BUILDING BLOCK
554
PROTECT
PARTITION
WAIT FOR OTHER BUILDING
BLOCKS TO PROTECT
556
I/l/
THEMSELVES AND
PARTITION
LOGICAL PORT
558
ID = BOOT ID?
REINITIALIZE HARDWARE
560
I/‘\/
AND SOFTWARE OF
BUILDING BLOCKS OF
PARTITION
I
562/1,
BOOT OS
564/»,
DONE
US 7,051,180 B2
NO
<_—
U.S. Patent
May 23, 2006
Sheet 9 0f 20
US 7,051,180 B2
FIG 6
HALT MEMORY
602
USAGE OF BUILDING
W
BLOCK
HALT I/O ACTIVITY
ON BUILDING
604
M"
606
W
608
W
BLOCK
HALT PROCESSOR
TASKS ON
BUILDING BLOCK
WITHDRAW
RESOURCES OF
BUILDING BLOCK
TURN OFF COMMIT
610
INDICATOR OF
W BUILDING BLOCK
600
U.S. Patent
May 23, 2006
Sheet 11 of 20
US 7,051,180 B2
0x7_FFFF_FFFF
3288-13
High Extended
Memory
l-IGB
llGB-lBNG
"GB-ZONE
llGB-32MG
High BIOS (PCI-U)
Processor Specific
Chinset Specific
0X1_0000_0000
0X0_FEOO_0000
PcI-o
PCH
PcI-2
MMIO Space
(512MB per Pcl)
PCI-3
0X0_8000_0000
2GB
Medium Extended
Memory
0X0_0100_0000
16MB
Low Extended
Memory
0X0_0010_0000
1MB
Segments 8 - F
(Manned to PCI or Mem)
512KB
Memory
F IG. 8
0X0_0008_0000
0X0_0000_0000
U.S. Patent
395
May 23, 2006
50:2
Sheet 12 0f 20
2:P59.;
N":
US 7,051,180 B2
153 28
553 c; HNDMEQ 32:?8$5 2E 85
.682
mm
A0?”;
0.QE
mm
ow
EM;l vml
U.S. Patent
May 23, 2006
.cmhz
02
Sheet 13 0f 20
1mmtm QE
IQtwma
F1:58amUJI‘ m? 35T82 iQ582M Al Il-[l
:pE uE “p5:8 2H58m
H
léfgims TILEiSQBU #LES
452631 :
US 7,051,180 B2
k
U.S. Patent
May 23, 2006
Sheet 14 0f 20
US 7,051,180 B2
70%~
N0 INPUT
140
Bypass
Even
InDut
MUX
\
50
N2 INPUT
‘
a2
>A
+
Bypass
Odd
r1433“
J
51
F IG. 1 1 A
U.S. Patent
May 23, 2006
,.____’
US 7,051,180 B2
Sheet 15 0f 20
external SRAHs
512K by 16 bit each
1: chips wide
Even Tag
Comparator and
Dispatcher
Diag. RIH access
Disp. Buffer
Even/Pipeline
52
TRID Logic
A4
53\
Odd Pipeline
Disp. Buffer
A
Update
Odd Tag
Comparator and
Dispatcher
external SRANs
512K by 16 bit each
lrchips wide
FIG.11B
U.S. Patent
May 23,2006
Sheet 16 0f 20
US 7,051,180 B2
Output MuxO
NO OUTPUT
U5
Output Muxl
1-
N1 OUTPUT
B4
Output Mux2
N2 OUTPUT
n7
Output Mux3
N3 OUTPUT
ll8
FIG.11C
U.S. Patent
May 23, 2006
2E5uow
2H:65
Sheet 17 0f 20
US 7,051,180 B2
D26H2535m55
QLwm
HKw“
NE:m:8H.t8:2:5
2:382?m whom
_
mg25m2:£2wEH2%._
2
3
H
5
9
5
5c2m>m
U.S. Patent
Port 0 part #
May 23, 2006
Port 1 part #
Port 2 part #
Port 3 part #
U.S. Patent
May 23, 2006
Sheet 19 0f 20
Port 0 Partition
Configuration Register
US 7,051,180 B2
/’89
31 Reserved
30
29
28
27
26
25
2!!
23
22
21
20
19
18
17
16
15
111
13
12
POEnable
v91
POPartWrProt
PUPortWrProt
PDConmit
POLOQ ID
~77
~78
~79
""‘-92
PDParNum
»-~--93
FIG. 14