Code and data structure implications when
patching without local storage… and resource
management overall.
Ken Demarest
[email protected]
[email protected]
Background on this talk
This is a thought experiment – it all started with Jon asking me
what hard problems I had heard about recently…
Patching consoles… Turns out I never did it, so…




Focus on high concepts, ignores numerous practical issues
Make assumptions about a future that may not occur
Goal is to drive discussion, out-of-the-box thought
Gather some current best practices relative to future improvements
And maybe… perspective for console guys who might not have
the depth of online experience PC developers have, and PC guys
who are unused to storage limitations
Initial Poll
Who has ever delivered a patch to their game?
Who uses late linking of data?
Who has edit-and-continue in their data path?


What areas cause level resets?
What areas invalidate edit-and-continue?
Whose products support mods / plugins?
Who is console only (vs PC or console+PC)?
Situation
Next gen consoles (PS3, XB2, PSP, DS) will provide connectivity
Sending data to console games post-sale will enable new
revenue opportunities





Patching (should never happen, of course)
Content unlock (fill every DVD)
Content extension (Lode Runner maps)
Community mods (console mods built on PC)
Marketing intrusion… I mean surveys, advertising, dynamic product
placement, up-sell, cross-sell
Storage may be limited


Never as massive as PC storage – MMO ‘Patch Kings’
Current PS2 no HD; Microsoft’s HD ‘nut’; trend to mem cards
Data Transfer Options
Require storage

But even then you won’t be the only app to utilize it
Send data every time



Plausible, but slow
Think of online connection as a ‘remote hard drive’
You’ve got ‘more bandwidth than storage’ (ex: 60%
broadband, 40% modem on PS2 today)
Conclusion: any data sent should be as small as
possible… but how?
Reality Check
North American PS2 install base is ~30M





3.2M connected users
Appears unlikely to break 30% connectivity in near- to mid-term
Europe still hamstrung
Japan uptake less certain - <1M today; Korea likely uptake
But all new PS2s have network adapter built in
Still, 30% of 30M is 9M users – the size of the largest PC game
penetration, and about 18x the size of the MMO audience
Assuming XB2 leverages MSN/Zone/Passport user base, we
could see big numbers here as well
So still a viable profit center
Considered as a patch problem, how do we
keep data transfer small?
Today’s code & data layout often resists efficient patch sizes


Code fixups
Data aggregation, indexes, etc
The good news:


Simple techniques can ease the problems
They (might) ripple into greater build efficiency
The bad news:



Many techniques imply slower loads
Might have to change your fundamental resource management
architecture
But hey, you’ve got a while to make these systemic changes
Code Options
Warning: Absurd ideas herein for the sake
of illustration
Stop fixup problems



DLLs (or equivalent – MS TDR)
Ship the obj files – link at run time
Fixed, predictable link order
 Imagine an order-retaining linker
 Post ship, name everything zz1, zz2, etc
Code hooks (like a plug-in architecture… or a mod)



In main loop, around mem allocations, root ancestor constructor/destructor,
DLL detection, kernel process insertion, etc.
Code intercept installation on load
All facilitated with a compact game state
Manual aspect-orientation

Crazy VMT redirect [applyDamage()]
Data has troubles too…
Sample data path:
Source
GameFriendly
Hot spots







Derived
Data
Link
Compress
Concat
Embed
Illustrative – not necessarily in order; each step may
also happen multiple times
Getting data game-friendly seldom has order guarantees
Some source data has Butterfly Effect on derived data
Derived data takes a long time to generate; often large (otherwise
you wouldn’t need to pre-generate it)
Compression may cross assets (text?)
Internal linkage lacks persistent references
Concat indices often resemble code fixups
Embedded concats only make it worse
Data Options
The biggies



Model data build on code build principles
Dynamic linking of data at (re)load time
Concat using relocatable, sector-based approach
 (the irony of FAT32; sector sorting; tell FAT to order files?)
The others


Modularity and predictable ordering in all data, to facilitate a
diff
Compression optional (or diff-friendly via window borders)
When Sending Data or Code…
Date-stamped resources make atomic changes clear,
QA-able
Diff then compress before send
Merge at load time (no local storage, remember?)
Only send when game-relevant
Veering to a related thought experiment…
Lets suppose that you never had to QA anything, or
worry about load times
Your end-users could theoretically get build results
just like the team does (suspend disbelief, please…
and Yes, MMOs appear to do this already…)
Hey, quality upload == quality build?
Conceptually, delivery of patches/upgrades/etc to customers is no
different than delivery of build updates to your team (except for QA,
load times, delivery infrastructure, blah blah)
HL2, Halo2, Oddworld all have facets of the data delivery techniques
discussed earlier


Trim resource management tends to yield faster builds
Tends to prepare you for efficient online delivery too
Mod community has had many of these techniques for years – yet
projects still let their build environment stray from the faith, often for
very valid reasons
How might these techniques point us towards ‘data delivery nirvana’
conjoined with ‘build environment nirvana’? Can TCP/IP distribution of
code and data be unified across target audiences (team/players)?
A few pathological cases…
Artist adds wrinkles to a character’s forehead, and all players see it
immediately
Dev team and the community make simo-changes to levels, and
everybody sees it at once
Content rights (ownership) is a characteristic of both the dev team and
players

i.e. what content you can see/play, as well as your rights to make changes
We’re heading over the edge, so why stop now…
Straw Man (1)
Warning: Blatant impossibilities ahead. No
throwing of rotten tomatoes, please.
All tools are also servers of their data and services via TCP/IP





Maya, Max, Photoshop, your crazy facial animation tool
Via a uniform format (SOAP? XMLRPC?)
Capable of evincing change deltas, not just entire data units
Via a perfectly uniform resource system with perfect pointer management
Game displays insta-results and multi-user simo editing on live levels
Late linking, and persistent symbolic links allow data-based ‘edit and continue’
All code and data compilation is always distributed and cached


Data validation is so good it isn’t possible for the team to get stopped
Oh, and automated code unit testing makes this true for code as well
Derived data is never required – only tasked out then later loaded when ready
Just-in-time auto-concatenation (by the game?) groups data for most efficient
load times
Straw Man (2)
OK, maybe one small tomato
Validation extends to code


A bot plays through the levels before you see them, finds bugs
before QA
Bugs are always fully repeatable, often across builds
Code and scripting both always link late, fast, dynamic re-link
The game never halts – the code is so robust it even saves
game state across major recompiles, effectively never halting
Publish to team members and publish to players only vary in the
amount of QA

In fact, player distribution is just a version control sync
Discussion
What in the straw man, ignoring constraints, is undesirable?
What are key obstacles to achieving the straw man?
What other high-concept, perfect-world systems would you like to see
in a next-generation resource management / data delivery system? (or
open discussion to asset chain overall)
Cost Analysis Poll
What % of tech time in your org is spent creating system (vs. leaf):
20%, 40%, 60%, 80%?
Data full recompile: <10 min, <30 min, <1 hr, <2 hr, more?
Code full recompile: <5 min, <10 min, <30 min, <1 hr, more?
Who runs metrics on code effort vs. lost team time?
Next Gen Tools Poll
Who journals, or uses techniques to improve bug reproduction?
Who uses unit testing? Expects to use it?
Focusing on techniques to debug multithreaded pain?
Fully generic layer for tweaking data values (with a GUI layer where
needed)?
Who sees integration of 3rd party tools as a major part of their next
product?
Other techniques?
Next Gen Resource Chain Stealth Talk
Ken Demarest
[email protected]