Scott Hanselman

I had a chance recently to chat with (Dr.) Douglas Beck, an old friend from my BBS days. Doug got his start in Mobile on early Windows CE systems as Digital Concepts and created some of the very best games ever seen on that platform and now develops for the Playstation Portable (PSP).

Since most readers of this blog are .NET folks, I thought it'd be interesting to talk to Doug and understand the differences between our two worlds. I used to develop on the Palm Pilot in C, so I have a tiny bit of experience on "Tiny Systems." However, the PSP isn't a tiny system at all, it's a freaking beast (in the best possible way.)  Let's see how different life is on a PSP.

Scott: So, Doug, we've known each other for how long? 15 years? But lost touch for 10?

Doug: Yeah, right around 15 years =)  It's was refreshing to see how much you'd built your resume since we'd last talked.  It's always good to see someone excel in their field.

Scott: Thanks, but seriously, you're a Doctor now?

Doug: I was definitely a career student until I ran out of degrees to get.  My Ph.D. is in Electrical Engineering, I suppose I could go back and work my way up the Computer Science degree ladder, but doing software dev in the real world is too fun to actually consider going back to school...

Scott: You have a successful causal gaming company, specializing in games for Windows CE. Did you do this do put yourself through school?

Doug: Fortunately higher degrees in Electrical Engineering are typically paid for by research so I was in pretty good shape there.  Being a student I had lots of free time and when PocketPC devices first hit the market I was so enamored with the device (iPAQ 3600) and how easy it was to develop for it that I found myself spending lots of time coding.  I eventually started writing my first game for the platform with the intention of only spending a few weeks on it.  Three months later I found myself releasing my first game which included an ambitious feature list, including internet multiplayer.  After the game achieved almost immediate commercial success I lost interest in Electrical Engineering.
Approximately 3 years later I had filled my offering out to around 25 products spanning 4 platforms..

Scott: Windows CE machines weren't initially known for their graphics power. Did you find the Windows CE Graphics primitives lacking? How much did you have to layer on top before you found it usable?

Doug: You'd be surprised how powerful the initial StrongARM 206MHz processor is when compared to today's 620+ MHz XScale. The StrongARM was more than twice as fast as it's successor, the first generation XScale (~420 Mhz) in my testing. Intel made some mistakes in their initial architecture of the XScale processor from what I understand that made the second generation PocketPC's performance underwhelming.

Doug: In order to develop any graphically compelling application on PocketPC, at least in the past, you need to circumvent any graphics primitives included on the platform and just access the frame buffer directly.  More recently there are things like OpenGL ES, and the late to the scene DirectX mobile stuff. There are also several 3rd party libs that people have written that are very nice.
One of the interesting things about the Pocket PC platform is that the display buffer in memory can have a completely arbitrary memory layout (and some did).  You'd think that on all PocketPCs the first byte of memory on the frame buffer with be in the upper left of the screen and then it would progress in the same fashion that you'd read a book until you hit the bottom right of the screen.  In practice this was not the case.  Some devices, like the iPAQ 3800, the first byte was in the lower right of the screen and then it would increment vertically from bottom to top, and then right to left....crazy.
My trick back in the day was to orient all my surfaces and my backbuffers in the same memory orientation as the frame buffer itself so I could do very fast blits. My first title was easily above 100 frames/second on the old 206MHz StrongARM. (135 FPS with overclocking =)

Scott: No wonder your stuff stood out! In order to support your memory orientation trick, did you #define your way around?

Doug: Since it was my first product, I never planned to see it running on anything but the original device so I had hard coded the memory orientation in the code (and the iPAQ 3600 had a funky memory orientation as well).  As new devices started to come out I used #defines to compensate but within only a few months I had to make a more general surface description (that removed all the #defines) and kept track of key metrics like size, x & y span directions, etc.  This allowed me to work around any frame buffer memory layout and I could still put all my surfaces in the matching memory orientation.
My first title was written entirely in C, it was all contained in one massive file, and I didn't use a debugger at all during development. *gasp* As I progressed, I made my game engine support multiple platforms so I could develop on my desktop and test on the device only when needed. I started using a debugger (which is very handy I might add), split my game into multiple files, etc. My later titles were written in C++ and entirely object-oriented.
I rev'd my engine several times and really had something that I could crank out software that supported virtually all windows mobile devices. If you're smart about it, you can have a Windows Mobile for Smartphone and a Windows Mobile for Pocket PC application in a single .EXE and share graphical assets between the two where possible, etc.
It's really important to come up with a scheme to minimize the size of the assets, sounds, graphics, etc. because when it's time to distribute your application there are several distribution mechanisms out there, over-the-air, for example that carriers are really fixated about keeping the size of the installer 'under a megabyte' for example.
I have several titles where I went through every asset and minimized it to the absolute minimum and then wrapped everything with zlib to make it even smaller.  Only to eventually go back and squeeze another 50% out of the size so I could get the application into a specific distribution channel

Scott: Did you use Microsoft tools? It seems like "Tiny Dev" *is* a hassle...so you used the emulator?

Scott: So your games ran on Windows, natively, and you developed them mostly on Windows while thinking about the device as the ultimate endpoint?

Doug: I fortunately had the luxury of having many devices.  I've got a stack at home now of nearly 3 dozen of windows smartphones and pocket pcs...heh
Yes, for ease of development I highly suggest just developing and testing your applications right on your desktop. There is a fair amount of overhead depending on what libraries and features you have in your game engine to get it all running on windows natively--but it's worth it.

Scott: What's the breakdown of everything under the hood of your windows mobile game engine?

Doug: Initially it was just a graphics engine, a sound engine, and a game all munged into one.  
Eventually it matured to a game sitting on an engine with the following features:
 
- A highly optimized 2D graphics engine with an integrated 3D software renderer.
- A GUI system and plenty of GUI widgets.
- A compressed file system that could mount files based on device, display resolution, etc.
- A memory management system.
- Sound and music using MikMod.
- A user input system.
- A font system with optional integration of Freetype for TTY support.
- A network and multiplayer lobby system.
- Zlib for real-time compression/decompression. 

At least, that's the feature set off the top of my head. For games, the real gotcha's seem to be getting overall device compatibility because of so many minor details between devices, getting a solid graphics lib, and the input system.  That's where I spent most of my time.

Scott: What's the status of this engine, can it be licensed? How it work within the context of the new Windows Mobile stuff? What about Origami gaming - have you look at that as a potential market for your engine? There's potential for kick ass gaming on Origami...

Doug: The engine is something used internally, never made available for licensing.  I never made a push to make it available because of the availability of other solid engines and the general weakness of the Windows Mobile games market as of late. It's works great with the latest Windows Mobile versions, including square-screen and VGA support. All of my software at dig-concepts.com has been updated to support Windows Mobile 2005 of course =) I've talked with MS about Origami and it was pretty obvious that that's not where they see Origami heading. MS does have a number of other things in the hopper for the future though.
The funny thing about kick ass gaming, is that you need more than a platform that is capable of it.  You need someone like Sony, Microsoft, or Nintendo to get behind that platform, make a huge marketing push to get the sell-out numbers to the point where developing a 5mil dollar title is possible...
For the Playstation Portable (PSP) the average budget for a title is between 4mil and 6mil with some titles going over the 10mil mark. So yes, now I've moved over to the PSP platform.

Scott: What's the OS for the PSP?

Doug: It's running Sony's ever-updating OS.  In just over a year Sony has updated the OS via firmware updates (required to play the latest games) approximately 5 times.

Scott: What kind of stuff are they changing? The public thinks they are chasing security stuff, but what's it mean for the dev? Do functions change?

Doug: The PSP itself is a remarkable piece of equipment.  Not only does it have an amazing screen, and optical DVD drive, Playstation 2-like graphics hardware, and 802.11 wireless networking, it's sexy to boot! They are addressing security issues and updating the main component apps (photo, video, music, games) of the device in the patches, but they are also making major additions to the platform.  They've added a web browser, RSS feeds, a 'sling-box' like client that allows you to stream live or recorded TV from home.  They've also added support for more video formats and general user interface improvements.  But the unannounced improvements that are coming to the PSP platform are the most impressive.  By next year at this time, everyone's PSPs will be transforming to much more than they are now through firmware updates.
As the firmware updates there are a few growing pains for developers but it's generally minor changes to support the latest firmware. When they jumped to the 2.0 firmware they made some particularly large changes to certain systems that required studios like ours to invest a fair amount of time getting everything working.

Scott: Example?

Doug: Releasing titles on the latest firmware is one of the requirements of releasing software on the platform. An example of a change that was required when migrating was a major change to the way the network modules were dynamically loaded during run-time.  Ultimately the new implementation was much easier and robust than the previous but during the time I was working on the migration the pressure was on to get multiplayer functionality restored so production could continue.

Scott: What IDE and languages can one use on PSP for official game dev? Do you sit down and go "File|New Game"? Does every house have to decide things like what engine are we going to use?

Scott: Debugging a psp game is interactive when it's running on the psp? there's pdbs over there? and is there a debug console for logging?

Doug: There's a fairly good debugger for the PSP.  It includes disassembly, multiple TTYs that you can print to, debug breakpoints/watchpoints, memory dumps, etc.
Unfortunately there isn't a build-in 2-way console which would have been nice.  For the work I've done with debugging multiplayer I've had to implement my own 2-way console.  In addition to all the game's networking, for debugging I integrated code to have each PSP connect to an IRC server.  From there I can join IRC with my favorite client and talk to all the PSPs individually or to entire groups of PSPs.  In addition I have little mini-apps that connect from my desktop computer and show real-time bandwidth and latency metrics and other statistics useful for debugging multiplayer games.

Scott: What kind of primitives are there to start with? Do you use their "directx" or write your own?

Doug: There's a few other things required, but you can basically get a hello world app running and displaying to the screen in less than 50 lines. Sony has it's own highly-optimized graphical library which is a great starting point.
I can't speak for other studios, but I believe most of everything they would need is there.  For studios looking to really push the envelope or perhaps for a specific genre, there are certain areas in the graphics library that may need to be improved for their specific app.
By the way, the Sony graphics libs are an OpenGL-like interface.

Scott: Since you've created your own layers for graphics before was your knee jerk reaction to pull a "not-invented here" and start layering, or did you have more self control?

Doug: Hehe, they put me on networking immediately when I hired onto this studio so I have never touched a line of graphics code for our PSP engine/game.

Scott: They include a p2p stack?

Doug: I implemented 'Headset Support' for our PSP title which was essentially VOIP across a UDP P2P layer implemented in one our network libraries developed internally to (Sony Computer Entertainment International) SCEI. So yes, there is some P2P stuff in there somewhere 

Scott: Is the biggest challenge on a psp things like memory leaks? Most of my readers have had garbage collection for >5 years so the idea of cleaning up after themselves may be foreign. It seems like one might chase memory leaks with a PSP and struggle when things get tight.

Doug: Typically for games a memory manager is developed to sit on top of the kernel's malloc/free.  On the PSP in particular, it's nicely laid out so you have a nice contiguous block of something like 25megs that you can allocate, plus an additional volatile area of 4megs that you can also utilize for certain types of data.
Our studio has some pretty comprehensive tools for managing memory, allocs, and frees.  We monitor the way memory is allocated throughout development and watch for things like islands that are created in the alloc/free process.  Islands are totally unacceptable in our memory space because we have as little as a 100K of allocatable space once the level is loaded and the gameplay is underway.
We have our memory partitioned into different blocks so that we can enforce hard limits on different teams like sounds, graphics, assets, code, etc.  In addition certain blocks of memory are blown away completely after a level plays out.  So essentially you don't need to even worry about doing a 'free' on something you've allocated in some instances.  It seems like a lazy and unclean approach to memory management in some cases, but it works remarkably well when it comes to ensuring that you'll be able to get a nice repeatable memory layout level after level and not suffer from a diminishing heap space over time which would eventually result in the game crashing.

Scott: This layer your studio has, is this something every studio is responsible for? I mean, isn't it kind of sub-optimal that there's not a pattern/library from Sony for this kind of thing? Or is malloc/free the appropriate place for sony to have left the developer?

Doug: It's certainly not something that every studio would need.  In our particular case we packed memory so tight that we were totally intolerant of any leaks or islands.
The funny thing about game developers is that they like to be in ultimate control of their own destiny which gives rise to including something like this in a game engine.  Even in a time when developers rely more and more on libraries and larger systems outside their control, there is still the desire to reel that stuff in as much as possible and control the associated risks.

Scott: Do you or can you "cheat" and jump into the hardware like you did on the CE machines? or will Sony not certify you if you do that?

Doug: Unfortunately you can't "cheat".  As you predicted never get certified.  Additionally, you application would likely stop working when the next firmware update is applied.  I've got an example on this point.  Early in the development we poked into memory to overclock the CPU from 222MHz to 333MHz.  This gave us a huge boost to frame rates during early stages of development.  Unfortunately it would break each time we updated the firmware.

Scott: Doh. I did a similar thing with my Palm application and when the new hardware came out, it was all over. I heard that you can't make UMDs, that only sony does? How do the ensure that you're gold disc will work? is there any chance that you'll get a bad gold and go, oh, snap....?

Doug: Correct, UMDs are only made internally at Sony and even first party developers don't have a way to make UMDs The DVD built into the dev kits is designed to emulate the UMD but it's about twice as fast as a UMD, so it's not really a good approximation.  Fortunately, all developers can send off a 'gold candidate' and get a dozen or so UMDs back to test  on an actual PSP.  The turn-around time is very quick as well.

Scott: So the only thing that would be iffy might be your level load times?

Doug: The UMD drive on the PSP has a very good read speed but a horrible seek time.  So you have to employ tricks to avoid doing seeks if you want to have quick load times.  Our studio put significant effort into this and our absolute worst case level load time is around 20 seconds.
There was a video circulating on the web of a PSP title that took over 6 minutes to get into the game and start playing--which is ridiculous, those guys should be ashamed of themselves, and I'm not even sure how the title passed certification.

Scott: How did you do the tricks with the turnaround time on UMDs? Try, order, try again?

Doug: Oh, by tricks for the UMD I meant by doing things like organizing data sequentially so that the UMD doesn't need to seek, data redundancy so the drive doesn't need to go different places to get stuff, etc. But yes, you can also go round and round with UMD samples of your software although it's usually not necessary.
The dev tools do a remarkable job of emulating the production device, so you pretty much know what you're going to get if you have some UMDs made.

Scott: What's the typical team size?

Doug: As for a typical PSP studio, the team sizes vary although they tend to be pretty impressive considering they are developing for a handheld device.  I'm speculating a bit here, but typically you're going to have a hard time finding a PSP team smaller than 15 or 20 people.  Our team was nearly 70 people strong with 20 software engineers on the project.

Scott: Sony's got some intense certification, I heard?

Doug: Sony holds titles to a very high level of quality before they are released for the platform.  There is a very long list of requirements and rules that guide and shape development. The upside of all this is that Sony has done an amazing job building out a platform, putting real marketing behind it, and getting the devices in customer's hands.
Essentially they've built a new platform that can sustain a sizable team and yield profitable title and profitable title.Good stuff really.

Scott: Do they assign a human to your game so there's always an individual you can talk to? Sony has a rep for being a faceless beast.

Doug: We tend to have all the support and connections we need.  I can't speak for all developers unfortunately.  I can guarantee that if you have a killer game, you'll get all the attention you need =)

Scott: Doug, thanks for taking all this time and for typing your fingers. I hope the readers will enjoy this peek inside a world they are not (I know I'm not) familiar with.

Doug: No worries. Peace.

About Scott

Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.

About   Newsletter

Hosting By