An old poem, for an old flame.

I used to dabble in poetry, but much of it was too maudlin to deal with, so I started to focus on fun stuff, like Ultimate Frisbee rhyming cheers, or limericks.

This my last ‘serious’ poem, from 20+ years back, for a proto-girlfriend’s birthday. She was having issues about the big 3-0 birthday for athletes; it was hitting at the same time her knee was giving her recurring problems. This is the moment you intellectually realize you’re on the wrong side of the bell curve for sports, and don’t know what to do about it. I had hit the same wall earlier,  multiple times from multiple injuries at multiple ages, so I wrote this advice.

As time grows long and wines age true, when lovers laugh and lives renew.

Your day at birth is brought to hand, and heavy weighs time’s dreaded sand.

But if heart and soul combine their will, then the body must its role fulfill.

So Carpe Diem! Seize the day! Follow your heart and let come what may.

Perhaps this was overkill for a birthday card, especially that early in our very tentative relationship 😉 But when I’m in, I’m in all the way.

A Surreal Cab Ride at GDC

So we’re heading out from the convention center, popping up to The Stinking Rose, a garlic themed restaurant in North Beach, to celebrate a great GDC showing. The driver starts off, and innocently enough, asks if we’re in town for a convention. When we mentioned games, he started going off about how games were ruining children and society. From his rant and appearance, this guy looked to be a gold mine of ‘far end of the bell curve’ sample data; a type of fellow I rarely get to use as a data source. I challenged his view on games. He was receptive when I pointed out people were playing more games and reducing their TV time, and that games were a social interaction event versus the solitary experience of the television. But he then got on the ‘computers in general were bad’ rant. I asked him if he had a computer and to compare his TV time versus surfing around on the web. He looked pensive and shrugged “alright, fair enough. I spend a lot of time surfing around, and I learn a lot each time.”
Now that I knew I had an interesting person to talk with, I was concerned about the short interview window; the restaurant was perhaps 15 minutes away. So I started digging in with some questions. He tended to ramble a bit on his answers, so as soon as I had what I needed out of one answer, I would politely cut him off and throw in the next question. It was a ton of fun! I learned a lot and the guy seemed cool; overall, fantastically successful cab ride. So I over-tipped him, we smiled and shook hands and I turned to the two guys I’d ridden up with, as the taxi drove off. Kris stomped his foot on the ground and said “Oh my God! That was the most terrifying cab ride I have ever had!!” Brian explained matter-of-factly, “I thought we were going to die. Several times.”
I was quite surprised, pulled back and looked at them. What could have been the issue? I thought it was a great ride, one of the best ever.
Kris said, “Well, I was a little worried that you would get us killed by challenging this crazy guy’s worldview, on his turf. But you convinced him that he was looking at the problem the wrong way, and I started to think we’d get out of there alive.” Brian interjected with “but the real problem was that he was a totally insane driver.” Brian continued, “then I saw you drop into scientist mode, and I knew it was all over.”
I knew immediately what Brian meant. I live for that sweet spot when you are totally immersed in a problem. And it’s true, sometimes I lose track of time and space, but I was shocked that the three of us could’ve had such a radically different experience in the same cab ride! I pressed for details and they rattled off incident after incident: driving through lights; driving through stop signs; driving without hands; driving while eating; driving while turned around in his seat and talking to me, etc. They had a pretty extensive list of some pretty valid complaints!
I paused and went to the mental videotape; slowly stepping through the cab ride conversation, and trying to observe the situation, not just the data. And sure enough, the guy was completely whacked! He had a big Styrofoam container of noodles that he would hold in one hand, while driving with one knee, and using the other hand to scoop noodles into his mouth, all while speeding down the already crazy streets of San Francisco. He would also do this while turning around and looking at me, regardless of the current traffic situation!

To help deal with this sort of problem (zoning in and missing what is going on around me), I have a little helper thread to track the overall environment; the only tricky part is remembering to feed it cycles frequently enough to catch such problems! As a little test thread, I’ve been forcing myself to take a guess at the current time, then I check the clock to see how close I was. After several months of this, I can now reliably feed background processes — at least, some of them — even when immersed. And it has also improved my tracking of time in general, leading to a much improved on-time behavior!

Accidental overdose: community involvement

When Michelle and I were living in Arlington – we bought our first house and moved in together – I accidentally got involved in community improvements, and learned a lot of very valuable lessons. We had bought a house on the street that had the occasional speeder going through, making it very loud and a little dangerous when they did. The dips in the road restricted visibility considerably, and you could easily come over the hill at high speed and find somebody in the way with very little time to react. My neighbor told me there was a neighborhood improvement plan that was proposing a stop sign, or a center median, or something to slow down the high speeds on our street without overly restricting normal speeds. The idea sounded very sensible to me, so when he asked me to come to a community organization meeting to look at the proposal, it was a no-brainer to say yes.
It was a very odd meeting. A handful of people took forever to very boringly go over some minor changes to the proposals. I was shocked when the president said that they could probably get a revised version of the document out in just a few months time! We’re talking like a very few edits of a very unsubstantial scope, and her timeframe for doing it would cause us to miss the summer funding session, pushing the projects back by at least six months. So I volunteered to do the rewrites, just to get things moving. Proposal goes in, I’m thinking things are smooth, and I get on with my life. Then the next community meeting is posted with details of the proposals, and the shit hit the fan like you wouldn’t believe.
Hundreds of people showed up, almost all incredibly opposed to the proposals, even the no cost, beautification projects! It turned out the community “leaders” hadn’t bothered to ask any of the residents what they thought of these proposals! There was the usual assortment of change-is-bad folks but most had some pretty valid points. For example, a proposal that I thought was fine would have directly affected a dozen homes, and the proposal had been written without their knowledge or consent. The pitch sounded innocuous: convert an abandoned alley way into a walking path. But the alley had been abandoned for decades, and almost all of the houses had adapted their landscaping and fences to use this open space. Granted, it wasn’t their open space to use; the land belonged to the county. But you can’t slap something like that in as a surprise to people. At least, not if you want their support.

Side note: a similar issue ironically occurred in our Walnut Creek neighborhood. The city planners had quietly decided to tear down some open space and put in some high density, low income housing, and had gone considerably out of their way not to allow the neighborhood to find out. Fortunately, Michelle and I had been through the problems, similar problems, and Arlington, and knew how to engage our neighbor’s outrage into an actionable plan, and the rogue city planner ran into a buzz saw.

But back to Arlington. The community overwhelmingly voted no to the proposals, and things were looking grim. Arlington had opened up considerable funds for community improvement, as long as they got community buy-in to the proposals. Everything was now at risk because the community association hadn’t gotten by in from the community before hand; they sprung it on everybody, all at once. So long story short, I ended up becoming president of the Civic Association, teamed up with the irresistible energy force of a displaced New Yorker, Lewis Bromberg by name.

Lewis had recently moved into our neighborhood, a few houses down the block, and right in the zone where Waverly Hills transitions from a nice set of suburb housing to some pretty Third World housing and a small commercial strip. Lewis was on board with the notion of community improvement, but by no means was able to knuckle under to a flawed plan put together by flawed leadership. He somehow drafted me into helping out, and between the two of us, plus some key support from a few other people in Waverly Hills, we completely redid the community improvement plan, pretty much from scratch. Shockingly, it turned out that asking people about what they would like to see in their community was a much more successful approach than coming up with a plan and surprising everyone with it. And when we delivered the new plan to the Civic Association meeting, again with hundreds of people attending, it passed with overwhelming support; 90% plus. We not only structured the plan around ideas generated by the community, we phased the developments in tune with changes to project funding structures we knew were in the works from the county. And with surprisingly little effort to me – Lewis did by far the most leg work of anyone involved – we were able to pull in over $3 million in government funded projects to improve our community. Anywhere from colonial style streetlights, infrastructure improvements to the local park, or traffic calming measures on the major streets. One potential drawback of the plan really concerned Lewis and I: there was a lot of legwork involved to make a project happen, both at the community level, and at the county level. And while we were willing to invest the time in a few starter projects, there was no way we were willing to do all of the work, all of the time.
To make it scale better, we redid the proposal to require all new projects to have two block captains involved, located in the affected area. The block captains were on the hook to get local buy-in to potential improvements, not us. Then we’d help them bootstrap the project with the county: which department to apply to, what they’re looking for, how much is potentially available, etc. Essentially a primer on how to make a project happen. Then the block captains would take over the legwork from that point, pushing things forward with the county. It was this organizational scalability that allowed us to tap into so much county funding.

Arlington County loved the approach. In fact, they took our proposal and made it the baseline standard for other communities looking to get involved in the programs!

TDD project summaries (KIXEYE & EA)

I recently spent two years at KIXEYE, building a test engineering group and integrating automated testing into every game team. Significant improvements in development time, scalability and operational cost/quality were directly attributed to the load-testing work I led, driven by metrics for the engineering, QA, producer and executive teams.
After the initial success wave of KIXEYE, operational costs skyrocketed and server failures during peak revenue hours became commonplace; the infrastructure and development processes simply weren’t scaling. The KIXEYE executive team issued a priority one mandate to implement load testing tools and test-driven processes for all titles.
To staff the testing team, we set up intern programs with several schools and hired the best interns out of each batch into our QE group, where we trained them in load testing, application metrics and how to work with each game team. We then embedded these QE engineers into each game team, working with the game teams to establish priorities and continuing to run daily scrums for mentoring purposes and fine-tuning direction.

This approach overcame each game team’s strong initial resistance to load testing: engineering decisions and launch decisions became largely driven by results from the automated tests and embedded performance metrics probes. We then grew the work into performance testing for mobile and web clients.
A quite interesting project for me was Robbie; a friendly front-end to the Maxis automated testing systems. After the success of automated testing in The Sims Online, building a second-generation system, integrated into all teams, became a top studio priority. We first tackled build stability, performance and finally content regression for the studio’s next major release: The Sims 2.0. By embedding myself into the Sims 2.0 team, we found tremendous opportunities for turning test and measure tools into task accelerators for the content team, and by aggregating performance statistics into an executive level dashboard we were able to help focus engineering resources and provide clear cross-team communication. For example, the studio head had been able to start a non-confrontational, very directed conversation on performance, as shown in this simple email: “I see on the dashboard that your frame rate is not where it should be at this point in your project. How are you planning to address this risk in the remaining time before launch?”

“I feel the need, the need for speed!” the value proposition for rapid II cycles and metrics-driven development

The key to success in building any game is how quickly a team can see a game in action, quickly evaluate the state of the players and the state of the game, and quickly react to the new information.
The key to success in online games is to quickly repeat the above evolutionary cycle, over and over again, without significantly damaging what attracted and retained your existing customers, or significantly damaging your platform and brand, while adding material to attract and retain new customers and finding new ways to make money, all while players continuously do the unexpected and vultures show up to pick at your established customer base. All of these constraints are difficult individual problems. Combined, they present an incredibly difficult problem.
The barrier to success is achieving the required development speed, visibility and flexibility in the multiple-dimension complexity of online games. New, continually shifting problems require more than just new solutions, they need a new way to solve them, over and over again. And with the development speed, scale, complexity and longevity of online games, your business needs game production tools, metrics and agility more than ever before.
Introduction, 1.1: A Darwinian shift in what games are requires a similar shift in how we build them.
Online games are the most intriguing shift in media since we first learned to scratch on cave walls. A product oriented industry is migrating to service business models that rent persistent, continually evolving virtual worlds, populated by studios and customers alike. You’re trying to attract and retain long-term customers, not point-of-sale customers. A service-oriented business has different cost and revenue models than packaged goods. Further, players require the flexibility to explore their own paths, build relationships and expand the shared virtual world. The more bonds players have to your game and to your quality of service, the longer you retain customers and the greater your revenue and the better your chances of expanding your customer base. These shifts in the relationship between the players of games and the producers of games change how we think, what our business goals and priorities are, and how we build software.
Online games are a combination of long-term experiments that seek and field innovative approaches to solve complex, continually changing problems:
• expanding the virtual world and the supporting infrastructure;
• exploring new types of fun;
• improving how you attract and retain players;
• improving your revenue models and lowering your recurring expenses.

That’s a lot of stuff to get right, quickly, over and over again, especially with the high quality of service required to attract and retain long-term customers.
• Your software and your processes must be designed for a continual, incremental, cost-effective growth pattern, with rapid prototyping to continually shape the growth direction and continual pushes of new content to paying customers.
• The heart of an iterative development process is the pace of iterations and your ability to see actionable results of iterations.
• Adding metrics to your incremental development cycle provides guidance in your experiments, visibility into your sources/sinks of money, visibility/predictability into your schedules and visibility into the speed, agility and stability of your platform and your processes.
Metrics are not a silver bullet in and of themselves [reference: Frederick Brooks, “No Silver Bullet”]. To meet the demanding requirements of online games, you need an integrated set of development approaches that, in aggregate, lower the cost and risk of fielding an online game service.
• Metrics point out anomalies for you to investigate or find high level, aggregated patterns of data too complex to absorb individually. Automating this triage / aggregation phase is essential to a rapid experimentation/reaction model: finding the problem area is the slowest part of debugging large, complex systems, such as gameplay, player behavior or nondeterministic servers. You need speed of analysis.
• If you can’t quickly react to what you’ve observed, you’ll be more frustrated than successful. You need speed of execution.
• To achieve these levels of agility and speed, a backlog and burn down charts just won’t do: you need your software, your tools and your processes to have ease of observation and ease of change built-in at the genetic level. You need more than process agility, you need system agility.
• Breaking things slows you down and complex, continually evolving software is easy to break. Your experimentation capacity is effectively zero when your software doesn’t work. You’re also forced to squander resources and time on something that worked just fine yesterday and you risk burning your customers with unreliable service. You need production and operational stability.
• Incorporating your long-term business success metrics into your day-to-day decision-making keeps you on the right track, and helps decide which answer to a complex question pushes the ball forward: a completely reaction driven process tends to thrash, not advance. Similarly, critical but less visible infrastructure features fall by the wayside until they are very visibly in the way of forward progress. You need simultaneous focus on both long-term goals and short-term milestones.
• Finally, continual process improvement is a hard people problem. Once the basic infrastructure and processes are in place, people tend to focus on other, more visible problems, such as gameplay. Unless infrastructure is measured and tracked against at least at least at least timeshifting up a short-term needs and long-term business goals, the quality, speed and stability of your infrastructure tends to fall below the radar, until it is visibly and painfully in the way of production and/or customers. In fact, QOS and speed of content refresh are critical game features in your online service business; they need the same level of attention and resources as any other feature, if not more so. And even then, human nature is unwilling to change how work is done in the middle of a project. You must be able to measure reality and adapt your software, your development processes over and over again. You need facts and willpower.

Purple’s Three Laws of Analytics
1. What gets measured, gets done. Getting the right thing done is an entirely separate problem.
2. Individual metrics aren’t worth spit. Context is everything.
3. Change is tough. Without willpower and a continual, evolving path that is easier to use than not to use, you won’t change.

Abstract: Iterative Innovation in Games and Gardens

Games and gardens are immersive environments that are designed for aesthetics and enjoyment: goals that are difficult to quantify and harder to engineer. They take years to complete, and it is hard to evaluate success without the user’s viewpoint from inside the immersive environment.

The hurdles: quickly, constantly and cheaply acquiring user, system and process data; preventing defects and implementation gaps from compromising the data; and continually shifting from rapid prototyping to fielded, highly complex systems over years of development and operations.

A cyclic, incremental evolution strategy that draws metrics from the current cycle to guide the next cycle is discussed. This Measure / Change / Measure model provides better focus on tangible problems than the traditional Guess / Change / Hope model, more predictable progress and manages continual growth.
The integration of several agile development strategies is proposed, ranging from test driven development, short, instrumented sprints, metrics driven development and Kaizen process improvement. The hypothesis is that tightly integrating agile strategies into the core of your project via Automation, Architecture and Analytics produces a hyper-agile software system: one can quickly identify problems and quickly shift both code and process, while also casting meaningful projections of future behaviors from current and historical data.

Iterative Innovation is a force multiplier that augments your entire team. Get more done, in less time, with less blood.

Low cost, low time investment gardening (1 of N)

Having a large garden is beautiful but can be time consuming. I’ve done a lot of experiments to reduce the amount of time required to implement and maintain an effective garden. Some factors I already optimized for our Walnut Creek garden were: hummingbird, butterfly and bee habitat; others include low-weeding requirements, drought tolerance and bloom time for cut flowers to bring the garden inside.

We have enough room in our Walnut Creek garden to have interesting shapes of foliage as well as bursts of color here in, but the San Francisco garden is significantly smaller, so I’ve increased the emphasis on color and sculpted the trees for interesting shapes.

Getting a sculpted bonsai style lemon tree is surprisingly easy. The tree was an absolute mess; no visible structure, a lot of diseased stems, etc. This basically took 4, 15 minute sessions of pruning to establish the basic tree shape. Now that doesn’t count the countless minutes sitting staring in the tree’s basic direction, trying to imagine future shapes while either just enjoying the weather or doing some writing. You don’t want to trim off too much at once; that can send a tree into shock. So I did three of the 15 minute sessions just getting the diseased and crossing branches and the usual crap out of the way. Then I let the tree cover for a couple of months, and now that I could see it the shape of the tree, I could safely proceed with the fine-tuned pruning, designed to show as much of the shape as possible.

Here’s where I hit a significant handicap; the tree actually had no interesting structure to it! So after a lot of contemplation, I kind of gave up on finding the ideal solution in favor of getting any direction going on now, then improving later. So all I did was to tie some bricks to the tree! I pulled the tree top towards the ground, using two different attachment points to pull it over further in the direction it was already leaning; kind of an exaggerated wind affect. Then I used more stringy bricks to pull other branches into a more open arrangement, especially to lower branches that looked like I could develop them into minor lobes of a new tree structure. It’s really important not to go to overboard at this stage, particularly in time investment, because you never know what the tree is going to respond with.

Expect lots of little sucker growth coming up in a couple of months and you want to be able to either incorporate those into your design, or pluck them off while it is still easy to do so. Those six stringy bricks out there, each taking about three minutes to install, burned less than 20 minutes, and presto! Bonsai shape in motion! Now we just need fertilizer and time. Once a month, I pull the bricks further out to re-tension the strings and exaggerate the proto-shapes more.  When you’re ready to cut the strings, expect some bounce-back, hence the exaggeration times.

This Bonsai approach is the best chance for the tree’s survival. When the owner gets their permits, this whole property is a tear down and start over. But a mature, well-shaped lemon tree is enough of an asset that even a ham-fisted renovator would probably keep it. I hope… I also sculpted the bushes that serve as privacy screening; that may save them as well.

The other parts of the garden I went with two approaches. The first was the traditional weed out what you can, sprinkle with water, rake everything to kill the germinating weed seeds, repeated twice. But there were so many weeds that the wildflower seeds I put down were quickly choked out by grasses and crap. So for the other beds, I just laid down some cardboard and put some weed-free, store-bought dirt on top of that. WAY FASTER! And much less maintenance thus far.

This is the first garden I’ve done on a tight budget and a tight schedule, so I restricted myself to a hundred dollars of butterfly native habitat plants, some cuttings from Walnut Creek, and maybe twenty bucks of wildflower seeds. The weed hassles led to a couple of blighted areas: that drove up the cost by, say, fifty bucks of bagged dirt over free cardboard. Then to make things portable, I have the expensive plants in partially buried containers, with some rooting holes cut into the sides and bottoms. The sprouting wildflowers will hide the containers, and the extra height from the containers gives an line of sight accent to the plants. In Walnut Creek, I found three tiers of growth height worked the best, but here in the Mission garden, I’m only doing two.

Butting Heads with a Butthead

I’m the type of guy who plays hard, within the context of what the rules allow. Wrestling and rugby are not violent games per se; there is a strict set of rules to follow that minimize chances of damage. Which is not to say nobody’s getting their bell rung a few times, just that it should only happen accidentally.
Until you meet up with some Schmuck who decided to expand the role set, presumably towards his advantage.

Enter the wrestling schmuck. A very few wrestlers will try to screw with your brain; intimidation tactics, designed to throw you off your game in either confusion or anger; the point is to disrupt the player, and usually outside the context of the rules. One guy I was matched up with seem to have had that concept as the only move in his repertoire.

Right from the opening whistle, he stepped way into me, leading with his head. Now you can’t immediately assume somebody that has had butted you is automatically guilty of being a jerk. Sometimes accidents happen, even with the most skilled opponent. Or he could simply be clumsy; poor body control combined with bad coaching explains a lot of the unnecessary contact in high school wrestling. So no harm no foul; I just took a step back and shook it off then moved back in. Thud! Exact same move, exact same result, except louder. I could actually hear our heads ringing.

Now I’m a little wary at this point. Regardless of the guy’s intent, two resounding head butts in under a minute is indicative of something, usually that the guy has no business being out there. Either he’s a jerk, or has so little body control that somebody’s going to get hurt. So you take a little extra time to focus on exactly what’s going on. I close again and like clockwork, Clunk! He’s clearly driving in with his head to make contact, either to wear me down or to intimidate me. I took a few steps back and gave the referee the classic shrug and “what the hell” expression. The ref looked even more bewildered than me; his return shrug indicated that he knew the guy was doing it deliberately, he just couldn’t think of a particular penalty that he could use to stop the behavior.

So when somebody’s wants to up the ante, that usually leads to me shrugging “okay” and playing with within the context of the new rules that he’s just established. I didn’t have a plan of what to do about it, but the ref made it clear that head butts weren’t going to be a called penalty. So the next time the guy stepped in leading with his head, I grabbed him by the shoulders and swung my head back and forth once (Thwack!), twice (Thud!) and thrice (Goonnggg!), using the ridge of my skull against his forehead. It was getting to me a little, but it had clearly taken the other guy out of his game, mentally and physically. His intimidation tactic wasn’t working, but more importantly I think I just could take more head-banging than he could!
He more or less slid matt-wards; all I had to do was follow him down and flip him partway through the process. He smacked down to the mat, shoulder blades first. An excellent takedown and even in my somewhat befuddled state, it was easy to hold on for the pin 😉

An even worst thug existed in Calgary’s Rugby circles. The little sociopath seemed to actually enjoy inflicting damage outside the rules! He was a known bad egg who was eventually banned from the league, but not before I played him several times. One of the many times I was at the bottom of a ruck pileup, the little shit came over to my side. He carefully looked around for the ref and then started to kick me in the head, over and over!
My arms were trapped in the ruck so all I could do was shift my head around to distribute the damage. Things were looking pretty grim until one of the guys saw the kicking. He jumped free of the ruck and flattened the little thug, which finally brought the ref into the equation; problem solved.

Iterative Innovation, Part Two: the Value Proposition for a comprehensive metrics program

Is a concerted metrics program necessary to achieve a rapid II cycle? Or will some spot checks, when needed, suffice? Why is measuring anything more than basic player behavior important?

Raise your hand if you’ve been haunted by some variant of this question: “Is all of this metrics stuff really necessary for success, or is this a post-launch luxury that we can do without, at least for now? Other online games have been built without it, we have a strong team, we already know what our users want, and our publisher is pushing heavily on game features, not tools. What’s the difference?”

People react to how their success is measured. To generate the correct behavior, you need to measure the right things. If everybody keeps chanting “game features, game features, game features”, then game features take priority even when project schedule, cycle rates, engineering efficiency, system stability and infrastructure all suffer. Without automated, accurate metrics, it is difficult to evaluate such priorities. All measures become “probably” based, and the tendency is to assume the optimistic side of probably, especially when you’re running late, as everyone always is. This leads to sacrificing less visible, less quantifiable problems, such as cost of frequent build failures, to show more progress on game features: the primary factor on how success is measured.

This in turn results in slow, inefficient development until the system breaks down so badly that cycle times and QOS become barriers to forward progress on anything. Slap some bandages on the worst of the spurting arteries, then straight back to game features, because now you’re out of time, too, Right?

All features, game play and game service, need to be evaluated by their impact on CA/CS/CR/LVT, and on the II cycle rate. See also: [The Probably Problem and Case Study (QA test suites)]. People tend to measure what is easy to measure, not necessarily what will generate the correct behavior. Gameplay features are easily seen, easily compared to the last version, and are definitely required for success. But a successful online game also requires many service-based features, such as QOS, quick response times, and cost of service per customer. When fuzzy definitions of infrastructure requirements such as “good enough service” or “good enough cycle speed/stability” exist, or they are not easily observed, they will continue to receive less attention than game features until something breaks.

To make infrastructure as important as game features, make infrastructure features as visible, fresh and trackable as gameplay features, and tie those infrastructure features to your core business success metrics: CA, CS, CR, LTV. Game teams live in the here and now: if you can’t visibly see the impact of work, right now, and how the project has advanced, it is going to get less priority than game features. Similarly, you need to tie specific quality levels to project milestones, and expose your true overheads, particularly dead build rate, go-back costs and generally things that get in the way of people trying to get their jobs done each day. And of course online games are a recurring service business, not build once, ship one once, sell once and move on. Online means customer retention, speed of new content development, operating costs etc become a big part of the decision cycle: things with a high recurring cost for a years/long service quickly show their value just from a $bottom-line perspective. But when you fold in the added creative edge on polish and innovation, that’s where TDD really pays off for the engineering, production and operations teams. To change the behavior, you have to expose the importance and progress on all components you are working on, right now, to the same level of “I can see that it’s important” and “I can see we’ve made progress this day/week” and “I can see that we’ve moved forward against what everyone has agreed is important.” Further, you have to have the strength of will to continually push for improvement on this, and you need the software/environment to expose the change/growth you need to fill the void of “thank Ghod, we’ve made progress today”.

When asked “do we really need a lot of metrics?”, start by reframing the argument. The correct question is “how does lack of metrics hurt our chances of success in a risky business market?”

• Historically and culturally, game features have been considered much more important than tools like metrics, or even a stable development environment.

• To succeed in an online game service, you need to change the mindset of your team. Metrics, development speed and stability are not luxuries, or something that can be deferred until “we’ve caught up on the important stuff”, where “important” is ultimately defined by how you’re measured and rewarded by money, social status and peer approval, but you can never catch up on the important stuff enough so you can do something not easily visible right now: it’s Catch-22. A stable, fast development cycle and the ability to see inside the myriad black boxes you’re creating are not luxuries: they are required, mission-critical features when building and running an online game service!

• Your team needs metrics because you’re in a service business, and need to understand your customers in order to attract more customers, retain them longer, increase revenue and decrease recurring costs.

• Additionally, your team needs fast visibility into experiments, schedule progress, and where the problems are in code, gameplay, production and monetization models. You also need a fast reaction time both for experimentation and live operational problems.

• Great game play features are a necessary but not sufficient condition for success in a continually evolving online game service. In an online game, stability, speed at runtime, speed of new content production and low recurring costs are also necessary conditions for success: they have a tremendous impact on the player experience and your ability to grow your business.

o A rapid IIC is also a necessary but not sufficient condition for success. Without a structured, evolutionary process to effectively utilize speed, your wheels may be spinning very quickly, but your game might not be advancing at all.

• You can’t achieve all of these customer and business requirements without a strong development environment. That means that scaffolding, metrics, automation, testing tools, content tools and agile architectures/processes are all mission-critical features of your online gaming service. Without them, you might survive development and launch, but don’t bet your lunch money on it: there are more unsuccessful online games than successful online games. What you can count on is slow, inefficient growth of your game once you’ve launched, and continual, chaotic firefighting to stay running at all, let alone turning a tidy profit.

• Instead, if you prioritize cycle speed, visibility and stability from day one, you’ll get a faster, more effective development cycle, a stronger customer experience and lower recurring costs. Bet your lunch money on this instead.

• A Measure/Change/Measure approach is far more efficient, agile, stable and predictable than the traditional Guess/Change/Hope process. And by having a fast cycle time, and staged comb filters in your decision cycle, you don’t have to worry if MCM is slower than GCH. Finally, if push comes to shove, the exact same infrastructure you need for MCM also allows GCH!

• An fast, incremental evolution strategy, driven by metrics, lets you start small and then iterate yourself a hit game.

• Also required for success is a low cost, high turnaround model for new content and affordable, scalable recurring costs.

• MCM is essentially the classic scientific method, sped up for online games. If it was good enough for Isaac Newton and good enough to kick-start the Age of Reason, it’s a good enough place to start with, as we improve how we build, optimize and expand games, it just needs to be faster and easier. Automation and rough, back of the envelope math are the keys here; you’re not looking for the final answer, you just want to know where your people should focus their valuable time.

• Describe what improvements to tools/process/architecture you need to do, and what will be the impact? There’s no need to get excessively accurate or precise in this analysis: that takes a lot of time and attaches the focus of people on the details, not the overall effect, and you’re not likely to have very accurate metrics of a chaotic process in the first place. For the first pass of optimizing gameplay, game service and production processes, high/middle/low buckets are all you need. o Case study: Jade linker

• Production / Platform metrics automatically show you which of your recurring costs are not affordable and what is the cost / speed for generating new content, while Player metrics show you the direction for new content.

• The customer experience is just as important, if not more so, as game features: the customer experience has a very strong impact on your primary business metrics, CA, CS, CR and LTV. [Insert WOW quote] o Deferring the tools that directly impact the customer experience is a dangerous decision, given the impact on your primary business metrics. o Further, the same tools that control the customer experience also control the developer experience. The same iteration speed and stability you need for a strong customer experience also act as force multipliers for your development team Then proselytize the message, repeatedly, across the entire team: the faster you can iterate and the more visibility you have, the more you can innovate, polish, and grow.

• The more visibility you have into your production processes, development environment and software architecture, the more you can accelerate your Iterative Innovation cycle (II cycle).

• The earlier you can measure your production processes, implementation variants and player’s reactions to your changes, the more likely you are to find a successful path and avoid schedule-busting dead ends.

• Production metrics in particular are critical for a rapid II cycle. Identifying and removing bottlenecks in your production processes not only speeds up your cycle time, your production processes increase in stability, efficiency and predictability.

• Metrics help you quickly triage problems, which is often the slowest part of problem-solving; metrics help you evaluate your changes on the fly and metrics help you decide when your current task is done.

• Changing how a team works is hard. If you can’t clearly show cause and effect, you’ve got an uphill battle. When you can quantify what changes brings what benefits, it’s no longer a battle at all. Increasing speed and flexibility without hurting stability and predictability has market value, making it much easier to change how people work.

• My advice: get some early adopters going, on things that are demonstrably better with metrics than without, because changing how people work is tough, even when it’s easier… Your mantra: a poor iteration rate, poor visibility and poor stability slap handcuffs on your entire team, increases your cost and schedule risks and trashes your reaction time when things go awry. Don’t do that.

You should have a minimal metrics system up and running by the time the first client connects to the first server, and/or when you first start prototyping game play. You should be using metrics as part of your task description and task completion steps from this stage of development forward. It is much easier to establish metrics-driven development early than it is to establish MMD late, and it sets the tone for the team from day one: metrics are not something that can be pushed off to launch, metrics are how your company does business.

Agile engineering in gardens

When I left the family farm behind for the big city University life, I swore a mighty oath that my hands would never touch dirt again. Fifteen years later, I had to reverse my stance after Michelle and I bought our first house together. We were doing the home renovation thing, and as part of that, we re-factored the gone-wild planting beds into something attractive. And we both fell in love with sculpting living things into art. Michelle’s the best color arrangement and plant placement of the two of us; I do more on the hardscaping, line of sight issues and usability analysis. Then we geek out in the middle on aesthetics. Sometimes I’ll enhance the look and feel, sometimes she’ll make a usability improvement that I hadn’t thought of.
The more I worked incrementally improving our garden, the more it reminded me of software engineering in a rapid prototype environment. Yes, I am that geeky! But most of all, it reminded me of trying to design and build a game. You have to have the full user experience available before you can immerse the user in it and evaluate their actions, likes and dislikes. So when you’re building something complicated that costs a lot of money, it’s really hard to come up with a schedule and a plan for something that is, by definition not truly understandable until it is already up and running!
Our Walnut Creek garden is the best of our efforts. A complete, Desolation of Smaug piece of crap when we started the overhaul. With a moderate amount of hard work, and a large amount of lazily lounging and evaluating potential design decisions in the comfort of our wonderful garden, we came up with some pretty nifty designs!

Here is the process we used, and before/after pictures:

There is always an interesting effect as we broke in new sections of the garden. Basically, our rule was is that whoever paid the price in time and sweat to recondition an area for a mini garden got to completely control the implementation. In other words, if you dig up the ground, you get to decide what to plant! We filled in the different mini garden areas, and an interesting secondary effect came up; what do you do about the seams between, say, a new Larry project area that borders a Michelle project with a different theme? This turned out to be perhaps the most exciting piece, where we got to riff a bit off of each other’s ideas, and incorporate the view of the different mini gardens into the overall line of sight plans. And even on my pieces, she’ll grok something I missed, leading to further bouncing delights 😉
We have, I think seven mini gardens? A mini garden I define as a reasonable sitting area that provides a different look and feel from the other areas, and different patterns of sun and shade. Across the aggregate, you could be outside in the sun or shade anytime of day, any season of the year, just by moving your chair to a different seating area. And to Chris Kosmakos’ unending delight, I grew tired of dragging chairs around, and simply pre-populated all the sitting areas with chairs. Chris and I did a lot of software design work for The Sims project in the garden, as well as just kicking back and having a lot of intriguing conversations on software, people and whatever else came up that day! Chris was also outrageously surprised one day when he commented on the number of fruit trees that we planted, saying that we clearly love fruit. Well, I explained, I actually don’t really like fruit at all. And Chris righteously squawked: “What! Then why do you have so many bloody fruit trees?”
I do make a lot of fruit juice themed drinks, but the primary purpose of the fruit trees is the citrus scent in the aroma therapy sections of the garden, with the secondary benefit of hummingbirds loving to dance in and out of the shapes. Also, there was an existing lemon tree, near death in the garden we took over. It had some gnarled twisty shapes that I was able to slowly reveal with pruning and training the tree as years went by. Essentially, imagine an eight foot high bonsai tree. I loved the crafting process, so I added more fruit trees to shape. The third purpose is privacy; their density of leaves, year-round, greatly contribute to our privacy screen. But the biggest reason was I guess just the sheer coolness of being a Canadian with fruit trees in your backyard and not freezing in the winter, plus if you consider them flowers, their unusually large color blossoms that stay on the tree for very long period of time, adding color and depth to the yard. So as it turns out, we do highly valued the fruit trees, just not on the same variables that Chris does 😉

Details and pictures of the processes: