A Technical Debtor

Architectural Renovation – 2 of N

Jeff Certain — Wed, 14 Jul 2010 02:10:26 GMT

One of the things that I failed to mention in yesterday’s post is that this series is going to be descriptive, not prescriptive. The biggest reason for that is that my experience is rather narrow in some regards, and I have some advantages that other developers and architects don’t enjoy. For example, the VP of Software at CCI (who I don’t report to, but who makes the calls about letting me borrow the developers!) was one of the earliest developers on our product… in the days when it was migrated to Access 2.0.

I mentioned briefly an example of what I refer to as “guerilla” architecture – that is, something that can be accomplished by a single person without a great deal of impact on, or effort required by, the rest of the team. (This was the modification of the code generation templates to use inheritance to reduce the amount of duplicated code.)

However, not all problems are this small, or can be accomplished without some sort of tacit approval. So, I’m going to touch briefly on some of the things I did to build consensus for putting such a massive effort into improving the architecture.

One of the things that became apparent very early was that there was an architectural pattern around the data that was causing developers at least twice as much work as was needed. I knew this, the developers knew this, the team leads knew this… but management wasn’t aware of the details. The first thing I did was bring the team leads on board, talking to them about the issue and explaining a proposed solution. This was a long-term effort, with a few words about it here and there. Part of the issue was that it was such a daunting change, and with such wide-reaching tendrils, that it honestly took me about 10 months to have the a-ha moment on how to attack the problem. (I mentioned in another post that I was on salary… it’s a good thing, too… I don’t think many consultants get the opportunity to take 10 months to propose a solution.)

Even before I had the solution in mind, however, I had sent a survey to all our developers. Among other things, it asked what the number one thing was that made their jobs more difficult. It really wasn’t any surprise when the unanimously indicated the very architectural pattern I wanted to change – as I said, pretty much everyone knew it was a problem. However, this gave me a concrete result that I could point to when discussing with management. I also have the benefit of having a leadership team above me that understands, not only software, but what happens when you keep a code base alive for a decade or more. By explaining that the architecture we had made development difficult, and would make maintenance even harder, I was able (along with the survey results) to present a compelling argument for change.

Next post, I’ll talk about what decisions ought to be considered once you’ve got approval to make massive changes.

Architectural Renovation?

Jeff Certain — Tue, 13 Jul 2010 03:29:40 GMT

Mere minutes after I posted on “architectural refactoring,” Keith Nicholas has commented about the unsuitability of the term. He said:

you'd be better off calling it "evolutionary design"
The dilution of the word refactoring is disturbing, there's a certain rigor to "factorization" which shouldn't destroy the product.
refactoring != a cool way to say "I'm gonna change stuff"

There’s a fair bit to be said on this, so I thought I’d add another post rather than try to reply in an undersized comment box.

I agree that the use of "refactoring" in this context makes me a little squeamish. Keith is absolutely right that there's a lack of rigor here -- partly because the scope of the changes ensures that enough changes that this isn't really a refactoring in the Fowleresque sense.

However, as Fowler defines it, "evolutionary design means that the design of the system grows as the system is implemented". In the case I'm describing, the design isn't growing. Rather, suboptimal design decisions are being rethought, reworked, rebuilt. In an evolutionary sense, I suppose you could argue this is "growth," since the new architecture is "fitter" than the old.

However, to my mind, the word "evolutionary" brings with it a connotation that the improvement is a small forward step, building on what is already there -- not a massive reworking of the underpinnings of the architecture.

The scope of this effort is what's a little unusual. Kyle Baley, for example, talks here about architectural changes and how you know when it’s time to start over. Now, Kyle is a fellow Canuck and an all-around decent fellow. I even agree that, from what he describes, they made the right choice. However, he describes an effort that was scoped at a maximum of 2 developer-weeks. I had on the order of twice as much time personally into proof-of-concept for the changes that were made; all told, we invested something like 120 developer-weeks into the changes.

I have trouble calling a step of this magnitude “evolutionary.” Perhaps architectural renovation would be a more apt term.

(And, as a minor aside, when I said earlier that “I’m not even leaving the functionality intact,” I should have been clearer. The original architecture made heavy, brilliant and infuriating use of refactoring. That trait, among others, were ripped out. After the renovation, the code base had been reduced by close to 40%, and virtually every line of code had been touched, or at least inspected. However, the business functionality remained essentially unchanged – essentially, since we were able to identify and remove a fair number of bugs during the process.)

Thanks for keeping me honest, Keith – and for the great point you made.

New York City, Here I Come!!

Jeff Certain — Tue, 13 Jul 2010 02:45:45 GMT

I’m delighted to reveal that I’ve been accepted to speak at the International Association of Software Architects (IASA) IT Architecture Conference (ITARC) in New York City, September 22-24th. I’ll be speaking on effective architecture refactoring… or, in other words, telling people about the mistakes I made, in the hopes that they won’t have to repeat them.

For those in NYC, this is well worth attending. The keynotes will be delivered by some people you may have heard of: John Zachman, Scott Ambler, George Paras, Len Bass and Angela Yochem. Eric Evans has been relegated to only giving a talk this year, as is Paul Rayner. The inimitable Doc List will be moderating an Open Spaces session. All in all, the sort of conference that takes about a month to fully sink in. Hope to see you there!

Speaking at Heartland Developer Conference

Jeff Certain — Tue, 13 Jul 2010 02:32:53 GMT

Yet another in my frenzy of posts for tonight!

I’ve been accepted to speak at the Heartland Developer’s Conference in Omaha, NE (Sept 8-10). This is an amazing conference, put on by Joe Olsen. An absolutely star-studded cast of speakers – some of the names you might recognize include (in no particular order) Chris Williams, Amanda Laucher, Don Demsak, Jason Bock, and Jason Beres. (Yes, I’m still surprised Joe is going to let me speak.)

The price is very reasonable for a two-day conference. If you’re anywhere in the midwest, this is a great conference to attend. I went last year, and it was absolutely spectacular. Knowing Joe, he’s only going to outdo himself this year.

Architectural Refactoring… Why Now?

Jeff Certain — Tue, 13 Jul 2010 02:23:50 GMT

You’ll notice for the next little while, I’m going to be talking about architectural refactoring. These posts will be a reflection on the past couple years as a software architect, specifically centered around what some people call “brownfield” development. That is, taking something that is in a less than ideal state, and improving it… not starting from scratch as in new (or “greenfield”) development.

Leaving aside philosophical discussions about whether all software development is “brownfield” or not (no plan survives contact with the enemy; no architecture survives contact with the writing of code), what I’m really interested in sharing is the experiences I’ve had with architectural refactoring. If Martin Fowler stumbles across this blog, I’m sure he’ll take exception to my use of the term “refactoring.” After all, I’m not making a series of small changes, validated through unit testing. I’m not even leaving the functionality intact. In at least one of the cases that I’ll describe, what I did was more akin to lifting the house off the foundation, blowing the old foundation into rubble, pouring a new basement, and then setting the house back down.

(I say “I” in this case… but really, I was the one that pushed the button that blew the foundation apart, then handed a new set of blueprints to the development team – they did the work rebuilding the business logic to deal with the changes I made.)

I had the privilege to give a talk about this, at Paul Rayner’s encouraging, for the IASA ITARC (I don’t know what it stands for either – it’s a regional conference) in Denver in May. The talk was incredibly well received, and George Fairbanks was gracious enough to encourage me to write about what I presented in that talk. So, here I am… writing about it, and also doing a fair bit more talking about it.

Architectural “Refactoring” – Part 1 of N

Jeff Certain — Tue, 13 Jul 2010 02:13:06 GMT

I’m in a bit of an odd position, in some ways. CCI brought me on as a “software architect” a little over two years ago. That role wasn’t really defined – in fact, I got to spend several months doing what I felt would be a good use of my time, and then I wrote my own job description. (Perhaps that’s not exactly what other people remember, but it’s sure how things felt to me.)

Like anything, this approach has benefits and drawbacks. On the plus side, I was empowered to go where the smoke was – to find the areas where I could have a great deal of impact, quickly. (One of my favorite comments of this time comes from Kathleen Dollard, who recommended I apply at CCI. She told me once that “you’ve had more impact at CCI in the last three months than I thought anyone could in three years – and they haven’t fired you yet!”)

This was the time-frame in which I was modifying XSLT code generation templates to put repetitious code into a base class, rather than repeating it over and over and over and over and… well, you get the point. They were the days where “minor” changes like this resulted in dropping nearly a million lines of compiled code through the use of inheritance rather than code generation. (I tease one of our team leads that my job is to remove code faster than his team can write it… and I’m winning.) While this seems like a cool number to quote, the real business value actually didn’t have all that much to do with our code. True, there was a lot less code generated on a nearly-daily frequency. However, what was really valuable to CCI was the fact that the background compiler stopped crashing!

It turned out that our application was big enough that it taxed the background compiler in the VB IDE just a little bit. Just enough to find a pesky memory fragmentation issue in the background compiler. Just enough to ensure that Visual Studio would die around once per hour for every developer in the company. (For those of you that noticed this problem went away in VS2008 SP1…. you’re welcome! The leadership team at CCI was kind enough to let me ship our code base to Microsoft, where they easily reproduced the problem, and fixed it.)

These were the good parts of the early days. There were also some pretty rough parts. I mentioned earlier that I’m in an odd position. There are several different roles a software architect can play. One of them is the “gunslinger” or “hit man.” This is the outside consultant who comes in, declares that the architecture is a massive failure, and that everything has to be started anew. Often, this means firing the person that created the old architecture, having the consultant lay out a new one, and then walking away. A bad day for everyone. Lots of resentment, and lots of work scrapped.

So, what does CCI do? I mean, this is a company that has a 95%+ retention rate, and whose more senior developers have been there a decade or more.

They hired me. Not as a consultant, but on salary. They kept their corporate knowledge intact, added an opinionated young upstart to the mix, and put me at the desk beside the most senior developer in the company. Then, to heap craziness upon craziness, they invested heavily in “refactoring” the architecture. By heavily, I mean dedicating the development teams to making deep architectural changes for a few weeks.

And, you know, what? It’s been a great learning experience for everyone… and it’s working. More about that later.

Agile Like Jazz

Jeff Certain — Wed, 26 May 2010 15:03:01 GMT

(I’ve been sitting on this for a week or so now, thinking that it needed to be tightened up a bit to make it less rambling. Since that’s clearly not going to happen, reader beware!)

I had the privilege of spending around 90 minutes last night sitting and listening to Sonny Rollins play a concert at the Disney Center in LA. If you don’t know who Sonny Rollins is, I don’t know how to explain the experience; if you know who he is, I don’t need to.

Suffice it to say that he has been recording professionally for over 50 years, and helped create an entire genre of music. A true master by any definition. One of the most intriguing aspects of a concert like this, however, is watching the master step aside and let the rest of the musicians play. Not just play their parts, but really play… letting them take over the spotlight, to strut their stuff, to soak up enthusiastic applause from the crowd.

Maybe a lot of it has to do with the fact that Sonny Rollins has been doing this for more than a half-century. Maybe it has something to do with a kind of patience you learn when you’re on the far side of 80 – and the man can still blow a mean sax for 90 minutes without stopping! Maybe it has to do with the fact that he was out there for the love of the music and the love of the show, not because he had anything to prove to anyone and, I like to think, not for the money. Perhaps it had more to do with the fact that, when you’re at that level of mastery, the other musicians are going to be good. Really good.

Whatever the reasons, there was a incredible freedom on that stage – the ability to improvise, for each musician to showcase their own specialization and skills, and them come back to the common theme, back to being on the same page, as it were.

All this took place in the same venue that is home to the L.A. Phil. Somehow, I can’t ever see the same kind of free-wheeling improvisation happening in that context.

And, since I’m a geek, I started thinking about agility. Rollins has put together a quintet that reflects his own particular style and past. No upright bass or piano for Rollins – drums, bongos, electric guitar and bass guitar along with his sax. It’s not about the mix of instruments. Other trios, quartets, and sextets use different mixes of instruments. New Orleans jazz tends towards trombones instead of sax; some prefer cornet or trumpet. But no matter what the choice of instruments, size matters.

Team sizes are something I’ve been thinking about for a while. We’re on a quest to rethink how our teams are organized. They just feel too big, too unwieldy. In fact, they really don’t feel like teams at all. Most of the time, they feel more like collections or people who happen to report to the same manager. I attribute this to a couple factors. One is over-specialization; we have a tendency to have people work in silos. Although the teams are product-focused, within them our developers are both generalists and specialists. On the one hand, we expect them to be able to build an entire vertical slice of the application; on the other hand, each developer tends to be responsible for the vertical slice. As a result, developers often work on their own piece of the puzzle, in isolation.

This sort of feels like working on a jigsaw in a group – each person taking a set of colors and piecing them together to reveal a portion of the overall picture. But what inevitably happens when you go to meld all those pieces together? Inevitably, you have some sections that are too big to move easily. These sections end up falling apart under their own weight as you try to move them. Not only that, but there are other challenges – figuring out where that section fits, and how to tie it into the rest of the puzzle. Often, this is when you find a few pieces need to be added – these pieces are “glue,” if you will.

The other issue that arises is due to the overhead of maintaining communications in a team. My mother, who worked in IT for around 30 years, once told me that 20% per team member is a good rule of thumb for maintaining communication. While this is a rule of thumb, it seems to imply that any team over about 6 people is going to become less agile simple because of the communications burden. Teams of ten or twelve seem like they fall into the philharmonic organizational model. Complicated pieces of music requiring dozens of players to all be on the same page requires a much different model than the jazz quintet. There’s much less room for improvisation, originality or freedom. (There are probably orchestral musicians who will take exception to this characterization; I’m calling it like I see it from the cheap seats.) And, there’s one guy up front who is running the show, whose job is to keep all of those dozens of players on the same page, to facilitate communications.

Somehow, the orchestral model doesn’t feel much like a self-organizing team, either. The first violin may be the best violinist in the orchestra, but they don’t get to perform free-wheeling solos. I’ve never heard of an orchestra getting together for a jam session.

But I have heard of teams that organize their work based on the developers available, rather than organizing the developers based on the work required. I have heard of teams where desired functionality is deferred – or worse yet, schedules are missed – because one critical person doesn’t have any bandwidth available. I’ve heard of teams where people simply don’t have the big picture, because there is too much communication overhead for everyone to be aware of everything that is happening on a project.

I once heard Paul Rayner say something to the effect of “you have a process that is perfectly designed to give you exactly the results you have.” Given a choice, I want a process that’s much more like jazz than orchestral music. I want a process that doesn’t burden me with lots of forms and checkboxes and stuff. Give me the simplest, most lightweight process that will work – and a smaller team of the best developers I can find. This seems like the kind of process that will get the kind of result I want to be part of.

RemoveHandler Issues with Custom Events

Jeff Certain — Tue, 04 May 2010 17:37:03 GMT

This is a case of things being more complicated that I thought they should be. Since it took a while to figure this one out, I thought it was worth explaining and putting all of the pieces to the answer in one spot.

Let me set the stage. Architecturally, I have the notion of generic producers and consumers. These put items onto, and remove items from, a queue. This provides a generic, thread-safe mechanism to load balance the creation and processing of work items in our application.

Part of the IProducer(Of T) interface is:

        1: Public Interface IProducer(Of T)

       2:     Event ItemProduced(ByVal sender As IProducer(Of T), ByVal item As T)

       3:     Event ProductionComplete(ByVal sender As IProducer(Of T))

       4: End Interface

Nothing sinister there, is there?

In order to simplify our developers’ lives, I wrapped the queue with some functionality to manage the produces and consumers. Since the developer can specify the number of producers and consumers that are spun up, the queue code manages adding event handlers as the producers and consumers are instantiated.

Now, we’ve been having some memory leaks and, in order to eliminate the possibility that this was caused by weak references to event handles, I wanted to remove them. This is where it got dicey. My first attempt looked like this:

       1: For Each producer As P In Producers

       2:     RemoveHandler producer.ItemProduced, AddressOf ItemProducedHandler

       3:     RemoveHandler producer.ProductionComplete, AddressOf ProductionCompleteHandler

       4:     producer.Dispose()

       5: Next

What you can’t see in my posted code are the warnings this caused.

The 'AddressOf' expression has no effect in this context because the method argument to 'AddressOf' requires a relaxed conversion to the delegate type of the event. Assign the 'AddressOf' expression to a variable, and use the variable to add or remove the method as the handler.

Now, what on earth does that mean? Well, a quick Bing search uncovered a whole bunch of talk about delegates. The first solution I found just changed all parameters in the event handler to Object. Sorry, but no. I used generics precisely because I wanted type safety, not because I wanted to use Object.

More searching. Eventually, I found this forum post, where Jeff Shan revealed a missing piece of the puzzle. The other revelation came from Lian_ZA in this post. However, these two only hinted at the solution. Trying some of what they suggested led to finally getting an invalid cast exception that revealed the existence of ItemProducedEventHandler.

Hold on a minute! I didn’t create that delegate. There’s nothing even close to that name in my code… except the ItemProduced event in the interface. Could it be? Naaaaah. Hmmm….

Well, as it turns out, there is a delegate created by the compiler for each event. By explicitly creating a delegate that refers to the method in question, implicitly cast to the generated delegate type, I was able to remove the handlers:

       1: For Each producer As P In Producers

       2:     Dim _itemProducedHandler As IProducer(Of T).ItemProducedEventHandler = AddressOf ItemProducedHandler

       3:     RemoveHandler producer.ItemProduced, _itemProducedHandler

       4:  

       5:     Dim _productionCompleteHandler As IProducer(Of T).ProductionCompleteEventHandler = AddressOf ProductionCompleteHandler

       6:     RemoveHandler producer.ProductionComplete, _productionCompleteHandler

       7:     producer.Dispose()

       8: Next

That’s “all” it took to finally be able to remove the event handlers and maintain type-safe code. Hopefully, this will save you the same challenges I had in trying to figure out how to fix this issue!

IASA ITARC – Denver May 6th

Jeff Certain — Tue, 06 Apr 2010 18:40:19 GMT

The Denver chapter of the International Association of Software Architects (IASA) is holding an IT Architect Regional Conference (ITARC) in Denver on May 6th.

The speaker list for this conference is amazing. Paul Rayner, Dave McComb, Randy Kahle, Peter Provost, Randy Stafford, George Fairbanks – all great speakers, and from Colorado. Brandon Satrom (who also happens to be the president of the IASA Austin chapter) will also be speaking, as will some other heavy hitters (for example, Ted Farrell, Chief Architect and Senior VP of Oracle). This is an amazing line-up, and the conference is quite reasonably priced ($150 for IASA members until April 10th, including a catered lunch).

I also have the privilege of being a presenter at this conference. If you’ve ever heard any of the previously named speakers, you know that they set the bar quite high. Sounds like I’m going to have to step up my game.

What I get to talk about is really cool stuff. The company I work for – Colorado CustomWare – brought me on board nearly two years ago. To say there was some technical debt is somewhat… understated. Equally understated would be that management is committed to doing the right thing. Over the past two years, we’ve done significant architectural refactoring – including an effort that took the entire team offline for most of a month. We’ve reduced the application size by 50% without losing functionality. As you can imagine, this has reduced the complexity of the application, making development faster and less prone to bugs. We’ve made many other changes – moving to an agile process, training developers, moving towards a more OO architecture. The changes we’ve made reveal, in some ways, just how far afield we were.. and there are still more changes to be made.

Amazingly enough, our leadership team is eager for me to share these experiences with other architects. I’m really looking forward to being able to do so.

Renewed

Jeff Certain — Thu, 01 Apr 2010 18:03:12 GMT

I just got a nice little e-mail from Microsoft. Despite the timing, it’s not an April Fools joke… I’ve been renewed as an MVP for another year.

Congrats to all the other MVPs being renewed today.