Ok, one more human interest post before diving into the technical stuff. Let's have a look at the Falcon team as it stands today. Kevin Lewis, Falcon Team Lead
Kevin joined MySQL two years ago, following a ten-year run as an engineer on the MicroKernel Database Engine team at Pervasive Software, (Btrieve, Pervasive.SQL) in Austin, Texas. He ascended to team leadership when Calvin, our project manager, left MySQL in January.
I worked with Kevin for five years at Pervasive, where he started six months before me, just as he did at MySQL. Kevin's a good friend and has at various times served as something of a father confessor to me, patiently indulging some of the more, shall we say, unsettled phases of my life.
Kevin walks his talk and is supremely ethical. He's a year older than me but looks ten years younger, and twice introduced me to the humbling rigor of backpacking in the Colorado Rockies. (T-shirt idea: "I got b****-slapped by a fourteener.")
Kevin's risen nicely to the challenge of running the Falcon team, and has managed to navigate the turbulent waters of leading a high-visibility team within an open source company. I wouldn't say that leading Falcon is like herding cats--we're not finicky anarchists--but it might be compared to producing the Juilliard senior class play for a performance before the U.N. General Assembly. Or something like that.
Anyway, Kevin's the kind of guy you want to lead a team because he has steady nerves, an assured demeanor, and is physically larger than any of us.
Jim Starkey, Principal Software Engineer, Server Architect
Falcon is Jim's baby, and he is, by all rights, alpha dog on the project. He wrote most of Falcon before MySQL acquired his company, Netfrastructure, in 2006.
I won't provide much of Jim's history here, suffice to say that he's been in the database industry longer than most MySQLers have been alive (seriously), and that he has a proven track record of developing and shipping databases that people pay for and use.
(This is not a minor point. I know many bright, confident and very skilled developers, in and out of the open source world, who lack the practical experience of shipping and supporting a product. If you'll forgive the war metaphor, it's like training for combat but never experiencing it.)
To his credit, and to the occasional dismay of his more reticent counterparts, Jim kicks up a lot of dust inside MySQL, frequently challenging the "It Has Always Been Thus" mindset that tends to accrete in any organization. And, in keeping with the war metaphors, Jim will not hesitate to toss a conversational hand grenade into a quiescent forum or otherwise unremarkable exchange, but only because that's what's on his mind at the time.
All of this is a good thing, in my opinion, because Jim's unsubtle energy is usually countered and productively engaged with corresponding intensity--there is no lack of intensity at MySQL--and some pretty good ideas get tossed about as a result.
Ann Harrison, Lead Software Engineer, Server Architect
Ann inhabits several roles on the Falcon team: architect, developer, documentarian. She is the go-to person for certain elements of the current or future Falcon architecture, Foreign Keys being a good example.
Actually, Foreign Keys is a really good example, because Ann must not only hammer out agreements with the other architectural heavy-hitters at MySQL, she must also comprehend (or convincingly pretend to comprehend) the limitless tracts of arcane Foreign Keys documentation.
Ann is also Cher to Jim's Sonny. If you are unfamiliar with this American cultural reference, let me paint the picture this way. Imagine Jim holding forth on a topic about which he feels strongly, whether on the phone, during a presentation or at the hotel bar. At some point, a clear, authoritative yet uncontentious voice will arise in challenge:
"Uh, no, Jim. Actually, that's not the case at all."
NMI triggered. Handler routine executes.
"Excuse me? What's not the case?"
"You said ABC. It's actually XYZ."
"No. It's ABC. And how on Earth can you possibly know that? I wrote the code myself."
"Yes. And I pointed out to you that it was wrong, and then you fixed it."
"Oh. You're right. Sorry. Ann's right. It's XYZ, not ABC. Anyway, as I was saying..."
Error corrected. Runtime processing resumes.
Trivia: Jim and Ann sail boats. Both are both private pilots. Ann has an instrument rating.
Hakan Küçükyılmaz, Senior Software Engineer, Falcon QA Lead
Hakan has Turkish citizenship, is of Georgian ethnicity and lives in Germany. Like all Europeans, Hakan speaks twenty-two foreign languages. Hakan developed some of his database chops working with (if not for) SAP.
Hakan is responsible for Falcon QA--an enormous challenge--and has been at MySQL longer than any of the Falcon team, since 1948, I think.
Apart from adapting the existing InnoDB test cases, the real challenge for Hakan has been to develop test cases that exercise elements of the Falcon engine that are uniquely Falcon--high concurrency and a heavy reliance on abundant RAM, for instance.
Hakan's efforts have been supremely valuable in helping us pin down exactly what this Falcon thing is relative to other storage engines and relative to what we expect it to be. Last summer, for example, Hakan started a weekly DBT2 run which, after no small amount of haggling over the test configuration, eventually became the primary engineering measure of Falcon performance.
[N.B. I must emphasize here that we use DBT2 as an internal metric with which to measure Falcon performance relative to InnoDB. It is by no means regarded as the best representation of Falcon performance, and, in fact, was chosen precisely because it tends to favor InnoDB, especially at low concurrencies. However, DBT2 is a measure of something, and judging by the number and quality of bugs resolved as a consequence of its use, DBT2 has proven to be an outstanding tool for evaluating Falcon performance.]
With the recent Sun acquisition, we now have the kind attention of the Sun Performance and Availability team, whose sole charter is to measure and improve performance on Sun hardware, which brings to mind a quote from Terminator: "That's what he does. That's all he does! You can't stop him!"
Anyway, I like Hakan. He is outspoken. He keeps us real. He's probably the hardest working member of the team. Hakan brings to the Falcon team a kind of weary, fatalistic edge that serves as a fitting balance to my own eternal optimism.
Imagine Friedrich Nietzsche. Now imagine Eeyore. There you have it.
The picture is of Hakan doing something really cool, I'm just not sure what.
Vladislav Vaintroub, Senior Software Engineer
Vlad joined Falcon in January. He's Russian, lives in Germany, has a math degree. Like most Europeans, Vlad speaks twenty-two foreign languages, many of them English.
Vlad got much of his database experience at Software AG. At MySQL, he hit the ground running and was fixing Falcon bugs before he got his first paycheck.
Vlad bravely or naively took over the Supernodes feature, a kind of meta-index (Kevin doesn't like that description, but I'm sticking to it.) He reworked the Supernodes code a bit, fixed the bugs, checked it in, demonstrated it to our beloved VP of Engineering and finally measured a significant performance improvement (more on performance later.) Not bad for his first three months on the job.
Vlad's soft-spoken and unobtrusive nature has a somewhat calming influence on the team, I think. He also possesses a rare talent (rare in the context of MySQL) that is shared by all Falcon developers, which is the ability to develop on Windows.
At right is Vlad affecting ironic detachment. The photo was taken shortly after the other Beatles had crossed the road.
Kelly Long, Senior Software Engineer, Performance Architect
I said this to Kevin at the MySQL Users Conference:
"I had a great discussion about Falcon performance with Kelly last night at the bar. This guy's a sleeper, man, I'm telling you."
First, just let me say that all of my great conversations at the Users Conference took place at the bar. Not so much because of the drinking (no, really), but because that's where everyone lands at the end of the day.
Kelly Long works for Mikael Ronström's architecture group and is not, strictly speaking, a member of the Falcon team. However, since late last year, Kelly's primary focus has been to measure and analyze Falcon performance. To give you a sense of how seriously he takes this job, at right is a snapshot of the gear this cat has at his home.
My remarks to Kevin that evening were, in part, inspired by some outstanding analysis Kelly did on the Falcon SyncObject implementation, and the subsequent fix. I'll save the details for a near-future post, but the fix closed a timing window whereby sleeping threads were not awakened when the object for which they were waiting became available.
The problem typically manifested at higher concurrencies (64 to 256 clients) on an 8-way system. The symptom was a perplexing and, frankly, frightening standard deviation. For example, at 64 clients, InnoDB might exhibit an STD of 1.8% whereas Falcon would sometimes have an 80% STD, rendering the results effectively worthless.
Apart from his recent SyncObject discovery, Kelly had also performed in-depth analysis on other sacrosanct components of the Falcon core, including some of the page cache algorithms, and was in the process of trying out code changes to verify his conclusions.
Now, in the context of our small and busy little team, I regarded this news as an absolute gift, because it was precisely the type of analysis we've needed but had neither the time nor the resources to perform.
I should also note that, as a seasoned senior engineer, Kelly had the good sense to establish some empirical evidence before announcing his findings, because changes to the Falcon core are not considered lightly, and we just don't have time to chase every "what-if" without probable cause unless it exhibits prima facie evidence of potential awesomeness.
Manyi Lu, Engineering Manager, MySQL Server 6.0
Manyi Lu is an engineering manager from the Sun Database Technology Group. She is from Beijing, lives in Norway and speaks at least two more languages than me. Manyi very recently became project manager of the MySQL Server 6.0 and related groups, including Falcon, Maria, Backup and the Subquery optimization teams.
Manyi introduced herself to Kevin and me at Marten's party during the UC. At the time, I was blathering on to Kevin and Kelly about some damn thing or another, when Manyi and Rune Humborstad, a Sun software architect, drifted over to us and started asking questions about Falcon's stability and performance.
I did not know Manyi or Rune, but was intrigued by their interest in Falcon's stability, which was a bit rocky at the time. Kevin and I were quite happy to oblige, and spoke candidly about the project, neither of us realizing that Manyi would soon become the MySQL Server 6.0 project manager.
We were sort of awkardly positioned around a pesky little five-foot tree in Marten's back yard, but it was a very interesting and positive conversation, and Manyi offered to have some of the DBTG group help with Falcon QA and possibly bug fixing. To me, this was the first tangible evidence of how the Sun acquistion can make a real difference in the outcome and success of the Falcon project.
I left Marten's party really psyched.
The MySQL Falcon team at Heidelberg Castle, September 2007 .
L-R: Hakan, Kevin, Calvin, Ann, Jim, Christoffer Hall (now on the Cluster team), Chris Powers.
