The second decade of the cloud

In my talk at Bio-IT World this year, I made some comments about “cloud” technologies that I think bear repeating.

2017 is somewhere in the middle of the second decade of the cloud.

Of course, when I say “cloud,” I mean much more than mere virtualization. You don’t get the 2017-benefits of “going to the cloud” by just hosting your legacy architecture in Amazon’s east-coast-1 availability zone. Nor do you get them by putting your one-server-per-service enterprise on a fancy VMWare / ESX system, no matter how “hyperconverged” it may be. That’s the kind of misuse of the technology that has kept the “on-prem vs cloud” boondoggle alive so far past its expiration date.

Virtualization, of course, is a very good idea. Depending on how you define it, we’re in at least the third decade of OS level virtualization. That’s even more of a solved problem than the cloud.

The benefits of the cloud in 2017 accrue when you adopt cloud-native architectures. This entails substantially more work than porting a system to a hosted platform. It is also absolutely worth it for all but the longest of the long tail of legacy systems.

A bit of history: Amazon Web Services launched as a platform in 2002, and re-launched with EC2 and S3 in 2006. At the time, I worked for BioTeam. Less than a year later, in 2007, we noticed that every single member of the technical team had independently chosen at least one AWS based solution for a customer need. There was no corporate mandate – it was the right way to do the engineering.

At the time, I was responsible for many aspects of Bioteam’s “Inquiry” software product. By early 2008, we had ported our software to AWS and were offering it under license terms that still read pretty well, 9 years on.

While the FAQ above has aged well, that 2008 port of Inquiry looks pretty dusty in the bright lights of 2017. We took a legacy HPC / batch computing architecture and we virtualized it to run on AWS. There is certainly some forward-looking stuff in there, hosts that spin up and down in response to backlogs of work, and also some cleverness around staging data to and from S3. However, it bears little resemblance to the approach that one might take today.

Chris Dagdigian put it well at Bio-IT World: Many cloud-native system architectures do not have a direct “on-prem” analogue. In particular, Lambda and serverless architectures are challenging to explain in terms of the systems that we built in 2006.

As just one small example: On the Inquiry port, we spent a lot of time convincing our old faithful HPC job scheduler, Sun Grid Engine to be okay with hosts appearing and disappearing all the time. In our hosted, legacy architecture, SGE interpreted many aspects of the cloud as repeated failure. Compare that with even the most basic autoscaling architectures – to say nothing of the wizardry behind tools like Amazon’s Athena. Athena is frankly a bit mind-bending for somebody who made a good living less than 10 years ago making less-usable systems to do less-robust data analysis.

I find it clarifying to think about “cloud” from the perspective of a non technologist. When the CFO, COO, or CSO think of “cloud,” or articulate a “cloud first” strategy, they almost certainly have business or scientific metrics in mind, rather than technical niceties like where the metal happens to live. When executives ask for “cloud,” in my experience they are asking for things like:

  • Remove an entire category of off-mission task from the in-house team.
  • Make technology updates totally seamless and automatic.
  • Vastly simplify licensing and budgeting – budget in terms of headcount, not opaque version numbers and product families
  • Scale without limit, even in the event of a “success catastrophe.”

Note that merely virtualizing a legacy architecture onto Amazon, Google, or Microsoft (or yes, even one of the at-least-six-way-tie-for-fourth-place other public cloud providers) provides zero of the benefits for which we are sent off to “cloud.”

The good news: These benefits are, in fact, possible for scientific and high performance computing. It will not be as easy as it was with human resources or office productivity tools, but we will do it. And it will not be as simple as moving everything to AWS east-coast-1.


Over the summer, I have the opportunity to think deeply about the ecosystem of technologies that go by the name “Blockchain.” I’m focusing particularly on how these might apply in a couple of different scientific and healthcare contexts. I plan to post snippets here from time to time, as much to force me to clarify my thinking as anything else. As Hyman Rickover said, "Nothing so sharpens the thought process as writing down one's arguments. Weaknesses overlooked in oral discussion become painfully obvious on the written page."

One challenge when trying to talk about blockchain is that it is massively hyped – sitting right at the peak of Gartner’s hype cycle. Most of the meetings I go to these days include at least one person who asks, regardless of the topic at hand, “what about blockchain?”

Another challenge is that Blockchain is strongly associated with Bitcoin and other cryptocurrencies. The hype brings a certain breathiness to the conversation, while the finance connection brings associations with fraud and nefarious dealings. Neither of these is entirely merited – but I’m finding it important to keep in mind as I explore.

For all that, the foundational documents are remarkably crisp, lucid, and readable. The original bitcoin paper, written under the pseudonym “Satoshi Nakamoto,” is only eight pages long – plus a half page of references.

It’s clear to me that there’s important work to be done in this space – and I’m thrilled to have the time to take part.

Bio-IT World 2017

Last week, one of my favorite conferences celebrated its 15th year: Bio-IT World. This is one of only a few forums where my particular professional community comes together to share our experiences, re-connect, and to try to catch a glimpse of what the future might hold. I’ve been showing up since at least 2004, and it’s always great to see old friends and to meet new ones.

I’m on the committee that awards “Best Practices,” and also “Best In Show.” This year I didn’t actually get to walk around and see the teams present their work. I missed that – it’s a private tour of the most interesting products and exhibitors. We were amazed at the strength of the entries this year. We saw projects from clinical diagnostics, agricultural genomics, semantic reasoning, and so very much more. The choices are never straightforward, but this year turned some sort of a corner for me in terms of direct impact and technical maturity.

The most fun two hours of the show was moderating the re-imagined “Trends from the Trenches.” Chris Dagdigian has been giving that talk since 2008, and he finally won his multi-year battle to expand it beyond just him. We re-framed the session as a panel / mini-symposium including five Bioteam people. It was a thrill to try to keep up with that crew, live and in real time. We tried a couple of experiments in audience interaction while we were at it, Slido to submit and vote on topics, and a pair of Catchbox throwable microphones. There was immediate synergy between the two systems: The very first Slido question was “can you reach the back of the auditorium with the Catchbox?” I couldn’t, but got an audience assist and it worked out okay.

I also got to give a solo talk in the Data track, focused on a couple of practical points about cloud technologies and data management. I’ve put those slides online at Slideshare.

Thanks to everybody at CHI who worked so hard to put together a great show. See you next year!

The Undocumented Years

Back in the day, I had a Livejournal blog where I kept in touch with a few dozen friends and acquaintances. I used it to push out updates that I thought people might find interesting, when they had the time to read. I used to post about once a day. Reading back through those posts gives a decent feel for where I was and what I was doing at the time. The signal to noise ratio was decently high. Compare with the triviality of facebook and the raw static of Twitter. Sure, there were plenty of posts about what I had for lunch – but I can also piece together a lot of my life during those years.

Having a standalone blog like this one is different. There’s no community – and if there was a community it would be all about me and my posts. People would be checking on me in particular rather than checking in on what “the crew” was doing*. I think that’s why I feel no particular urgency to put anything here at all. The crew is gone.

I’m sort of sad that the world has moved on and I’m no longer living a documented life. I’m in a mood to get all retrospective, and I lack the tools.

Or more loosely:

You may ask yourself, what is that beautiful house?
You may ask yourself, where does that highway lead to?
You may ask yourself, am I right, am I wrong?
You may say to yourself, my god, what have I done?

*: Yes, I’m aware of RSS feeds. I offer them. The point stands.

Tools I use

I work with computers for a living. Here are some of the tools that I use all the time:

Macbook Pro: I’ve used a Macbook Pro as my primary workstation since they were introduced, and I haven’t looked back. I’ve had one significant hardware issue in all that time, and Apple fixed it for me at no charge. As JWZ says: I don’t buy computers based on how fast they are, I buy them based on how easy it is to get things done with them, and Apple is the hands-down winner on this pretty much across the board. OS X is not Linux and it is not Windows. If you really want to run Linux or Windows, I advise you to run the OS you actually want to be using.

That said, OS X appears to be devolving into an OS designed primarily for use on a phone with a sidebar of workstation features. If that slide continues, I’ll probably jump back to Linux. And cry, because Linux productivity software still really sucks.

Postbox is a commercial email reader. Given that reading mail is one of the first use cases of the internet I would have expected a good, free solution to be out there. No, Pine does not cut it. Postbox is commercial software, but so worth it. I don’t even use 75% of the features (integration with social media, RSS aggregation, etc). All I want out of a mail reader is “many IMAP accounts with really good sorting functionality and not too crashy.” Apple‘s used to be a pretty solid piece of code, but I gave up on it as of about 2010.

Chrome: Web browsers are sort of a sore point for me because so many websites are designed to fail. At this point the cardinal sin in a web browser is blocking me or stalling my computer. When I want google, I want it NOW NOW NOW so I can get a fact without hitting a context switch in my brain. Chrome is fast and not very crashy. When it fails to render some particular site, I don’t struggle, I just pop open another browser. In order of “likelihood to work,” I go: Firefox, Opera, and finally Safari. The obvious exception is when I’m looking at Apple’s movie trailer site, which only works in Safari.

Microsoft Office 2011: I do not use productivity software as a political statement. I do not use document editors merely to write things that only I will read (that’s what VI is for). In terms of getting by in the workplace of 2012, I cannot sacrifice a couple of minutes fighting with my word processor every time I try to add comments to a document that somebody else wrote.

OmniGraffle: For making pictures. So very much superior to PowerPoint. Then I dump the pictures into PowerPoint.

Google Docs: The above notwithstanding, Google Docs got it exactly right in terms of collaborative editing. I run a small consulting group based on a few google spreadsheets.

Google Reader: Great RSS aggregator. I think Postbox does this too – but all my links are in Google already.

Fax Zero: Sometimes you’re dealing with someone whose business process is stuck in 1996 and who needs a FAX sent to them. You cannot argue them out of this fact. In those cases, I print to PDF and send them a FAX.

OneBox: A great little company who provide an “800” number that I can point to whatever actual phone I want, a voice-to-text-to-email service, good conference calling, and so on. If you send a FAX to my onebox number, It shows up as a PDF in my email – as God intended.

Adium: Apple finally broke iChat beyond redemption, so I switched to Adium. I have the feeling that it’s broken at some level, but I don’t care enough to dig into it. Of course, I also use Skype because everybody else does.

Amazon’s Cloud: I finally moved this blog over to Amazon’s cloud servers. Next step, turn off the server in the basement and cancel the IP address that I’ve been paying for.

Of PAX and the Greater Internet

This weekend I spent three days at PAX East. PAX stands for the Penny Arcade Expo. PAX is a convention / exhibition of games and gamers. It’s also something of a movable-feast nerd mecca. This year, like last year, it was at the Boston Convention and Exhibition Center (BCEC) – the biggest venue in town short of the football arena. If the rumors are true, they sold out the entire space for Friday and Saturday, and had some tickets at the door on Sunday. I heard numbers like “20,000 attendees.” I have no idea whether that’s at all accurate or if it’s inclusive of the exhibitors and staff. In any event, it’s large. I started seeing PAX badges on the trolly that I take into town. I decided to walk from South Station to the site because of the mass of gamers waiting for the shuttle bus.

PAX briefly deforms the mass transit user profile of Boston. Is that big? I think it’s big.

Games and gamers have gained much broader acceptance in the past decade or so. Computer games in particular used to be the realm of pasty faced basement dwellers. Now, the gaming console is a staple of the main feasting halls where the cool kids dine and dance. Musical / social beat matching games like Rock Band and Dance Dance Revolution have added athletic and group participatory aspects to games. Phone based social games like Lexulous and Draw Something mean that adults can and do play parlor games with their parents and siblings before bed – maintaining a social connection sorely lacking since the Great Error of single-family housing in subdivisions of the 50s.

PAX brings all of those people together in a massive convention hall.

I don’t want to sound like too much of a shill here, but I’ll go for it anyway: I see in PAX a microcosm of a different and better sort of society. The core of it is that PAX people (at PAX – I have no idea how they act at work or in their daily lives) are, by and large, accepting of each other in a way that I do not usually see in broader society. Everyone knows at some level that when you show up at a gamer convention, there will be gamers there. Most of us pasty faced basement dwellers figured out early in life: Everybody is weird. Everybody, without exception, has a freak flag that is flown on occasion. Each of us requires special handling from time to time. Most of us have read Stranger in a Strange Land and felt its uncanny truth. Hell, there are entire subgenres of scam built around the universal feeling of being just a bit different – just a bit outside.

There’s also a flavor that I picked up in the Martial Arts. There is a default to humility because in all likelihood, the person you just met is really very good at whatever game they do play. Sure, I’ll take you apart at Gears of War – but I don’t even know how to work the controls for … what are you playing again? It looks really cool. Also? Rad costume, bro!

So we had 20,000 people, give or take, all packed into a convention center – and we were all weird together. More important was that the vast majority of us are used to having weird friends. Even the superficially normal people who showed up had to admit that – yeah – they like to play the same games as the geeks.

So what does that mean in practice? The exhibitionist extroverts wear costumes and pose for pictures with each other. Introverts like me quickly tire out and then wind up in a safe little corner watching from a safe distance. There’s a dance stage where people who want to play the dancing games do so. People who want to watch, watch – and we applaud even when someone doesn’t know what they’re doing.

So we’ve got something like 20,000 people doing what they actually want to be doing, and accepting what other people are doing – for a whole weekend. It feels fundamentally different from my day to day interactions with most people most of the time. We can laugh and say “that’s why they have to pay you to be at work,” but I fall back on Jane Mcgonigal‘s comment last year: “The opposite of play is not work. The opposite of play is depression.” Obviously we cannot play games all day every day – but bringing a gamers attitude to day to day life might help a lot of people a lot of the time.

Sure, there’s a corporate flavor to the expo floor. That’s where the funds are centered. We’re still in an economic model where large amounts of resource are pooled and re-distributed via corporations. In 2012 you don’t get to rent the BCEC unless you have the resources of several major corporations at your back.

Don’t like Microsoft? Fine. Go ahead and dislike Microsoft. However, they did rent out a thousand or more square feet of the convention floor at full fare to host your party. Yes I know that you paid admission. You think that your measly registration pays for the hall? You thought perhaps that PAX is expensive for you to attend? Let me assure you that there would not be a party there without the corporate sponsors because none of us could afford to throw it.

There are a number of reasons for this culture. The biggest one is the attitude that Mike and Jerry bring to the direction of the thing. Wise and well heeled publications pontificate on how “leadership defines the culture of an organization,” but to see it in action is mind-bending. Mike and Jerry stand up on stage for an hour plus, each day of the convention, and take questions from the floor. I’ve been to the large corporate all-hands, and this is different. Some of the questions are cringe-worthy – but they invariably take them in stride. At the Saturday session, they got asked rather specific details about their sexual preferences and proclivities. Jerry, in particular, laughed it off and then gave rather specific answers. He closed with something along the lines of:

“I think that if we don’t talk about this stuff, then people sit around thinking that they may be weird and unacceptable, and they don’t know because they cannot talk about it. Perhaps they are weird. Probably not. Probably better to know for a fact whether or not you’re weird. The only way to know is to talk about it.”

My favorite response was to a questioner who asked them to use their powers to resurrect the TV series Firefly. Again, paraphrasing:

“I think that the idea of technology enabled participatory democracy, where massive numbers of people directly effect the behavior of corporations and governments, I think that’s not some sort of sci-fi fictional future. It’s today. It’s happening over and over again in 2012. You people got together and caused a company to change the content of a game that was already pressed to plastic and distributed to stores! You called the government on its SOPA bullshit! So you’re asking me to do something? I say go for it! What are you asking me to do, exactly? You look at the same internet that I do! Make it so!”

Anyway, I wandered the floor, I sat in a corner and watched the exhibitionist extroverts. I caught some music in the JamSpace. I picked up one new game for my phone. I played a couple of console games that I’ve wanted to check out. I even watched some friends play tabletop role playing with many, many dice. I wound up tired but happy.

PAX was fun. Would attend again.

Decent Charities

I find it difficult to select charitable organizations to support with either my dollars or my time. Everybody has a hand out. Federal “not for profit” status is no indicator of good intent, much less effective action in the world. Neither is appearing in the news any kind of a clue about whether or not you ought to support someone. The recent kerfuffle over the Invisible Children organization is a great example of why you need to do your due diligence. Kony is a terrible criminal, but sending cash to Invisible Children appears to be a rather limited way to address that.

On the opposite front, there are just too many good causes and good people out there to support them all. I find that I have to make a choice and accept that, at least today, there are going to be a lot of perfectly good efforts in the world for which I have no dollars or time left.

Anyway here are a few groups that I support. I would love to hear how you decide where your limited charity budget goes.

Boston Area Charities

  • The Birth Sisters at Boston Medical. These folks provide comfort, support, and good advice to women in urban Boston who really, really need it. This is a great charity. I also heard last night that their budget is simply not making ends meet.
  • AIDS Action of MA: Within my lifetime, AIDS has gone from being a death sentence to being a chronic but survivable disease. AIDS Action has been a leader in both clear, honest information and effective support for more than 30 years.
  • Greater Boston Food Bank
  • Rosie’s Place: A Boston sanctuary for poor or homeless women.
  • WGBH: I think that democracy only works if you have an informed citizenry. Public radio provides a critical service. Also? I like the weekend shows.

Women’s and Reproductive Rights: Yeah, I’m savagely pro-choice. Also pro-women. Actually, let’s go with “pro-sentient-beings.”

International, with a focus on Haiti

  • Family Health Ministries: These are the folks with whom I’ve traveled to Haiti. They’re good people with their heads in the right place, and they work hard.
  • Partners in Health: Paul Farmer’s organization. Highly, highly effective and visionary group addressing the root causes of disease in some of the poorest places in the world. If you want to be horrified and inspired at the same time, read some of Farmer’s books. I recommend The Uses of Haiti as a starting point.
  • Amnesty International: Amnesty does direct action, but they also serve to keep certain unacceptable horrors in the world front and center in the news. They’ve been fearlessly speaking truth to power for a very long time.
  • The International Campaign for Tibet: I’ve been to Tibet, I’ve seen the ongoing, deliberate, focused efforts by China to destroy Tibet as a culture and Tibetans as an ethnic group. ICT are the most effective group in the world trying to protect that culture.



So, contrary to all good opinion and intention, I find myself back in the guts of the 10 year old C / C++ monstrosity that is BLAST. For those who aren’t bioinformatics geeks, BLAST is the Basic Local Alignment Search Tool. It’s the very most popular piece of software ever created for analyzing genetic sequence data. It’s also a big pile of C, written by biologists for biologists. One of those pieces of software inside of which you really don’t want to look.

So anyway, we have one architecture (Alpha E40, running Redhat 7.2, thanks for asking) for which BLAST barfs on a particular alignment. If I feed it two very specific sequences it dumps core. So far, I’ve rebuilt without optimization, upgraded gcc to the very most recent version, and examined the core file. This showed me that it’s an “Arithmetic error” in a particular function. Turns out it’s the line:

return searchsp * exp((Nlm_FloatHi)(-Lambda * S) + kbp->logK);

Everything is cool until that outermost multiplication by a very small value. Example idiot prints:

Entering BlastKarlinStoE_simple
^^ searchsp: 351649.000000 Lambda: 1.374063 S: 489 logK: -0.341642
^^ -Lambda * S: -671.916867
^^ exp((-Lambda * S) + kbp->logK): 1.101132e-292
^^ searchsp * the above: 3.872119e-287
Leaving BlastKarlinStoE_simple
Entering BlastKarlinStoE_simple
^^ searchsp: 351649.000000 Lambda: 1.374063 S: 518 logK: -0.341642
^^ -Lambda * S: -711.764697
^^ exp((-Lambda * S) + kbp->logK): 5.446862e-310
Floating exception (core dumped)

The part that gets me is that this works *fine* on other architectures. Unless I get this done in pretty short order, it would have been cheaper for me to upgrade the system on which the code runs than to spend my time debugging it.


Phrases I’ve Heard Enough This Week

“Space”: Our application is poised to become market dominant in this space
“Grid”: Our application is grid compliant, grid aware, and grid integrated to compete in this space
“Best of Breed”: Our best of breed grid services application will control the P2P space
“Web Services”: The web services / grid convergence represents a best of breed synthesis for this space
“Pattern”: Emergence of distributed, grid, web services applications follow the emerge – hype – fail
pattern with shocking regularity (in this space)

In other news, some buzzwords seem to by dying a merciful death. I didn’t hear anyone talk about new paradigms, synergies, and there was only a little blather about objects.

In other news, my fruit smoothie is tasty.

Public Policy

I heard a conversation on the radio this morning which was a classic example of how to lose (win) a political debate. The topic was transportation policy. The occasion was an ongoing strike of the bus drivers (which sucks all around). The participants were the president of the “taxpayers league” (or some such activist/advocacy group) and a member of the Minneapolis transportation board.

It was very one sided, because the taxpayers league guy would never give any plan or statement beyond vague platitudes. He was therefore the winner. The transportation board woman had specific ideas: “Increasing the number of lane-miles will decrease congestion.” His response: “Sure, for the rich white folks who drive on the highway. Instead of catering to your rich cronies, couldn’t we give the working class, the people who make your sandwiches and clean your home, couldn’t we give them a chance to get to work too?”

At that point, the correct (and only possible) response is: “I think we agree on that goal. I really do. We agree that fair, balanced transportation policy for all people, regardless of income level is a goal. I’ve given one possible approach, and you’ve raised an objection to it. Now I’d like to hear your proposed approach, so that we can discuss them and come to some sort of middle ground.”

As it was, she rose to the bait and was destroyed on air by someone who would never give anything more specific than “I like babies. I disagree with you. Therefore you hate babies. Baby hater, how can you hate babies!” “Wait, what are you talking about? I just said…” “BABY HATER!” It was excellent rhetorical technique, and I’m sure that a lot of listeners were left with the impression that the taxpayer guy is very smart, and the transportation board lady was very stupid.

This happens *all the time* in political discourse, and it makes me sad. I would much rather deal with people with real, testable opinions than with good debaters.

Jamie Zawinski has a page somewhere (which I can’t find at the moment) dedicated to the fact that if he was going to carry a sign with a slogan on it, you’d be able to deduce his stance on some issue from the words physically present on the sign. “A living wage for everyone!” or “Peace, not war!” don’t qualify. Instead “Raise the minimum wage” or “Never sell weapons to another country again!” See how this works?

A simple rule of thumb: Imagine the opposite of the position you’re arguing. The dead, absolute opposite. Now ask whether there is even one person in the room who supports that opposite. If not, maybe your argument needs a little refinement so we can tell what you’re talking about.

Then again, if you just want to win arguments…this is a great technique.