Data driven health decisions

I just had a personal experience with how timely, personal measurements can drive better health and lifestyle decisions.

Unfortunately, it wasn’t related to any of the times that I’ve been genotyped, nor was it in the context of care by any physician. In fact, I had to cross state lines in order to get it done.

More on that later.

The punch line, for the curious, is that I have elevated levels of mercury in my system, and I should probably eat a bit lower on the food chain when I order sushi.

Genomics fanboy

I’ve been a genomics fanboy for years. I enrolled in 23 and me when it first came out. I did the exome add-on that they offered briefly in 2012. I signed up with the Personal Genome Project around that time, and one of the exomes you can download from their site is mine. I drove an hour and spat in a tube for the Coriell Personalized Medicine Collaborative.

Coriell has been the most satisfying for me personally, since they occasionally email a PDF of a manuscript that is based on analysis using data derived from my (and many other’s) saliva. For me, at least, getting to read science papers and think, “I helped!” is much more motivating than cryptocurrency based micropayments.

While it’s all been fun and interesting, I haven’t learned very much that was terribly actionable. Without putting too fine a point on it, I have basically re-verified that I don’t have any of the major genomic disorders that would have already shown up by middle age. My standard line describing what I learned is I’m most likely male, almost certainly of northern European descent, likely brown hair, likely brown eyes, etc.

A question of focus

One way this shows up for me is that I don’t really know where to focus my health and lifestyle efforts. Sight unseen, one might tell a person like me that I should work out a little more, mix it up with cardeo and weight bearing exercise, eat a mostly vegetarian diet, don’t smoke, drink in moderation if at all, maintain a regular sleep schedule, use sunblock, floss, don’t sit too long at work, meditate, never read re-tweets, practice test driven development, etc, etc, etc.

None of it appeals and more or less because I know that all this advice is generic. I.e: It doesn’t really apply to me. I’m pretty healthy, so who cares, right?

On the opposite side, I’ve written before about my frustrations in convincing my physicians to screen me for colorectal cancer. I have a family history on both sides, genetic markers, and a medical history that all point in the same direction: Elevated risk. The current state of clinical practice is that men don’t need screening before age 50. I’ve been getting screened since my late 20’s, and I persist in thinking that it’s a really good idea. This is one of those cancers that is easily treatable with early detection and lethal without it.

So there we have it: Advice is either so generic that I ignore it, or else when I do have actionable information it’s a challenge to convince my physician to act on it.

Personalized bloodwork

Enter Arivale. They are a relatively recent addition in the direct to consumer health and lifestyle offerings that are cropping up this year. I heard about them through professional connections (thanks Dave!), and I’ve been excitedly waiting for them to offer services in Massachusetts.

The Arivale process involves a battery of bloodwork, genetic testing, and a gut microbiome (which is a novel experience if you haven’t provided a laboratory with a stool sample before). They combine this with coaching from people trained in nutrition, genetic counseling, and behavioral modification.

Because of the niceties of paying for lab work, I had to leave Massachusetts in order to reach a lab who could actually accept my money to draw the blood. Bright and early on a morning in the middle of February, I made a pre-breakfast, pre-coffee commute into New Hampshire to be stuck with needles.

Let me pause and say that again: Our health care system is so screwed up that I had to cross state lines to get bog-standard bloodwork done, entirely because I was paying out of pocket for it.

I also filled out a battery of health and family history questionnaires, as well as some about personality and lifestyle.

Show me the data

A couple of weeks later, I got an email and logged into a slick web dashboard. I went ahead and did the integration with my Fitbit account. I disabled GPS location sharing but enabled the rest. Let’s hear it for granular access control. Because Fitbit connects to my wireless scale, my Arivale coach was suddenly able to access five years of weight data on top of the four years of info on my pulse, sleep, and walking habits that my Fitbit devices have accumulated.

Let me pause and say that again: I logged into a slick web dashboard and integrated years worth of data about myself in the context of a new battery of lab tests. At no point did I have to write down my previous physician’s FAX number on a piece of paper.

It felt normal and ordinary, because I’m used to these integrations everywhere except health care. I do this sort of thing with my bank, my utilities, my news feed, and all sorts of other places.

That is a different rant, but come on!

Ahem.

Retrograde

Anyway, I logged in and saw (among other things), this:

It honestly gave me pause. I’m pretty robustly healthy. I don’t expect to see any of my biological metrics “in the red,” but there it was.

So I did a quick Google search, top hit, I feel lucky:

A bit of refinement:

Which led me to look at my last few Grubhub orders.

Yeah, every time I order, I bolt on that mackerel. That’s for me. That’s my treat. It’s worth noting that February 15 was the night before I made that hungry, grouchy drive. I know that mercury accumulates in tissue and lingers there over time, your milage may vary, but it’s a pretty clear signal in my book.

And it showed up in my lab work.

Fallout

So there you have it. All of a sudden, I’ve picked something actionable to do for my health – out of the incredible variety of good advice at my fingertips. Because, well:

Converged IT and the Cloud

I promised that I would post a summary from our closing panel at the Converged IT and the Cloud thread at the Molecular Medicine Tri-Conference.

Unfortunately, I was having so much fun in the session itself that I didn’t take any notes at all. Please forgive errors and omissions. My slides are here, but they’re the very least part of the conversation.

I opened up the session with the question that my friend Carolyn posted in the comments of the last post: “What are the biggest barriers to immunotherapy becoming translational (FDA, funding limits, enrollees in clinical trials)? How can patients best support future immunotherapy developments?”.

It sobered the audience considerably, especially when I pointed out that her interest is as a current patient of the system that we all acknowledge has tons of room for improvement.

My point in starting with that question was to move the conversation up a level from IT people talking about IT stuff – and to provide both motivation and urgency. It is very unlikely that a session on “converged IT and the cloud,” would be able to answer Carolyn’s question. That said, we would be remiss to sit around talking about network speeds and feeds, regulatory frameworks, costs per gigabyte, and other technical details without ever engaging with the high level “why” that drives our industry.

Each of the four panelists prepared a brief summary on a specific topic:

Jonathan Sheffi (@sheffi) is the Product Manager for Genomics and Life Sciences within Google Cloud. He spoke about the convergence that he sees in data structures and standards as customers bring different data types like health information, outcomes data, and so on to the “same” cloud. This was pretty exciting to me – since it is the infrastructure groundwork that will support some of the things we’ve been saying about collaboration and integration in the cloud.

Aaron Gardner is with Bioteam, and shared an absolutely whirlwind review of machine learning and AI for our field. The coolest part, to me, was the idea of AI/ML as a de-noising tool. The hope is that this will allow us to take unwieldy volumes of data and reduce them to only contain the necessary level of complexity for a certain task. It took me back to a dimly remembered time when I would talk about “Shannon Information Content” and similar concepts.

I first heard Saira Kazmi speak at the 2017 Bio-IT World, when she was still with the Jackson Laboratory. She had earned a reputation as Jax’s “queen of metadata.” She combined a handful of deceptively simple techniques with an impressively diplomatic tenacity to create a sort of ad-hoc data lake – without ever pausing to go through the most painful parts of the data lake process. Instead they chose to archive first, scrape file headers into a JSON format and stuff it into a NoSQL database, and (my favorite) stored checksums of large primary data files in a database to identify duplicates and support provenance tracking.

Finally, we had ā€¸Annerose Berndt (@AnneroseBerndt) – who has just finished standing up a genome sequencing center to serve the UPMC hospitals. I asked her to hold forth a bit on security, compliance, quality systems, and other absolutely necessary bits of process discipline.

We shared a wide-ranging and illuminating conversation building on these topics. It was a blast.

As I said from the stage: I really cannot believe that it’s somehow part of my job to have conversations like this, with people of this caliber. How cool!

Molecular Medicine Tri-Conference

I’m in San Francisco this week, attending the Molecular Medicine Tri-Conference. I’m specifically focused on a new track called Converged IT and the Cloud. I’m paying particular attention and taking notes – both because it’s very interesting and exciting, because also because at the end of three days I get to moderate a panel discussion. We’ve set ourselves the challenge of accepting any and all conversation topics and questions – which … even given the talks so far … is going a very broad landscape of awesome, interesting, challenging, and occasionally scary stuff.

In an attempt to break the tyranny of locality and provide a bit of access – if you have a question or topic for the panel, please send it to me (comment, direct message, email, text message, or whatever). I will commit to summarizing the topics here after the conference.

On a personal note, it’s amazing to reflect on how important this community has been in my life and career. Kevin Davies opened the session. Seeing him brought back memories of the uppity startup magazine he created, called Bio-IT World. That magazine developed a mutually supportive relationship with a scrappy new consulting company called Bioteam. By hook and crook, hard work and happy accident, we’re all still bumping along the road, showing up at the same conferences, and working to improve the world together. Along the way, I’ve seen collegial work relationships turn into deep and lasting friendships.

It’s pretty cool. I’m happy to be here.

The Multi-Protocol Fantasy

There’s a particular class of problem that I’ve grappled with regularly over the past decade. I got to go another round just before the holiday break. I figured that this was as good a time as any to share some thoughts.

Multiple representations

Some file servers are able to present a single filesystem via two (or more) different protocols. The most common use case is to support both Linux and Windows clients from the same underlying data. In the usual setup, Linux systems see a POSIX compliant filesystem via their old reliable Network File System (NFS) protocol. Windows clients use the Windows native CIFS/Samba and see a friendly, ordinary NTFS filesystem (sometimes mapped as “The L: drive” or similar). In recent years, vendors and developers have begun to add support for newer systems like S3 and HDFS.

We use this sort of thing all the time in the life sciences. Lab instruments (and laboratory people) write data to a Windows share. The back end heavy lifting of the high performance computing (HPC) environment is done on Linux servers.

The reason that this seems like a good idea is that it’s kind of a pain to convince Linux systems to mount via CIFS / Samba, and it’s also kind of a pain to do the reverse and mount NFS from Windows. While it is certainly possible, all of us of a certain age will respond with a grouchy “harumph,” whenever the idea comes up. I’ve spent many happy hours navigating the complexity of the “unix services” tab on the active directory master (assuming that a lowly Linux admin is allowed access to the enterprise identity management service) or else googling around (again) to find that one last command to convince Linux to “bind” itself to an LDAP server.

The underlying problem

POSIX and NTFS are just different. As soon as we stray beyond the very simplest use cases (writing from a lab-user account and reading from a pipeline analysis account, for example), we encounter situations where it is literally impossible to give correct semantics on both sides. One filesystem has to present an invalid answer in order to provide correct behavior in the other.

Under POSIX (the Linux / NFS flavor) we use read, write, and execute permissions for a single user (the owner), a single group, and a conceptual “everybody.”

Under Windows, we have “access control lists” (ACLs). Any entity from active directory (either users or groups) can have read or write permissions. The idea of a file being executable is handled in a different way.

It’s straightforward to create permissions under windows that cannot be directly applied on the Linux side. The simplest example is a file for which exactly two users have read access – but where there is no group comprised of only those two users.

Note that this is true even if all of the users and groups are already correctly mapped between Windows to Linux.

Note further that as you start creating utility groups to overcome this “bug,” you will rapidly discover that NFS only honors the first 16 groups listed for a user – even if those groups are served up by an NIS or an LDAP service.

There are plenty of other examples. The rat’s nest I found myself in last month involved setting default permissions on files created by Windows such that they would be sensible on the Linux side. Under Linux, each user has a single default group (specified by number in a file called /etc/passwd). That concept, of a default group for each user, simply doesn’t exist under Windows.

Go around

Back in November, I wrote a post titled Go Around, about how, sometimes, when you hit an intractable obstacle it just means you’re looking at a problem from the wrong angle. That’s exactly what’s going on when you spend time yelling at a vendor or deforming your active directory forest to solve these protocol collisions. It’s asking the impossible.

As an engineer or informatics person, it is up to you to find a way to frame problems such that they are actually solvable.

There is no such thing as a free lunch on this one. If you want to use POSIX semantics, you have to accept the limitations of POSIX. The same is true of Windows, S3, HDFS, or any other data representation.

Vendors: Multi-protocol permissions blending will never ever work. It just looks like buggy, flaky behavior on the client side. Stop trying.

Users: Go around.

Addendum

The way that you “go around” on this one is to decide which protocol the majority of usage will come from and engineer for that.

If it’s going to be used mostly from Linux – then don’t set complex permissions under Windows, and tell the users that their fancy ACLs are not going to be honored. If it’s the other way around (mostly Windows use), then create a utility group on the Linux side (just the one) and make sure that all the appropriate users are in that group.

Trust me, it’s much easier than trying to merge fundamentally incompatible semantics.

Two clouds and a bicycle

Yesterday, I got to solve a puzzle that I’ve solved more than a few times before. “How do we get a moderate amount of data from here to there in reasonably short order?”

The specifics in this sort of puzzle change all the time. What constitutes a moderate amount of data? What sort of distance matters between “here,” and “there?” Is it miles? vendors? representation?

Other factors, like the timeline, remain constant. “Reasonably short order,” usually means “by tomorrow morning,” give or take. It’s a short, but achievable timeline where smart decisions to reduce latencies in the real world (start immediately!) matter as much or more than decisions about transfer protocols or software tools that might give a more perfect use of resources.

It’s a case where the perfect is the enemy of the good, and where having seen puzzles like this before confers an advantage. As is true throughout life, having friends helps too.

Here’s the story:

500GB on a jump drive

Diamond Age Data Science is a consulting company in the business of computational biology and bioinformatics. The founder texted me in the early afternoon: They had agreed to do a a particular analysis, “before the end of the year.” They planned to use Amazon’s cloud, and the source data was on a 1TB USB disk at LabCentral in Cambridge, MA. The customer had tried a sensible thing – plug the disk into a laptop and use the GUI to drag and drop the files.

The time estimate was FOUR DAYS from now.

In real terms, this would have consumed one of the two weeks remaining before the end of year holidays. This would put the deadline at risk, and had the potential to interfere with a laptop-free vacation.

In addition, the customer wanted to take their laptop home in the evenings even during the data transfer. That meant stopping and restarting, or perhaps lashing together an impromptu workstation.

It was only about 500GB of data. 500 gigabytes is 4,000 gigabits. At one gigabit per second (a relatively standard wired connection in a modern office), this was not less than 66 minutes worth of transfer. Even accounting for inefficiencies on the order of a factor of two or three – this shouldn’t have been an overnight problem – much less a week of delay.

I happened to be about a 10 minute bicycle ride from the lab in question, and this is a game I like to play.

To make it more fun, I decided to up the ante from “tomorrow morning” to “before dinner.”

Could I move the data and still get home in time for dinner with a friend from out of town? I figured that it was reasonably possible, but I couldn’t afford to go in the wrong direction for even an hour.

As an aside, overhead and protocol challenges are a really big deal in large scale data transfers in clinical medicine, finance, satellite, media/entertainment, and so on. For a graduate education on this topic, read up on Michelle Munson’s Aspera. Start with the FASP protocol.

Which way to the cloud

There are a lot of places in Boston and Cambridge with good connectivity. I figured that I could call in a favor with friends at Harvard’s research computing, at the Broad Institute, or even at one of the local pharmaceuticals. Any of these could have worked out fine.

However, all of those groups use fiber optic connections that terminate at the same building: The Markley Data Center at Downtown Crossing. It’s perhaps the single best connected building in New England. Also, the people who run the place are smart and like to play this sort of game.

So I picked up the disk and biked across the river. It took about 20 minutes.

My bicycle was about a 3Gb/sec transfer device – with latencies on the order of 1,200,000 milliseconds.

Linux Servers are Free

There’s an old saying – you never eliminate a bottleneck, you just move it around. Since I wanted to be home in time for dinner, I wanted to think through, rather than discover those bottlenecks. One of the big potential bottlenecks was the wireless card on my laptop. Most wireless networks are not full 1Gb/sec. My Apple laptop, for all that I love it, does not come with a traditional RJ45 network plug. I was going to need a wired connection.

So I texted ahead and asked the folks at Markley if I could borrow a laptop or a workstation to plug the disk into. This was also a hedge – assuming that we could get the data transfer rolling, I wouldn’t be in the same situation as the folks across the river – waiting on a data transfer before unplugging my laptop.

In the same conversation (stopped by the curb on the Longfellow bridge), we decided to create a virtual machine on Markley’s private cloud for use as a staging machine. I’ve been trapped in enough data centers over the years, babysitting some dumb process while my friends ate dinner without me. So I requested a bit of infrastructure. About 90 seconds later, I had an email in my INBOX including some (but not all) of the credentials needed to log into a dedicated internet connected server with a terabyte of attached disk.

The big advantage of the VM was that it would be trivial for me to reach it from home. Also, I could install whatever software I needed on it without inconveniencing the Markley staff or forcing them to re-image the loaner laptop. The in-building network is extremely “clean” with almost zero packet loss and sub-millisecond latencies (better than a bicycle by about 7 orders of magnitude). I figured that it was very likely that we could get the data off of the disk to something in-building in a couple of hours without any drama. The wider internet is not necessarily so clean and consistent.

I was, at this point, assuming that something would go wrong, and that I would be logging in after my guests went home to bump the process along.

Besides, it’s a 90 second task to configure a virtual machine on a private cloud.

So by the time I got to Downtown Crossing, there was a workstation waiting for me, as well as a virtual machine that I could log into. The nice folks there had also found a Thunderbolt to RJ45 adapter, in case I didn’t want to bother with the loaner laptop. On a whim, mostly curious how fast we could make this happen, I plugged the disk into my own laptop and started a simple “scp” of a test file.

Too slow!

I was getting 3 megabytes per second.

3 megabytes per second is 24 megabits per second. About 2.3% of the number that I had been using for my estimates. That put me right back into the “several days” estimate from before. My friend, sitting next to me, tried the same simple copy from the internal disk on his laptop to the server (connected via an identical connection) and got consistent performance of 80 megabytes per second.

In the spirit of not overthinking it (and getting home in time for dinner!) we tested two things at once by swapping the disk over to his machine and starting the in-house copy to the VM. Why mess with it? It was moving into mid-afternoon now, and we wanted this first phase done before we were anywhere close to stressed.

I should note that, at that point, it could -totally- have been the disk. In that case, we would have been stuck. Fortunately, the connection screamed to life at 80MB/sec. He set to work starting a few batches of copies to run in parallel, which squeezed another 10MB/sec or so out of the connection. 90 megabytes per second is 720 megabits per second, or 72% of the theoretical maximum of the connection. Our theoretical 66 minute transfer was going to take 90 minutes to get to the VM.

Duh

It wasn’t until half an hour later that I realized the problem with my laptop: I still had my VPN configured. After all that bicycling and thinking I was so clever – my laptop was doing exactly what I had set it up to do, and spraying packets all over the country, from Seattle to Georgia, so that whatever coffee shop, airline, or hotel I happened to be connecting from couldn’t snoop on me.

After a good laugh, we agreed that I have a really good VPN, all told. Also, we let the copy run, it was already about 1/3 of the way done.

The Last Bit

The final step was almost laughably easy. I wanted the data to wind up on Amazon’s S3. My friend at Diamond Age had configured a user account for me, and had picked a name for the bucket.

It turns out that enough people have solved this problem before that it was mostly copy-pasting examples from the Amazon Command Line FAQ. There was the tiniest bit of Yak Shaving to get Python, PIP, and the Amazon CLI configured on the VM – but after that I ran a command very much like:

aws sync /data s3://my-awesome-bucket

And was rewarded with transfer rates on the order of 66MB/sec.

I verified that I could get ~60 megabytes per second consistently from the private cloud to the public cloud, and also that we could do both transfers concurrently. The link from the laptop to the VM was still contentedly chugging along at 90MB/sec.

I also verified that I could log into the VM from outside of the data center. I would have felt pretty silly to have to come back the next day to do any necessary cleanup.

My friend and I caught up a bit about our plans for the holidays. Then I biked home as the light faded.

By the time I got home, the initial sync command was done, and all the data from the disk was present on the VM. I ran the sync again, with the options to do a detailed check for any differences that might have crept in by copying files in these two stages.

It finished up while I was writing this post, after dinner.

DeepVariant

Earlier this week, Google published DeepVariant, a machine learning (ML) based tool for genomics. The software is now available on the DNANexus platform.

This is kind of a big deal, and also kind of not a big deal.

Does it matter?

It’s a big deal in the same way that ML systems exceeding the performance of radiologists on diagnostic classification of images is a big deal. Sure, it’s a little creepy and intimidating when a computer program exceeds a respected and trained professional at one of their tasks. On the other hand, it would take a spectacularly naive and arrogant person to claim that a radiologist’s only job is image classification.

It’s not a big deal because there is still so much domain expertise required to derive scientifically meaningful results from genomic data, and because these methods are still changing all the time.

The DeepVariant team took one of the points in the genomic analysis workflow where scientists have historically used eyeballs and intuition to identify subtle patterns in the data. Prior variant callers were built atop that intuition, coding it into complex algorithms. That’s why there was a well characterized image format (Pileup) already available as a starting point for the project – scientists still want to look at the results of their callers to see if the results align with intuition.

That’s why there was a contest for the team to win. Because we’re still figuring this stuff out.

It was a good place to start, and the system performed much as we might expect.

Much to Learn

I saw a preview of this technology at the Broad Institute, sometime in mid to late 2016. We were all really impressed. I remember that someone asked exactly the right question: “Can it -discover- a new sort of biological artifact or feature? One that we haven’t seen before?”

The team was unambiguous: Of course it can’t. Until the patterns are present in the training data, there’s nothing there to learn. Further, this particular approach will never suggest that, maybe, we’re looking at the problem sideways.

Put another way: There is a lot of genomic biology still to be learned.

Every year that I’ve been in and around this field, there has been at least one discovery that has up-ended major pieces of institutional knowledge and dogma. Formerly safe assumptions get melted down, combined with new insights, and formed into superior alloys all the time.

The more subtle challenge

There is a more subtle challenge in this particular case: We’re dealing with measurements rather than facts here. The process of DNA sequencing is complex and subtle, with biases and omissions and room for improvement throughout. The way that this particular test was framed up assumes that there is one unambiguous correct answer to the question of variation, and that we already know that answer.

A genomic biologist – or scientist of any stripe – has to hold two truths in their head at the same time: They must gather data to answer questions, and they must also accept that the data may suggest refinements to the question itself. Those refinements to the question, the ones that call existing knowledge in question – that’s where the real innovation happens.

Given enough data, machine learning now excels at answering well formed questions. The task of questioning our assumptions and changing the question itself remains much more subtle.

The take home

The short version is that computers are here, right now, to take away any part of any job that involves memorizing a large corpus of data and then identifying new examples of old categories based on patterns in that data. This is just as true for eyeballing pileup images as it is for reading ZIP codes or license plates.

Machine learning is also here for any part of your job in which you merely turn the crank on a bunch of rules and formulas. This has already impacted a bunch of different jobs: Tax preparation, law, real estate, and travel planning have all undergone radical changes in the last decade.

One final thought: This is also a big deal because while it takes massive computation to create a recognizer like DeepVariant, it is trivial to use that recognizer on any particular input. Variant calling in the old model takes up a lot of CPU power – which can now be turned (hopefully) to more subtle questions.

Consultants vs. Contractors

A few weeks ago I wrote about the distinction between bioinformatics and computational biology. My point in that post was not to be absolutely correct about the definitions. Rather, I hoped to encourage conversation around the differences. I hoped to light up the importance of considering this distinction when planning and executing on projects.

There is a similarly valuable distinction to be made between contractors and consultants.

The core of it (for me) is that contractors are paid to do the project itself, where consultants are paid for their opinion, advice, and support – usually framed in highly actionable recommendations.

Opinions will, of course, vary. Please leave comments, particularly if you disagree!

With that framing, you can see why it’s important to know which kind of engagement you’re signing up for, either as customer or provider. I’ve had projects go rather radically off the rails because of mixed signals on the role that I was expected to play.

Note that many of us walk in both worlds. Most people spend the early part of their career doing projects, and transition to opining and advising once they have the benefit of that experience. There are, of course, exceptions, particularly when the services are closer to coaching than engineering.

This also ties into the question of how to set rates.

Finding the appropriate value for a contracting engagement tends to be relatively straightforward. The customer usually has an idea of the value of having the project complete, and has an idea of what it would cost them in staff effort and wall clock time to do it in-house. If that value (to the customer) is greater the amount that the contractor needs in order to make it worth their while, then there is space for negotiation.

Consulting roles are harder to quantify, though it is possible. One fruitful approach is to look at the expected costs / returns of various project outcomes. One need only look at the costs of a ransomware attack, data loss, or HIPAA violation to see that there is plenty of space in the information security field, both for good advice and also for robust operational practices.

As usual, I’m excited to hear your thoughts on this – doubly so if you think I’ve missed something important!

Setting rates

This is a post about money – specifically about how to set consulting rates. It builds on two prior posts about the legal mechanics and some business development practices of my one person consulting shop.

People tend not to talk about the specifics of compensation. Because of this, most of the advice that I’ve read over the years has been vague to the point of uselessness. There are various reasons for that – mostly centering on cultural squeamishness about money. My opinion is that while discretion and humility around the absolute numbers are good things, we need to be able to talk about this sort of thing far more openly than we usually do.

The majority of this post is about figuring out what you need – a minimal baseline that will cover expenses. At the end, I offer a bit of a vague tease about figuring out what the market will bear. It amounts to “until you have data based on your own experience, you have to ask your friends.” This will take emotional energy and trust – especially at first – but it’s absolutely essential to go into negotiations knowing both common practice and your baseline requirements.

As usual, this is based on my experiences and those of a few trusted friends. Your milage will almost certainly vary. I’m curious to hear your comments.

The Salary Comparison

A consulting rate is very different from a salary. In fact, comparison with a salary is among the worst ways to figure out an appropriate rate to charge for your time:

Consider the old shorthand:

There are approximately 50 weeks in the year, and 40 hours in a work week. That means that there are 2,000 hours in a work year. If your aspirational annual salary is $100,000 (a round number makes the math easier) you can just divide by 2,000 to get an hourly rate of $50.

The problem with this approach is that it’s misleading and causes you to under-charge.

It is unwise to bet on being 100% billable, particularly over the long term: I’ve been consulting for a while, I’ve got a strong professional network, and I treat business development as a core component of my job. My experience is that I can sustain between 70% and 80% billable time (against the arbitrary 40 hour week). A safer assumption until you have data on your own network and capacity is 50%.

Vacation: Even if you personally decide that you don’t need to take breaks – many customers will go completely dark on you during national holidays and also from mid November through early January. It’s also challenging to get the right people in the room during the summer.

There is a lot of non-billable work in consulting: I’ve written before about “business development Fridays.” If you want to have billable work, you must do non-billable work. This inevitably cuts into your ability to meet that notional 40 hour, 50 week schedule.

There are a lot of non-salary expenses that an employer would usually cover: Health insurance is the big one, and the self employment tax works out to about 7.65% right off the top. Also consider any sort of employer contribution to a retirement fund, transportation benefit, funding for education or conferences, and so on.

Business expenses: Distinct from the non-salary benefits, you also incur direct expenses. Even the small ones add up: Computers, printers, shipping, an attorney to review contracts, an accountant to prepare taxes, liability and other insurance, a co-working space to host meetings … these things all need to be funded out of the consulting rate.

Multiple budgets

A better approach is to start with a budget, or rather at least two budgets. Even though I’m a one-person shop and the IRS doesn’t really care whether I keep dollars in this checking account or that one – I still make a strong distinction between my personal and my business finances.

Start with a personal budget. Keep it high level, and be sure you get all the major expenses in there. Housing, health insurance, utilities, transportation, childcare, saving, etc. In order to keep from getting distracted by the details, I set an arbitrary requirement that it had to fit on one sheet of paper.

Note that taxes are not in the personal budget – I’ll get to that in a second.

The bottom line on the personal budget was the number that I decided to transfer from my business account to my personal account every month. That’s what I decided to pay myself.

With that in hand, I had the first line for my business budget.

This second budget is similarly high level. It includes liability insurance, all the expenses listed above, and also a line for taxes based on an assumption that a little less than 50% of every dollar I receive will somehow go out the door to the city, state, and nation.

The bottom line of the business budget is the actual amount of money that I need to bring in every month in order to make this independent consulting thing work.

That’s my number.

The tyranny of the number

When I started out with BioTeam in 2004, Stan insisted that each of us work to a ‘number,’ an amount of revenue attributable to our work. At the time I resisted it – how could a number capture the grace and beauty of my engineering contributions to the team? How base and unseemly for a software person to be tied to a (yuck!) sales goal.

Technical people, I am speaking to you, are you listening? Get over that squeamishness and learn to love your number. It is how you get paid.

Now the only thing left to do is to figure out how to turn that number into a rate.

Setting a rate

Having done the work above, you know the total amount of dollars that you need to bring in (on average) every month. The next question is how to chop that up into hours, days, and deliverables to make it work.

I made a little chart breaking out my required rate under various assumptions. This gave me a baseline – the minimum that I could possibly charge, under various conditions. Because of the bottom-up approach, I also had a good feel for what I might need to charge for fixed fee deliverables like blog posts and technical assessments.

Then I went out and did some market research.

What the market will bear

Of course, you shouldn’t simply charge the minimum. Instead, you should charge an appropriate and fair rate for your services. There are several ways to define “fair” in this context. All of them involve talking to people about money.

The very best thing to do is to ask peers who either provide or consume consulting services about what rates seem reasonable and common to them. This is best done one-on-one with people you trust. Over time, you will get more comfortable both asking and offering this level of detail.

Negotiation is a whole different topic for a different blog post, so I’ll just summarize here:

Figuring out if an exchange is “fair” requires that both parties understand the relative values involved. If you’re going to be an effective partner in negotiations, you must understand your requirements before walking in the door. The whole top portion of this post was required just to be able to hold up your end of the business conversation.

If you don’t have information both on your needs and on the common practice in your industry, the other person might suggest that you take a reasonable salary number and divide by 2,000. Seems “fair,” right?

Go Around

When I was working with the NY Genome Center from late 2011 through early 2014, I took the opportunity to drop in and train at Oishi Judo Club in southern Manhattan. It’s a great school filled with strong, friendly, skilled judoka of all ages and backgrounds. It was the perfect place to balance out the stresses of living in hotels and working at the edge of my capacity.

One of the people at Oishi, Paul Virtue, taught me a great life lesson that I apply frequently in my professional life.

The summary / mnemonic is: “Go around.”

Paul is vastly stronger and more experienced than me at Judo. He’s a lifelong athlete who competes internationally in several martial arts. He’s also a careful training partner – I was never worried that he would injure me. I always considered it an honor that he would train with me. One of the things that I love about the Judo community is that it is filled with people who are so generous and careful as they share their expertise and experience.

Anyway, Paul and I were grappling one day, and I found myself frustrated that I couldn’t get past his “guard.” I was pushing hard, tiring myself out, but to no avail. He was just too strong. Our different levels of physical conditioning meant that he was going to remain stronger than me as we both tired out. After a few minutes of this, he looked me in the eye and said:

“That’s never gonna work! Go around, man, go around.”

Note, he didn’t say “give up,” “you’re not strong enough,” or anything like that. He also didn’t make a point of flipping the situation around with a cruel, “time’s up, n00b.” Instead, he gave me exactly the tip that I needed at that moment – rather than either shutting me down entirely or encouraging me to continue on a hopeless path – he gave me permission to try a new approach – to be curious and experiment.

What does it mean to “go around?” It’s just that since I couldn’t move him from where I was – I had to move myself. with a new perspective, it was a different problem.

Note that this wasn’t a miracle cure. There was no rapid training montage where I was suddenly just as strong and as skilled as him. Instead, we rapidly evolved to a new stalemate – but then there was another and another. In that series of transitions I discovered a joyful new level of judo practice. It was no longer a test of strength on strength – being strong enough to beat the other person – but rather one of agility and flexible perspectives.

People say that training with the founder of Judo, Jigoro Kano, was like “fighting an empty jacket.” By the time you would bring force to bear, he would have already moved out of the way.

The best engineers I’ve worked with have had this same supple tenacity, turning problems over and around until they suddenly become simple enough to solve.

Recognizing when a particular approach is never going to work and adapting around it isn’t failure (though it frequently feels like failure at the time). Instead, this is the higher level of practice – whether you’re grappling with a gnarly technical problem or an experienced athlete.

It’s a lesson that I took back to my work at the genome center – the team worked around strange configurations of fiber in the street, remarkably complex construction delays, evolving funding sources and strategic imperatives, and even a hurricane that flooded southern Manhattan. In every case, “going around” turns out to have been much more effective and powerful than merely powering through.

If you can’t budge the problem from where you are – then you have to move yourself.

Go around.