Month: March 2019

25 Rules For Sons

A few days ago, one of my professional contacts shared a list titled “rules for sons” on LinkedIn. It was filled with advice like, “the man at a BBQ grill is the closest thing to a king,” and “carry two handkerchiefs. The one in your back pocket is for you. The one in your breast pocket is for her.”

Lists like this are always making the rounds. This one may have started with a 2015 book titled “Rules for my Unborn Son.” There are other versions online, but they’re all the same story. Manhood is about wearing sport coats, working the grill, asking the pretty girl out, marrying the woman, playing team sports, and maybe serving in the military.

I scrolled past, but found that it was still bugging me after a couple of minutes, so I went back and left a two word comment: “Misogynist claptrap.”

He (you knew it was a guy who shared the post, right?) responded almost immediately that I had clearly not read rule 23: “After writing an angry email, read it carefully. Then delete it.”

LOL, right?

I severed the LinkedIn connection. No harm, no foul – but I don’t go to LinkedIn looking for irritation, and arguing in the comments section has never, even once, changed anybody’s mind.

I shared the story with my spouse, and she said, “You should tell him, and you should tell his employer too. Those people scare me. They can’t hurt people like you, but they can and do hurt people like me.”

So in the spirit of “hey bro, not cool,” here’s the deal:

Truth in Advertising

My contact is the regional sales lead for a new company. His job is to open doors, get meetings, develop relationships, and eventually to make sales.

For a person in that position, LinkedIn is a marketing tool. This guy is an experienced professional. He knows what he’s doing here, at least at an unconscious level. His list – like this blog post – is signaling to his community about what kind of a person he is and what he expects of the rest of us.

No matter the title, this is not about any notional “sons.” Instead, this is how he expects the men in his professional network to act.

The message is that my contact is a certain kind of businessman. He has a firm handshake, looks you in the eye, and is an experienced negotiator. You know he’ll close the deal and then you can both go home to your wives and kids.

Under the hood, though, the inverse message is also clear: We’re supposed to think less of men who don’t make strong eye contact, who wear nontraditional clothing, who (for whatever reason) don’t marry the girl or work the grill. Those people aren’t up to this guy’s standards.

He also keeps a clean hankie on hand in case one of the ladies is overcome with emotion. Good dude, right?

So What’s The problem?

Lists like this rise from a nostalgia for a time when gender and relationship roles were supposedly simpler. Men were men, women were women, and there was a well defined and correct way to fill either role.

Of course, those roles were radically asymmetric when it came to the workplace. Women were (and are) paid less, under-promoted, subject to outright abuse and subtle neglect, and generally treated like second class human beings. We’re going to be grappling with the fallout of those antique and chauvinist ideas for the rest of our lives.

Even worse: The idea that there is a single correct way to experience gender is incredibly toxic. Our society is slowly and haltingly coming to grips with the diversity of human experience – and lists like this, while superficially innocuous, are a step backwards.

Things weren’t actually simpler back then. Rather, people who didn’t fit into the dominant patterns either adopted an ill-fitting persona at great emotional and mental cost, or else they were excluded, ostracized, and subject to violence and even questionable medical procedures aimed at correcting them because they were somehow wrong at being themselves.

The problem with pushing this as some kind of misty eyed ideal in a professional / business context like LinkedIn should be apparent on the face of it.

The Inappropriate Thing

The thoughtful reader might go back, look at the list, and say that this blog post is a bit of an uptight overreaction. There is no particular word or phrase that stands out as inappropriately crossing some clear line. That’s how this sort of signaling works. The inappropriate stuff emerges gradually as we establish some spaces (the grill, the locker room, perhaps the board room or the industry event) as masculine and therefore subject to different rules.

This is the gateway to some really nasty stuff. Once we start down this road, we’re just a fraternity induction and an MBA away from the @GSElevator twitter feed.

More on that at the end of the post, but first allow me to share another example:

The All Male Conference

A few years back, I was invited to speak to a meeting of the US sales and engineering teams of an early stage technology company. I was already a customer, and my team was in the middle of a proof of concept evaluation of their new product.

When I arrived, I was struck by the massive gender imbalance. It was an all male event with at least 50 men in attendance. The two women at the conference center were the receptionist who gave me my badge, and the person who served the coffee.

The thing had a weird and macho vibe: When the national sales lead finished his presentation, his last slide was a picture of some bird, perhaps a duck, that he had run over on the way to the meeting. The room laughed, some uncomfortably. He left the grisly picture lingering on the projector while he took questions.

After my talk, I had an opportunity to meet with the executive team. I asked about the total lack of women at the event, and they laughed and said that they had just been talking about that.

LOL, weird, right?

I pushed, and they told me that they were working on it, but had to take it slow. Dead-duck guy? He brought in amazing sales numbers. He apparently saw any effort at diversity as diluting his talented team with charity cases and low performers. They didn’t want to alienate him, so they had to tread carefully.

I cancelled the proof of concept and insisted that they go only through me for future communications I don’t know what other tricks dead-duck-guy had on offer, but I knew I didn’t want him talking to my team.

This particular story has a happy ending. The company did some soul searching and then hired a global head of diversity, who was the most forceful and intersectional person you’ll ever meet. They made a sustained effort to fix their biased and unbalanced team. Dead-duck guy may still be there, but I certainly never saw him around again.

Along the way they discovered something really important: Their product had a much larger potential audience than they had realized.

The company had been blind to that larger market because so many potential customers had been unwilling to take an initial meeting with dead-duck-guy and his team. They never showed up as qualified prospects.

Let me say that again: The macho, hyper-masculine approach of their best sales guy was alienating half of their target audience. The people who didn’t want to deal with him didn’t call back and explain themselves. They just moved on.

Maybe they took Rule 23 to heart.

Conference Season

I mentioned that the inappropriate thing usually shows up later. Let’s talk about that:

Conference season is starting. That means lots of mix-and-mingle events. The goal is relationship building. There will be coffees, breakfasts, lunch-and-learns, bar nights, and boozy steak dinners. There will be private presentations back at the Air B&B, invitations to travel and speak at the national convention, and so on.

As these invitations ramp up, my experience is that they move more and more into masculine spaces that exclude women. Once there, we always tend to see a bit more of the old “locker room” banter. It’s a ratchet that goes in only one direction.

This happens gradually to avoid anybody getting all weird and uptight when the enticements on offer depart from what we talk about in mixed company.

I mean seriously, you know why they put all the big industry events in Las Vegas, right? It’s not for the child care facilities, I can tell you that much.

Why Speak Up?

Real talk: I’m pretty nervous about posting this article.

I know that my contact will see it – I plan to send him a link (seems only fair). I know at least a few other people who will think I’m talking about them. I feel social pressure against rocking the boat and upsetting anybody.

I felt exactly the same way before I spoke up at that all-male team meeting. It’s super stressful go to somebody’s party and tell them that they are doing it wrong. I was the invited speaker. I checked all the boxes of gender, race, and personal presentation to be welcome, and I still very nearly censored myself.

The thing that pushed me over the edge, then and now, is that this is the same pressure that keeps women silent in the face of uncounted insults and indignities. It gave me just the briefest glimpse of what it’s like to be on the unpleasant side of social pressure to conform, stay quiet, and obey. That brief glimpse was enough to motivate me to speak up then, and it continues to do so today.

As the saying goes: If you see something, say something.

I’m saying something.

Not cool, bro.

The network is slow: Part 1

Let me start off by agreeing that yes, the network is slow.

I’ve moved a fair amount of data over the years. Even when it’s only a terabyte or two, the network always seems uncomfortably slow. We never seem to get the performance we sketched out on the whiteboard, so the data transfer takes way longer than expected. The conversation quickly turns to the question of blame, and the blame falls on the network.

No disagreement there. Allow me to repeat: Yes, the network is slow.

This post is the first in a series where I will share a few simple tools and techniques to unpack and quantify that slowness and get things moving. Sometimes, hear me out, it’s not the entire network that’s slow – it’s that damn USB disk you have plugged into your laptop, connected over the guest wi-fi at Panera, and sprayed across half a continent by a helpful corporate VPN.

True story.

My point here is not to show you one crazy old trick that will let you move petabytes at line-rate. Rather, I’m hoping to inspire curiosity. Slow networks are made out of fast and slow pieces. If you can identify and remove the slowest link, that slow connection might spring to life.

This post is about old-school, low-level Unix/Linux admin stuff. There is nothing new or novel here. In fact, I’m sure that it’s been written a bunch of times before. I have tried to strike a balance to make an accessible read for the average command line user, while acknowledging a few of the more subtle complexities for the pros in the audience.

Spoiler alert: I’m not even going to get to the network in this post. This first one is entirely consumed with slinging data around inside my laptop.

Endless zeroes

When you get deep enough into the guts of Linux, everything winds up looking like a file. Wander into directories like /dev or /proc, and you will find files that have some truly weird and wonderful properties. The two special files I’m interested in today both live in the directory /dev. They are named “null” and “zero”.

/dev/null is the garbage disposal of Linux. It silently absorbs whatever is written to it, and never gives anything back. You can’t even read from it!

energon:~ cdwan$ echo "hello world" > /dev/null 
energon:~ cdwan$ more /dev/null
/dev/null is not a regular file (use -f to see it)

/dev/zero is the opposite. It emits an endless stream of binary zeroes. It screams endlessly, but only when you are listening.

If you want your computer to spin its wheels for a bit, you can connect the two files together like this:

energon:~ cdwan$ cat /dev/zero > /dev/null

This does a whole lot of nothing, creating and throwing away zeroes just as fast as one of the processors on my laptop can do it. Below, you can see that my “cat” process is taking up 99.7% of a CPU – which makes it the busiest thing on my system this morning.

Which, for me, raises the question: How fast am I throwing away data?

Writing nothing to nowhere

If my laptop, or any other Linux machine, is going to be involved in a data transfer, then the maximum rate at which I can pass data across the CPU matters a lot. My ‘cat’ process above looks pretty efficient from the outside, with that 99.7% CPU utilization, but I find myself curious to know exactly how fast that useless, repetitive data is flowing down the drain.

For this we need to introduce a very old tool indeed: ‘dd’.

When I was an undergraduate, I worked with a team in university IT responsible for data backups. We used dd, along with a few other low level tools, to write byte-level images of disks to tape. dd is a simple tool – it takes data from an input (specified with “if=”) and sends it to an output (specified with “of=”).

The command below reads data from /dev/zero and sends it to /dev/null, just like my “cat” example above. I’ve set it up to write a little over a million 1kb blocks, which works out to exactly a gigabyte of zeroes. On my laptop, that takes about 2 seconds, for a throughput of something like half a GB/sec.

energon:~ cdwan$ dd if=/dev/zero  of=/dev/null bs=1024 count=1048576
1073741824 bytes transferred in 2.135181 secs (502880950 bytes/sec)

The same command, run on the cloud server hosting this website, finishes in a little under one second.

[ec2-user@ip-172-30-1-114 ~]$ dd if=/dev/zero  of=/dev/null bs=1024 count=1048576
1073741824 bytes (1.1 GB) copied, 0.979381 s, 1.1 GB/s

Some of this difference can be attributed to CPU clock speed. My laptop runs at 1.8GHz, while the cloud server runs at 2.4GHz. There are also differences in the speed of the system memory. There may be interference from other tasks taking up time on each machine. Finally, the system architecture has layers of cache and acceleration tuned for various purposes.

My point here is not to optimize the velocity of wasted CPU cycles, but to inspire a bit of curiosity. While premature optimization is always a risk – I will happily take a couple of factors of two in performance by thinking through the problem ahead of time.

As an aside, you can find out tons of useful stuff about your Linux machine by poking around in the /proc directory. Look, but don’t touch.

[ec2-user@ip-172-30-1-114 ~]$ more /proc/cpuinfo | grep GHz
model name : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz

Reading and writing files

So now we’ve got a way to measure the highest speed at which a single process on a single CPU might be able to fling data. The next step is to ask questions about actual files. Instead of throwing away all those zeroes, let’s catch them in a file instead:

energon:~ cdwan$ dd if=/dev/zero  of=one_gig  bs=1024 count=1048576
1073741824 bytes transferred in 7.431081 secs (144493358 bytes/sec)

energon:~ cdwan$ ls -lh one_gig
-rw-r--r--  1 cdwan  staff   1.0G Mar  5 08:57 one_gig

Notice that it took almost four times as long to write those zeroes to an actual file instead of hurling them into /dev/null.

The performance when reading the file lands right in the middle of the two measurements:

energon:~ cdwan$ dd if=one_gig of=/dev/null bs=1024 count=1048576
1073741824 bytes transferred in 4.222885 secs (254267367 bytes/sec)

At a gut level, this makes sense. It kinda-sorta ought-to take longer to write something down than to read it back. The caches involved in both reading and writing mean we may see different results if we re-run these commands over and over. Personally, I love interrogating the behavior of a system to see if I can predict and understand the way that performance changes based on my understanding of the architecture.

I know, you were hoping to just move data around at speed over this terribly slow network. Here I am prattling on about caches and CPUs and RAM and so on.

As I said above, my point here is not to provide answers but to provoke questions. Agreed that the network is slow – but perhaps there is some part of the network that is most to blame.

I keep talking about that USB disk. There’s a reason – those things are incredibly slow: Here are the numbers for reading that same 1GB file from a thumb drive:

energon:STORE N GO cdwan$ dd if=one_gig_on_usb of=/dev/null bs=1024 count=1048576
1073741824 bytes transferred in 75.596891 secs (14203518 bytes/sec)

That’s enough for one post. In the next installment, I will show a few examples of one of my all time favorite tools: iperf.