Covariance and Contravariance

The concept of covariance and contravariance is fairly straightforward for basic datatypes, but it gets a lot less intuitive when you start dealing with generic collections. I recently, finally, got my head around it.

Here are some simple cases with simple classes in C# (for brevity, I’m leaving out the rules for basic datatypes). A covariant conversion is when you convert from a derived class to a class that it inherits from. It requires no casting and makes logical sense:
string foo = "foo";
object o = foo; //you can assign an string to an object

A contravariant conversion is the opposite, and assignment requires a cast (which could throw an exception if it can’t be converted):
object foo = "foo";
string s = (string)foo;

Enter Collections

Though it’s obvious that you can convert a string into an object, shouldn’t you also be able to do this?
var strings = new List<string>();
List<object> objects = strings;

The short answer is no. Though converting from string to object is covariant, converting from a List of strings to a List of objects is not. A cast won’t compile either.

Want to see why? You can actually try it for yourself with an array, because an array is covariant (considered by some to be a mistake):
string[] strings = new string[2];
strings[0] = "foo";

object[] objects = strings;
objects[1] = new Object(); //things go BOOM.

This throws an OverflowException because you just tried to put an object into an array of strings. So, when Microsoft implemented generic lists, they didn’t allow you to expose yourself to this mistake, and they catch it in the compiler. That’s where covariance gets tricky.

In fact, contravariance gets tricky as well. To steal part of Eric Lippert’s example here, shouldn’t this compile? (Assuming Giraffe inherits from Animal)
void Foo(Giraffe g) {...}
Func<Animal> action1 = Foo;

This code is trying to make the assertion that the Foo method, which accepts a Giraffe as a parameter implements a delegate which accepts an Animal as a parameter. The problem is, if you actually called action1 with an Animal aside from a Giraffe, Foo wouldn’t be able to handle it and would produce an exception.

But, aren’t there other cases that don’t cause problems?

There are, which is why a number of changes were made to C# 4.0.

In our list/array example… what if you never added elements to the list? Wouldn’t the covariant conversion be safe? You’d never be able to add an object to an array of strings. The answer is yes, which is why the IEnumerable interface is marked covariant. This is perfectly valid (same example as above, swapping out List for IEnumerable):
var strings = new IEnumerable<string>();
IEnumerable<object> objects = strings;

IEnumerable is now considered covariant since it’s interface only exposes methods related to reading.

Similarly useful capabilities were added with regard to contravariance (example taken from here):
static string GetString() { return ""; }
Func strFun = GetString;
FuncobjFun = strFun;

The last line used to throw an exception, however this is actually a safe operation – any caller expecting an object to be returned will also be able to handle a string.

Symptom Solving

There are many things to dislike about American politics, but possibly my least favorite is the degree to which we’ve completely given up on solving problems, and spend time debating solutions to the symptoms. Even during an election year, when there’s no scarcity of big talk, our politicians are largely still unwilling to talk about problems – only symptoms.

Some recent examples:

Health Insurance

There is no shortage of debate about paying for healthcare. There is a very serious shortage of debate around why healthcare is so expensive. There is considerable work that could be done to lower the cost of providing health care and all we’re doing in this election year is assuming the cost is fixed, and fighting about how to pay for it.

Paying for Higher Education

Similar to heath care, this is another case where we assume costs are fixed. However what upsets me the most about this discussion, which also made it’s way into the State of the Union Address, is that we continue to frame it as a conversation about Federal Student Loans. It is absolutely mind-boggling, in the wake of the housing crisis, that someone can make the logical argument that inexpensive, low-interest rate loans keeps costs down. Some (non-politicians) are calling it a Higher Education Bubble. Even if you disagree that the ROI on a college degree is decreasing (that is, what students get out of it is not increasing proportionate to the cost) it is undeniable that making money cheaper drives prices up. Giving students access to cheap money makes it easier for institutions to raise prices. This fact also got a nod in the State of the Union, but it’s been happening for decades and it would be blasphemous for a politician to suggest that the government should subsidize less. No one is talking about the cause of high tuition. They’re only trying to solve the symptom.

Income Inequality

The president called this “the defining issue of our time” and history will regard it as the central argument of the Occupy Wall Street movement. On this issue I’ve heard some discussion on the causes, but by and large even this argument degrades into one about taxation. That is, after the income is already unequal, how do we re-equalize it? Unfortunately this problem is actually really hard, and an across-the-board AMT-style tax like was mentioned in the speech is completely unrelated to the root cause. I don’t believe most Americans #1 issue is income inequality, it’s mobility – their ability to advance (we’re all “temporarily embarrassed millionaires“) which would mean that the conversation has almost no connection to the problem.

These are just a few of the most recent hot topics for which there’s no solution in sight… because the problem is not part of the conversation.

"Easy" Problems

In software, you wind up spending most of your time solving easy problems. Problems that have been solved before and, thanks to Google, have solutions available in an instant. Thousands of well-defined, no-risk, completely solvable problems.

Virtually everything having to do with a user interface, if you’ve seen it done before, is “easy.” Gmail has a drag-and-drop attachment feature, so clearly that problem has been solved. Facebook has a live-scrolling ticker of events – piece of cake, just do what they did. Also, anything that has an open source solution is easy. Thank you Solr/Lucene, search is easy. Thank you Hadoop, doing computations on huge datasets is easy. Thank you Ruby on Rails, building complete websites can be done in 15 minutes.

I actually do think all of these things are easy, to the extent that “easy” means you’re going to be able to solve them without needing to hire an expert and you won’t need to dust off your old Algorithms textbook at any point in the process. But there are a few things “easy” doesn’t mean:

Easy != Fast

Just because you know how to do it, that doesn’t mean it can be done quickly. We all know how to mow a lawn. Can you do 10 acres in 5 minutes?

Depending on the problem you’re solving, usually one of two reasons explains why Easy != Fast

1. The problem is clear, but tedious to solve

Building a sign-up form is simple – some text boxes, and a submit button. Of course, what sign-up form would be complete without validation, autocompletion/correction, tooltips, markers for optional/required, anti-bot/spam protection… and each of those things is easy too, but you’ve just got a lot of lawn to mow. (photo from: smashing magazine)

2. The problem is solved by a 3rd party tool, and it’s got it’s own… problems

Solr does, in fact, make it very easy to build a search engine that produces fast, relevant results. Now, bone up on schema.xml, db-data-config.xml, the various query parsers and their associated syntaxes, then review what the best/current libraries are for whatever language you’re using. It’s true, you don’t have to write any algorithms. You’ve also saved yourself time over building this thing from scratch – no question. But, you’re still about to spend a considerable amount of time on this.

Easy != Well Defined

There are still the fundamentals of software engineering at play. The example of the registration form above was also partially an tale of vague requirements.

“We want a sign up form just like has” is the easiest way to spec a feature – you just point to something else and say “do it like that.” Even though that may seem less vague than “we want a sign up form”, unless you truly understand the thing you’re referring to you may have accidentally added more requirements than you meant to. Did you notice the has a live, password strength calculation and doesn’t let you sign up unless the password is strong enough? Did your developer just lose a few hours researching the best plugins/schemes for that, then implementing them, all to build something you didn’t actually need?

Easy == Solved, but what does Solved mean?

Google makes billions of dollars on CPC ads. Paypal makes billions of dollars in processing payments. They each have plenty of competitors doing the same things, so the technologies behind processing CPC ads and sending payments are surely mature, and these are cases of solved problems.

But how much time and effort do these companies spend fighting fraud every day? Google has an Ad Traffic Quality team. Paypal has a Fraud Investigation Team.

Sometimes a solved problem only looks that way because there is a metric ton of ongoing work making it appear that way.

Want to know why Nordstrom has beautiful, consistent photography of all of their products on their shopping pages, with the ability to zoom in on, and see a product at multiple angles? Solved problem: simply unbox every single thing you sell and photograph it the exact same way from the exact same angles. After that little exercise, adding a zoom feature is pretty straightforward.

Piece of Cake

I don’t object to using the word “easy” in software. It’s useful to acknowledge that something has minimal technical risk. But, be careful when it creeps into discussions about scheduling or operations. Time is not measured in units of difficulty.

How to Enjoy American Football

I love football, but I know it’s hard for a lot of people to get into. More than once I’ve heard it described as “just people hitting each other.” The scoring is also a bit strange, and there are a lot of odd rules, so if you want to enjoy football that’s not where you should start.

Think of Football as War

Not in the hyperbolic sense, but in the strategic, battleplan-scheming, territory-grabbing sense. If you watch it with this in mind, the sport is a lot clearer.

Each team has 11 players on the field. One team has the ball. The team with the ball is trying to move into the opponent’s territory, and the opponent is trying to stop them.

The field is 100 yards long. You get 4 tries to move the ball 10 yards forward. If you succeed, you get another 4 tries. If you fail, you give the ball to the other team.

On television, they draw a blue line on the screen to indicate where the play starts (the line of scrimmage). The yellow line is where they have to get to in order to earn another set of 4 tries (tries = ‘downs’). On the screen above “3rd and 5” means “this is my third try, and I have 5 yards to go.”

So, what would you do?

If your job is to move down the field, how would you do it? If you were commanding an army, how would you draw up a scheme to advance on your enemy?

In football, you have two basic options:

  1. Throw the ball.
  2. Run with the ball.

Pretty simple.

If you throw the ball, you can move a lot of yards in a hurry. But, it’s risky. The ball’s in the air. If the defense grabs it, it’s theirs.

If you run the ball it’s comparatively safe, but you may not move very far. If the defense is expecting that, they will tackle you right away.

One thing about football that differs from other sports: those 11 guys on the field are specialists in what they’re doing for each scenario. When a team is playing offense, they have 11 completely different players on the field than they do when they’re on defense.

Wide Receiver #81, Cornerback #24

For every strategy you have on offense, the defense has a counter strategy. For all of your talented, specialized players, the defense has players that specialize in stopping them. You have a few wide receivers – tall, fast guys with great hands, who’s #1 job is to catch passes. They have cornerbacks and safeties – guys who are often even faster than wide receivers, who specialize in sticking on them.

You have to mix it up

It’s not uncommon to think running = boring and passing = exciting. But, strategically, one doesn’t work without the other. If a team just did one or the other, the defense could easily shut them down. If you mix it up, you keep the defense on their toes. A great example was last week, in overtime, when the Steelers defense was sure the Broncos were going to run the ball… and they were wrong:

Though running usually only lets you move forward a few yards, there’s always a chance that something amazing can happen: Marshawn Lynch vs. Saints.

Ok, but Kicking is still Boring

When you don’t move the ball ten yards after your first 3 tries, you have a decision to make. You can try again, but if you fail the other team will get the ball right at the spot. This is key, because if you’re close to your own endzone (the area your opponent is trying to get to) they won’t have very far to go in order to score.

So you have the option to punt. You put a guy out there who will kick the ball to your opponent, but he’s going to do his best to make sure they have to start as far back as possible. A punt can routinely push your opponent back 45 yards.

During a punt, the play on the field is rarely exciting. Punts are interesting strategically – they move the line of scrimmage dramatically, so the other team will have to spend more time and effort to advance the ball back towards you. As with offense and defense, when the punter comes on the field this is yet another set of 11 guys (called ‘special teams’) who specialize in this aspect of the game. As such, there are players who are exceptional punt-blockers or punt-returners who can make a punt exciting.

The game is too slow

The game’s cadence is driven by the fact that the offense is given 4 tries to do anything. They get 40 seconds to meet, decide what they want to do, then run a play. Whether or not they succeed, they stop and get 40 seconds to plan the next one. This is the time when the strategy is formed. Pass or run? Who’s running where? While the offense is deciding what to do, the defense also mulls over how they’re going to line up and stop them. This time itself is also strategic – if the offense can figure out what they want to do very quickly, the defense doesn’t have any time to think. If the offense takes a long time, they are giving that same amount of time to their opponent.

As a viewer that cadence conveniently meshes well with drinking. But, if you’re thinking like the offense and defense, you can also use that time to think about what kind of play the offense is going to run, and see how the defense organizes itself to try and stop it – since they’re guessing just like you are. I think this mindset makes the entire game much more interesting.

Gmail’s new scrollbars subtly prevent me from sending mail.

In the last 2 weeks, I’ve noticed on several occasions I’ve thought I’d sent mail only to find it sitting as a “Draft” in Gmail. I began to think there was a horrible bug in Gmail and was having flashbacks to the days of figuring out why Outlook would leave mail stuck in your outbox.

Today I figured it out. It’s the scrollbars.

When you start replying to the message, assuming you scroll down a little into the body (which I seem to do often on my laptop’s screen, but less often on a desktop screen) the “send” button scrolls off. But, it’s not obvious that you’ve lost the send button, because there is still a top row of buttons. I wind up, subconsciously, clicking the “back arrow” button as the closest thing to “send” or “reply.”

If the scrollbars were the traditional browser-standard ones, it would be obvious to me that I’m in a frame that had scrolled down. For example, it’s obvious inside the message pane itself. As it stands now, I don’t even notice that a row of buttons has scrolled off of the page, since only part of the page is scrolling, and it’s doing so very subtly.

Windows Resource Monitor

Windows has always made it very easy to see what your CPU is doing, and how much memory you’re using, via Task Manager. However, the bottleneck in many cases is disk I/O, and it’s not nearly as obvious. Though you could add additional I/O-related columns in the task manager process list, I/O isn’t summarized in a graphical form. Windows Resource Monitor does it though, and it’s actually a click away once you’re in Task Manager:

If you fire it up, the UI is oriented around the resource you’re looking to monitor – CPU, Memory, Disk, and Network. The Disk panel has graphs of Disk Queue Length, which you’ll recognize if you’ve used Perfmon before as one of the most useful counters for spotting I/O bottlenecks:

After discovering it, I dropped a second drive in my machine to do builds on. There is still a lot of activity on the C: drive that I’ve yet to pick through, but moving the builds to a different drive has made a noticeable difference in the responsiveness of my system while the compiler is running.

NFL Playoffs 2010: Week 2

Wow, what a crappy round of predictions last week. I blew 3 out of 4.

This week I’m picking all upsets except the Patriots over the Jets. So: Ravens, Packers, Seahawks. If the Seahawks upset Bears, it would tee them up for a massacre in the NFC Championship. But, I still hope they can pull it off.

NFL Playoffs 2010: Week 1

Let’s see if I can do better than last year. Shouldn’t be hard.

  • New Orleans over Seattle – I’d love to see this go down differently, and Seattle is a much better team at home, but a team with a losing record has no business being on the field with last year’s Superbowl champions.
  • Indianapolis over the New York Jets – This game as it should be close… but the Jets have lost a lot of momentum, and the Colts have Peyton Manning.
  • Baltimore over Kansas City – Would be great to see Kansas City win, but Baltimore is a tougher team. I really don’t want to see a Ravens-Patriots rematch.
  • Philadelphia over Green Bay – This should be a great game as well, Michael Vick just seems unbeatable right now.

Howto: Copy a Certificate out of IIS and into a Coyote Traffic Management Sever

No one probably wants to read this, but I couldn’t find the full end-to-end walkthrough on the internet, so I figured it should be there. Here are the steps (please suggest a faster way if you know of one!):

1. Export the Certificate using the MMC Snap In Tool (instructions found here but summarized below.

There are a bunch of ways to export a certificate, each of which misses various parts that you’ll need. This makes sure you get the cert chain and the private key:

  1. Start > Run type in “MMC” and click OK
  2. Go into the File Tab > select Add/Remove Snap-in
  3. Click on Certificates and click on Add.
  4. Select Computer Account > Click Next
  5. Select Local Computer > Click Finish
  6. Click OK to close the Add/Remove Snap-in window.
  7. Double click on Certificates (Local Computer) in the center window.
  8. Double click on the Personal folder, and then on Certificates.
  9. Right Click on the Certificate you would like to backup and choose > ALL TASKS > Export
  10. Follow the Certificate Export Wizard to backup your certificate to a .pfx file.
  11. Choose to ‘Yes, export the private key’
  12. Choose to “Include all certificates in certificate path if possible.” (do NOT select the delete Private Key option)
  13. Enter a password you will remember
  14. Choose to save file on a set location
  15. Finish

2. Convert the binary pfx file into a .pem file with openssl (tip here, but reproduced below)
openssl pkcs12 -in publicAndprivate.pfx -out publicAndprivate.pem

3. Strip the password out of your private key
openssl rsa -in publicAndprivate.pem -out private.pem

4. Put the password-less private key into your .pem file
Open your “publicAndprivate.pem” file, and replace the private key section (marked with —–BEGIN RSA PRIVATE KEY—–) with the contents of your private.pem file.

That new file now has your cert, and password-less private key for uploading into a Coyote box. There may be a faster way (openssl may have a different combination of flags) but that did it for me.