Navigate / search

Azure WTF#

This is my contribution to the 2015 F# Advent Calendar.

I recently set myself a challenge. Build a web-service in F#, and make it run on Azure. Nothing too fancy, just serve up some resources as Json, and allow me to add, update and delete them.

How hard could that be?

Well, as it turns out, hard.

I scavenged body parts from a few blog posts and talks. Beginning with Tamizhvendan Sembiyan’s excellent and detailed post on building web-services with

Suave is a really neat web framework that lets you work with HTTP Requests and Responses using idiomatic functional code. Which, Tomas Petricek points out is how it should be. The web is essentially a functional model. It is ideally a stateless function that accepts a Requests as input, and produces a Responses as output.

Tamizhvendan showed some nice handlers for serialising to and from Json, and for mapping urls to functions. I’m not going to go into all the details that he has covered. Seriously, read his post.

I do want to mention one place where I took a slightly different approach to his. He represented a resource as a record containing functions for the typical CRUD methods. The signatures are nice, like the ‘a -> ‘a option for Update, signifying that the resource in question may not exist to be updated.

type RestResource<'a> = {
  GetAll : unit -> 'a seq
  Create : 'a -> 'a
  Update : 'a -> 'a option
  Delete : int -> unit

I’m not a fan of the ‘record of functions’ approach to implementing interfaces. But the point of the post was to demo Suave and it did that. I was up and running and quickly had a basic webservice working locally. That word, ‘locally’ is about to get really significant.

The First Problem
In line with Tamizhvendan’s post, I built my first web service as a console app, ran it locally and it worked. Then, I tried to get it running on Azure.

I’m sure this is possible, but every example I could find for F#/Suave on Azure involved using Scripts instead of an App. Chief among them this one by Scott Hanselman.

Ok, I’ll just convert my Console app to a script for now.

How hard could that be?

Well, as it turns out, hard.

Hello Scripts
Before I actually try to convert my existing code, let’s see if I can get a simple hello world web service running as a script on an Azure Website. Let’s start with Scott’s post, mentioned above.

Ok, there’s a lot of new ground for me to cover here. Suave, Fake, Kudu, Paket, heck even F# scripting was pretty new to me, I don’t think I’d ever used it for anything that required more than one fsx file.

I have deployed a C# ASP.Net MVC app to Azure from github and each push triggers a new deploy. Setting that up was simplicity itself. Getting an F# script to deploy and run on Azure was starting to look like an ordeal by comparison.

And keep in mind, a Script isn’t even what I want. That’s a temporary compromise because I couldn’t get a console app to work. I thought F# was supposed to make life simpler.

The Second Problem
Now, what I should have done is just deploy Scott’s code directly from his repo, or forked his code and deployed it. But where’s the fun in that. No, I tried to recreate his work from scratch, to really understand it. And that cost me in the short term. But I think it was worth it.

I’ll do a separate post on the things I managed to get wrong just trying to recreate and deploy his simple example. Basically if you have any extra files in your repo like .sln or .fsproj files, you’ll get some confusing errors. And if you’re missing any, like oh say the .deployment file then weird stuff happens or doesn’t happen.

Sometimes nothing happens and it’s not your fault. You try to deploy and everything just hangs. I’ve noticed that three separate nights from about 11pm until after 1am (Irish Time). And then it works itself out. I need to investigate that more, but it’s really annoying when you’re already unsure of the code you’re deploying.

Now that I had something running, it was time to expand it to host my full Web Service.

How hard could that be?

Well, as it turns out, hard.

The Third Problem
Remember, I have to build my full Web Service as a script. The thing is, F# scripts are fine for building small scripts. If you’re building an app, with types and modules, you really want to be using the compiler and making real apps and libraries.

The reason for this is that when scripts reference each other, you can end up loading the same file more than once, because it’s a dependency of more than one other script. When this happens FSharp loads the file into two different namespaces. See the FSI_0006 in the following example.

[Loading /Users/richarddalton/Projects/EloRateApi/site/Model.fsx]
namespace FSI_0006.EloRateApi.Model

What this means is that the same ‘Type’ can exist in memory as two different incompatible types.

To put it bluntly, you can’t build complex apps as F# scripts. And on the off chance that you manage to, you probably shouldn’t. But, for now I wanted to get something running, so scripts would have to do.

I have good news, and bad news. The good news is I did manage to build a Web Service that serves up Json and lets me Create, Update and Delete resources.

The bad news is it only has two resources, Players and Games, and because it’s fsx files I don’t see it as a viable approach for a real web service with any more resources or any more logic.

So, I need to get back to the drawing board and either get console apps running on Azure, or put the bulk of the code in a Library, and use a single, simple Script to launch the Web Service. Which means I have to learn a little more about Fake so that I can build the Library and Deploy it.

The Code
The code in full is available here

Forgive me, it doesn’t yet have a nice welcome readme, but it will. The code also isn’t quite the way I like it. Partly because of the limitations of scripts, and mostly because I haven’t finished simplifying it yet.

The code does run locally, just clone the repo and run webserver.fsx using fsharp interactive. The Azure deployment is still a little hit and miss.

If you read Scott’s post mentioned above, you’ll understand how the deployment works, but for now, let me mention one or two pieces of interesting code.

webserver.fsx defines the routes and runs the server. It’s the starting point for the project.

restful.fsx is basically a stolen and slightly modified version of Tamizhvendan’s code, which allows me to be a little more flexible in the methods I define for a resource, because I don’t use an ‘interface’ as such.

model.fsx contains my types representing Players, Games and the State of the service. It also relies on a script called CQAgent that I’ve written and which I reference directly from Github using Paket.

CQAgent uses F#’s MailboxProcesser to provide a generic agent for managing State, and querying it using query methods, or modifying it using commands. It’s worth a blog post in it’s own right, and I’m indebted to Tomas Petricek for getting me over the line in writing it.

Here we see the power of suave. We compose handlers for incoming requests, so if a request is a HTTP.DELETE to the “games” resource, it will fail to match on any of the Player routes, and it will fail on all but the last game route.

let PlayerRoutes = choose [
                        Get "players" (Player.GetAll model)
                        GetById "players" (Player.GetItem model)
                        Post "players" (Player.Create model)
                        Delete "players" (Player.DeleteItem model)
                        Put "players" (Player.Update model)
                        PutById "players" (Player.UpdateById model)

let GameRoutes = choose [
                        Get "games" (Player.GetAll model)
                        GetById "games" (Player.GetItem model)
                        Post "games" (Player.Create model)
                        Delete "games" (Player.DeleteItem model)

let routes = choose [
              (NOT_FOUND "Huh?")

Each of those choices aren’t just a way of matching, they are calls to functions in restful.fsx that
will get the work done. If we look at the Delete function in restful.fsx, it’s a bit intimidating.

let Delete resourceName delete =
    let resourcePath = "/" + resourceName
    let resourceIdPath =
        new PrintfFormat<(int -> string),unit,string,string,int>(resourcePath + "/%d")

    let deleteResourceById id =
        delete id

    DELETE >>= pathScan resourceIdPath deleteResourceById

Don’t worry about how the function works, let’s just look at the signature. It takes a string resourceName. As you
can see we pass “players” or “games” to that. The third argument ‘delete’ is actually a function that will do the work of deleting the resource.

It’s signature isn’t annotated but it is ‘a -> unit, which is exactly the signature Tamizhvendan used.

We pass the following to that argument

(Player.DeleteItem model)

Let’s look at the Player module.

let public DeleteItem (model: CQAgent<State>) id =
    model.Command (fun (s: State) ->
        ((), { s with Players = removePlayer id s.Players }))

We can see now that what actually happens is we partially apply the DeleteItem method, passing it just the model,
producing an int -> unit function that matches the required signature of a Delete method.

Json etc.

All of the serializing to and from Json, and the parsing of urls us handled by restful.fsx. All of the handling of system state is handled by CQAgent.fsx.

The Ghost of Christmas Future

Where to from here?

This has been a really frustrating project because with Suave and with some of the other code, it really feels like F# is a great language for building web services. But the tooling and infrastructure either isn’t there or my ambition is writing cheques my knowledge can’t cash.

I’ll stick with it. But I’m really hoping I get a comment saying, I’m doing it all wrong, and it’s really simple when you know how.

Pride and Prejudice

My last post was in support of functional programmers. This post tries to get to the nub of the ‘Arrogance’ issue. Where does it come from? What sustains it? Why Functional Programmers in Particular?

There are real problems. There are enthusiasts who are hostile to newcomers. There are newcomers who are hostile to functional programming. I’m going to ignore these extremes. I’m interested in people who engage in good faith, but don’t manage to connect.

There’s seems to be a kind of dual reality happening. Newcomers and the enthusiasts can have different perspectives of the same interaction. Some of this confusion plays out in a world of anonymous forums and 140 character tweets. We are not designed to deal with complex issues through these new media. Things escalate, fast.

Here are a few of the more common disagreements.

Functional Programmers moralise about side effects and purity.

Functional programmers are cast as the vegetarians of programming. Side effects and mutable variables are off the menu. Some go full vegan and learn Haskell.

To a newcomer this can seem a bit puritanical. “Thou Shalt Not” lectures don’t sit well with programmers.

This is where we touch on the difference between easy and simple. Easy is a close cousin of familiar. Adding a lock around mutable state is easy if that’s what you know how to do.

Simplicity is the absence of complexity. Learning new skills to eliminate mutable state is hard, but the resulting solution can be simpler. As familiarity increases, difficulty decreases and you get simple AND easy.

If you learn a new paradigm, you are by definition ok with the unfamiliar. You have a different perspective than some of the newcomers that you are trying to convince.

That difference may lead newcomers to see you as puritanical, and may lead you to see newcomers as lazy. In that situation, everybody loses.

Functional Programmers Dismiss OO, which has served us well.

Every functional programming enthusiast I know has a background in Object Oriented programming. Many will admit to having struggled to do OO well.

The deeper they get into functional programming the simpler it seems by comparison. That is one of it’s big attractions.

It’s not uncommon to equate OO with the familiar complex solutions I describe above. And to think of functional programming as the unfamiliar but simpler alternative.

I’ve been guilty of thinking in those terms. It’s understandable. You’ve struggled with OO. You find a paradigm that seems to address your specific issues. It’s all too easy to think you’ve found ‘a better paradigm’.

Presenting functional as a better alternative to OO doesn’t sit well with people who know and like OO. It’s hostile, and it’s wrong in any case. Just as it’s wrong for newcomers to claim that OO is better or more intuitive.

I think of OO and functional as the longitude and latitude of software development. When you learn both you don’t explore two different paths, you explore an entire area mapped out by two axes.

Functional programmers obsess over clever unreadable code

Some programmers enjoy exercises like ‘Code Golf’. They cram entire algorithms into a single tweet. These recreational programming challenges are popular with with both functional and imperative/OO programmers.

Functional code does tend to be terse. If it’s unfamiliar to you, it can look a lot like a code golf exercise. The gleeful reactions of programmers to such code don’t help.

Idiomatic functional code is at a level of abstraction above imperative code. It tends to avoid loops, conditionals and temporary variables. It has unfamiliar concepts like pattern matching, sum types and partial application.

It can be a shock to see actual abstraction done well. Most of what we call abstraction is just indirection.

Object Oriented code done well should also offer abstraction over imperative code. That’s a story for another day.

Yes, code golf happens, there’s nothing wrong with it. But, not all programmers cackling over some terse code are playing code golf. They are not all smug and obsessed with their own cleverness.

Sometimes they’re writing good production code. They’re delighted to see abstraction working like it should. Every programmer, regardless of paradigm should celebrate that.

I just don’t get it / Functional Programmers are smug because they “get it”

Most programmers with a bit of experience can pick up a new language in a few days. It’ll take longer to understand it’s peculiarities. But you should see progress in days, if not hours.

We can do this because languages have common roots. This won’t happen if you try to learn a new paradigm.

Learning a new paradigm is akin to learning programming all over again. You’ll feel lost. For some, that is evidence that functional programming is “not intuitive”.

There is debate about whether OO is ‘intuitive’ because it models the real world? Or whether it’s just ‘familiar’ after years as the dominant paradigm?

For me, the lost feeling is the best part. I’m back to the days when new knowledge came in BIG jumps. When unconnected ideas joined together and unlocked completely new ideas.

It can be hard to articulate “how” you learned functional programming. It’s easy to give newcomers the impression that it is something that you just “get”. Try not to do that. There are eureka moments, but they come after working though that lost feeling.

Functional Programmers ramble on about maths and lambda calculus

I missed two years of school which left me with some big gaps in my maths knowledge.

Trust me when I say, you don’t need maths to grasp functional programming. But, functional programming can be a gateway drug to maths.

If functional programmers seem enthusiastic about maths it’s because maths is cool. When mathematical ideas start to make sense, it’s the same rush that we get from programming.

To work in a new paradigm and see it not only make sense, but tie back to mathematical underpinnings. It’s exciting.

Getting Involved

If you are interested in learning functional programming, the one piece of advice I would offer above all others is make contact with real people. Stay away from forums.

Watch talks, read blogs, follow people on twitter, see who they follow in turn. Get a sense of the personalities, the sense of humour etc.

People who are enthusiastic about functional programming will be glad to help anyone who shows a genuine interest. They will also be able to suggest others who can also help you.

We all tend to mirror the people we interact with. If you are enthusiastic and genuine in your enquiries you will find people who are enthusiastic an genuine in their efforts to help.

Some Functional Programmers Are Arrogant

On Sunday, I was preparing to fly home after Progressive .Net Tutorials in London. The conference had seen a big chunk of F# content. I’d spent a couple of days chatting with amazing people. Recharging the enthusiasm battery pack.

Apart from some heavy breathing on the mic for the first few minutes of my talk, and my inexplicably saying immutable when I meant mutable (multiple times), I was ok with how my talk went, feedback had been good.

The F# sessions were well attended, spirits were high, and to top it all, it emerged that Don Syme was to be awarded the prestigious Silver Medal by the Royal Academy of Engineering, for his work on C# Generics and the creation of F#.

It was a great week.

Then, a tweet.

Not this again. Functional Programmers are arrogant. Functional programmers drive newcomers away from functional programming.

It reminded me of Bill Hicks’ story about turning on CNN and seeing WAR, FAMINE, DEATH then going outside and hearing birds chirping. Where is all this stuff happening? Where are these arrogant functional programmers attacking people eager to learn, driving them away.

It reminded me of Bill Hicks’ story about turning on CNN and seeing WAR, FAMINE, DEATH then going outside and hearing birds chirping.

I’ve been programming since 1984 and professionally since 1996. So, yes, I’m a noob compared to Bob Martin, but dammit I’ve been around a while and nothing I’ve experienced in programming comes close to Functional Programming for sheer fun and a supportive community.

We started the Functional Kats monthly meetup in Dublin and specifically kept it non language specific to make it as welcoming as possible, and to allow users of different languages to learn from each other. Sceptical of Functional Programming? Then come along and use C#, Java or Javascript, or whatever. All welcome here.

Here’s the thing about people who are enthusiastic about functional programming. Most of them are just so relieved to find someone showing a genuine interest that they will bend over backwards to help you. To build you up. To make YOU feel smart.

I know this because I was on the receiving end of this generosity from the moment I showed an interest. This is why it’s frankly baffling when this charge of arrogance is levelled. Why functional programming? Of all the areas of software development, why is the arrogance of functional programmers a thing?

If it’s just a comment on the internet, a random tweet with no specific details, like this one

Then I just ignore it. I really don’t have the time or energy to deal with all the people that I think are wrong on the internet.

But this wasn’t some random tweet. This was Robert C. Martin. This was someone with a following, a reputation. This was someone speaking with authority. Worse than that, this was someone with a foot at least partially inside the world of FP. He has given talks on Clojure and even (I hope jokingly) suggested that we should all switch to Clojure and call programming languages done.

Worse than that, this was someone who has been on the receiving end of exactly these kinds of remarks. Craftsmanship is elitist. TDD’ers are arrogant. And not even random unsubstantiated claims. When Ted Neward (rightly in my opinion) raised questions about the link between Craftsmanship and the treatment of Heather Arthur, Bob took to his blog to defend Craftsmanship and swat back Ted.

At least Ted cited a specific incident. At least he called out specific bad behaviour, and it was bad.

Here’s the thing about people who are enthusiastic about functional programming. Most of them are just so relieved to find someone showing a genuine interest that they will bend over backwards to help you.

Bob issued a general tweet saying that arrogance is a “significant barrier to entry”. And I couldn’t believe it. He was legitimising random nonspecific comments about arrogance, with a random nonspecific comment about arrogance. (Yay..Recursion!)

Bob is better than this I thought. Bob should be better than this. Bob has to know that this tweet can be called on any time someone wants to lazily accuse functional programmers of being especially arrogant, without giving an example. More arrogant than OO programmers, more arrogant than DBA’s or DevOps advocates. More arrogant than C++ programmers or Agile Coaches.

Bob didn’t say some programmers are arrogant. He said some “functional” programmers are. He didn’t say some software craftsmen (or women) are arrogant. Actually he said they’re a great bunch helpful welcoming people. The Manifesto says they are.

Uncle Bob has jumped on the “Functional Programming/Arrogance” bandwagon, rather than use his considerable knowledge of computing history, and his considerable reputation within the industry to say, hold on, let’s all take a second and see if there really is a special arrogance problem in this particular field, or is this just received wisdom that people repeat without question.

Bob knows that arrogance and ego has been a feature of virtually every discussion in programming, from getting rid of GOTO to Static vs Dynamic typing. From how or whether to TDD, to how or whether to automate deployments. From Waterfall to Agile. From Pragmatism to Craftsmanship to Just Ship It. Hell the Javascript community almost imploded into an argument about semi-colons.

So, THAT is why I tweeted a reply

And that is why Bob asked me to elaborate, and that is why this post exists.

Is there arrogance in Functional Programming? Sure there is. Why? Because it’s programming.

Fred Brooks nailed it in the year I was born.

We work “only slightly removed from pure thought-stuff”

We build “castles in the air, from air, creating by the exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures….”

Of course all of us are arrogant some of the time, and some are arrogant all of the time. Of course. How could it be otherwise?

Google ‘Functional Programming Arrogant’. You’ll see some examples. Nothing really bad I have to say. I’d even posit that a lot of people dismissing FP as a fad or a bandwagon, or a bad idea without ever looking into it are showing as much or more arrogance.

Google ‘My Little Pony Arrogant’ there are far more results.

I’ll write another (much shorter) post about why I think this arrogance issue really exists, and what we can do about it, but tweets like that one from Uncle Bob are not part of the solution. They are part of the problem.

That’s for another day.

Fun with Charts and the REPL

PUBLIC SERVICE ANNOUNCEMENT: This post uses WinForms to display charts. It’s fine, but might pose some cross platform issues. You might like to consider XPlot as an alternative.

Apart from immutable by default, expressions instead of statements, easier parallel code, less noise, higher order functions, expressive type systems and borderline magic type inference what has functional programming ever done for us?

My new favourite part of the whole story is the REPL, and the freedom it (and the features above) gives us to experiment with code.

Power up an FSharp script file and use NuGet to get FSharp.Charting and FSharp.Collections.ParallelSeq. Once you’ve installed the packages, load them in the script file, and open the necessary namespaces.

#load "packages/FSharp.Charting.0.90.10/FSharp.Charting.fsx"
#r "packages/FSharp.Collections.ParallelSeq.1.0.2/lib/net40/FSharp.Collections.ParallelSeq.dll"

open System
open System.IO
open FSharp.Charting
open FSharp.Collections.ParallelSeq

Now we’re ready to go. We’re going to throw various numbers of dice and then chart the results. Let’s focus first on throwing one dice.

let random = System.Random()
let randomDice = Seq.initInfinite(fun _ -> random.Next(1,7))

This is an infinite sequence of individual dice rolls. For Monopoly we’d roll two dice at a time, for Yahtzee we’d roll 5. So our next job is to roll a number of dice at the same time. We can use Seq.take for this.

let throwAtATime n = Seq.initInfinite(fun _ -> (Seq.take n randomDice) |> Array.ofSeq)

We’ll be adding the ability to save dice rolls to file and load them again, so here are a few helper functions that’ll make things a little easier later.

let NumsToText (nums: int[]) =  nums |> (fun d -> d.ToString())
let Join (separator: string) (array: string[]) = String.Join(separator, array)
let TextToNums (text: string) =  text.Split(',') |> Int32.Parse

NumsToText converts the throws (integers) into an array of strings. Join wraps the String.Join function to allow us to partially apply it (and use it with pipelining). TextToNums splits a comma separated string of throws (read from a file) and converts it back into an array of ints. I’ve also wrapped the File.WriteAllLines and File.ReadLines so that we can use them in pipelined code.

let WriteToFile file lines = File.WriteAllLines(file, lines)

let ReadFromFile file =
    |> TextToNums

Whether we generate throws randomly or load them from a file, we have them in the same format, a Sequence of Arrays of integers. Lets write a function that takes that data structure and plots it as a chart. Here’s a function to plot the throws using FSharp Charting. Note that we use a ParallelSeq to speed things up a little.

// Plot To Screen
let Plot (throws: seq<int[]>) =
    |> Array.sum
    |> PSeq.countBy id
    |> PSeq.sortBy fst
    |> Chart.Line

We sum the dice in a given throw. Then we group by the total, sort by total and Chart the results.

As expected the result is a bell-curve, although this little experiment shows how easy it is to visualise data in situations where you might not necessarily know the pattern in advance. We’ll see some more charting shortly, but lets quickly look at how the same data structure can be persisted to a file.

// Save To File
let Save file (throws: seq<int[]>): Unit =
    |> NumsToText
    |> (Join ",")
    |> Seq.toArray
    |> WriteToFile file

And that’s it, we now have a suite of function that will allow us to do some cool things with very little effort. I love being able to use a simple data structure and a few helper functions and then use the REPL to “play” with the data. It allows us to answer questions about the data more quickly, but more importantly than that, the experimentation suggests questions we might not have otherwise thought to ask. Here are a few examples of getting stuff done in the REPL in just a few lines of code.

//Generate 1000000 random throws of 6 dice and save to file
throwAtATime 6
|> Seq.take 1000000
|> Save @"C:\Temp\dice.txt"

// Generate 1000000 random throws of 6 dice and plot a graph
|> Seq.take 1000000
|> Plot

// Read From file and plot a graph
ReadFromFile @"C:\Temp\dice.txt"
|> Plot

// Read From file and save to another file (No, I don't know why you would either).
ReadFromFile @"C:\Temp\dice.txt"
|> Save @"C:\Temp\dice2.txt"

As you play with data, questions come to mind. Like, can we visually see what more dice does to the bell curve? Of course we can.

let threeDice = throwAtATime3 |> Seq.take 100000
let tenDice = throwAtATime10 |> Seq.take 100000
Chart.Combine [Plot threeDice; Plot tenDice]

Are you pondering what I’m pondering? Could we just use to generate a whole slew of samples and plot them all together? Let’s see.

// Generate samples for 1 to 10 dice and plot them all 
|> (fun n -> throwAtATime |> Seq.take 1000000) 
|> Plot |> Chart.Combine 

Here we’ve seen a simple use of FSharp Charting. We’ve covered reading from and writing to a file. We’ve also seen how a couple of helper functions and paying attention to function signatues allow for some very readable code.

Minimize mental overhead for code readers

This week at our FunctionalKats meetup in Dublin, tackled the Luhn checksum algorithm.

This is a simple programming task, the challenge was to try and make the code readable.

I rattled off some code that worked, but I wasn’t at all happy with it. It implemented the problem as described. Partitioning the numbers into two groups and dealing with each group in turn.

let Luhn s =
    let double x = x*2

    let sumDigits = function
        | 18 -> 9
        | n when n > 9 -> n%9
        | n -> n

    let odds, evens =
        |> (string >> int)
        |> Array.ofSeq
        |> Array.rev
        |> Array.partitioni (fun i -> i%2 = 0)

    let oddSum = Array.sum odds
    let evenSum =
        |> double
        |> sumDigits
        |> Array.sum
    (oddSum + evenSum) % 10 = 0

The Array.partitioni function doesn’t exist, I had to write that too.

module Array =
    let partitioni f a =
        let first, second =
            |> Array.mapi (fun i x -> f i, x)
            |> Array.partition (fun (i, _) -> i)
        ( snd first, snd second)

I didn’t like the idea of partitioning the numbers. That may match the problem description but I think we can do better. I like functions that take in something and transforms it, perhaps through many transformations, but without splitting or branching the logic.

If a function splits the data and does separate transformations, there’s mental overhead involved. Every variable in a function is an extra concept for the reader to understand.

My second attempt tried to rectify this

let Luhn s =
    let double x = x*2

    let sumDigits = function
        | 18 -> 9
        | n when n > 9 -> n%9
        | n -> n

    let mapOdd = id
    let mapEven = (double >> sumDigits)
    let mapNumber i = if i%2=0 then mapOdd else mapEven

    let total =
        |> (string >> int)
        |> Array.ofSeq
        |> Array.rev
        |> Array.mapi mapNumber
        |> Array.sum

    total % 10 = 0

Instead of partitioning the data we map the entire array. The function used in the mapping is different for odd and even positions. The mapNumber function handles the toggling of the two functions.

This is better, but it’s still not right. Those mapOdd and mapEven functions, that IF expression selecting between them. It still feels like two trains of thought. We can do better.

let sumDigits n = Math.DivRem(n, 10) ||> (+)
let isMultipleOf x y = y % x = 0

let Luhn: string->bool = (string >> int) 
    >> Seq.rev
    >> Seq.map2 (*) (Seq.cycle (seq[1;2])) 
    >> sumDigits
    >> Seq.sum
    >> isMultipleOf 10

Now we’re getting much closer to the ideal. The function accepts a string, produces a bool. It gets from string to bool through series of simple transformations. No IF expressions, no splitting of the numbers into groups, no toggling of functions.

The trick lies in generating a cycle of 1’s and 2’s. When you multiply a sequence by 1,2,1,2,1,2…. You double every second item in the sequence. With that done, the rest of the function falls into place.

There are a few other subtle improvements. The sumDigits function is simpler thanks to the use of Map.DivRem and ||>.  The Luhn function itself is a composition of transformations. I also created the Seq.cycle and Seq.rev.

module Seq =
    let rec cycle s = seq { yield! s; yield! cycle s}

    let rev s = s |> Array.ofSeq |> Array.rev |> Seq.ofArray

I’m happy with the function as it stands. The ideas I wanted to get across with this post are

  • Don’t let the description of a problem constrain your thinking. It’s often possible to find solutions that are simpler than the problem would imply.
  • Try to reduce the moving parts, the “concepts” in your functions. Aim for functions that read straight through. A reader should have to push as little as possible onto their mental stack.
  • Try to avoid splitting your data and working on different subsections of it. A function should only accept the data it needs to work on. Keeping that data together as you transform it isn’t always possible. If you can achieve it, your code will usually be simpler.

Decorating Immutable Christmas Trees


This post is the fifth in the 2014 F# Advent Calendar series.

As much as I love having a real Christmas tree, I hate buying it and dragging it home. I hate decorating it. I hate taking it down again. Untangling lights, tinsel and baubles.

The problem with Christmas trees is that once you’ve decorated them, there’s no easy way to get back to the bare tree that you bought. You have to get in there and undo all the work.

An immutable tree sounds appealing.

“Sorry kids, we can’t decorate the tree. Don’t shoot the messenger, I don’t make the rules.”

That’s all fine until one of the kids discovers that if you do decorate the tree you get a new tree. “TWO TREES” they scream, then, “THREE TREES” and the whole thing descends into a Dr Seuss story.

How I learned to reject change and love immutability

Not too long ago if you asked me to create a data structure like a binary tree, it would be written in C# and it would look like this.

    class BinaryTree<T> where T : IComparable
        public T Value { get; private set; }
        public BinaryTree Left { get; private set; }
        public BinaryTree Right { get; private set; }

        public BinaryTree(T value)
            Value = value;

        public void Add(T value)
            if(value.CompareTo(Value) == -1)
                if(Left == null)
                    Left = new BinaryTree(value);
                if (Right == null)
                    Right = new BinaryTree(value);

Notice that the Add method doesn’t return anything. It “changes” the tree. You use it like this:

    var t = new BinaryTree<int>(5);

Back then, creating immutable data structures seemed like more trouble than it was worth. I knew how to do it, but it was easier to convince myself that I had the discipline to not need immutability.

I still see that push back against immutability in many of my C# colleagues. They’re not wrong. If you look at software development through a C# or Java lens immutable data structures can seem like a bit of an indulgence.

The Path Of Least Resistance

The thing is, immutability is easy. The problems I had were down to the tools and the path of least resistance that they encouraged.

If you asked me to build a binary tree today it would be immutable unless there was a good reason for it not to be. A big part of that change is down to language choice and the switch to Functional Programming.

Here is the same data structure and functionality as the C# code above, but this is F#. There isn’t much extraneous code here. We are immutable by default.

module BinaryTree

type BinaryTree<'a> =
    | Tree of 'a * BinaryTree<'a> * BinaryTree<'a>
    | Empty

let rec add value tree =
    match tree with
    | Empty -> Tree(value, Empty, Empty)
    | Tree (x, left, right) when value < x ->
        Tree(x, add value left, right)
    | Tree (x, left, right) ->
        Tree(x, left, add value right)

With this implementation we can use our binary tree as follows

    let t = Empty
    let u = BinaryTree.add 5 t
    let v = BinaryTree.add 3 u

And t, u and v will each represent different trees.

The second argument of the add method is the tree to add to, so we can pipe the output of one call into another, as follows:

    let t = Empty
            |> BinaryTree.add 3
            |> BinaryTree.add 4
            |> BinaryTree.add 2

Immutable is Easy

Lets see how you add something to an immutable tree. The result of adding a value to a tree is always a new tree. That’s important so keep it in mind, we are all about creating new trees here.

If we have an empty tree and we want to add a number like 5 then it’s easy, the new tree has one node containing the value 5 and no sub-trees.

    | Empty -> Tree(value, Empty, Empty)

If we add a value to an existing tree, we have two possibilities. The new value is less than the root of the existing tree, or it isn’t. In both cases we create a new tree, but what will it look like?

In both cases the new and old tree will have the same root value. Remember the new value gets added to either the left or right sub-tree.

If the new value is less than the existing value, we add it to the left sub-tree. This means the right sub-tree will be the same as that of the original tree.

    | Tree (x, left, right) when value < x ->
        Tree(x, add value left, right)

Look at that. Create a new tree with the same value as the old tree, the same right sub-tree, and with the new value added on the left sub-tree.

Of course that’s a recursive call so it will compare the new value with whatever values exist down the left sub-tree.

The only remaining scenario is a new value that’s equal to or greater than the value in the existing value. As you might expect it’s the mirror of the previous case.

    | Tree (x, left, right) ->
        Tree(x, left, add value right)

As you can see with a handful of lines of code we not only get immutable binary trees but we get trees that share common nodes. The new tree doesn’t need to copy the unchanged sub-tree of the original, it can just point to it, safe in the knowledge that it is immutable.

Expressions vs Statements

Look again at the Add method in the F# example. All we have is a 1:1 mapping of conditions to expressions. The expressions evaluate to the new Trees.

Compare this with the nested if…else statements in the C# example.

The imperative (statement style) code describes how to change an existing tree. It uses a few nested questions. This kind of code requires you to keep your wits about you. How did I get here? What answers to what questions brought me to this line?

The functional (expression style) code is much simpler (in my opinion). There are three conditions that map 1:1 to expressions. Each expression produces a new tree.

We don’t write code that describes how to construct the new tree, or change the old one. We just construct the new Tree and we’re done.

Gifts Under The Tree

One aspect of Christmas Trees that I do like is the gifts. Here to finish are a few random snippets, questions, and ideas that relate to the F# code above.

Folding into a Tree

We can use the add method to create a BinaryTree by folding a collection, using an Empty tree as the starting point.

    let nums = [5; 8; 3; 5; 3; 7; 4; 8; 4; 7; 89; 4;]
    let t = List.fold(fun acc n->BinaryTree.add n acc) Empty nums

As an exercise, reverse the arguments in the definition of the Add method so it looks like this:

let rec add tree value =
    match tree with
    | Empty -> Tree(value, Empty, Empty)
    | Tree (x, left, right) when value < x ->
        Tree(x, add left value, right)
    | Tree (x, left, right) ->
        Tree(x, left, add right value)

With that done, how can we simplify the List.fold above?

What impact does that have on this earlier code:

    let t = Empty
            |> BinaryTree.add 3
            |> BinaryTree.add 4
            |> BinaryTree.add 2

Proper Decorations

What kind of tree has numbers or even strings? Let’s get some Tinsel, Lights and Baubles on there.

type Decoration =
    | Bauble
    | Lights
    | Tinsel

let decorations =
    [Tinsel; Lights; Bauble; Bauble; Tinsel; Bauble; Lights; Lights; Tinsel; Bauble; Bauble; Bauble]

let t = List.fold(fun acc n->BinaryTree.add n acc) Empty decorations

That will work, but the tree won’t look good. It segregates the different decorations into their own sections of the tree.

How could you create a tree that distributes the quantities of baubles, tinsel and lights evenly around the tree?

Removing A Node

Removing a node requires a little more thought than adding. Try and write an FSharp method to remove an item from a Tree. Here’s one approach.

Now try to write a remove method in the mutable C# BinaryTree. Which did you find easier? Was there any difference at all? Have you more confidence in one or the other?

Traversing The Tree

You can use a sequence expression to traverse a tree and output the results as sequence.

let rec inorder tree =
    seq {
      match tree with
          | Tree(x, left, right) ->
               yield! inorder left
               yield x
               yield! inorder right
          | Empty -> ()

Note that calling inorder on the left and right sub-trees will produce sequences of their own. We use yield! to elevate those into the sequence we’re creating.


How would you you change the add method so that it throws away duplicate values rather than adding them to the tree? Go on, it’s easy.

This post is the fifth in the 2014 F# Advent Calendar series.

Curried vs Uncurried Functions

Over a year ago I explained Memoization in this post. If you’ve read that, you should be comfortable with the following.

open System.Collections.Generic

let negate x =
    printfn "Negating..."

let cache f =
    let cache = Dictionary<_, _>()

    fun x ->
        if cache.ContainsKey(x) then
            let result = f x
            cache.[x] <- result

let fastNegate = cache negate

We’re pretending that ‘negate’ is an expensive function that we want to avoid calling unnecessarily, so we use the ‘cache’ function to wrap ‘negate’ with some caching behaviour. Negate takes one argument. What happens when we try to cache a function that takes more than one argument, like ‘add x y’?

On first glance it appears to work.

let add x y = 
    printfn "Adding two args"
    x + y

let fastAdd = cache add

val add : x:int -> y:int -> int
val fastAdd : (int -> int -> int)

But if you try to use it the results are, disappointing.

> fastAdd 1 5;;
Adding two args
val it : int = 6

> fastAdd 1 5;;
Adding two args
val it : int = 6

The underlying ‘add’ function is called every time. We can check to see if anything is added to our cache with a ‘printfn’.

let cache f =
    let cache = Dictionary<_, _>()

    fun x ->
        if cache.ContainsKey(x) then
            let result = f x
            cache.[x] <- result
            printfn "Added Item to cache, Items now in Cache: %d" cache.Count

> fastAdd 1 2;;
Added Item to cache, Items now in Cache: 1
Adding two args
val it : int = 3

> fastAdd 1 2;;
Adding two args
val it : int = 3

> fastAdd 2 1;;
Added Item to cache, Items now in Cache: 2
Adding two args
val it : int = 3

That’s odd. The first time we call the function, something is added to the cache. The second time, using the same values, nothing is added to the cache. When we try different arguments another item is added to the cache. This suggests the cache is being used.

Remember that functions in FSharp are curried, which means that a function which takes two arguments (like add) is translated (curried) into a series of functions which each take a single argument.

So, when we try to cache

add x y

what we actually cache is the partially applied function

add x

We can test this. As long as we keep the same value for x, the fastAdd function will use the cached version. As soon as we change x, we get a new partially applied function that is different to the one in the cache. We can see this in the example above. It is the change in x from 1 to 2 that triggered the addition of a new item to the cache.

So, we’re not actually caching our *expensive* add function at all. We’re just caching the partial application of it.

We could rewrite our ‘add’ function so that it takes one argument, a tuple.

let add (x, y) =
    x + y

let fastAdd = cache add

> fastAdd (1, 2);;
Added Item to cache, Items now in Cache: 1
val it : int = 3

> fastAdd (1, 2);;
val it : int = 3

> fastAdd (1, 3);;
Added Item to cache, Items now in Cache: 2
val it : int = 4

We’ve ‘uncurried’ the function. Now everything works. But it’s still no good. We’ve duplicated our add function, and remember this all relates to functions that are a lot more complicated than addition.

We could wrap the original curried function so the logic isn’t duplicated. We’re really just putting an adapter around it to change it’s signature.

let add x y = 
    printfn "Adding two args"
    x + y

let uncurriedAdd (x, y) =
    add x y 

let fastAdd = cache uncurriedAdd

val add : x:int -> y:int -> int
val uncurriedAdd : x:int * y:int -> int
val fastAdd : (int * int -> int)

> fastAdd (1, 3);;
Adding two args
Added Item to cache, Items now in Cache: 1
val it : int = 4

> fastAdd (1, 3);;
val it : int = 4

That’s still rubbish though. The signature of ‘fastAdd’ is different to ‘add’, so it isn’t an alternative. If we think in terms of higher order functions, ‘add’ and ‘fastAdd’ have different types. We can’t pass ‘fastAdd’ to a function that is expecting ‘add’ and expect things to just work.

We could write a ‘cache2’ function that follows the same pattern as ‘cache’ but works with functions of two arguments. That will work but it’s a shame not to use the working cache function that we already have.

Feel free to stop reading and figure out a solution. What follows is one way of doing it.

The solutions above weren’t good enough, but we were on the right track. Uncurrying the ‘add’ function to make it compatible with our cache is fine, although we shouldn’t have to write a specific function each time we need to uncurry a function. Through the magic of higher order functions we can write the following.

let uncurryTwoToOne f (x, y) = f x y

That allows us to write the following.

let uncurriedAdd = uncurryTwoToOne add

That saves us doing the uncurrying individually for each function. It still leaves us with a function that isn’t compatible with the original signature of Add. So, caching it is still no good. Remember the ‘cache’ function doesn’t change the signature, it takes a function that accepts one argument and it wraps it to add caching, but it maintains the same signature. That way, the cached version can be used anywhere the original is used.

It looks like we need to wrap the cached version of ‘add’ in yet another function to convert the signature back to curried form. Another higher order function to the rescue.

let curryOneToTwo f x y = f (x, y)

And now we finally have a working solution.

let fastAdd = add |> uncurryTwoToOne |> cache |> curryOneToTwo

> fastAdd 1 3;;
Adding two args
Added Item to cache, Items now in Cache: 1
val it : int = 4

> fastAdd 1 3;;
val it : int = 4

Note that in the ‘fastAdd’ function we are not piping the result of ‘add’ through the chain of functions, the actual ‘add’ function itself is the value. Each function in the chain is passed a function which it wraps before passing it on. Like building up the layers of an onion.

It seems like we should be able to make ‘fastAdd’ into a general purpose higher order function too.

let curried2 f = f |> uncurryTwoToOne |> cache |> curryOneToTwo

Putting it all together. We get the following

open System.Collections.Generic

let uncurryTwoToOne f (x, y) = f x y

let curryOneToTwo f x y = f (x, y)

let cache f =
    let cache = Dictionary<_, _>()

    fun x ->
        if cache.ContainsKey(x) then
            let result = f x
            cache.[x] <- result
            printfn "Added Item to cache, Items now in Cache: %d" cache.Count |> ignore

let cache2 f = f |> uncurryTwoToOne |> cache |> curryOneToTwo

let add x y = 
    printfn "Adding two args"
    x + y

let fastAdd = cache2 add

> fastAdd 1 3;;
Adding two args
Added Item to cache, Items now in Cache: 1
val it : int = 4

> fastAdd 1 3;;
val it : int = 4

> fastAdd 1 4;;
Adding two args
Added Item to cache, Items now in Cache: 2
val it : int = 5

> fastAdd 1 4;;
val it : int = 5

Aim for DRY, but be willing to fall short.

I’m not going to rehash the reasons why “Don’t repeat yourself” (DRY) is a good thing, I’ll assume you know why duplicating code is bad.

Strictly speaking DRY isn’t really about repeating code, it’s more about representing the same “Information” or “Knowledge” in different places. For that to happen the code doesn’t actually have to be the same.

Conversely, duplicated code doesn’t always represent duplicated knowledge or information, completely different concepts can sometimes be represented by virtually identical code.

Mathias Verraes recently wrote a really good blog post on this very issue, it’s well worth a look.

This post is about aiming for DRY, but deliberately coming up short.

I’ve been adding some NUnit tests to an existing system recently and I kind of liked the way they turned out, despite the fact that I duplicated some code.

The following is heavily modified code, I can’t show the original, but it gets the point across. Please don’t read into anything you see about credentials or security, this code is one step away from Widgets and Wotzits, it’s NOT REAL CODE.

    class Describe_Club_Membership
        public void A_User_Can_Join_The_Club()
            UsingCredentials("appName", "appKey");
            var userId = GetUserId("");



        public void A_User_Can_Leave_The_Club()
            UsingCredentials("appName", "appKey");
            var userId = GetUserId("");



The entire contents of the tests are written using functions that exist only for the purpose of making the tests readable. In a sense these few functions define a Domain Specific Language. Code within any function (but particularly within a test) should always be written at the same level of Abstraction, would should never have code like the following in a test.

    class Describe_Club_Membership
        public void A_User_Can_Join_The_Club()
            var credentialsFactory = new CredentialsFactory("Membership");
            var credentials = credentialsFactory.CreateDefaultCredentials("appName", "appKey");

            var registrationRepo = new RegistrationsRepository(credentials);
            var userId = registrationRepo.GetUserByEmail("").FirstOrDefault().Id;

            var binding = new BasicHttpBinding();
            var membershipEndPoint = new EndpointAddress(ConfigurationManager.AppSettings["..."]);
            var membershipService = new MembershipServiceClient(binding, membershipEndPoint);

            ... etc...   

You get the idea, everything that needs to be done, is in the test, from pulling information out of Repositories, to reading Configuration Files, to configuring services endpoints to actually calling services, and on and on.

I don’t know about you, but I prefer the first code sample to the second.

Those functions that make up the Domain Specific Language in the first tests, have to go somewhere, and as you can see from the second example, some of them involve a little bit of work. Look again at the first code sample, you’ll see that I have a function called ‘UsingCredentials’ but it doesn’t return credentials. If it did I could pass them into the subsequent functions that need credentials, but in my DSL it’s enough to state the credentials I want to use. Passing the credentials and all the other dependencies around would unnecessarily clutter up the tests.

So, clearly the credentials are being saved somewhere by the ‘UsingCredentials’ function and then subsequent functions can use them.

    class Describe_Club_Membership
        private Credentials _credentials;

        public void A_User_Can_Join_The_Club()
            UsingCredentials("appName", "appKey");
            var userId = GetUserId("");



        // Test Fixture DSL Commands
        private void UsingCredentials(string appName, string appKey)
            _credentials = CredentialsHelper.GetCredentials(appName, appKey);            


The UsingCredentials function is actually just a one liner, the heavy lifting is done elsewhere in a helper class. The job of getting credentials may be needed by lots of TestFixtures so I want that logic in a shared component, not duplicated. We’re heading in the right direction of DRY.

So, what then is the point of the UsingCredentials function we see here?

Put simply, I like the way it makes the tests read. I could have used this one liner, and others like it in the test, but a test made up of a series of these kinds of calls is more cluttered and it gives the impression that these helper classes may be significant to the test. I don’t want to give that impression.

I want to declutter the test so that only the logic being tested remains. Wrapping this and other one liners in tiny DSL style commands sends a message that the details of how the helpers are called is not as important than the logic being tested.

But what then if I want to use the same Helper from other Test Fixtures, aren’t I going to have to duplicate the little functions that wrap the one liners?

Yes I am.

This is where I deliberately fall short of DRY.

For one of my fixtures I have 7 of these tiny functions. They vary in complexity but none contain more than one line of code. Code that I do not want to see directly in my tests. In another fixture, I use three of those same wrapper functions, and I’ve duplicated the three that I need.

How could I make use of these DSL functions without duplicating them? One benefit of the wrapper functions is that they can access the credentials or any other member of the fixture without it being passed as a dependency. If I move the wrapper functions out I’m basically back to calling the helper functions and needing to pass all dependencies. In short, the wrapper functions no longer serve any purpose.

I could define the helpers and the credentials in a base class and inherit my test fixtures from that. That is in fact a really nice solution, it’s DRY, the Tests look just as nice and the Test Fixtures don’t even have the tiny wrapper functions in them.

If you wanted to go with the inheritance option I’d be hard pushed to make a case against it. I just don’t like inheritance and I really don’t like it in test suites.

Irrational as it may be I prefer to have these tiny functions so my tests can be as clean as possible, but I’m willing to fall a little short of DRY rather than use inheritance.

If someone is curious when looking at a test and wants to see what a function does or where credentials come from, it’s right there, in the fixture, no climbing an inheritance hierarchy.

As soon as you start using inheritance it sucks you in, you start adding more to the base class, or an extra level of inheritance here and there. It gets messy real fast.

What will happen when not all of the test Fixtures need Credentials? When some subset of fixtures need some other shared member? The Base class starts to become the union of all needs rather than the intersection. And for what? So we can avoid duplicating a handful of one line functions that are unlikely to change. Even if they do change it’s going to be trivial, perhaps a new argument to the helper function.

The key points for me are:

  1. Keep your tests as free of clutter as possible, nothing should remain but the logic being tested, written at a consistent level of abstraction.
  2. If necessary define a tiny Domain Specific Language for your tests as simple one liner functions which call to the real shared code, kept elsewhere.
  3. Keep the DSL functions you need right there in the Test Fixture, duplicate them if necessary. Don’t try to over engineer your DSL using inheritance or any other clever tricks.

The third point about inheritance is maybe up for debate, but I don’t thing the first two points are.

One Less Thing To Worry About

In the closing scene of the movie Forrest Gump, we learn that Lt. Dan helped Forrest invest in some kind of “Fruit” company (Apple Corp.).

“So then I got a call from him, saying we don’t have to worry about money no more. And I said, that’s good! One less thing.”

That quote comes to mind whenever I think about immutability.

Let’s make a list of Int’s and display it.

    var numbers = new List<int> { 1, 2, 3, 4, 5, 6, 7, 8, 9 };

    Result: 123456789

Now, let’s reverse the list. In CSharp lists are mutable so we can do the following:

	var numbers = new List<int> { 1, 2, 3, 4, 5, 6, 7, 8, 9 };

    Result: 987654321

If we just wanted to show the numbers in reverse order, but keep the underlying list in order, this wouldn’t work. The Reverse method screws with our underlying data. The following would be nice.

	var rev = numbers.Reverse();

It’d be great if rev now pointed to a reversed list while numbers was still in order. That would be how an Immutable List would work, the Reverse method wouldn’t change anything, it would create a new List.

The result of the List object does the same, if you Sort, you sort the underlying data, you don’t get a new list. Add or Remove items, you change the underlying list.

There is a solution. There is an ImmutableList, just find Microsoft.Bcl.Immutable in Nuget and use System.Collections.Immutable, and you’re all set.

Except that’s a little problematic too. You lose that nice initialisation syntax that the old mutable list had.

	var numbers = new List<int> { 1, 2, 3, 4, 5, 6, 7, 8, 9 };

Under the hood CSharp relies on mutability to make that magic work.

So, you need to use NuGet and you need to write extra code, and all to stop you accidentally mutating a list. This is never going to be the default choice for CSharp developers.

There is another choice, if we call the Reverse method as follows, we get the behavior we want.

	 rev = numbers.AsEnumerable().Reverse();

Problem solved. Well no. There’s an even bigger problem. Thanks to Jonas Elfström for writing up this post on IEnumerable.

Take a look at this code and see if you can figure out where the result comes from.

    var numbers = new List<int> { 1, 2, 3, 4, 5, 6, 7, 8, 9 };
    var rev = numbers.AsEnumerable().Reverse();

    Result: 123456789

As Enumerable isn’t really giving you immutable behavior. The variable ‘rev’ is still pointing to the same underlying data, just through the lens of a lazy operation. The AsEnumerable().Reverse() won’t happen until we use the results. Which means reversing the underlying numbers will cause them to be reversed back again when we eventually use the IEnumerable view of it.

This is why I would prefer data structures that are immutable by default. You avoid this kind of silliness. I’m sure many will have no problem whatsoever with the example above, but I find it annoying, unintuitive and confusing.

In CSharp I never get to put mutability aside as one less thing to worry about. You have to work really hard to make anything immutable, really immutable.

There is a place for mutable data structures and a place for immutable data structures. It seems logical to me that if one has to be harder work than they other, immutable should be the easier of the two. It should be the default.

The challenge today is overwhelming complexity

Here’s a really great post by Tom Moertel on squeezing every last ounce of performance out of machines back in the day.

It was a time when unrolling a loop to save the few clock cycles or seeing a unique way to use the registers of a chip could take a game from clunky to classic.

Stories of making machines do the impossible are the stuff of legend. The closest we mere mortals came was rejigging our config.sys and autoexec.bat files to free up a little memory. If you’re under 30 even that may sound alien.

Tom’s post is a great read, but there is a real gem of wisdom towards the end, under the heading “To the old days!”.

“The challenge wasn’t overwhelming complexity, as it is today. The challenge was cramming your ideas into machines so slow, so limited that most ideas didn’t fit.”

Turn that quote around. The challenge today isn’t cramming your ideas into machines so slow so limited that most ideas don’t fit. The challenge today is overwhelming complexity.

The challenge today isn’t cramming your ideas into machines so slow so limited that most ideas don’t fit. The challenge today is overwhelming complexity.

I would guess that at this point less than 5% of developers ever need to sacrifice maintainability for performance. I’m probably being generous at that. If you are taking on complexity in order to speed up the execution of a method, or reduce it’s memory footprint, you’re more likely doing harm for no good reason.

But, this post isn’t about code. It’s all too easy to focus on “Clean Code”, have code reviews, refactor relentlessly etc. But, if you deploy that code into an overly complex environment, or using an unnecessarily complex infrastructure, or manage the whole thing using unnecessarily complex ceremony then your code cleaning efforts may be wasted.

It’s interesting how many books are available on how to write “good code”, or how to refactor “bad code”, but there are relatively few if any books about refactoring “bad environments”.

Refactoring should extend to the entire development process. If you can refactor away an unneeded Class, but can’t get rid of an unneeded tool, you’re on the road to ruin. No amount of intention revealing Interfaces or SOLID code will save you.

The phrase “Technical Debt” has been coined to mean the deliberate taking on of subpar “code”, with a view to fixing that problem later. I don’t believe it’s a satisfactory definition.

All too often “Technical Debt” means “We have this one line fix, we’ll come back later and implement it correctly in all of the Controllers, Views and Models that should be affected. I wonder about the logic of treating a change in multiple places as preferable to making a change in one place.

I also wonder whether that one liner change is where time will be lost in maintenance. Even if there is a more “correct” way of fixing an issue, is the code really where your team is losing productive hours?

Or, is it the time needed to set up a development environment? Or figure out how to find a test environment that’s “close enough” to production to be useful. Perhaps you have so many tools that nobody really understands how everything fits together any more.

If there is a “Knack” to getting your code base to build, then clean code is the least of your problems.

When the complexity of the solution exceeds the complexity of the problem, THAT’s technical debt. By “solution” I mean the WHOLE package. Code, Tools, Frameworks, Process, Infrastructure, Documentation, Planning, Review.

Very few of us need to find “clever” solutions any more. The capabilities of machines far exceed almost anything we could possibly want to do. The challenge now is complexity and we should be devoting a lot more of our attention to it.