How much time should you spend automating a routine task?

Today's XKCD really tickles me. "Is It Worth the Time?" is a handy chart showing how much time you can invest in automating any recurring task in order to save time, on balance, over five years. I am an inveterate automator of recurring task, always looking for ways to shave seconds.

On the other hand, I think I'd halve the figures Randy gives in this chart, because many of the routine tasks you automate will change in some significant way in less than five years and require further work. Also, the chart fails to account for the losses in innovation and serendipity you suffer when you over-optimize a routine task so that you effectively can only do it in one highly constrained way.

Finally, there's the opportunity cost of clearing a relatively scarce large block of time to spend on automation, which may be a better bargain than giving the task more time overall, where that time comes out of a pool of more abundant small snips of time.

In other words, a five day block of time given to automating a task might cost more (that is, might crowd out more productive work) than ten half-day blocks of time or 40 one-hour blocks.

Still: this is crack for me.

Is It Worth the Time?


  1. The one major confounding factor(that I’m sure he’s aware of; but it wouldn’t really fit on a handy chart) is that automating things can be(though isn’t always) an interesting and rewarding activity, a bit of novel challenge, a chance to think about the problem, etc.

    Just plowing through it, clickety, clickety, click, again, and again, by contrast, tends to etch at your sanity like a grain of sand under your eyelid.

    When I can get away with it, there is definitely a class of problems that I’ll automate, even in the face of expected time loss, just because writing the script is more interesting than clicking ‘OK’ 150 times.

    A second confounding variable, less important; but the countervailing factor to the ones you mention, is that ‘automated’ can mean ‘highly resistant to stupid once-off mistakes that people make when bored, tired, etc.’

    Across a thousand runs of an automated process, you aren’t likely to see much innovation; but you also aren’t too likely to see ‘places where I missed a spot because my phone rang’.

    1. You also have to take into account automating may help you get out of doing a task you really dislike, no matter how much time you save.  Plus it allows for more time to read BB.

    2.  I came to the comment thread to post just this!  I’ll gladly spend seven hours automating an hour-long mundane task that I have to do five times, just to make things interesting/challenging for myself.

      But there’s also a third confounding variable.  Once I’ve automated the task, I can now run it a lot more than the 5 times I was originally planning.  Maybe the mundane task produces an interesting snapshot of some data, and I needed 5 of these snapshots to make a decision.  Now that I’ve automated the task, I can run it 5000 times.  Or I can modify it, and use it in other contexts.  Sometimes these advantages don’t appear until after you’ve gone ahead and automated it.

      1. Modifying and reusing in other contexts, exactly!

        Once you’ve solved the problem for one task, your time-to-solve for the next similar problem is far smaller. So really you should be totalling the time saved for both tasks. Except, you didn’t know about the second task at the time, so couldn’t, when doing your cost-benefit-analysis.

        AutoHotKey, how I love you thee, you have automated away so many computing annoyances for me.

        Another confounding issue is that time has different values. So automating a process can move the time for that process out of “crunch mode” time and into “development” time. That is; to build a new version of the software I’m working on used to take a day, now it takes one click and does itself while I’m commuting to or from work, costing effectively no time at all.

        By definition, every time I used to roll a new client, I’d be at the deadline for that client. So I have saved myself a day on every deadline!

        And it has the knock-on effect that I can roll out test clients *every single day* on the way home from work, completely redefining the way we do testing.

          1. Playing devil’s advocate: should he be payed less because he has made efforts to become more efficient?

          2. No. But businesses generally approve of slow, stupid workers who take 50 hours per week to get the job done and detest smart, efficient ones who get the same amount of work done in 10 hours. cf. Families where the loser child gets all the parental attention while the functional ones are utterly ignored. Also cf. GW Bush presidency. Some human societies have a strong preference for incompetence.

    3. Minecraft example: in the game, it is absurdly easy to plant a little garden and have all the food you could ever need.  Yet some players delight in designing insanely complex automated factory farms, which produce, at the click of a button, more than anybody could ever, ever use.

      Why?  Because it’s fun!

      1. Harvesting food by hand is tedious when you’re running animal farms, the automation can be a real timesaver. 

  2. Its a never ending problem in software and a classic example is where projects are coded to the future proof by moving much of the implementation into configuration files, often in XML. So you end up with ten times as much XML as code, the now editing the configuration is an specialised field in its own right, its just not as easy because XML was never intended to be used in that way.

    Another one is with plugin frameworks, so that future development only needs to impact the plugins, but before long the framework has to change to accomodate new requirements and then you have to rewrite 300 plugins so that keep working…

  3. Report automation is my specialty at the company I work for. My general philosophy at work is that if there is any task that I have to do manually more than once, I will automate it.

    Automation not only saves time but reduces the chance for error. Plus, as an earlier commenter mentioned, it is also an interesting challenge to automate things.

  4. For people with a horrible memory (like me), sometimes the automation is more about preventing you from being an idiot and missing that one little thing. That can increase the worth of the automation work considerably.

  5. I thought about automation today as I examined the radio astronomy grad students’ lab spectrometer setup to see if it could be improved, as it was designed 20 years ago. I thought about giving them an automated tuning system, but remembered that the professor really wants the students to know how to do it using the manual method. Even though it’s tedious, it’s good practice for later on when they get to use automated frequency control systems. Right? right?

    1.  Yes, and all first-year computer science undergrads should be forced to complete assignments using nothing more than a needle, a magnet and a steady hand…

    2.  When I was learning stats I had to do a number of t tests and ANOVA by hand. It was incredibly tedious, and introduced many opportunities for error, but it sure did help me understand what was actually happening when I click a few buttons in a stats program (until I learnt that the stats program is actually just doing a regression analysis and pulling out stats that I then report as t and F values. Oh well, regression is more useful anyway).

  6. Fifteen minutes a day totals up to one work week per year.  An hour a day on the internet every day equals a month of 8 h days of lost work every calendar year.

    There aren’t many work tasks that take 15 minutes a day.  I can think of a lot of things that take me half an hour twice a month.  It would be worth no more than a couple hours of tinkering.

    The flip side of this is that in many organizations, a process has been automated over and over and over again by different people.  If it had been done once, it would have been a big deal, but there’s not a lot marginal benefit in doing it the twelfth time.

  7. Some other things people haven’t mentioned:
    If you can automate the tedious stuff, you are more likely to be aware enough to pick up on the stuff you need to use your brain for.
    Automating one area can give you the skills to automate analogous areas in the future, thus saving you more time.
    If you are good enough at automation, you may end up getting a job that doesn’t involve so much tedious work. 

  8. I find that the time spent automating a task often equals time spent fully understanding it and then learning the skills to do the automation. Very valuable stuff there, even if the automation itself isn’t so much.

    1. I like to think that is the case, but sometimes the skills are rendered obsolete. There’s a wonderful British movie of a factory automating the production of vacuum tubes in the sixties, just in time for them to get replaced by transistors. The machinery designed for that task didn’t have an equal in semiconductors.

      1. The funny thing is, that factory might be profitable if it were to restart today. There’s a small, but strong market for tubes for things like guitar amplifiers.
        A similar thing happened with denim looms in the US. I’m not sure of the timeline, but some smart people from Japan bought the old (obsolete) denim looms, retooled them, and started producing denim with them again. 

  9. If a thing is going to take me longer than 2 weeks to automate, then it’s a thing that someone who has no automation ability can do.  Forever.  Because paycheck.

    Seriously, if it takes more than 2 weeks to go gold, you need to hire a consultant.  I have shit to do.

    Source: I am a crappy DBA.

  10. Incredible amounts of time are spent by software staff automating things without ever considering, even in a cursory fashion, if there’s any payback at all.  Usually it’s just an excuse to avoid boring things like meeting with customers or testing your code.  Often things get automated to the hilt just in time for them to be scrapped.

    In general the software industry is abysmal at thinking about the ROI for anything.  Entire programming paradigms are thrown out because someone who never heard of the idea of “design trade-offs” notices a single, isolated drawback in it.  After all, it’s more fun to rewrite a bunch of tools than to do actual work.  People obsess over coding efficiencies that won’t matter in a couple of months due to Moore’s Law when their projects have no specs.  People will spend days figuring out how to (illegibly) code things with two fewer characters in an idiom but won’t put a single comment in anything they do because that’s boring.

    The construction industry is much more mature.  When you ask them to build a house they don’t set up a forge in your backyard and start making optimal hand tools.  (Or rather get into a fight about what an optimal tool for all possible jobs would be.) They often even have a good idea of what they’re building!

    1. Right. Construction projects are rarely late and always on budget, right?

      The analog for your house example would be a company that builds websites for businesses. They build the same thing over and over and so the work is very predictable. The software business might make some custom tools just like a construction company might build some custom jigs.

      The thing that kills both is a change once the project is underway. Tell your builder you want another room is about the same as telling your developer team that the application needs some new capability. The software world has responded with agile methodologies. 

  11. There’s plenty of reasons to automate rather than just flat time saved.

    1) Interruptions. If some non-automated task interrupts my train of thought every time I have to do it and automating it means it no longer interrupts my train of thought that’s actually a huge savings in time as getting back into the flow can than anywhere from 10-30 minutes after an interruption.

    2) Automation often prevents errors. Every time I do something manually there’s a chance I’ll make an error. Automation reduces that chance thereby saving more time than the flat difference between manual and automated.

    1. input and output error, doing 10 hrs of work once only to find you  were given incorrect data is frustrating enough to spend the next 10 automating the task…

  12. I’d double it, because automating is much more fun than doing repetitious tasks “by hand”! =)

  13. Another thing to consider is the danger robust automation plays in reducing the robustness of the people using it (and thereby overall robustness).   If you automate a task that is currently done weekly but it will fail once a month and require human intervention, there isn’t  much of a problem (other than it sounds like you did a crappy job on the automation).  

    If you then reduce the failure rate to a 2-3 times a year, you will likely see the time taken to resolve the problem grow a bit as the details have become less familiar to the folks tasked with fixing it.  

    Make your automation even more robust and reduce the failure rate to once every two years, and there is a good chance that no one in the organization has the first clue about how to fix the problem.

    It’s a bit of a paradox, but robust automation can reduce overall robustness.

  14. I just think it’s a pity the chart has blanks, because it was an opportunity for some XKCD-ness.

    Take the 1-Day saved line: The last gray box is just an obvious omission: if you could save one day a week, you would save 260 days in 5 years, so you could spend nearly nine months working on the automation to save time. 

    More interestingly, the next gray box to the left would represent saving one day per day, or essentially taking the one task you do all day and reducing it to zero. For this, you could obviously spend all five years working just on the automation, and at the end of the five years you would snap your fingers, and all five years-worth of work would be done instantly. Then you go off and do fun things for the rest of your life, because then next 50 years-worth of work would also be done.

    Moving further to the left, of course, requires some idea of negative time, or perhaps relativistic speeds.

  15. If only there were some means of automating tasks in such a way that *other people* could also use the automation!  Just think of all the time you could save collectively!

  16. Oh man, I hate routine.  It absolutely destroys my attention span.  Automate, make it smart and self-checking, and fast.

    The xkcd guide is practical, but my loathing of repetition is so scathing that I couldn’t pay attention to it if I wanted to!

  17. When you’re a developer, it’s not just your own time that you’re saving. Is it worth investing a week in shaving 5 seconds off a task performed 5 times a day? Yes, if you have more than 28 users of your software. 

  18. Nobody writes good procedure documents, nor keeps them updated. Even where they are written, nobody reads them, and those few that do, don’t stick to them.
    Automation will often formalize and document the procedure en passant, and ensure that everyone always sticks to that procedure. That’s a big win in itself.

    1. Which is why documentation is often enforced by contractual or legal requirements to be effective.  That way when the procedure stops working there is a paper trail, since the developer may be long gone. Undocumented code is a bottleneck, and everyone has seen the rising young programmer become a roadblock.

  19. Automation also serves as self-documentation, esp if you have the script write a date-stamped log entry (or logfile). If you click through by hand, 6 months hence, you will never remember how you did it or if you missed a step. I run a data analysis group, and the task isn’t done until you’ve checked in a script that reproduces everything from the raw input to the final graphs.

  20. Great and perhaps surprisingly on-topic discussion.

    Another consideration: most jobs entail two kinds of tasks: tasks that feel like the work you’re supposed to do and tasks that feel like interruptions from work.  For example, logging hours in time sheets seems to be taken as an inconvenience and distraction by many people who must do so.  These “disruptive” tasks cause more stress and dissatisfaction than tasks that feel like “real work”.

    So I guess that I’m saying besides the pragmatic concerns already discussed there is also a psychological factor.  The worker might become more efficient overall by automating tasks that make the worker feel distracted and dissatisfied even if the automation entails a net loss of time spent performing that task.

  21. Outside of preventing human error, the primary reason I spend time on automation is so that untrained technicians can complete complicated tasks.  

    Need to review all of the server log files to see if the backups were completed? Click here.  Need to update a config file on every server? Put the file in this directory, an automated script will verify the file is formatted correctly and either email you an error if not or update the config on all of the appropriate servers.

    Need to pull all of the web server apache logs, convert them to our reporting format, archive the old database and post them to the SQL database, generate a few standard reports and post them to the Intranet server? Click here.

    Automation is about more than saving time.

Comments are closed.