Analysis Paralysis: Over-thinking and Knowing Too Much to Just CODE
I read a post on ArsTechnica today called "I know too much to program quickly. What can I do?" that is summary of a StackOverflow question by Zilk, who says:
Lately, I've been noticing that the more experience I gain, the longer it takes me to complete projects, or certain tasks in a project. I'm not going senile yet. It's just that I've seen so many different ways in which things can go wrong. And the potential pitfalls and gotchas that I know about and remember are just getting more and more.
Trivial example: it used to be just "okay, write a file here". Now I'm worrying about permissions, locking, concurrency, atomic operations, indirection/frameworks, different file systems, number of files in a directory, predictable temp file names, the quality of randomness in my PRNG, power shortages in the middle of any operation, an understandable API for what I'm doing, proper documentation, etc etc etc.
This really hit me because THIS IS ME. I was wondering recently if it was age-related, but I'm just not that old to be senile. It's too much experience combined with overthinking. I have more experience than many, but clearly not enough to keep me from suffering from Analysis Paralysis.
I have two side projects I'm doing on vacations and in the evenings when the house is asleep. One is a port of popular iOS application to Windows Phone, the other is a iOS app with a cloud service startup with my buddy Greg. Both projects have had awesome beginnings and then stalled when things just got overwhelming.
I kept starting features, the stalling. I felt like I was thrashing to disk, spending more time swapping ideas around in my head rather than just doing them. I'm still getting lots of things done, in general, I'm productive, but when I code I just thrash.
I'm overthinking stuff. "Write settings to a file" turns into a mess of paranoia around concurrency situations, upgrading settings from previous versions of the app (that don't exist, mind you), and it just snowballs from there. It's not exactly scope creep, but it's a kind of architectural paranoia. I see so many issues and possible bugs that I've learned over the years that could derail a feature that I end up derailing the feature.
The answer, they say, is You Aren't Gonna Need It. "Perfect is the enemy of the good" reminds user Telastyn. These are easy to intellectualize but hard to internalize. User Mouviciel says:
Looks like you are not experienced enough :). The next lesson is: stick to requirements, not more.
I get that, but me, I often need another brain to complement my own.
How I Solve Overthinking
I learned about Agile from James Shore while I was working at Corillian some years ago, but it's Pairing that resonates with me the most. With a good pair, you'll get 3 times the work, not double.
I worked my way through both these startup issues by bringing in another brain. I'm not the best programmer, but I do OK. But somehow we are both better when we pair. I paired with Greg on the iOS and my new friend Jan Hanneman on the other. They are both clearly better coders than I, which is intimidating, but I'm still sure I provide value. What they gave me was a fresh perspective and a focus to say "YAGNI" and just get features done. The ironic part is, if I'm brought in on a project to pair, that's what I bring also.
My wife thinks this is hilarious. It's the old relationship joke where your partner says something for years and years, then one day you rush home from work to share this amazing new "insight" from a stranger...the same insight your partner has been sharing all this time.
Since I work remotely, all my Pair Programming has to happen over video chat and screen sharing. I use Skype, Lync, Join.me, and whatever else works. We take turns working through features in Trello boards, sharing one person's screen, talking and coding, designing and brainstorming, then commiting to Git, syncing, and switching the share.
This seems to work well for sessions as long as 3 hours, but after that, we get pretty wasted. However the feeling of accomplishment when you work through a problem with a partner is also magnified.
Does your coding life get paralyzed? How do YOU work through it?
* Photo courtesy of FOCUS100
Scott Hanselman is a former professor, former Chief Architect in finance, now speaker, consultant, father, diabetic, and Microsoft employee. He is a failed stand-up comic, a cornrower, and a book author.
Also, as a single developer it's just minimum viable feature for me, improving as necessary, and writing tests for the requirements. With tools like resharper if you make a big architectural mistake, it's almost not a problem.
I just saw this by helping the Wife with some chores: what sounded like half an afternoon of work for her alone was suddenly done in half an hour. It may have seemed less efficient - stepping around each other in the kitchen - but the rapid completion of jobs, the extra help with distractions (ie. the Kids), and a dialogue that focused on goals, all helped achieve a better overall result.
It didn't just seem quicker. It WAS quicker. And now we're both free to do other things.
As to the analysis paralysis, I just decided there were more things in the world that I don't know than I do know. No matter what you do you're not going to please everyone, but doing nothing will definitely please no one. Organise, prioritise, execute. Do the best you can, and keep learning so your results keep improving.
It really does seem that experience and the knowledge gained from it, can sometimes come with a negative side-effect...ANALYSIS PARALYSIS.
Thanks again for the very close-to-home post.
I don't have a team. I work alone. This whole article is so *me* it's spooky.
at least now I feel better about it. Thanks, Scott.
Even if we know nothing we are trapped and I think it applies to everything we do and not just coding....
Or maybe you need to write cross platform accessible GUI code that's going to work for both WPF and iOS via Xamarin - but for whatever reason nobodies sure if iOS is mission critical or a nice dream. Start with the one you want, but keep your standards up and abstract. You'll wind up with lots of little methods, and can split the non-OS dependent ones into a nice view model that will work for both with little effort.
For each new row of code I write I find it easier and easier to find a good and (hopefully) readable solution - "good enough, not perfect, but mainainable. People will understand.".
Pair programming is awesome! Awesome if the pairs have about the same level of understanding of programming in general. If too unequal, it is probably a great learning experience for the not-so-experienced, and fun (but very stressful) experience for the more experienced person (that would have solved the problem much faster if not pair programming with the newbie).
Then again, pair programming is probably a really good tool for exposing problems: unbalanced teams, unrealistic timebox (estimates).
You also learn how to actually make things reusable which is, in my eyes, one of the mayor steps towards becoming a great Engineer.
To do the interleaving, I take advantage of the fact that I commute to work by bicycle on quiet country roads, and the 50-odd minutes each way gives me the time AFK that I need to work through the analysis issues enough that when I'm actually back in the office, I already know what I'm going to do for the next stage.
For me, projects move at a slow pace or do not get completed when there is no clear deadline, which is the case with most personal projects. This is a side-effect of optimising productivity and relegating less urgent tasks to the bottom of the to-do list.
At work, I design and move to File.New in a very short time. Then, experience takes over as the project gathers pace. The same experience that seems to be the cause of analysis-paralysis becomes invaluable in keeping the project moving, especially in situations where junior developers get stuck from uncertainty or their own analysis-paralysis.
What has worked for me to lessen analysis-paralysis in the absence of a hard deadline (as in most moonlighting projects) is to have milestones with simple goals; the milestones become deadlines and keep me focused.
Some features may never get revisited and as such will always stay in their simplest form. Others may become core parts of the system, but then if I am doing it correctly, after about three iterations I should have a good abstraction that is easily extensible.
In addition to this, I find regularly discussing the underlying architectural goals of the system with the other engineers on the team means that we all have a basic understanding of when to innovate something new and when to refactor something that already exists so that it can be extended to support new requirements
Knowing what can co wrong is not a bad thing — fear-driven development is.
Your example of writing a file is a very good one. While one shouldn't build an entirely new file system access layer, one should definitely be prepared to handle the case that a file couldn't be written for various reasons. Likely errors should be caught and handled (if possible) to increase robustness, as long as that doesn't get mixed up with crazily defensive programming.
I find myself taking forever to get started in the initial phase of a project. Too much thinking-ahead, remembering best practices (both institutional & learned), architecting for longevity etc.. I think back jealously of my ignorant days when I could just dive in and code from the first minute of a new project.
The sad thing is, common sense says here that the better architected & planned solution will produce a higher quality product. But in reality that polished solution may not get delivered quick enough, or the client isn't so fussed about having all the things we think are important.
Perhaps the answer is to create a solution with just enough seperation of concern that it can be iterated for future mods & scalability, but no more.
1. Set a time limit to completing the next logical piece of work. Never more than 2 hours. Usually 1.5hrs.
2. Write the commit message for the work first. This draws a rock solid finish line. A bit extreme but I find it helps stay focused.
3. Start. Design, write tests, code. Whatever is needed to get to that finish line in time.
I find I veer from the plan a lot, but never too much. I leave technical debt, but never crappy code each time through this exercise.
Most importantly, the given problem is solved.
Then I can decide if the next move is to settle debt or move forward.
This is really true. I started a personal project 6 months ago. I was losing control in that project because of over doing, organizing, what patterns to use? which installer I am going to use? What are the portability options? etc. Then my deadlines got closer and I lost interest. I started thinking to drop the project and run away.
Few weeks back I got the control back by saying this to myself. Don't follow any rules (but basics), patterns, architectures. "Get the feature done, test and ship". I plan to refactor the code that I wrote once after every 2 weeks ( next iteration ). So, that I have a broader perspective of how the code can be re-used or re-factored.
Thanks again for the great article, it motivates me !
With so many languages, frameworks, architectures, design patterns, IDEs, source control repositories etc. it can really drive you batty.
Do I use code first? Entity framework? Test driven development? NodejS? MVC? Scaffolding? Razor? C#? Java? Ruby? Azure? AWS?
And don't even get me started on design patterns (Gang of Four).
+1 for @Steve and @Marius Sometimes you just need to ignore the fear and just code. Get the idea out of your head and then refine it. Iterative development. Short release cycles.
If you over analyze everything you'll never write a line of code.
Same rule applies to Legos. Don't spend all of your time sorting out the pieces by shapes and colors. Just build it!
I experience a sort of terror when, at the moment of setting to work and finding myself before the infinitude of possibilities that present themselves, I have the feeling that everything is permissible to me… Will I then have to lose myself in this abyss of freedom?
I experience a sort of terror when, at the moment of setting to work and finding myself before the infinitude of possibilities that present themselves, I have the feeling that everything is permissible to me… Will I then have to lose myself in this abyss of freedom?
I’m for maximum freedom and minimal pain.
Sometimes you just have to write it. That's it. We are not living in perfect world(maybe it is but we can't understand), and our lives not so complete. So same idea for our codes. It hasn't have to be coded complete too. The Unfortunate Reality. I believe open source is the saver of our imagines/visions. Do not expect well structured companies to create well written softwares.
Before I can know what I don't need, I need to know what's *really* needed! And that's a much bigger problem nowadays. Clients always want something totally simple. Until it turns out, that they want a complete CRM for a few thousand euros.
The other question: Do I need a technical abstraction (ORM, Interfaces, MVVC) or a logical abstraction like: Do I have customers with Address-lines or Organizations that relate to multiple Locations, (shipping- and billing addresses). But if you implement a "Client"- vs. a "Natural Model" then it's like feeding your kids with Chocolate, just because they want it. We all know, there is no such thing like a single Address per Customer. YAGNI or not.
However the other extreme happens with enterprise-clients. The more money they burn the happier they seem to be. If you want them to get their software done, never do what they say, do what you know they want. However, if you want to keep your (well payed) contract as long as possible just follow every brainless guideline you can find. Even accept to design object models collaboratively using Excel!
So funny: A big company I worked for years ago said, they will definitely have only ONE client, everything else was YAGNI. 6 months later they had three. And at the end of the year they had 431 clients! With minimal differences but drastic consequences for the code! Thankfully we did not do what they said. Because using T4 we could generate the Code for as many clients they wanted for all the "silos" at once. SQL, Biztalk, Exchange, Active Directory and Web. Copy, and edit XML, build and done (almost). And even today they wonder who saved their asses. Needless to say that I got fired, so they will never know. ;-) Oh yes: sometimes "you will gonna need it".
1. Make it work
2. Improve it
3. Apply best practices
This is one of the development approach I try to push as much as possible to my students too. Using the KISS (Keep It Simple Stupid) principle as much as possible in the step 1 is crucial. In step 2 you think about improvements, which usually comes with experience or by reading what others do in similar cases/scenarios (if you have no experience). In step 3, if you have time and resources on disposal, you apply best practices by following standards, patterns or some other advanced ways of how to handle many possible scenarios, potential side-effects that may occur, optimizations, etc. etc., but still stuff which are inside the scope of the problem requirements you're trying to find solution for.
A similar outcome can be achieved by setting context and boundaries for the task when working alone. When this is done and one gets stuck in analysis paralysis, one can re-align everything again through the set context/boundaries. For me, I like to do this by walking away from the desk and talking to myself while having a walk, basically stopping and restarting what I'm doing and validating against the context/boundaries set - Pairing with myself?
I feel happy now, because i am not the only programmer feel like paralyze
Although I feel your post title is highlighting the wrong part of the post.
The post is about the value of pair programming. Analysis Paralysis is one of the things that is solved by it.
At my new job there is no peer programming and now I realize how much I took it for granted. Lacking the figures to back it up, but I'm pretty sure I'm not even half as productive as I used to be.
I often know all the pro's and con's of a given approach, probably better than many other people who might come along after - but I worry that they will get hung up on the "cons" aspect of what I've done and not realise there were many other factors at play that they will be unaware of a couple of years down the line.
Then I shake myself down and realise that the quality of what I've done is probably better commented and better reasoned than the majority of code I see written and I try not to fret about it so much!
The curse of being your own worst critic!
This way I can focus on the objective of release as with minimal necessity and over the time all the possibilities.
Aweseome blog post. I am glad that you are talking about such things. Sometimes they help much more than some extremely nifty grifty tech-talk.
If the different tools have their own requirements, it even gets more complicated. If you choose this, you also have to work with that and so on.
What happened since the good old days when you had "a", or at least only a few standard templates and you'd use the time on the real problem, instead of thinking more about HOW you implement a feature?
Thanks for your insights, Scott.
Btw: I had to lough hard when you talked about your wife having told you the solution for some time. This is soooo true and soooo spot on.
I was paralyzed by the idea of being a craftsman, that everything I did needed to be immaculate. What needs to be great is the finished product, not the process.
It's all too easy for me to forget that perfect is the enemy of good, even in this age of agile techniques and rapid deployment.
I know what you mean by potentially overthinking issues when it comes to file access, for example. One of my favorite quotes is "making good decisions come from experience, and experience comes from making bad decisions" so you don't want to purposefully use built-in fuctions/patterns for temp files because you know they will throw when you hit asynchronous ops or heavy load. But having another person with you (in person or virtually) can get you somewhere between your "gold plated" idea and their "minimally viable doneness". 20 minutes of talk (or reading Stack Overflow opinions) can save an hour a lot of times.
* Our team had an awesome Scrum Master and all dev teams had certification-level training in the process. Not a typical set of ingredients for most shops, I know.
My days of starting a new project by setting out coding are done. Now I have to write out what I want my program to do. I follow the old seven habits mantra - begin with the end in mind. Otherwise I would over produce my project and have it end up like Kevin Costner's Water World. A movie that is full of promise yet over produced, over budget and often delayed for a finished product that was underwhelming with reviews and revenue.
During planning a product focus only on quadrant 1 tasks. Implement and release. If demand is meet, update focusing on quadrant 1 and pick one item off of your quadrant 2 list. Implement and release. The goal is to release often and build on the important features.
Sorry if I went a bit off topic.
In software, many things that sound theoretically possible struggle in practice (or are just plain impractical). On the other hand software is very malleable, so it's easy to customize the cut later (making it shorter, larger, etc).
So in software is more of make a rough measure and cut.
(Incidentally there's a good one from Dilbert: http://dilbert.com/strips/comic/2008-03-16/).
Personal software ideas for me always turn into a "Requirements Big Bang" from both the User and Technical perspectives; the pattern of requirements exploding from a simple concept into an entire universe of options.
I look back fondly at my earliest days in college, developing an entire web portal of features in PHP and MySQL extremely quickly. What happened? I learned more about all the perspectives in software development... I no longer had a #2 pencil and eraser to draw with. I have oil-based paints, watercolors, selections of pencils and pens with varying widths, and more with various means of display and an array of lighting techniques with considerations for the art buyer, the photographer, the gallery guest, caretakers, art historians, and on and on.
How do I solve it? Usually I need to physically split my responsibility for either the requirements or development. Either I write the user stories and someone else does development or vice versa. For personal projects, I've begun to write my feature requirements very explicitly using BDD technologies like SpecFlow for C# and either TickSpec or SpecFlow for F#. This has certainly helped, but without a deadline and clear requirements, I still get lost in the "How" behind the code and unit tests.
I can only hope that one day... one day... I will have an open-source project that reflects the experience of my LinkedIn profile. :) Until then, open-source for me is simply about an attempt to share some ideas, some technical approaches, and some code and nothing more.
Seth Godin, author of Linchpin and The Icarus Deception: How High Will You Fly? talks about analysis paralysis throughout those books as he talks about how your life is art, but when the art overhwhelms then you don't move.
The way to solve it is: Ship It! Get 'er done and ship!
You can fix things in iterations.
Also, great quote: "No one (or any software) ever became perfect through inaction."
In other words, failure isn't making a product with bugs. Failure is not making.
I think one needs to write down the issues burning in your brain as you design and code your modules.
Write it all down.
Come back to it and ask those ol' questions: Do I need this? Do I really have control over it? What is the probability of "that thing" actually affecting my code?
For example, with regards to writing a file to disk, my first iteration would be the quickest possible code I could write to get the job done. Then, after that, and all considerations had been taken into account regarding the simplest of requirements, I start adding more, such as concurrency, file-systems, and other support. Write for the common case first, and then address other issues with your remaining time.
My latest epiphany was while watching Iron Man. I was thinking, I really need that Javis to keep me focused on the current problem/feature and stop tangenting off, but how did people do this ages ago? Versioning through Marks. So, as this is my project; what if I create it 2, 3 or 5 times? Other than my OCD fighting to say "can't waste a second of my time", it really doesn't as I've wasted more so far. So the plan of attack...?
- Mark 1: One project app, basic interface of main screen and 10 minutes of hand created data.
- Mark 2: Start again with data in another project. Second time should be quicker to code and fix minor changes.
- Mark 3: Either start again on separating a part into 2 or add a feature if you are happy with the parts.
- Mark 4: Repeat prior Mark.
Also pair programming had a very positive effect on focus, preventing the undesired yak shaving.
What I found an a-ha moment with this post and comments, is what several have mentioned as the fear in doing it wrong or inefficient. This starts the yak shaving. The pair programmer helps to recognize that happening, and also can point things back on track with the help of specs/wireframes, to focus on what really matters.
Consider you had those experiences for a reason and once learnt cannot be unlearned. (Unless you really do go senile).
So your experience tells you that you should add numerous contingencies to code that you once considered simple. You worry that you take too much time writing your code. Did you used to worry that you took too much time debugging faults you had written?
When I start to work on any component, I almost immediately start thinking about how it can be extensible and engineered to handle almost anything that's thrown at it and before you know it, I'm thinking about writing something that's way different and almost certainly overkill for what needs to happen.
I've learned to grab a hold of myself when this starts to happen and question whether it really needs to be as massive as I'm making it out to be. That's when I start thinking about simpler alternatives and before long, I'm back on track and on my way to where I was originally headed.
Chess players have to struggle with this same issue when deciding which move to make within a certain time. A recommendation from one Grandmaster was to start by picking a set of candidate moves, analyse each one in turn, then make a decision. You need to be disciplined so as not to analyse the same move twice, and need to realise your limitations that you have a finite time and must make a decision that might not be perfect. If in doubt, the first move you thought of is often the right one (statistically this was case with strong chess players).
I have a saying that helps me, "Keep Pressing Buttons!"
Just keep pressing buttons. Have a design problem? Add to the design doc by pressing buttons. Distracted? Stop surfing the net and keep pressing buttons. Writers block? Keep pressing buttons. You get the picture.
And I tackled the cross-browser compatibility issue by designing and hand-coding Single Page Application ( SPA ) sites using a single code base for all the browsers, IE 6, 7, 8, 9, 10, and 11, and Firefox, Chrome, Opera, and Safari, and techniques that eliminate differences between browsers. I call this design method Single Master Application Resource Templates, or SMART pages.
The single master application template page is loaded just once, and then resources such as new content are loaded into the template in the background with AJAX. This keeps the size of the initial page that's loaded small, and avoids users waiting for huge pages with many included files to load. And the extra content is loaded as needed, for example when users' click submenu items.
All this is described in the book I wrote, "A Practical Guide to Developing Web 2.0 Rich Internet Applications", published by Amazon.
Iteration will sort out the details.
The subconscious works in the background to clarify the things that nag at you when you're not in the moment.
I am PROUD of the the fact that it takes me longer to solve development problems. I think that means I am finally getting somewhere. It means that I do not blindly jump without thinking. It means that I anticipate problems before they occur. It means, I am a "BETTER PROGRAMMER".
ANALYSIS PARALYSIS IS A GOOD THING!!!!! (IMHO)
I've been using Screenhero when I have to pair with someone working remotely. The app has some of the best latency out there for screensharing, it lets both people be in control (mouse and keyboard) at the same time, and the dev team is always releasing updates. I highly recommend it, even just to try it. They are also very receptive to bug reports. I submitted one a few days ago, and I just received a response from their CEO with a custom build that fixed the bug.
Sorry if this feels like a plug, but I felt this app has worked wonders for pairing remotely.
And then forever more you have to stare at something you know could be better and leads to nothing but technical debt. New work comes along that has to then build on that which inevitably means compromising (/*cough*/ hacking) which just frustrates even more.
I know things can't ever be perfect but so often it comes down to what's the quickest, cheapest way to get the job done which ultimately leads to the above.
I can totally recognize myself in what you write. Someone wrote that it takes him 2 weeks instead of 1, in my case I think it's even worse (5 or 10 to 1 ratio, maybe). This led to so much un-satisfaction from my employers point of view that I now work as a IT architect / administrator (/ and much more Mc Guyver stuff) and that I almost don't code anymore.
The situation was all the worse that I am unable to take a decision and stick to it, as I am always in search for perfection (and indeed, I write very little bugs and what I do tend to work smoothly with very little debugging). What appears to me as the best solution one day is not the best anymore the day after, and I loose time and get depressed about my lack of efficiency.
In fact, as others have said, I don't want to be caught having forgotten something, so I'm condemned to anticipate everything. That's terrifying, depressing, exhausting. I've tried to sell my overall performance is not that bad, considering what I do is problem-free, but managers hate people who can't honour deadlines.
I've almost never pair-programmed, but I loved the little I did. Maybe I should try to find a company that promotes it, I'm sure it's not easy to find here in France, but it's obviously worth trying.
Thank you anyway, everyone.
Ivan (sorry for the alias, I'm a man :o) )
One thing that helps me in situations like this is thinking back to my early undergrad engineering course work and revisiting the principle of Occam's Razor - "All things being equal, the simplest solutions tends to be the right one". :)
In my experience, most of the coding that occurs in applications is around the maintenance. I have heard it estimated that 90% of the time spent in an application is maintenance coding - fixing bugs, adding new features, security fixes.
Clearly, good well constructed code is easier to maintain. When you are thinking that you are spinning wheels - when you eventually do write some code - perhaps you are reducing the ongoing management lifecycle costs. Perhaps spending more time up front - reduces the costs overall for the products lifecycle.
Obviously that's not the approach for every software - it probably gets in the way of hacking up a quick proof of concept. I think you need to consider what you peers think - since that might be a more objective measure.
I think it is the "fault" of the web. You should go and interview some SW engineers from back in the 80s and ask if they had similar issues and I doubt they did.
For every line of code I write today, I can google if there's a way to do it better or differently. And even if I say to myself that it doesn't have to be perfect, just good enough, I don't really live it.
2) Recognize that not every solution needs every bit of complexity
3) Write modular code (i.e. get the behavior allocation correct) so you can refactor piecemeal without affecting the rest of the code.
(3) is the hardest to do, but once you get it, you find that coupling with (2) allows you to start your code simple and add the complexity piecemeal rather than feeling like you have to do the birth of Athena thing.
That is the very first time I frequented your web page and up to now?
I amazed with the analysis you made to create this particular publish extraordinary.
As things developed, I assume I need professional help, i.e. an able statistical programmer to 'pair' with. However, I became an academic loner already.
I work with people that just stick to what they think is the basics and then so many field issues down the line.
Comments are closed.
On a previous project I switched MVVM libraries 3 times because ultimately I wanted something which allowed me to get the app eventually working on iOS and Android using Xamarin. In the end the app was only launched on Windows 8 and Windows Phone 8 and all that time spent on porting to a new MVVM framework was wasted.