In Adrenaline Junkies and Template Zombies, the six principal consultants of The Atlantic Systems Guild present the patterns of behavior they most often observe at the dozens of IT firms they transform each year, around the world. The result is a quick-read guide to identifying nearly ninety typical scenarios, drawing on a combined one-hundred-and-fifty years of project management experience. Project by project, you'll improve the accuracy of your hunches and your ability to act on them. The patterns are presented in an easy-reference format, with names designed to ease communication with your teammates. In just a few words, you can describe what's happening on your project. Citing the patterns of behavior can help you quickly move those above and below you to the next step on your project. Not every pattern will be evident in your organization, and not every pattern is necessarily good or bad. However, you'll find many patterns that will apply to your current and future assignments, even in the most ambiguous circumstances. When you assess your situation and follow your next hunch, you'll have the collective wisdom of six world-class consultants at your side.
This is a poll asking the Stackoverflow community what non-programming books they would recommend to fellow programmers.
Please post only ONE BOOK PER ANSWER.
Please search for your recommendation on this page before posting (there are over NINE PAGES so it is advisable to check them all). Many books have already been suggested and we want to avoid duplicates. If you find your recommendation is already present, vote it up or add some commentary.
Please elaborate on why you think a given book is worth reading from a programmer's perspective.
Note: this article is similar and contains other useful suggestions.
by Dale Carnegie
Although this was first published in 1936, the advice contained within is still as fresh and appropriate as ever. Don't be put off by the name. This isn't some underhand guide to having your way with unsuspecting victims, but rather common sense advice on how to get on with people, how to nurture relationships and make the most of yourself and your fellow man (and woman).
It is well known that technical folk (including programmers) are often thought of as not being terribly 'people oriented' (whether this is a justified stereotype or not is subject of another discussion) and so this book is an invaluable resource for teaching you the finer points of human interaction.
It's warm, heartfelt, sturdy, straightforward and timelessly written. Highly recommended.
by Eliyahu M. Goldratt and Jeff Cox
To elaborate: It is a book on how to approach problems. To identify bottlenecks in your system and work on them. So in short, it isn't a programming book, but shows (in novel format) how to problem solve -- and is thus very valuable to a programmer.
[Update Gishu] It's an eyeopener on how the throughput of your entire system depends on the bottlenecks. Optimizing other stages/operations will not produce any results. Although this is ingrained in developers who have had experience optimizing a scenario in their app; however zooming out to a more higher level and applying this can have profound gains. Beck's XP Book has a dedicated chapter on the Theory Of Constraints. Programmers who move onto Leads/PMs will find this a valuable addition to their toolkit.
This book will inspire anyone to think and be original.
for those more into game development.
This might not be a popular one, but
by Thomas Pynchon
Gravity's Rainbow is my favorite book of all time. I read through the whole thing last summer, and I'm in the process of reading it again. From a writer's point of view, it's pure, beautiful art. However, I recommend it here because it really forces the reader to think and make a lot of mental connections.
Of course, this book has a reputation for being impossible to finish. It's definitely the hardest book I've ever read. Pick it up at your own risk.
Don't Make Me Think by Steve Krug. An essential book about web usability. As Krug says, "Common sense isn't always obvious."
(Hint: Amazon.com has good usability)
Update: This is now part of the library at work. I've gotten about five people to read it so far. 100% positive reviews, predictably.
Here's a strange one for you all to think about.
It's a modern classic that everybody should read, and I'd be very surprised if English or Media Studies students weren't recommended to read it at some time. Reading should not only be informative and educational, but enjoyable as well. If you're not going to read a book for pure fun now and again then you'll only end up frustrated with the books you need to read as a programmer/developer.
This book is a real eye-opener; a book that'll really make you think about your own life, and for a programmer whom spends their day dealing with pure thought-stuff it's a great way to get you thinking on a different track.
It's a very fast paced action story that's excellent to clear your head with. It's a fun book to read (will make you laugh), and the characters are rather tragic (will make you feel more satisfied at work). This is one of those books that is hard to explain the content without fear that people will think you different and odd, but all the same you must tell everyone about (I.e. makes you talk to people).
The Humane Interface by Jef Raskin.
Questioning the Unquestionable and Thinking the Impermissible
by James P. Hogan
God programmer meet God marketing guy, and no it's not Steve Woz and Steve Jobs, but it's the Johns, Carmack and Romero.
Business, gaming and programming all rolled into one. a definitely page turner all the way until the end.
by Fred Brooks
My personal opinion is, apart from programming, in life we need to find a balance, about everything (or keep striving for it). Many times, I have found myself getting too immersed in one aspect of life (frequently programming/work) at the cost of others. Over the years I have learnt to recognize this and act accordingly.
In work, sometimes I have come across pretty difficult people, making it hard to work with them (not just my opinion, but also of other team members). Previously I used to try hard to convince them, make them more helpful, etc. and get frustrated when I don't succeed.
But this book Tigana, by Guy Gavriel Kay helped me understand that sometimes a person can be inherently complex, hard to work with, without he/she helping it. It is a science fiction novel, and it may not be completely appropriate here, but it helped me work better with my team, so I am linking to it here. It helped me become more objective in dealing with people I work with.
Jeffrey K. Liker - The Toyota Way (Amazon link). A good if at times semi-boring read, but loads of information from the company which invented Lean.
Games People Play by Eric Berne.
IMHO it is a very useful aid to understand and deal with office politics (among others).
We think we’re relating to other people – but actually we’re all playing games.
Forty years ago, Games People Play revolutionized our understanding of what really goes on during our most basic social interactions. More than five million copies later, Dr. Eric Berne’s classic is as astonishing–and revealing–as it was on the day it was first published. This anniversary edition features a new introduction by Dr. James R. Allen, president of the International Transactional Analysis Association, and Kurt Vonnegut’s brilliant Life magazine review from 1965. We play games all the time–sexual games, marital games, power games with our bosses, and competitive games with our friends. Detailing status contests like “Martini” (I know a better way), to lethal couples combat like “If It Weren’t For You” and “Uproar,” to flirtation favorites like “The Stocking Game” and “Let’s You and Him Fight,” Dr. Berne exposes the secret ploys and unconscious maneuvers that rule our intimate lives. Explosive when it first appeared, Games People Play is now widely recognized as the most original and influential popular psychology book of our time. It’s as powerful and eye-opening as ever.
As well as the mentioned Gadwell's Tipping Point, Blink is a good choice.
A Brief History of Everything by Ken Wilber.
In the ambitiously titled A Brief History of Everything, Wilber continues his search for the primary patterns that manifest in all realms of existence. Like Hegel in the West and Aurobindo in the East, Wilber is a thinker in the grand systematic tradition, an intellectual adventurer concerned with nothing less than the whole course of evolution, life's ultimate trajectory—in a word, everything. . . . Combining spiritual sensitivity with enormous intellectual understanding and a style of elegance and clarity, A Brief History of Everything is a clarion call for seeing the world as a whole, much at odds with the depressing reductionism of trendy Foucault-derivative academic philosophy.
The Fountainhead by Ayn Rand. Atlas Shrugged is already on this list, but the Fountainhead deals more with craftsmanship and integrity, rather than supply-side economic theory. Definitely worth a read for anyone in a creative field.
Could not put this one down, The Evolution of Cooperation by Robert Axelrod. Its a fascinating read and as game theory books go it's pretty accessible.
Snow Crash By Neal Stephenson
Ender's Game by Orson Scott Card
An analysis of the 12 worst Supreme Court decisions
Great and interesting book about how our liberties are being trodden on by the government. Libertarian viewpoint, but objective.
Burton G. Malkiel
Nothing else will teach you better how to get a handle on your money.
Niccolo Machiavelli's The Prince. After wondering why people acted so strangely at work, this book was the first of many, that taught me why.
How was JPod not posted? It's like a (already posted) Microserfs with internet. It's typical Coupland novel, must read for every techie, geek, webz hipster.
Here are some quotes
"You googled her?" "Of course I did. Didn't you?" I'd somehow forgotten to perform this essential task.
“After a week of intense googling, we’ve started to burn out knowing the answer to everything. God must feel that way all the time. I think people in the year 2020 are going to be nostalgic for the sensation of feeling clueless.”
“It turns out that only twenty percent of human beings have a sense of irony – which means that eighty percent of the world takes everything at face value. I can’t imagine anything worse than that. Okay, maybe I can, but imagine reading the morning newspaper and believing it all to be true on some level.”
Nothing too programmer-specific, but being in the industry that we are, it helps immensly to have a positive mindset depicted in this book.
Note: I had to move this book from my previous answer to here, to comply with the question's specific rule that one post -> one answer
This is probably not going to be popular, but "If liberty means anything at all it means the right to tell people what they do not want to hear."
by Neal Stephenson
It's very dated, but I have yet to find a single book (or essay for that matter) that gives a quasi-outsider's view of an industry that the public is apathetic to understand. The insights and descriptions are spot-on, even though the conditions have dramaticly changed over time.
by Alan Cooper
It's about using the right language to talk about projects - using stories (and personas) instead of 'features' to talk about stuff that needs to be realized. Also a lot of emphasis on interaction design and related activities. Delivering what is needed instead of what is asked for.
Also read these free manifestos
(Note: moved the other book to a separate answer)
Awaken the Giant Within by Anthony Robbins.
This book as about why stock markets are not predictable like casinos and the lottary. It is very readable though querky. It will help you to understand when statistical techneques do not work, why math is not understanding, why project managers can't predict schedules and how they can with less effort.
The book does not go into heavy math but will give you a feal for when the math can and more ofter can not be used.
Author: Nassim Nicholas Taleb. Also wrote 'Fooled By Randomness'
Read it if you work with mathematics, statistics or finance - Or if you have a pension.
Simon Singh's Fermat's Last Enigma is one of the greatest books I have ever read.
This non-programming book has taught me a lot about running after the solution of a problem, no matter how old and complex it is.
by Roger Penrose
Somehow in the line of Godel, Escher, Bach but, I think, easier to read.
Dealing with people you can't stand:
Dealing with People You Can't Stand: How to Bring Out the Best in People at Their Worst (Paperback) ~ Dr. Rick Brinkman (Author), Dr. Rick Kirschner (Author), Dr. Rick Kirschner (Author), Dr. Rick Brinkman (Author)
Sensation & Perception by E. Bruce Goldstein will really pull a lot of software engineers out of their comfort zones. I found it to be fascinating when I started thinking about effective scientific visualization techniques with the user's physiology and psychology in mind. Issues with the user's potential for color blindness, visual acuity, attention span and information processing abilities are just some of the reasons why I keep going back to this book.
If you're a Python developer, you will not get around viewing Monty Python stuff. But to quickly look up a quote you find in any Python doc, I really recommend those:
(as well as part two, they're great; Amazon) and
Reading doesn't give you the great look of a puzzled Michael Palin or the anger of a furious John Cleese, but it still is a worthwhile lecture.
by Tracy Kidder
"The Ultimate History of Video Games" of course!
Why? Because in one book you get history, fun, anecdotes, business decisions, project management, opinions, wonderful quotes, the hardware and the software ... all in all portraying an industry that went through numerous cycles, ups and downs, deaths and reincarnations. But most of all: Steven Kent managed to make this book a very entertaining read, you'll be captivated by each chapter.
This is similar to another question. Here is a link to my answer over there.
Now, Discover Your Strengths is my favorite personal/career development book. It teaches the most successful people become successful by focusing on building on their strengths, rather than covering up weaknesses. This book helps you find out where your strengths lie.
by Tom DeMarco and Timothy Lister
Great background on what managing risk means and lots of good tools for quantifying risks. They discuss a risk estimation tool which uses statistics to produce a pragmatic and reality-based understanding of the effects that risks will have on a given projects completion date and confidence level.
The prologue on "The Ethics of Belief" is not to be missed.
Concise, bare essential and time-less!
The First Quarter : A 25-year History of Video Games. Unabashed old-school video game geekery.
by Gabriel Garcia Marquez
by Steven Levy
Does a great job of outlining some of the eras in computing, from the enviroment that sprung up around the Tech Model Railroad Club at MIT, to the Homebrew Computing in the bay area, to the story of the game companies of the early 80s. Especially the MIT section has wonderful descriptions of hackers at work, doing what they do best (in a wholly non-technical writing style), bumming instructions, making the machine do their bidding, and in the mid-seventies, it describes the self-made community of hardware hackers (including Wozniak), who built their own computers. Hugely entertaining, and a good way to understand where some of these communities originate from (academics, hackers, tinkeres).
If you don't want your job to be outsourced (as have happened to many programmers) then you need to read this book, A Whole New Mind - Why Right-Brianers Will Rule The Future, actualize it, and put it into practice yesterday!
I would recommend: "Code" by Charles Petzold.
It completely opened my eyes on how computers actually work, explained and illustrated clearly. I learned that computers have no inherent understanding of numbers, letters, words or anything like that. These were human concepts and it was up to the computer programmer (at a very low level) to present they patterns of bits from computer memory to something users would find meaningful.
Despite its title, "Code" has nothing to do with coding, but explains how computers work at the electrical level.
by Tom DeMarco and Timothy Lister
This classic book encourages us to think about the people instead of the process. It's full of practical advice on team building, productivity and office environments. It's a must read, not just for managers, but anyone related to software development.
Get two copies, one for you and one for your manager.
by Robert M. Pirsig
This book is many things, but you could say it's sort of a philosophical take on what it means to "grok" something.
Commentry from Garth Gilmore:
I credit this book with teaching me more about software development than any programming book I ever read.
The central thread in the book is how our romantic (artistic) and classical (technical/rational) perceptions of the world are both derived from how we perceive quality in the environment around us. This understanding is then applied to apparently mundane tasks like motorcycle maintenance.
To give some examples of how this applies to coding:
Long story short its a good read :-)
Simon Singh's The Code Book is a great book about how cryptography was born and how people is always trying to challenge it.
by Douglas Adams
Life, the universe, and everything
"See first, think later, then test. But always see first. Otherwise you will only see what you were expecting. Most scientists forget that." -- Wonko the Sane
by Keith Ferrazzi
Comments from duplicate answer by Flory:
I did not think that I would like it before I got the book but I really enjoyed it. It is basically about how to build a relationships. Prior to reading it I expected it to be very trite and about how to use people for your own ends. Instead it was the opposite in how to be used to everyone's ends. Very interesting.
What is the name of this book?, by Raymond Smullyan. It is a wonderful book of puzzles about the intricacies of logic.
by Michio Kaku
There's a lot of space out there to get lost in.
-- John Robinson, Lost in Space
A Short History of Nearly Everything by Bill Bryson
by Neal Stephenson
This book follows parallel stories of a World War II code breaker and his present day descendant, and deals a lot with the development of computers (Alan Turing is actually a character in the book). A geek's must-read!
Stranger in a strange land because every programmer should grok the word "GROK".
Lessons Learned in Software Testing by Kaner, Bach, and Pettigrew. Brilliant book, easy to read.
The Thermodynamics of Pizza by Harold Morowitz.
This could have all kinds of morals, depending on how you take it. 1. You can use science to improve EVERYTHING! :-) 2. Make sure you choose the right level of abstraction when designing and coding. 3. You can really improve your life if you just take a few minutes to think about it.
by David Allen.
Written in 1966 this classic science fiction novel takes place on the penal colony Luna (the moon). The story is told by the only programmer/computer repairman on Luna, Manuel. Manuel has a secret. The master computer (Mike) that controls all of Luna has become a sentient AI and happens to have Manuel as its only friend. Mike is rough around the edges at first, its speech is fuzzy and it plays childish but dangerous jokes with its god-like abilities. As time wears on Mikes abilities fully develop into a mature being. With Manuel's guidance they will go on an adventure together that spurs the revolution of freeing Luna from Earth!
This novel is the first Robert A. Heinlein novels I have read but will certainly not be the last. The fact that this book was written in 1966 still astonishes me! It has barely any dated parts and could easily pass for a contemporary novel. It wont he Hugo award for best novel.
Truly one of the better "programmer" style novels I have read. Great adventure the whole way through. If anyone has a suggestion as to which Heinlein novel I read next, please leave a comment!
Written in 1950, Dianetics: The Evolution of a Science describes the optimum computer as an introduction to a science of the mind.
by George Orwell
by Mark Haddon
It will give you some perspective of your odd co-workers.
This is an amazing book that details some very counter-intuitive conclusions about the LACK of THINKING actually predominates our decision process.
Player Piano by Kurt Vonnegut
Beyond Fear by Bruce Schneier.
From Amazon: "Schneier provides an interesting view of the notion of security, outlining a simple five-step process that can be applied to deliver effective and sensible security decisions. These steps are addressed in detail throughout the book, and applied to various scenarios to show how simple, yet effective they can be....Overall, this book is an entertaining read, written in layman's terms, with a diverse range of examples and anecdotes that reinforce the notion of security as a process".
Or just consider it a straight read on understanding what security means - whether for computers or in real life. It can give you the tools to handle the ginormous amounts of FUD we encounter every day.... And it's entertaining, besides. (Even got my father to read it, and he's enjoying it...)
Universal Principles of Design, by William Lidwell, Kritina Holden, and Jill Butler
One of this biggest issues I have with many programs I have used is the lack of design put into the interface and into the product. This book goes in-depth describing how to enhance the usablilty within a interface. It also tells you all of the basic principals and rules of design, and they give many examples for many different applications whether its techinical or non-technical. The book reads a little like a college classroom book (and it probably is for many design schools), so it the not the most exciting thing to read, but I find the most informative when it comes to interface design.
One notable premise contained within this book reminds me of the saying "If you go far enough away, then you're on your way back home". For example, the Eastern and Western approaches to philosophy and science were so diametrically opposed for centuries but perhaps they're coming around the other side towards similar conclusions these days?
It may be 30 or so years old, but it's still very much worth the read.
I can't believe I didn't see this already listed:
by Frank Herbert
Dune is the pinnacle of Sci-Fi novels!
I've been really enjoying haiku recently. To that end, I'd very strongly recommend The Haiku Handbook: How to Write, Share, and Teach Haiku by William J. Higinson.
I recommend reading/writing haiku as a way to relax.
The Tipping Point is one of the best books that I have ever read.
by Lynne Truss
Becoming a better communicator in people language, I believe, makes you a better communicator in code. Punctuation is a very good place to start improving your writing.
Every programmer should read this book to learn how to pick up women.
Information theory, betting, value of information, etc.
by Mihaly Csikszentmihalyi
The best and most productive coding is done in a flow state. This is a psychological study of the phenomemon. Although the book is scientifically rigorous it remains accessible to the lay-person.
21st Century Jet: The Making of the Boeing 777, by Karl Sabbagh
From coffee cup holder to three-hundred-foot wing, this book is the story of how a group of people came to build a brand new aeroplane.
The book describes the development of the Boeing 777, from initial concept, through requirements gathering, design, development, testing, production, and delivery. The engineers and management implemented a new development system, overcame changing requirements, met strict safety requirements, and continually optimized the solution. It describes how the designers and engineers worked to make the aircraft easier, safer, and more intuitive for everyone who would come in contact with it (air crew, maintenence crews, and passengers).
Software developers can learn a lot from this book. It's very well written, it reads like a novel. I've read it twice and highly recommend it.
Boeing Computer Services president John Warner said, the Boeing 777 is "three million parts flying in close formation." Sounds like software to me.
Charles Perrow's "Normal Accidents" investigates what can happen when complex technology goes horribly wrong, and formulates his theory of the "normal accident": complex, tightly coupled systems will have accidents, because minor faults interact with catastrophic consequences. We see this all the time in programming and systems administration, and yet, as far as I know, few of these concepts are understood outside safety engineering.
(He also writes very well, and brings life to what could have been a rather dry book).
by George Lakoff
It's a book about how people categorize things, and about reasoning in general. It's long and extremely boring for some people, but it is still great.
If you live on the Unix side of the world, The Art of UNIX Programming by Eric Raymond (see also here). Despite its title, it is not a programming book, and it contains very few lines of code indeed. It's the best book I know about the Unix philosophy.
I found Zero: The Biography of a Dangerous Idea to be pretty decent. He has a followup to this called Decoding the Universe: How the New Science of Information Is Explaining Everything in the Cosmos, from Our Brains to Black Holes which I have but haven't read yet so I can't comment on how it is.
Secret Rendezvous by Kobo Abe. Abe's the frickin' man, man.
But seriously, if you like Murakami, you owe it to yourself to check out Abe.
Happiness is a Choice by Barry Neil Kaufman
It's a great book that can help you understand you can choose how to feel. Turns out you can be responsible for a lot more of your emotions than you think.
Another one from a different angle from prior posts: Gödel, Escher, Bach: an Eternal Golden Braid, by Douglas Hofstadter.
I can not believe this book has never been mentioned!! It is one of the best book about product management I have read in years. If you are working for a startup, it is a must read.
The Milkshake Moment: Overcoming Stupid Systems, Pointless Policies and Muddled Management to Realize Real Growth by Steven Little
The Right Stuff by Tom Wolfe
Walter Murch's "In the Blink of an Eye"
Or any other Speed Reading text. Learning my own quirks about how I read has helped me to be conscious of other aspects of how I think. Having the ability to control my reading speed has proven to be invaluable. I still choose to read books for pleasure at my previous reading speed with all its flaws.
The Illuminatus! Trilogy by Robert Shea and Robert Anton Wilson. In many ways, this book changed the way I do my thinking. Not sure whether it is good or bad to completely distrust anything and everything, but at least it keeps ones mind critical instead of automatically accepting something as truth without questioning.
The book also introduced me to the concepts of discordianism, which I find having quite a few interesting points.
I just bought it on Audible last week and I can't stop listening to it. It goes through the factors of successful people (ex: Bill Gates, Bill Joy, The Beatles). Fascinating!
by Hugh Sebag-Montefiore
Having a bad week at work? Well at least when you can't figure out some algorithm people aren't dying in their hundreds in the freezing North Atlantic waiting on you to work it out.
As well as being a great read about the dawn of the modern computing age, this book can help with perspective.
A great book about the development process. It also highlights how developers are doomed to keep repeating the same mistakes over and over again
Juggling is mandatory. All programmers must juggle. Sorry, it's a rule.
Rick Cook - The Wiz Biz
This is a compilation of the first two novels in a series, called 'Wizard's Bane' and 'Wizardry Compiled', respectively.
It all began when the wizards of the White League were under attack by their opponents of the Black League and one of their most powerful members cast a spell to bring forth a mighty wizard to aid their cause. What the spell delivers master hacker Walter "Wiz" Zumwalt. With the wizard who cast the spell dead, nobody can figure out what the shanghaied computer nerd is good for--because spells are not like computer programs.
Lots of in jokes for the Unix/Linux crowd to enjoy. Pretty much anybody in the software industry will enjoy it, I think.
I'm managing a project that has to be estimated, according to rough requirements and specifications. Because of that, the estimations on the specific features and tasks are set of discrete values, instead of just one discrete value (for example, between 10 and 20, instead of exactly 17).
I'm curious, if I want to get an idea of the approximate probability to finish some task within the lowest estimate, how should I approach this? Please, for the sake of the discussion, disregard factors like my estimation skills, used platform, etc.
I was thinking about using Poisson distribution, with λ = (low + high) / 2, assuming that the probability for each of the proposed values abides to the law of rare events / normal distribution. This doesn't account for the fact that going out of my estimation limits is more unlikely than likely, but still...
What do you think about that, and which approach would you choose for such experiment?
I recommend reading Waltzing With Bears by Tom DeMarco and Tim Lister - it goes into schedule estimating in some depth.
As a rule of thumb I would say the probability of finishing any project within the lowest estimated time is approximately zero. This is both from the analysis they give in the book, and from personal experience.
I am wondering, how "real" work day (7-8 hours) relates to "engineer hours" term used in estimation of the time necessary for completion of some task. I think that estimated effort in EH (if correctly estimated) cannot be simply translated into work days by dividing with 8, and that effective work day of a programmer is shorter than the time he spends in the building in which he works. This can lead to big errors in estimates when estimating small chunks of tasks (i.e. what is scope of one iteration in SCRUM) and when there are no best/worst case estimates, but planning is done based on individual task estimates done by programmers. When programmers need to estimate time necessary to do some individual tasks, they usually estimate the time from the moment they start working on it to the moment they complete it. Needless to say, it is insane to expect that someone will do 4 tasks of 2 hours each in one day.
I would like to know what are best practices in successful companies for relating effective time vs. time spent "at work" and are there some books or researches which estimate what is average time programmer spend working (focused on the tasks at hand), not doing other activities which are not included into estimate, like reading/writing mails (except ones included in estimate, if it is support task or similar), brainstorming, meeting, drinking coffee, estimating tasks and updating status in bug/task tracking tool.
If there are no books or researches, any links to articles about this issue by respected members of community will be of help also as I couldn't find any, these keywords are too generic.
Best quote ever:
“There's no sense in being precise when you don't even know what you're talking about.” (John von Neumann)
Also, experimental software engineering attempts to draw statistically relevant conclusions out of experiments in a controlled environment. Very roughly you get to design experiments applying the scientific method, choosing input variables (ie. experience of the programmers, time of the week, language/framework, etc.), and measuring the output.
The scientific method suggests a cycle of observations, laws, and theories to advance science. Experimental software engineering applies this method to software.
A good book I remember from a remarkable university class on the topic is “Experimental Software Engineering”. It assumes you have a grasp over statistical distributions and descriptive statistics, but it's a very nice read.
There are two things that we do when we program:
When we're repeating similar things, estimating - whether in points of relative effort / complexity or in hours - is usually fairly straightforward. Even when estimating in points, these can easily be correlated back to hours after a sprint or two, once the team has got up to speed.
The problem most teams face is that they're often doing new stuff. This is because:
Thus, a lot of what we do on software projects tends to be new. The usual same-old same-old frequently ends up being given to junior developers, for whom it's new.
The trouble with new things is that you have no idea how long they're going to take. You've never done them before!
I've seen a number of strategies for handling this, depending on how determined someone is to (force devs to) produce reliable estimates, how much trust you have with stakeholders, etc.
Pad the estimates horribly and then make the work fit them (this is what developers end up doing if PMs / SMs force them to produce reliable estimates, whether the devs realise it or not).
Kanban's "classes of service" allow you to mix some new things with some familiar things. This will tend to allow a less variable flow through the development pipeline, so you can base your estimates on that.
Do everything that's risky first (particularly if you don't know how risky it is!). Gradually the flow will stabilise. On a 9 month project, it took 3 months to start producing any kind of useful estimates, so careful with this one. The upside is that your stakeholders will see you addressing the things which keep them awake at night early, so trust can be high. Your PM / SM will go spare during this time. It's OK.
Accept that the old metaphor of "time spent = work done" is a myth in projects involving high amounts of learning and knowledge work. It might have been appropriate in the days when developers were treated like a factory - churning out the implementation of others' designs - but in the world of Agile and Scrum, it's completely inappropriate. Even Scrum.org have recently removed the necessity for estimation and velocity measurements from their guide. Find other ways to develop trust, manage risk, win budgets and have conversations around the work - since these are the only reasons you need estimates in the first place.