Nov, 2018

It Doesn’t Have to Be Crazy at Work by Jason Fried and DHH

The Big Idea: choose to have a calm, profitable, healthy workplace. 

  • Why is work so crazy? Physical and virtual distractions at work. And an unhealthy obsession with growth.
  • Sustained exhaustion is not a badge of honor, it’s a mark of stupidity.
  • How many hours at the office are really spent on work itself?
  • The answer isn’t more hours, it’s less bullshit. Less waste, not more production. Fewer distractions and less stress.
  • It’s time to give people the uninterrupted time that great work demands.
  • Basecamp has been profitable every year. Profitability alleviates stress.
  • Where does our money come from? Customers, not venture capital.
  • We don’t have a single employee in the Valley.
  • We put in about 40 hours a week most of the year and just 32 in the summer.
  • On balance we’re calm — by choice, by practice. We’re intentional about it.
  • Protect people’s time and attention.
  • 40 hours of work per week.
  • Ample time off.
  • Meetings are a last resort.
  • Asynchronous communication first, real-time communication only when necessary.
  • Sustainable practices for the long term.
  • Focus on profitability.
  • When you realize the way you work is malleable, you can start molding something new, something better.
  • We didn’t just assume asynchronous communication. We tested out everything and figured it out.
  • We found that paying for people’s vacations was better than cash bonuses.
  • Hustlemania has captured a monopoly on entrepreneurial inspiration.
  • You’re not very likely to find that key insight or breakthrough idea north of the 14th hour in the day.
  • Improve iteratively and continuously. Fewer explosions and more laying of bricks and applying another layer of paint.
  • Put in a good day’s work, day after day, but no heroics.
  • The business world is obsessed with fighting, winning, dominating, and destroying. But there is another way.
  • Do we have enough customers paying us enough money to cover our costs and generate a profit? Good. Then we’re successful.
  • What matters is that we have a healthy business with sound economics that work for us. Costs under control, profitable sales.
  • We’re serving our customers well, and they’re serving us well. That’s what matters. And just for the customers, we’ve invested in many softwares like CRM that help us in managing customers and have been trying to follow whatever Salesforce has professed.
  • “Comparison is the death of joy.” —Mark Twain
  • There’s no chasing others at Basecamp, just deep work and keeping customers happy.
  • We don’t do goals.
  • We don’t mind leaving some money on the table and we don’t need to squeeze every drop out of the lemon. Do we want to maximize value through constantly chasing goals? No thanks.
  • We are working on building a long-lasting sustainable business with happy employees.
  • How about something really audacious? No targets, no goals. And if you must have a goal, how about just staying in business? Or serving your customers well? Or being a delightful place to work?
  • Everyone wants to be a disrupter these days. If you stop thinking you must change the world, you lift a tremendous burden of yourself and your team. 9pm. meetings and weekend sprints are not as necessary.
  • “NO PAIN, NO GAIN!” looks good on a poster at the gym, but real life is not like the gym.
  • Most of the time, if you’re uncomfortable with something, it’s because it isn’t right. Listen to your discomfort. It was the discomfort of knowing two people doing the same work at the same level were being paid differently that led us to reform our payment structure.
  • It was discomfort working at companies that had taken large amounts of venture capital that led us to pursue a path of profitable independence.
  • Working 40 hours a week is plenty. During the summer, we even take Fridays off. If you can’t fit everything you want to do within 40 hours per week, you need to get better at prioritizing and focusing, instead of working longer hours .
  • Cut out what’s unnecessary.
  • Protect what’s both most vulnerable and most precious: your employees’ time.
  • Eight people in a room for an hour doesn’t cost one hour, it costs eight hours. Plus the cost of the interruption in concentration.
  • Instead of update meetings, we ask people to write updates daily, when they have a free moment. Others can read them when they have a free moment.
  • 60 minutes isn’t really an hour if it’s broken up into four 15 minute blocks.
  • Productivity is for machines, not for people. We believe in effectiveness .
  • Stop equating work ethic with excessive work hours.
  • Work doesn’t happen at work because of all the interruptions. To facilitate collaboration, we borrowed an idea from academia and have people schedule office hours. People are welcome to stop by and discuss work during office hours.
  • The shared work calendar is one of the most destructive inventions of modern times. Taking someone’s time should not be easy.  Meetings should be a last resort, especially big ones.
  • We don’t require anyone to broadcast their whereabouts or availability at Basecamp. Hours worked and butts in seats don’t matter; only actual work matters. The only way to know if work is getting done is by looking at the actual work. That’s the boss’s job.
  • We don’t require anyone to broadcast their availability and we reject the proliferation of chat tools invading the workplace. Know how to reach someone in an emergency but also recognize there are very few actual emergencies.
  • The expectation of an immediate response is the ember that ignites so many fires at work. Create a culture of eventual response rather than immediate response.
  • Instead depending on chat to stay caught up on work, catch up on what happened today as a single summary email. We also write monthly updates called “Heartbeats.”
  • We do care and we do help. But a family we are not. A family sacrifices everything for each other. We’re people who work together to make a product that we are proud of. You don’t have to pretend to be a family to be courteous. Or kind. Or protective.
  • The best companies aren’t families. They’re supporters of families.
  • A leader sets the example that everyone follows. If you value reasonable hours, plentiful rest, and a healthy lifestyle for yourself, then others will follow. If you, as the boss, want employees to take vacations, you have to take a vacation. Workaholism is a contagious disease.
  • The trust battery between the two of you is either charged or discharged, based on things like whether you deliver on what you promise.” A low trust battery is at the core of many personal disputes at work.
  • What the boss most needs to hear is where they and the organization are falling short. The boss needs to ask: “What can we do even better?” “What’s something nobody dares to talk about?” “Are you afraid of anything at work?” “Is there anything you worked on recently that you wish you could do over?” “What do you think we could have done differently to help Jane succeed?” “What advice would you give before we start on the big website redesign project?”
  • The CEO is usually the last to know how things are really going.
  • There’s no such thing as a casual suggestion when it comes from the owner of the business.
  • On low-hanging fruit: he further away you are from the fruit, the lower it looks. Declaring that an unfamiliar task will yield low-hanging fruit probably means the person doesn’t know what he’s talking about.
  • In the long run, work is not more important than sleep.
  • At most companies, work-life balance is a sham. If work can claim hours after 5:00pm, then life should be able to claim hours before 5:00pm to regain the balance.
  • CVs might as well be tossed in the garbage. Everyone know they are exaggerations. To work at Basecamp, you have to be good people.
  • To evaluate someone, we put a real project in front of the candidates so that they can show us what they can do.
  • Be wary of senior people from large companies. Trying to teach a small company how to act like a big one rarely does anyone any good. Unlearning can be just as hard as having to pick up entirely new skills.
  • But be patient. Unless you hire someone straight out of an identical role at an identical company, they’re highly unlikely to be instantly up to speed and able to deliver right away.
  • Talent isn’t worth fighting over. Someone who’s a superstar at one company often turns out to be completely ineffectual at another, so a superstar somewhere else is not worth fighting for.
  • Talent at Basecamp rarely comes from traditional war zones like San Francisco or New York. More likely, it’s Oklahoma, Tennessee, or Toronto.
  • We look at people’s actual work, not at their diploma or degree.
  • It takes patience to grow and nurture your own talent.
  • Most people just don’t enjoy haggling, so Basecamp has a fixed salary structure. Everyone in the same role at the same level is paid the same.
  • Once every year we review market rates and issue raises automatically.
  • The goal at Basecamp is to pay everyone at the top 10 percent of the market for their role and level.
  • We get the market rates through a variety of salary survey companies.There’s also no penalty for relocating to a cheaper cost-of-living.
  • We don’t pay traditional bonuses, just a generous salary.
  • There are no stock options at Basecamp because we never intend to sell the company.
  • We’ve vowed to distribute 5 percent of the proceeds to all current employees if we ever sell the company.
  • There is profit-sharing. Basecamp distributes 25 percent of growth in profits to employees in that year.
  • Basecamp isn’t a startup. Basecamp is a stable, sustainable, and profitable company.
  • Happiness and productivity are found in working with a stable crew.
  • Free dinners are a hoax. A free dinner for working late sounds more like a bribes than a benefit.
  • We don’t offer gotcha benefits, only relevant “outside the office” benefits. Benefits: fully paid vacations, 3-day weekends all summer, paid sabbaticals, continuing education allowances, charity matching, CSA (community-supported agriculture) shares, one monthly massage, a monthly fitness allowance.
  • Offices should operates by library rules. The office should be quiet and calm. Conversations should be kept to a whisper.
  • The purpose of a vacation is to get away. We used to offer unlimited vacation, but we eventually noticed that people actually ended up taking less time off.
  • When someone leaves, be honest and clear with everyone about what just happened. At Basecamp, an immediate goodbye announcement is sent out companywide.
  • Following group chat at work is like being in an all-day meeting with random participants and no agenda. It’s completely exhausting. Chat puts conversations on conveyor belts that are perpetually moving away from you. Chat is great for hashing stuff out quickly.
  • The two rules for chat at Basecamp is “Asynchronous most of the time. Real-time only sometimes.” And if it’s important, slow down and take it offline to think.” Important topics need time.
  • A deadline with a flexible scope will result in a healthy, calm project.
  • When we present work, it’s almost always written up first. Then it’s posted to Basecamp, so people can have time to digest and respond, with a written respond on Basecamp. We don’t want first impressions.
  • Friday is the worst day to release anything.
  • Culture isn’t what you intend it to be. Culture is what culture does. What we do repeatedly hardens into habits and that becomes your culture.
  • Right from the beginning of Basecamp, we insisted on a reasonable workweek. We didn’t pull all-nighters to make impossible deadlines. When calm starts early, calm becomes the habit. If you start crazy , it’ll define you .
  • Today we ship a feature when it’s ready rather than waiting until all features are ready.
  • If every decision has to be made by consensus, you’re in for an endless grind. Someone in charge has to make the final call. Instead, get used to saying “I disagree but let’s commit.” Then move forward.
  • Knowing when to embrace Good Enough is what gives you the opportunity to be truly excellent when you need to be. Separate what really matters from what sort of matters from what doesn’t matter at all. Be clear about what demands excellence.
  • When we spend six weeks on a project, we begin prototyping as soon as we can in those first two weeks. As we pass the mid-point, it’s time to focus in and get narrow. New ideas that arrive too late will just have to wait.
  • “Doing nothing” should always be on the table. It’s too easy to fuck up something that’s working well. “What if we did nothing?”
  • Calm requires getting comfortable with “enough.” If it’s never enough, then it’ll always be crazy at work.
  • Every mature industry is drowning in “best practices.” So much of it is bullshit. There are so many reasons to be skeptical of best practices.
  • Unless you’ve actually done the work, you’re in no position to encode it as a best practice.
  • Many best practices are purely folklore. No one knows where they came from, why they started, and why they continue to be followed.
  • All this isn’t to say that best practices are of no value. Some are helpful to get you going, at which point you can abandon them as you need.
  • You can’t develop a calm culture if you’re constantly fretting about what the best practices. Create your practices and your patterns.
  • “Whatever it takes” is the rallying cry for captains of industry and war generals. Reasonable expectations are out the window when we operate according to “whatever it takes.” There certainly will be rare moments when whatever it takes is truly called for .
  • Rather than demand “whatever it takes,” ask , “what will it take?” Then decides if it’s worth it. Discuss strategy, make tradeoffs, make cuts, or come up with a simpler approach.
  • Too much shit to do is the problem if you’re obsessed with productivity hacks.  The only way to get more done is to have less to do. Saying no is the only way to claw back time.
  • “There is nothing so useless as doing efficiently that which should not be done at all .” — Peter Drucker
  • At Basecamp, we’ve become ruthless about eliminating wasteful tasks.
  • Nearly all product work at Basecamp is done by teams of three people. The team is usually two programmers and one designer. We don’t throw more people at problems, we chop problems down to size. Big teams make things worse all the time.
  • Work expands to fill the time available. Work expands to fill the team available. You can do big things with small teams.
  • Rather than jumping on every new idea right away, we make every idea wait a while. First we finish what we started, then we consider what we want to tackle next. New ideas can wait.
  • Learning to say no is a required skill if you want to be calm.
  • We’ll take a risk, but we won’t put the company at risk.
  • When we make big changes to Basecamp, we give it six months and see how it turned out. We’ll tweak it along the way. We’ll be ready to revert if needed. We prefer managed, calculated risk with a safety rope attached.
  • In the summer, we work 4 day, 32 hour weeks.
  • In the autumn, we pay for a weekly community-supported agriculture share for each employee .
  • Until you’re running a profitable business, you’re slowly (or quickly) running out of business. We focus on keeping out our costs in check.
  • Being profitable means having time to think and space to explore. Being profitable means being in control of your own destiny and schedule.
  • When companies are in the red, employees worry about their jobs. When companies talk about burn rates, two things are burning: money and people.
  • Selling to small businesses and selling to enterprises are two very different approaches requiring two very different kinds of people.
  • Learn to launch. Generally, you just have to ship it. Do your best, believe in the work you’ve done, and ship it. After you ship, you can iterate on real insights and real answers from real customers.
  • We’ve never committed to a product road map. Promises in a product road map pile up like debt. Promises are easy and cheap to make, while actual work is hard and expensive.
  • We’ve been ripped off and cloned a hundred times. We’ve learned you have to move on.
  • People don’t hate all change. What customers and employee don’t like is forced change. We still run three completely different versions of Basecamp, so that customers don’t have to change if they don’t want to.
  • Things get harder as you go, not easier. The easiest day is day one of a new company. As you get bigger, you hire people, there is more competition, and there are increased costs.
  • When it comes to complaints, remember that, mostly, everyone wants to be heard and respected.
  • Companies are culturally and structurally encouraged to get bigger and bigger. But the good old days, the founders miss, are when their business was simpler and smaller.
  • We wonder why didn’t they just grow slower and stay closer to the size they enjoyed the most?
  • Our goal is to maintain a sustainable and manageable size. We still grow, but slowly and in control.
  • We chose calmness, so we cut back on products and features, even when times are great. Cutting back when times are great is the luxury of a calm, profitable, and independent company.
  • A successful business is healthy profits, increased benefits for our employees, and an environment where people can do the best work of their careers.
  • Choose to: protect people’s time.
  • Choose to: work reasonable number of hours.
  • Choose to: relieve people from the conveyor belts of information.
  • Choose to: give employees the focus that their best work requires.
  • Choose: contemplation and consideration prior to communication.
  • Choose to: give endless growth a rest.
  • Choose to: give teams control over what can be reasonably accomplished given the time.
  • Choose to: finish what you started before moving on to the next idea.
  • A calm company is a choice

Rework by Jason Fried

You don’t have to work miserable 60 / 80 / 100 – hour weeks to make it work. You don’t even need an office.

Ignore the real world. It’s a place where new ideas , unfamiliar approaches , and foreign concepts always lose .

Learn from your successes. Failure is not a prerequisite for success.

Long-term business planning is a fantasy. You have to be able to improvise. This isn’t to say you shouldn’t think about the future or contemplate how you might attack upcoming obstacles. That’s a worthwhile exercise. Just don’t feel you need to write it down or obsess about it. Working without a plan may seem scary. But blindly following a plan that has no relationship with reality is even scarier.

Why grow? Why is expansion always the goal? You’ll need a better answer than “economies of scale.” Grow slow and see what feels right. Premature hiring is the death of many companies. Once you get big, it’s really hard to shrink without firing people, damaging morale. Runs a business that’s sustainable and profitable.

Workaholism is stupid. Working like crazy just isn’t sustainable. Workaholics try to make up for intellectual laziness with brute force. Workaholics create guilt and poor morale among their coworkers. You end up just plain tired. No one makes sharp decisions when tired. The real hero is already home because she figured out a faster way to get things done.

Instead of entrepreneurs, let’s just call them starters.

To do great work, you need to feel that you’re making a difference. This doesn’t mean you need to find the cure for cancer.

Scratch your own itch.

Start by making something, now. What you start is what matters, not what you think or say or plan to start. Ideas are cheap and plentiful.

The most common excuse people give: “There’s not enough time.” There’s always enough time if you spend it right.

Keep in mind why you’re doing what you’re doing. A strong stand is how you attract superfans, not just customers. Also, when you stand for something, decisions are obvious. Ex: Whole Foods.

Be authentic with your mission statement. Having an authentic mission statement means truly standing for something.

Outside money is Plan Z. When you take outside money, you give up control. The pursuit of “cashing out” begins to trump building a quality business. Spending other people’s money is addictive. Customers move down the totem pole. Raising money is incredibly distracting. You start having meetings with your investors and / or board of directors.

You need less than you think. There’s nothing wrong with being frugal. Great companies start in garages all the time.

Start an actual business not a startup. Actual businesses worry about profit from day one.

You need a commitment strategy, not an exit strategy. You’ll be able to focus on getting customers to love you, instead of getting acquired.

Be the smallest, the leanest, and the fastest. Avoid: excess staff, meetings, thick process.

Constraints are actually advantages in disguise. Southwest — unlike most other airlines, which fly multiple aircraft models, flies only Boeing 737s. Because of this, Southwest has lower costs and a business that’s easier to run.

You just can’t do everything you want to do and do it well. Sacrifice some of your darlings for the greater good. Getting to great starts by cutting out stuff that’s merely good .

There’s the stuff you could do, the stuff you want to do, and the stuff you have to do. Find out what’s essential and focus all your energy on making it the best it can be.

Ignore the details early on. Nail the basics first and worry about the specifics later. We sketch out ideas with a big, thick Sharpie marker, instead of a ballpoint pen.

Long projects zap morale.

Be a curator. Constantly look for things to remove, simplify, and streamline.

Gordon Ramsay’s first step is nearly always to trim the menu. When things aren’t working, cut back.

Focus on what won’t change. The core of your business should be built around things that won’t change. Things that people are going to want today and ten years from now. Those are the things you should invest in. Eg. Amazon and Japanese automaker focus on core principles.

It’s not the gear that matters. Content is what matters.

Launch now. Once your product does what it needs to do, get it out there. When you impose a deadline, you gain clarity.

Instead of describing what something looks like, draw it. Get to something real (a prototype) right away because people need to see something to start working on it.

Sometimes abandoning what you’re working on is the right move.

Interruption is the enemy of productivity. Long stretches of alone time are when you’re most productive. Use email, over chat/meetings/call, as much as possible.

Meetings are toxic. The true cost of meetings is staggering.

If you must have a meeting, follow these simple rules: set a timer, invite as few people as possible, always have a clear agenda, meet at the site of the problem, end with a solution.

Good enough is fine. Aim for maximum efficiency with minimum effort. Problems can usually be solved with simple, mundane solutions. When good enough gets the job done, go for it.

Quick wins. Momentum fuels motivation. No one likes to be stuck on an endless project. The longer something takes, the less likely it is that you’re going to finish it. Small victories let you celebrate and release good news. Ask: “What can we do in two weeks?”

Don’t be a hero. If you already spent too much time on something that wasn’t worth it, walk away. You can’t get that time back.

Forgoing sleep is a bad idea.

We’re all terrible estimators. Estimates that stretch weeks, months, and years into the future are fantasies. The solution : Break the big thing into smaller things.

Start making smaller todo lists. Prioritize visually. Put the most important thing at the top.

Make tiny decisions by breaking up big decisions. Smaller, attainable goals like that are the best ones to have .

Don’t copy. Copying skips understanding — and understanding is how you grow. When you copy, you never lead, you always follow.

Make you part of your product or service. Zappos sets itself apart by injecting CEO Tony Hsieh’s obsession with customer service into everything it does. Polyface sells the idea that it does things a bigger agribusiness can’t do .

Being the anti – ______ is a great way to differentiate yourself and attract followers. Having an enemy gives you a great story to tell customers, too.

Underdo your competition. Solve the simple problems and leave the hairy, difficult, nasty problems to the competition.

It’s not worth paying much attention to the competition. Focus on competitors too much and you wind up diluting your own vision. Focus on yourself instead.

Say no by default. Use the power of no (to certain customers) to get your priorities straight. Recommend a competitor if you think there’s a better solution out there.

Let your customers outgrow you.

Don’t confuse enthusiasm with priority. The enthusiasm you have for a new idea is not an accurate indicator of its true worth.

To create a product that exceeds expectation, you might need to promise a bit less. Over-promising and under-delivering is like a one-night stand. You don’t want a one-night stand with your customers, you want a long-term relationship.

Being obscure is a great position to be in. When you’re obscure, you can try new things. No one knows you, so it’s no big deal if you mess up.

Build an audience. An audience can be your secret weapon. Every day they come back to see what we have to say. When you build an audience, you don’t have to buy people’s attention — they give it to you. Share information that’s valuable and you’ll slowly but surely build a loyal audience.

Teaching probably isn’t something your competitors are even thinking about. Etsy teaches. Gary Vaynerchuk teaches. Teach and you’ll form a bond you just don’t get from traditional marketing tactics.

Give people a backstage pass and show them how your business works. People love finding out the little secrets of all kinds of businesses.

Don’t be afraid to show your flaws. Imperfections are real and people respond to real.

Press releases are spam. Instead, call someone. Write a personal note.

Forget about the Wall Street Journal. You’re better off focusing on getting your story into a trade publication or picked up by a niche blogger.

Marketing is something everyone in your company is doing 24 / 7 / 365.

The myth of the overnight sensation. Dig deeper and you’ll usually find people who have busted their asses for years. It’s hard, but you have to be patient. Starbucks, Apple, Nike, Amazon, Google, and Snapple all became great brands over time, not because of a big PR push upfront. Start building your audience now.

Never hire anyone to do a job until you’ve tried to do it yourself first.

The right time to hire is when there’s more work than you can handle for a sustained period of time.

Pass on hiring people you don’t need, even if you think that person’s a great catch. Don’t worry about “the one that got away.”

Hire slowly. Hire a ton of people rapidly and a “strangers at a cocktail party” problem where everyone tries to avoid any conflict or drama. No one says, “This idea sucks.”

We all know resumés are a joke. The cover letter is a much better test than a resumé.

There’s surprisingly little difference between a candidate with six months of experience and one with six years. How long someone’s been doing it is overrated. What matters is how well they’ve been doing.

Don’t fall into the trap of thinking you need someone from one of the “best” schools in order to get results.

With a small team, you need people who are going to *do* work, not *delegate* work. Avoid hiring delegators. Delegators love to pull people into meetings ,

Hire managers of one. They don’t need a lot of hand-holding or supervision. They’ve run something on their own or launched some kind of project .

Hire great writers. If you are trying to decide among a few people to fill a position, hire the best writer. Clear writing is a sign of clear thinking .

It’s crazy not to hire the best people just because they live far away. To make sure your remote team stays in touch, have at least a few hours a day of real-time overlap. Meet in person once in a while .

Test out employees. You need to evaluate the work they can do now, not the work they say they did in the past. Hire them for a mini-project.

Own your bad news. When something bad happens, tell your customers.

Getting back to people quickly is probably the most important thing you can do when it comes to customer service.

A good apology accepts responsibility. If you’ve built rapport with customers, they’ll cut you some slack and trust you when you say you’re sorry.

Put everyone on the front lines. Good restaurants sometimes have chefs work out front as waiters for a stretch. Listening to customers is the best way to get in tune with a product’s strengths and weaknesses. The more people you have between your customers ’ words and the people doing the work, the more likely it is that the message will get lost or distorted along the way.

After you introduce a new feature , change a policy , or remove something , knee – jerk reactions will pour in. Ride out that first rocky week. Make sure you don’t foolishly backpedal on a necessary but controversial decision. Let them know you’re listening .

You don’t create a culture. It happens .

Culture is the byproduct of consistent behavior. If you treat customers right, then treating customers right becomes your culture. So don’t worry too much about it. Don’t force it. You can’t install a culture. Like a fine scotch, you’ve got to give it time to develop.

Don’t make up problems you don’t have yet. Most of the things you worry about never happen anyway. Optimize for now and worry about the future later. The ability to change course is one of the big advantages of being small.

Skip the rock stars. The environment has a lot more to do with great work than most people realize. Rockstar environments develop out of trust, autonomy, and responsibility.

When everything constantly needs approval, you create a culture of non-thinkers. Realize that failing to trust your employees is awfully expensive .

Send people home at 5. You shouldn’t expect the job to be someone’s entire life.

Policies are organizational scar tissue. Don’t create a policy because one person did something wrong once. Policies are only meant for situations that come up over and over again.

Talk to customers the way you would to friends. Avoid jargon. Don’t talk about “monetization” or being “transparent;” talk about making money and being honest.

Write to be read, don’t write just to write. Whenever you write something, read it out loud. When you’re writing, think of the person who will read your words.

Four-letter words you should never use in business: need, must, can’t, easy, just, only, and fast. Need: very few things actually need to get done. Can’t: you probably can. Easy: people rarely say something they have to do is easy.

Stop saying ASAP: when everything is high priority, nothing is.

Data Science for Business by Foster Provost and Tom Fawcett

The Big Idea: Invest in data and data science teams. Better data + better data scientists = better models = better business decisions = sustainable competitive advantage. 

Chapter 1: Introduction, Data Analytic Thinking

  • Data mining is the extraction of knowledge from data.
  • Data science is a set of principles to guide data mining.
  • Big data means datasets that are too large for traditional data processing systems and require new technologies such as Hadoop, HBase, MongoDB.
  • We are in Big Data 1.0 still. Big Data 2.0 will be the golden era of data science.
  • Building a top-notch data science team is nontrivial but can be a tremendous strategic advantage.
  • Ex: fraud detection, Amazon, Harrah’s casinos.
  • It’s important for managers and executives to understand basic data science principles to get the most from data science projects and teams.
  • Just like chemistry is not about test tubes, data science is not about data engineering or data mining.

Chapter 2: Business Problems and Data Science Solutions

  • There are a few fundamental types of data mining tasks: classification, regression, similarity matching, clustering, association grouping, profiling, link prediction, data reduction, causal modeling.
  • This book will focus on: classification, regression, similar matching, and clustering
  • Ex: churn prediction is a classification problem.
  • Supervised vs unsupervised. Supervised data mining has a specific target. Unsupervised data mining is used to learn and observe patterns in the data but doesn’t have a specific target.
  • It’s important to appropriately evaluate prediction models.
  • Your model is not what the data scientists design, it’s what the engineers build.
  • Data science engineers are software engineers who have expertise in production systems and in data science.
  • Data mining is closer to R&D than to software engineering.
  • Invest in pilot studies and throwaway prototypes.
  • Analytics skills (ability to formulate problems well, to prototype solutions, to make reasonable assumptions) are more important than software engineering skills in a data science team.
  • Useful skills for a business analyst: statistics, SQL, data warehousing, regression analysis, machine learning.
  • There is much overlap, but there is a different because understanding the reason for churn, vs predicting which customers to target to reduce future churn.
  • Ex. Who are the most profitable customers? SQL
  • Ex. Is there really a difference between the profitable customers and the average customers? Statistics and hypothesis testing.
  • Ex. But who really are these profitable customers? Can I characterize them? SQL, statistics, automated pattern finding. Classification.
  • Ex. Will some particular new customer be profitable? How much revenue should I expect this customer to generate? Predictive model. Regression.

Chapter 3: Introduction to Predictive Modeling, From Correlation to Supervised Segmentation

  • Predictive modeling is supervised segmentation. We have some target quantity we would like to predict.
  • A classification tree, or decision tree, is a method to classify data instances.
  • Tree structured models are a very popular data modeling technique and work remarkably well.
  • Tree structured models are also easy for business users to understand.

Chapter 4: Fitting a Model to Data

  • Tuning the parameters so that the model fits the data is parameter learning or parametric modeling.
  • The most common procedure is one you’re already familiar with, linear regression.
  • Logistic regression applies linear models to class probability estimation, and is one of the most useful data mining techniques.
  • Nonlinear support vector machines and neural networks fit parameters based on complex, nonlinear functions.
  • If we increase the complexity, we can fit the data too well, then we are just memorizing the data.

Chapter 5: Overfitting and Its Avoidance

  • Just because a model fits the data very well, doesn’t mean it is better at predicting. It could just be memorizing the data.
  • If you torture the data long enough, it will confess.
  • Fundamental tradeoff between model complexity and overfitting.
  • Always hold out data to test the model.
  • A fitting graph shows the difference between accuracy during training and accuracy during testing.
  • Overfitting is bad because the model picks up spurious correlations that produce incorrect generalizations.
  • A learning curve is a plot of the generalization performance against the amount of training data.

Chapter 6: Similarity, Neighbors, and Clusters

  • Similarity between data instances is described as distance between their feature vectors.
  • Nearest-neighbor methods predict by calculating distance between a new data and neighbors in the training set.
  • Similarity is used as the basis for the most common methods of unsupervised data mining, clustering.
  • Hierarchical clustering can provide insights that instruct further data mining.
  • A cluster centroid can be used as the basis for understanding clusters.

Chapter 7: Decision Analytic Thinking I, What is a Good Model?

  • Accuracy is too simplistic a metric.
  • A confusion matrix differentiates between different types of errors (eg. sensitivity vs specificity)
  • Expected value frameworks are extremely useful in organizing data science thinking and evaluating models.

Chapter 8: Visualizing Model Performance

  • A profit curve is useful for business user to evaluate classifiers.
  • A Receiver Operating Characteristics (ROC) graph is useful for evaluating models when class priors or costs/benefits are not known.
  • Since ROC curves are not intuitive, a cumulative response curve (or lift curve) is most appropriate for some business users.

Chapter 9: Evidence and Probabilities

  • Bayes rule is used for conditional probabilities which occurs frequently in business problems.
  • Naive Bayes rule is valuable because it is very efficient, practical to use, and can learn on the fly.
  • Naive Bayes rule should be avoided when costs/benefits are uses. Best to use when rankings are more important.
  • Bayes rule are the basis of evidence lifts. Evidence lifts are useful for understanding data like “Facebook Likes as a predictor of High IQ”

Chapter 10: Representting and Mining Text

  • Term frequency (TFIDF) is a simple and useful data mining technique for text.
  • Topic layers can also be used to assist with understanding text.

Chapter 11: Decision Analytic Thinking II, Towards Analytical Engineering

  • Expected value framework is a core approach useful in many data science scenarios.

Chapter 12: Other Data Science Tasks and Techniques

  • Not discussed in depth in this book: co-occurrence grouping, lift and leverage, market basket analysis, profiling, link predictions, social recommendation, data reduction, latent information, bias vs variance, ensemble models, causal explanations

Chapter 13: Data Science and Business Strategy

  • Understanding data science concepts leads to awareness of new opportunities.
  • Understanding the ROI of data science results in increased investment in data and data science teams.
  • Data science is a sustainable competitive advantage.
  • A culture of data science is valuable in building a data science team.
  • A top data scientist is worth many times an average data scientist.
  • Data science is learned by working with top data scientists, either in industry or academia.
  • A top data science manager understands the technical principles, understands the business needs, and manages people and projects well.
  • There is only one reliable predictor of success of a data science research project: prior success.
  • Top data scientists want to work with other top data scientists. Most want more responsibility. Most want to be part of a fast-growing, successful company.
  • Consider funding a PhD student for $50k/year.
  • Consider taking on a data science professor or a top data science consultant as a scientific advisor to guide projects and attract data scientists.
  • An immature data science team has processes that are ad-hoc.
  • A medium-maturity data science team employs well-trained data scientists and managers.
  • A high-maturity data science team focuses on processes as well as projects.