67

This question is meant to serve as a list of databases and their configurations that the major web sites use and would be a great reference for anyone thinking of scaling their web site to the size of Twitter, Facebook or even Google.

Please keep your answers to a minimum and be sure to cite any sources used.

EDIT:

Also, please bold both the web-site name and the database for easier scanning.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
niktech
  • 115
  • 1
  • 5
  • 12
  • 2
    I am surprised most of the big famous sites are not using Oracle except linkedIn as mentioned below. Is it because of the licensing cost or some specific drawback? – K Singh Jan 21 '10 at 05:54
  • 1
    The community should be able to vote if a question is correctly closed as off topic or not. – JordanBelf Feb 08 '15 at 05:53
  • @JordanBelf: http://stackoverflow.com/help/privileges/close-questions – zondo Jul 03 '16 at 04:50
  • relevant read https://www.8bitmen.com/what-database-does-facebook-use-a-1000-feet-deep-dive/ – underdog May 26 '19 at 14:36

10 Answers10

26

Facebook.com

  • MySQL with MyRocks. Used to store user info and social activities such as likes, comments, and shares.
  • Hive (Data warehouse for Hadoop, supports tables and a variant of SQL called hiveQL). Used for "simple summarization jobs, business intelligence and machine learning and many other applications"
  • Cassandra (Multi-dimensional, distributed key-value store). Currently used for Facebook's private messaging.

Currently running 610 (soon to be 1000) Hadoop nodes in a single cluster with Hive datastore. Both Hive and Cassandra have been open-sourced by Facebook.

Facebook stats:

  • More than 200 million active users
  • More than 100 million users log on to Facebook at least once each day
  • More than 30 million users update their statuses at least once each day
  • Average user has 120 friends on the site

Sources:

GG.
  • 21,083
  • 14
  • 84
  • 130
niktech
  • 115
  • 1
  • 5
  • 12
16

Stack Overflow - SQL Server.

Jeff Atwood wrote a nice blog post on this

https://blog.stackoverflow.com/2008/09/what-was-stack-overflow-built-with/

Community
  • 1
  • 1
ACP
  • 34,682
  • 100
  • 231
  • 371
15

LinkedIn.com

  • Oracle (Relational Database)
  • MySQL (Relational Database)

Databases replicated on multiple servers for high availability. Each specific Service uses its own domain-specific DB.

LinkedIn stats:

  • 22 million members
  • 4+ million unique visitors/month
  • 40 million page views/day
  • 2 million searches/day

Sources:

niktech
  • 115
  • 1
  • 5
  • 12
13

Flickr uses MySQL.

YouTube uses MySQL but they are moving to Google's BigTable.

Myspace uses SQL Server.

Wikipedia uses MySQL.

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Mohammed Nasman
  • 10,992
  • 7
  • 43
  • 68
  • 3
    Does this mean they have one database server handling a gazillion things everything? How does it not blow up?? – bigpotato Jun 24 '14 at 20:11
12

Microsoft.com

  • SQL Server (no surprise there)

Microsoft.com stats:

  • 250 million unique visits/month.
  • 70 million page views/day.
  • 15,000 connections/second.
  • Maintains an average of 35,000 concurrent connections to a total of 80 Web servers.

Sources:

Fredrik Mörk
  • 155,851
  • 29
  • 291
  • 343
11

Yahoo.com

  • PostgreSQL (modified) - A client can connect to any of the nodes in the cluster (or a policy restricted subset). A query flows from the client to the server it chose to connect with. The SQL compiler on that node compiles and optimizes the query on that single node (no parallelism).

Yahoo.com stats:

  • 24 billion events a day
  • 2-petabyte, claims largest database (Mar 2008)

Source:

KahWee Teng
  • 13,658
  • 3
  • 21
  • 21
  • 1
    It's described as "structured data, as opposed to unstructured data like e-mail and other documents." Hassan, the Data VP added, "It's about how people use our Web site, both from the advertising perspective and from the consumer experience perspective." – KahWee Teng Jul 11 '09 at 15:30
9

Digg

  • MySQL (Relational Database) for scaling out reads
  • MemcacheDB (Key-Value Store) for scaling out writes

Both data stores are distributed across multiple servers.

Digg stats:

  • 30M users
  • 26M uniques per month
  • 2 billion requests a month
  • 13,000 requests a second, peak at 27,000 requests a second.

Sources:

double-beep
  • 5,031
  • 17
  • 33
  • 41
niktech
  • 115
  • 1
  • 5
  • 12
9

Twitter.com

  • MySQL (Relational Database).
  • Cassandra (Multi-dimensional, distributed key-value store). Twitter is just "beginning to use Cassandra at Twitter" (see second source).

In May 2008, Twitter had 1 MySQL instance for writes with multiple MySQL slave instances for reads.

Twitter stats:

  • Total Users: 1+ million
  • Total Active Users: 200,000 per week
  • Total Twitter Messages: 3 million/day
  • 5% of Twitter users account for 75% of all activity
  • 72.5% of all users joining during the first five months of 2009

Sources:

niktech
  • 115
  • 1
  • 5
  • 12
5

Google uses BigTable: http://research.google.com/archive/bigtable.html

stribika
  • 3,146
  • 2
  • 23
  • 21
3

PlentyOfFish.com using Microsoft SQL Server:

https://blog.codinghorror.com/scaling-up-vs-scaling-out-hidden-costs/

Cœur
  • 37,241
  • 25
  • 195
  • 267
duffymo
  • 305,152
  • 44
  • 369
  • 561