Sean Quinlan, “Storage at Scale” (MP3 audio), O’Reilly Media Velocity Conference, IT Conversations, 2008/06/24

When massive volumes of data is expected to always be available on the Internet, how is that engineered?

A one in a million hardware fluke would happen every day at Google, so reliability must be handled by smart software. In this talk from the 2008 Velocity Conference, Sean Quinlan of Google describes the tools that Google uses to manage terabytes of data spread over millions of machines.

Because their needs are too big for a single machine, Google does not look at single machine performance. Rather, they look for the most performance bang for the buck and buy lots of it. They then layer software on top of it, which allows them to replicate data across multiple machines to compensate for the potential failure of any one machine. The two major systems they use are the Google File System (GFS) and BigTable. GFS is a cluster file system laid on top of a data center that stores chunks of data either as append-only sequences similar to log files or as read-only sorted tables of key/value pairs. These restrictions allow seamless and reliable storage at Google’s immense scale.

IT Conversations | Velocity Conference from O’Reilly Media | Sean Quinlan

[MP3 audio]

Advertisements
About

David Ing blogs at coevolving.com , photoblogs at daviding.com , and microblogs at http://ingbrief.wordpress.com . See .

Tagged with: , ,
Posted in Uncategorized

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Beyond this media queue
This content is syndicated to Twitter. For professional perspectives, look to Coevolving Innovations; for a photoblog, look to Reflections, Distractions.
  • Ubuntu L2TP VPN
    Now with Ubuntu 17.04, since Ubuntu 16.04, I’ve had problems with connecting to a VPN that uses L2TP.  I found a fix through a series of steps, beginning with “Enabling L2TP over IPSec on Ubuntu 16.04” | Zaid Daba’een | Aug. 22, 2016, updated Mar. 29, 2017 at http://blog.z-proj.com/enabling-l2tp-over-ipsec-on-ubuntu-16-04/ . sudo add-apt-repository ppa:nm-l2 […]
  • 2017/10/08 13:10 Alice Rawsthorn, “Good or Bad Design”, EditDX
    Keynote @alicerawsthorn @EditDX Introduction by Nina Boccia @nboccia Director of Programs, Design Exchange Alice Rawsthorn is authoritative voice on design Last book was Hello World: Where Design Meets Life (2013) Next book, Design as an Attitude, to be published in spring 2018 Based in London OBE (Order of the British Empire) for service to design and […] […]
  • 2017/10/02 13:05 Richard Florida, “Urban Futures Keynote”, EditDX
    Plenary @richard_florida @EditDX intro by @alexbozikovic This digest was created in real-time during the meeting, based on the speaker’s presentation(s) and comments from the audience. The content should not be viewed as an official transcript of the meeting, but only as an interpretation by a single individual. Lapses, grammatical errors, and typing mistake […]
  • 2017/09/29 16:35 “Future of Design Thinking”, EditDX
    Panel @EditDX @rsomerson  @CodeZebra  Luigi Ferrara, @jamescowan as moderator This digest was created in real-time during the meeting, based on the speaker’s presentation(s) and comments from the audience. The content should not be viewed as an official transcript of the meeting, but only as an interpretation by a single individual. Lapses, grammatical error […]
  • Dissertation committees and privilege
    Is a dissertation about the content, the student, or power dynamics? Dissertation writing is very stressful, what with the looming job market, the high cost of living in many cities, the balancing act of life and health — all while being expected to produce intellectual gold. To that, add the anxiety caused by those committee […]
  • 2017/09/10 13:55 Paul Cabaj, “Financing a Coop”, Disrupting the Disruptors
    Workshop @chickweedpatch @CoopsCanada #platformcoop, platformcoop.ca/, Beeton Room, Toronto Public Library This digest was created in real-time during the meeting,based on the speaker’s presentation(s) and comments from the audience. The content should not be viewed as an official transcript of the meeting, but only as an interpretation by a single individua […]
Contact
I welcome your e-mail. If you don't have my address, here's a contact page.
%d bloggers like this: