8

I am just a beginner in MySQL, I need to know how much data can be stored in MySQL. I am developing a web crawler, can I store all the data in MySQL, or do I need to use another Database? Which is more faster? What I mean is, which has the highest Writing/Reading Rate? Do I need to reconfigure to add more data?

Lazer
  • 90,700
  • 113
  • 281
  • 364
Alex Mathew
  • 1,534
  • 9
  • 31
  • 57
  • 4
    You mean the entire data of the web? :) – Daniel Vassallo Aug 05 '10 at 16:54
  • You can store as much data in MySQL as you have disk space and your OS can cope with. There's really not a lot of advice anyone can provide, given so vague information. – nos Aug 05 '10 at 16:55
  • @Daniel :yep @MarkJ:why? – Alex Mathew Aug 05 '10 at 16:58
  • @Tom Gullen sure it is(given the OS can account for all the disk space), you might not be able to store it all in one thable though. – nos Aug 05 '10 at 16:58
  • @Nos, you edited your comment and it is now correct. What you said previously however was not! :-) – Tom Gullen Aug 05 '10 at 17:05
  • A _very_ large amount can be stored in MySQL, but please keep in mind: unless you're very interested in some semantics of HTML-use, don't store actual HTML pages, store the stripped text-content. – Wrikken Aug 05 '10 at 17:10

2 Answers2

13

Depends on the operating system.

**Operating System**           **File-size Limit**
Win32 w/ FAT/FAT32              2GB/4GB
Win32 w/ NTFS              2TB (possibly larger)
Linux 2.2-Intel 32-bit      2GB (LFS: 4GB)
Linux 2.4+                      4TB
Solaris 9/10              16TB
MacOS X w/ HFS+              2TB
NetWare w/NSS file system      8TB

http://dev.mysql.com/doc/refman/5.0/en/full-table.html

Your write/read rate is of pretty much no concern to you, your bottleneck is going to be your internet connection.

Tom Gullen
  • 61,249
  • 84
  • 283
  • 456
  • i am using a win32/NTFS system,what will i do if the data goes higher than 4tb??its already 3TB – Alex Mathew Aug 05 '10 at 16:58
  • Find a different DBS, SQL server on a 64bit machine can have 16TB. http://msdn.microsoft.com/en-us/library/ms143432.aspx Just shop around and see what tickles your onion jar. Oracle can handle 8589541376 G on a 'BFT' addressing scheme. I don't think MYSql was designed with your needs in mind. – Tom Gullen Aug 05 '10 at 17:02
  • @Tom Gullen: So you are suggesting SQL Server,is there any Open Source DB for my needs?what DB did google using? – Alex Mathew Aug 05 '10 at 17:04
  • @Alex: google is using homegrown BigTable (http://stackoverflow.com/questions/362956/what-database-does-google-use). – Wrikken Aug 05 '10 at 17:08
  • @Wrikken : Thks for the info,in my case what all DB can use?what are the solutions? what about Hadoop?is it a DB,can i use that for a webcrawler? – Alex Mathew Aug 05 '10 at 17:11
  • .. and the MySQL manual actually suggests some solutions for a bigger table, you could just employ one of those. – Wrikken Aug 05 '10 at 17:11
  • @Wrikken:what are they?can you please say? – Alex Mathew Aug 05 '10 at 17:14
  • @Alex:they are at the link Tom provided. And as far as recommendation: I'd store the crawled files as documents & let sphinx (http://www.sphinxsearch.com/) index them, I do however not know about its size limits, but you can distribute them over several nodes. – Wrikken Aug 05 '10 at 17:21
  • @Tom : Can i use Hadoop as a DB? – Alex Mathew Aug 05 '10 at 18:00
  • i see a contradiction in the same page https://forums.mysql.com/read.php?22,379547,381106 says 32 TO per table can you explain – Zain Elabidine Feb 10 '19 at 16:14
0

https://forums.mysql.com/read.php?22,379547,381106

InnoDB Size Limits

  • Max # of tables: 4 G
  • Max size of a table: 32TB
  • Columns per table: 1000
  • Max row size: n*4 GB
  • 8 kB if stored on the same page
  • n*4 GB with n BLOBs
  • Max key length: 3500
  • Maximum tablespace size: 64TB
  • Max # of concurrent trxs: 1023

Nanda Kishore Toomula

Sr DBA,Nokia India

CMDBA 5.0

Sher Singh
  • 279
  • 3
  • 13