17
I’m still alive – preparing my presentation for ConFoo conference (about massive scalability), writing an article on load balancer in the free software domain, doing a technical review of a book on Nginx (it should be pretty good), and driving my company at crazy speed (some new contracts and operations for existing ones). Ho, and I’m preparing a wedding (mine) – yeah, I’m alive and living my life at the fullest.
In the midst of all that, I’ve got a mandate involving MySQL databases and the import of a big chunk of data (about 8gb raw). Yeah, I know, some peoples reading my blog will know whom I speaking of. The data come with the schema, the load data queries and the CSV files. Should be pretty easy, long, but easy.
Got that one wrong. I’ve been bitten by all the bugs that were on the road:
– MySQL LoadData (from CSV) use the TMPDIR variable (which was on root filesystem) and LoadData generate a temporary queries in that directory. In the present case, the 8GB raw csv represented over 40GB of required tmpdir space.
– MySQL Replication doesn’t live really well with importing large amount of data. The first (failed) import broke all the replication (ring of masters + slaves).
– and…
MySQL’s innodb log file cannot shrink its size. That’s pretty simple as a rule, but it creates lots of problems when you have a failed import due to free space issue. Whatever you do, MySQL will NOT give back this hard drive space unless you dump all and re-import. The current case being a database of over 20GB, it’s really a no-go (and since you are lacking free space, dump is a no-go).
The important things to take away from that encounter is: always enable innodb_file_per_table on MySQL. The performance overhead will be minimal (very) – you might even get better performance depending on your file system tolerance for fragmentation – AND – you will be able to drop table and regain the disk space… even if its only for a couple hours. The drop of a 10GB database should always give you some free space ;-). Question is: why haven’t MySQL put this setting as default ? Its been around for long enough… is stable and does help the general user (a sysadmin) experience.
Tags: free software, mysql | comments (1) | read more...
10
Tags: Asterisk, cloud computing, freesoftware, glusterfs, GNU/Linux, labsphoenix, Lustre, mailman, masi, mysql, opensource, personnal, sme, zabbix, zimbra | comments (0) | read more...
19
Tags: affinity, consultant, cpu, database, GNU/Linux, jobs, mysql, oracle, sysadmin, Technique | comments (0) | read more...
13
Tags: apache, ensim, mysql, opensource, php, sysadmin | comments (0) | read more...
25
Tags: consultant, labsphoenix, mysql, storage, sysadmin, Technique, zabbix | comments (0) | read more...
