Why do we need a JSON API for MySQL?
The real need sprang from a situation I was facing daily at Rackspace. We have a lot of production and pre-production environments which are in flux but we need a way to query data from various MySQL servers for multiple purposes. Some folks need data in ruby or python scripts while others need to drag in data with .NET and Java. Wrestling with the various adapters and all of the user privileges on disparate database servers behind different firewalls on different networks was less than enjoyable.

That's where this bridge comes in.

The bridge essentially gives anyone the ability to talk to multiple database servers across different environments by talking to a single endpoint with easily configurable security and encryption. As long as the remote user can make an HTTP POST and parse some JSON, they can query data from multiple MySQL endpoints.

How does it work?
It all starts with a simple HTTP POST. I've become a big fan of the Python requests module. If you're using it, this is all you need to submit a query:

import requests
payload = {'sql': 'SELECT * FROM some_tables WHERE some_column=some_value'}
url = "http://localhost:5000/my_environment/my_database"
r = requests.post(url, data=payload)
print r.text

The bridge takes your query and feeds it into the corresponding MySQL server. When the results come back, they're converted to JSON and returned via the same HTTP connection.

What technology does it use?
Flask does the heavy lifting for the HTTP requests and Facebook's Tornado database class wraps the MySQLdb module in something a little more user friendly. Other than those modules, PyYAML and requests are the only other modules not provided by the standard Python libraries.

Is it fast?
Yes. I haven't done any detailed benchmarks on it yet, but the overhead is quite low even with a lot of concurrency. The biggest slowdowns come from network latency between you and the bridge or between the bridge and the database server. Keep in mind that gigantic result sets will take a longer time to transfer across the network and get transformed into JSON.

I found a bug. I have an idea for an improvement. You're terrible at Python.
All feedback (and every pull request) is welcome. I'm still getting the hang of Python (hey, I've only been writing in it seriously for a few weeks!) and I'm always eager to learn a new or better way to accomplish something. Feel free to create an issue in GitHub or submit a pull request with a patch.

mysql-json-bridge: a simple JSON API for MySQL is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

My goal was to find out which commits existed in development but didn't exist anywhere in master. From there, I needed to find out which commits existed in master that didn't exist in development. That would give me all of the commits that needed to be in the development branch.

I constructed this awful looking bash mess to figure out which commits were in development but not in master:

I had a list of commits that existed in development but not in master:

965cf71 Trollface
acda854 Some patch 2
bf1f3e2 Some patch 1
db1980c Packaging

From there, I could swap MASTER and DEV to figure out which commits existed in master but not in development. Only a couple of commits showed up and these were the ones which were committed and pushed to master inadvertently. After a couple of careful cherry picks and reversions, my branches were back to normal.

Compare commits between two git branches is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Here's an example of what httpry's output looks like on a busy site like icanhazip.com:

2012-03-13 23:29:39 186.x.x.x	192.x.x.x > GET	icanhazip.com	/	HTTP/1.1	-	-
2012-03-13 23:29:39 192.x.x.x	186.x.x.x < -	-	-	HTTP/1.1	200	OK
2012-03-13 23:29:39 187.x.x.x	192.x.x.x > GET	icanhazip.com	/	HTTP/1.0	-	-
2012-03-13 23:29:39 192.x.x.x	187.x.x.x < -	-	-	HTTP/1.0	200	OK
2012-03-13 23:29:39 188.x.x.x	192.x.x.x > GET	icanhazip.com	/	HTTP/1.1	-	-
2012-03-13 23:29:39 192.x.x.x	188.x.x.x < -	-	-	HTTP/1.1	200	OK
2012-03-13 23:29:39 189.x.x.x	192.x.x.x > GET	icanhazip.com	/	HTTP/1.1	-	-
2012-03-13 23:29:39 192.x.x.x	189.x.x.x < -	-	-	HTTP/1.1	200	OK

You can watch the requests come in and the responses go out in real time. It even allows for BPF-style packet filters which allow you to narrow down the source and/or destination IP addresses and ports you want to watch. You can run it as a foreground process or as a daemon depending on your needs.

It's now available as a RPM package for Fedora 15, 16, 17 (and rawhide) as well as EPEL 6 (for RHEL/CentOS/SL 6).

New Fedora and EPEL package: httpry is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The guide: Redundant Cloud Hosting Guide

As always, if you find anything in the guide that needs improvement, I'm all ears.

A simple guide to redundant cloud hosting is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Quite a few distribution maintainers have gone to great lengths to keep Xen domU support in their kernels and it's been an uphill battle. Some kernels, such as Ubuntu's linux-ec2 kernels, have patches from 2.6.18 dragged forward into 2.6.32 and even 2.6.33. It certainly can't be enjoyable to keep dragging those patches forward into new kernel trees.

The paravirt_ops support for Xen guests was added in 2.6.23 and continues to be included and improved in the latest kernel trees. However, there are two significant problems with these new kernels if you're trying to work with legacy environments:

• the console is on hvc0, not tty1
• block devices are now /dev/xvdX rather than /dev/sdX

If you only have a few guests, these changes are generally pretty easy. Switching the console just requires some changes to your inittab or upstart configurations. Changing the block device names requires changes to the guest's Xen configuration file and /etc/fstab within the guest itself.

Considering the amount of environments I work with daily at Rackspace, changing the guest configuration is definitely not an option. I needed a way to keep the console and block devices unchanged so that our customers could have a consistent experience on our infrastructure.

Luckily, Soren Hansen offered to pitch in and a solution became apparent. Through some relatively small patches, the legacy console and block device support was available in the latest 2.6.32 version (2.6.32.12 as of this post's writing).

So far, I've tested x86_64 and i386 versions of 2.6.32.12 with the console and block device patches. It's gone through its paces on Xen 3.0.3, 3.1.2, 3.3.0 and 3.4.2. All revisions of Fedora, CentOS, Ubuntu, Debian, Gentoo and Arch made within the last two years are working well with the new kernels.

Legacy tty1 and block device support for Xen guests with pvops kernels is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

I've been a bit slow on new posts lately for two main reasons. First, I've been working hard with my new position at Slicehost, (I'm still with Rackspace). Also, I've thrown out the old code for Boxcheck.com and I'm re-writing it from scratch.

Most of you probably know that the first version of Boxcheck threw out some odd errors for some tests and caused confusion for those that depended on the results. The code is being overhauled to handle errors and special cases much more efficiently. I'm implementing multiple technologies on the back-end to make the interface simpler and easier to use. There may also be an API that folks can script against - but I haven't gotten that far yet.

In short - thanks for all of your e-mails! Boxcheck isn't going away at all. It's going to be back soon! If you have any questions or suggestions, feel free to give me a shout.

UPDATE 2010-03-26: I've abandoned the project for now, but if anything changes, I'll be sure to make another post.

Boxcheck v2.0 is on the way is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

As always, I'd like to personally thank everyone who has made this release possible. Your efforts have provided benefits from MySQL users worldwide, and they have shown me how rewarding it is to maintain open source software.

MySQLTuner 1.0.0 is now available is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Luckily, mod_deflate makes this easy, and the Apache documentation has a handy initial configuration available:

<Location />
SetOutputFilter DEFLATE
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
Header append Vary User-Agent env=!dont-vary
</Location>

This configuration will compress everything except for images. Of course, you can't test this with curl, but you can test it with Firefox and LiveHTTPHeaders. If you don't have Firefox handy, you can try a very handy web application that will give you the statistics about the compression of your site's data.

Compress your web content for better performance is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

I've discovered one of the most awesome applications for a Mac called ScreenFlow (#3 on my list after #1 TextMate and #2 Adium and I decided to put it to use for my first screencast.

This screencast will teach you how to write a basic PHP script that will query data in MySQL. It is designed for users who are familiar with Linux but are just getting started with PHP. Here are the basic parts of the screencast:

» Import a MySQL dump file
» Connect to MySQL with PHP
» Select a default database
» Issue a query and retrieve results
» Style the rows retrieved into a table with alternating row colors

Download the screencast in Quicktime format (80.9MB)

NEW: Uploaded the video to Vimeo as well.

As always, your comments are welcome. This is my first screencast, and I'm sure I can make some improvements on future videos. Please let me know what I can change to make these screencasts better for you.

Screencast: Write a PHP script to query MySQL is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

MySQLTuner is the Debian Package of the Day! is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Lots of folks have been asking for a summary of what's going on after my Twitter posts appeared earlier today. Here's the latest information I've seen and heard today:

Parallels Development Strategy
» Create an SMB edition, solidity the Application Packaging Standard (APS), and further integrate billing
» 80+ Research and development engineers currently work on Plesk products
» Increase density and make a more useable API
» Make Plesk the dominant panel through integration, SaaS deployments and scaling

Plesk 8 Series
» Plesk 8.4 brought DomainKeys and additional features
» Odd point revisions will contain minor bug fixes while even ones will contain new features
» Plesk 8.6 will have IPv6 support, full ModernBill integration and the APS software catalog

Plesk 9 Series
» Reseller level accounts will be available, and they will fit between the server administrator and client accounts
» Clients can be promoted to resellers
» Domain administrators can be promoted to clients (and resellers)
» Home and desktop pages will be merged
» Better rollover tooltips will be provided
» Hosting setup will be easier and setting limits will be more intuitive
» Server administrators will be able to hide certain services and functions
» Overselling will be allowed so that soft and hard limits can be set
» API improvements will be backward compatible
» Reseller billing will be integrated with ModernBill
» Choose to use Postfix rather than Qmail
» Greylisting is built into the panel

Plesk Future Ideas
» Vast majority of Plesk deployments have five domains or less
» SMB edition with support for one domain and one administrator (SaaS)
» Corporate fileserver and collaboration will be in the SMB version
» Use PHP via FastCGI rather than Apache module
» Lighttpd support for Plesk panel as well as hosted sites
» Multiple MTA's (in addition to Postfix/Qmail)
» Upgrade checks can be run prior to upgrading Plesk so that issues can be avoided
» Recovery tools will be available for broken upgrades
» Local Plesk mirrors can be used to do remote upgrades
» Multiple level backup/restore options for admin, reseller, client or domain administrator
» Migration of shared hosting environment into Virtuozzo container or Plesk Server container
» SiteBuilder 5.0 will have mobile web site design support
» Event calendar is coming

ModernBill
» Will be integrated into Plesk
» Support for 20+ payment gateways
» 100% ready in Q4 2008

Application Packaging Standard (APS)
» Applications will be available directly from Parallels within the Plesk panel
» Modules are easy to install and update
» Applications can be maintained by packagers and kept up to date
» Centralized license management
» Installation and configuration can be completely scripted
» Install scripts can get Plesk configuration data right from Plesk so the interface within the application matches Plesk's interface

Notes from Sergei's Keynote
» Hosting growth is slowing, but it mature
» Managed hosting is growing a lot, and SaaS has entered the marked
» Mentioned Rackspace's pending IPO as a sign of the changing market
» Large SaaS entries: Google's products, Amazon's S3/EC2, Microsoft's Hosted Exchange
» Google is a monopoly, the most dangerous, won't partner with anyone
» Microsoft works harder on partnerships, but was more dangerous without Yahoo!
» EMC quietly launched direct hosting, storage hosting, cloud computing
» IT is mocing off-premises, which is both a threat and opportunity
» Hosting companies must stay competitive via pricing, high levels of service
» Create stickyness with critical service offerings and owning the customer (and their data)
» Hosting companies can improve with broad sets of service plans, flexible upgrade paths, and close ties with customers/partners
» Parallels vision includes partnering, automation, virtualization, and standardization

I'll try to take good notes tomorrow and get an update together for you then!

Parallels Summit 2008 - Day 1 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

PHP and MySQL Web Development by Luke Welling, Laura Thomson
Barnes & Noble: http://snurl.com/265xp

Why do I like this book so much?

• Teaching by application - The book teaches fundamentals by showing how to apply techniques to an active website. There's not a ton of theory to wade through, and you feel like you're learning the material faster.
• Intertwined strategies - You learn how to get PHP working with MySQL, and then you learn how to optimize your code. It's important to know which work is best done by PHP and which is best done by MySQL. This book teaches both.
• Lots of examples - The CD-ROM comes with tons of code examples that actually relate to something you can use.

I'd be happy to loan my copy, but I've loaned it out and it never returned.

Best PHP and MySQL development book is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

I'll keep updating the blog as I get closer to releasing the next version. Also, as soon as a beta is available, I will certainly let you know!

Taking a couple of weeks off is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Here are the new features:

» Storage engine counts are shown

In addition to the actual size of the tables stored under each storage engine, there is a table count as well. The table count reflects the number of tables that exist on the server which use the specified storage engine.

» Minor changes

• Reduced overall code size
• Optimized subroutines to use fewer system calls and math computations
• Added storage engine disabling recommendations to the bottom of the output

» Bugs fixed

• Fixed an issue that caused incorrect statistics for storage engines with MySQL 3.23
• Corrected a logic bug that displayed odd storage engine statistics calculations

MySQLTuner Revision 29 is now available is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

To pick up the new script, visit the MySQLTuner site.

Here's some of the new features:

» Shows banner of enabled and disabled storage engines

Near the top of the MySQLTuner output, you'll find a line like this:

[**] Status: -Archive -BDB -Federated +InnoDB -ISAM -NDBCluster

Of course, this is shown in color within the terminal, but all enabled storage engines are shown in green with a plus sign at the front. Disabled storage engines are shown in red and are prepended with a minus sign.

» Recommends disabling unused storage engines

If MySQLTuner finds storage engines enabled that are not in use, it will recommend that they are disabled to save resources:

[!!] InnoDB is enabled but isn't being used

» Calculates total data corresponding to storage engines

Any storage engines in use are shown with the total amount of data stored using those engines:

[**] Data in MyISAM tables: 9G
[**] Data in InnoDB tables: 189M
[**] Data in MEMORY tables: 0B

» Displays exact counts in addition to percentages

For some results where percents are shown, exact counts are being displayed as well:

[OK] Slow queries: 0% (1/4M)
[OK] Highest usage of available connections: 56% (17/30)

» Initial InnoDB support

As an initial step towards InnoDB support, the innodb_buffer_pool_size is compared to the total amount of InnoDB data stored on the server:

[OK] InnoDB data size / buffer pool: 189.7M/384.0M

» Other minor changes

• Added additional section headers to further organize the output
• Merged the total buffers lines into one for more compact output
• Added MySQL 5.1 to the supported list to prepare for upcoming GA release
• For filesize amounts less than 1024 bytes, the "B" letter is shown to represent bytes
• InnoDB log file recommendation removed due to bad implementations based on recommendations
• Switches from spaced indents to tabbed indents

MySQLTuner Revision 26 is now available is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

You can read more about the package.

MySQLTuner is now a Debian package is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

» Changed how indexes are calculated on MySQL 5

Thanks to Jon Hinds, I found that when running the tuning script against MySQL 5, the following SQL statement caused MySQL to open all of the tables on the server, which of course caused the table cache hit rate to plummet each time the script is run:

SELECT SUM(INDEX_LENGTH) from information_schema.TABLES where ENGINE='MyISAM'

The script now calculates index size using du operations for all MySQL versions.

» Added checks for innodb_log_file_size

I'm working in some InnoDB support, and the script now checks to see whether the innodb_log_file_size is 25% (+/- 5%) of your innodb_buffer_pool_size.

» Added checks for 32-bit and 64-bit architectures

The script now determines if you have a 32-bit system with less than 2GB of RAM. You'll get a polite suggestion to move to a 64-bit OS so that MySQL can allocate more than 2GB of RAM safely. Also, if your maximum possible memory usage is over 2GB on a 32-bit system, you'll get a warning about stability issues. (Allocating more than 2GB on a 32-bit system can cause thread thrashing and a system crash.)

» Fixed a bug in the recommendations for temporary tables

I had a pretty ugly math error, and it's fixed now. You will see recommendations for increasing the size of the max_heap_table and tmp_table_size buffers as long as they are not at 256MB already.

» Fixed thread cache recommendations and warnings

If your thread cache is set to 0, you now receive a warning about a disabled thread cache. Also, a separate recommendation is made in that situation. If your thread cache is set too low, but still enabled, a separate recommendation will appear.

» Internal changes

Some of the commented lines are switched around a bit, and some of the arrays have been consolidated to speed up the script a bit more.

Ready to download the script? Go to http://mysqltuner.com/ and get it for free.

MySQLTuner Revision 22 is now available is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

More data can be found in the Mandriva Linux Archives.

As an aside, I find it quite interesting that the first package for my script popped up on Mandriva. My first adventure into Linux was with Mandrake many years ago.

MySQLTuner is now a Mandriva Linux package! is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

* Added NetBSD support (thanks to Dave Burgess)
* Switched username/password prompts to STDERR so they won't appear in printouts

Downloads are available, or you can do a new subversion checkout.

MySQLTuner Revision 20 is available is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

* Adjusted infoprint to use asterisks (cosmetic)
* Per-thread/global buffer counts are now displayed
* Key buffer increases are only recommended if the buffer is smaller than total indexes and hit rate is < 95%
* Dropped max_seeks_for_key checks
* Temporary table size increases are not recommended over 256M
* Aborted connection calculation and recommendation adjustments

You can download the latest copy on the MySQLTuner page, and you can get diffs for the new version as well.

MySQLTuner Revision 19 is available is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Some beta testers are already giving it a whirl, and I hope to have it released to the public later next week!

Boxcheck.com improvements is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

• Added aborted connection checks
• Added % reads/writes counts
• Adjusted recommendations for slow query logging, max seeks per key, and joins without indexes
• Added licensing data
• Added props for Matthew Montgomery

Coming up soon is InnoDB support and additional per-thread buffer checks.

MySQLTuner minor update is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

• Specific variable recommendations are made with suggested values as well
• Odd recommendations have been reduced
• Some math errors were corrected
• More configuration items are supported, like table locks, thread caching, table caching and open file limits.

To find out more and to download the script, head on over to mysqltuner.com.

Huge MySQLTuner overhaul is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Also, why should you pay for a web front-end for tools that people have on their servers already?

Check out Boxcheck.com and let me know what you think. I became so fed up, I decided to make my own site that is faster and guaranteed free forever. Period.

Sub-par DNS stuff at DNSStuff is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Query Optimization
I know - you were hoping I'd talk about hardware to start this thing off, but optimizing queries is the #1 way to get a MySQL server in gear. MySQL gives you great tools, like the slow query log, multiple status variables, and the EXPLAIN statement. Put these three things together and your queries will be on their way to a more optimized state. I'll go into great detail about query optimization in a later post.

Memory / System Architecture
We all know MySQL likes RAM, and the more you give it (to a point) the better the performance will be. If you consider the alternative to memory, which is swapping on disk, it's obvious to see the gains.

So why did I add system architecture to this section? Well, if you have 32-bit Redhat, you can only allocate 2GB per process with the standard kernel. If you jump up to the SMP or hugemem kernel (in ES 2.1, you need the hugemem kernel for this to work), you can allocate 3GB per process. There is a caveat - MySQL can only use 2GB per buffer in 32-bit land. In a 64-bit OS with an appropriate Redhat kernel, you can allocate much larger buffers, and this can be tremendously helpful to tables which use the InnoDB engine. The memory allocation abilities are a great benefit, but also keep in mind that you will also get a boost in math performance within MySQL due to the 64-bit architecture. It's a win-win!

Disk Performance
Running a critical database on IDE or SATA drives just doesn't cut it any more. A SCSI or SAS drive is required for the best performance. Although you hope that MySQL doesn't touch the disk much, it's important to remember that you need to make backups often, and you may need to restore data. Also, if your site is write-intensive, the disk performance is much more important than you think. It will reduce the time that tables are locked, and it will also reduce the time for backups and restores.

CPU
Although CPU comes last, don't forget how important it can be. If you run a high number of complex queries and perform a lot of mathematical operations, you're going to need a CPU that can handle this load. Dual CPU's or dual core CPU's will help out even more, since MySQL can use multiple CPU cores to perform simultaneous operations. Keep in mind that 64-bit will outperform 32-bit in MySQL, and also allow for greater memory allocations (look in the Memory section above).

Final Note:
Keep in mind that these are general suggestions, and these suggestions may not apply to all users. For example, on sites that are heavily read-intensive, you may find that CPU speed is more important than disk speed. Also, if you're not using all of the available memory on your server, but your performance is still sagging, adding more memory won't help. Consult with a DBA and find out where your server's slowdowns are, then make a change with your queries or with your hardware. Remember, throwing more hardware at the problem will not always solve it.

Speeding up MySQL is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.