What is the Modern databases
What is a contemporary database?
we’ve some terms that wander between marketing and technical descriptions – NewSQL, NoSQL. we’ve much needed work on write-optimized database algorithms – Tokutek, LevelDB, RocksDB, HBase, Cassandra. We also get reports of wonderful performance. i feel there’s an excessive amount of specialize in peak performance and not enough on predictable performance and manageability.
Building a DBMS for production workloads is tough . Writing from scratch is a chance to try to to tons better than the products that you simply hope to exchange . it’s also a chance to repeat many mistakes. you’ll avoid a number of the mistakes by getting advice from someone who features a lot of experience supporting production workloads. I worked at Oracle for 8 years, wrote some good code (new sort!) and glued tons of bugs but never got anywhere near production.
Common mistakes include insufficient monitoring and poor manageability. Monitoring should be simple. i would like to understand where something is running and not running (waiting on IO, locks). I also want to drill down by user and table — user & table aren’t just there for access control. i’m SQL-centric in what follows. While there are frequent complaints about optimizer making bad choices I can only imagine what proportion fun it’ll be to debug load problems when the query plan is hidden away in some external application.
The best time to believe monitoring is after spending an excessive amount of time debugging a drag . At that time you’ve got a far better idea about the info that might have made things easier. One example of missing monitoring was the shortage of disk IO latency metrics in MySQL. In one case not having them made it much easier to not notice that the oversubscribed NFS server was making queries slow via 50 millisecond disk reads.
Monitoring should be cheap in order that it can always be enabled and from this I can understand the typical costs and spot changes in load from the weekly push. But I also got to debug some problems manually so i want to watch both sessions that i do know are too slow (get the query plan for a running SQL statement) and to seek out sessions/statements that are too slow (dump things into the slow query when certain conditions are met). Letting me do EXPLAIN for statements in my session is beneficial , but i actually got to do EXPLAIN from statements in production – if the optimizer uses sampling i would like to ascertain the plan they get and if temp tables are involved I even have no idea what is going to be in their temp tables. MariaDB (and MySQL) recently added support to point out the query plan for statements that are currently running. are often “> this is often even more useful when the query plan and performance metrics can be dumped into the slow query log when needed.
The goal for monitoring performance problems is to eliminate the utilization of PMP. once I want to know why a system is running slower than expected I frequently check out thread stacks from a running server. I hope at some point that pstack + awk isn’t the simplest tool for the work on a contemporary database. i used to be debugging a drag with RocksDB while scripting this . The symptom was slow performance for a read-only workload with a database that does not slot in cache. I even have seen the matter previously and was quickly ready to find out the cause — the block cache used too many shards. Many problems are easy to debug once you have previously experienced them. this will be restated as many problems are expensive to debug for many users because they do not have full time database performance experts.
The focus on peak performance are often at odds with manageability. The faster thanks to peak performance is via tuning options and too often these options are static. Restarting a database in production to vary an option may be a bad idea. Dynamic options are an improvement. Using adaptive algorithms in situ of the many options is even better. And if you add options, confirm the defaults are reasonable.
Predictable performance is a component of manageability. How does your modern database behave when confronted with an excessive amount of load? It helps once you can classify your workload as high-priority and best-effort then shed load from the best-effort users. Alas this needs how to differentiate users. In theory you would like the hi-pri users to urge their work done then let best-effort users compete for the spare capacity. this needs some notion of SLA for the hi-pri users and spare capacity for the DBMS. These are hard problems and that i haven’t used an excellent solution for load shedding.
This could be a way larger rant/document but i need to return to my performance debugging.
Consider Reading to these articles: