Clear those hurdles

How to overcome 5 common database challenges

Oren Eini
© Shutterstock / Simona Balconi

Choosing the right database can be a challenge in itself thanks to the many options available today. However, once you clear that hurdle, more remain. In this article, Oren Eini broadly explores five of the most common problems for database engineers and talks about solutions for each one.

For decades, databases were a known quantity. In the same sense that you could get a Ford in any color you wanted as long as it was black, we always knew what database to use to solve the next challenge. Relational databases were the only answer for pretty much any question.

The size and scale of today’s data is changing at a rapid pace. Today there are over 3 billion smartphones and 20 billion smart devices in use all collecting, analyzing, and sharing data. In the next 5 years, both of these numbers will double, increasing the amount of data available to us exponentially.

Database models that have been reliable since the Bee Gees are struggling to fit into a world that is getting more distributed and reliant on ever increasing amount of data. With the explosive growth in capabilities, scope and complexity of what users expect from applications, it is no longer possible to settle for the same decisions of yesterday.

There are a large number of databases available today, each with their own distinct advantages and disadvantages. The best way to determine the right database for your needs is to ask yourself, what are the primary challenges I want my database to meet? Once you know the biggest obstacles to spinning your information into gold, you can more easily assess each solution based on how they scale these pitfalls and can push your organization forward.

SEE ALSO: Truth first, or why you should mostly implement database first designs

Here are the 5 most common database challenges and how a quality database will overcome them.

1. Data security

In the last two years, over 100,000 systems were hacked into because their database had been left completely exposed on the public internet. The irony is that these databases weren’t vulnerable because of a security issue, their security document for the database was over 60 pages long. How many people actually got to reading and implementing proper security practices!

Today’s database must come with security that’s easy to set up, quick to notify users if something is wrong and protects both data at rest and in transit. Given the critical nature of losing access to your data, or worse, having the data at the hands of a (nefarious) 3rd party, delayed detection is not an option. The decision should really be either using the database’s built-in security, sticking with what your organization is already using, or reinforcing your application with both.

2. Performance

Databases have always faced harsh performance criteria. The amount of data you are taking in may be expanding, but to satisfy your users you need to process it faster, without making them jump through hoops to get there. Superior performance involves deployment with good default, being able to recognize an operational environment and adjusting automatically and getting to a place where good performance doesn’t necessitate a full-time babysitter. A database must be built for high-performance, regardless of the hardware. If a database performs well on older machines, and smaller machines like Raspberry Pis, it has good “native” performance.

SEE ALSO: “The database development can be built into DevOps!”

3. Data safety

Data needs to be processed in a way that ensures nothing is lost. Even in a nonrelational database or in a distributed cluster, there must be ACID guarantees both across the database and throughout the cluster. The challenge for fully-transactional databases is performance. ACID costs time. A good database is one that can give you both superior performance and transactional guarantees.

4. Resource utilization

It used to be assumed databases would run on the best hardware available and they behaved accordingly. Today, with the prevalence of containers and VM deployment, that isn’t the case. To keep up performance in the face of limited computing power and handle larger amounts of more complex data, a database must get 100 percent out of the resources it is working on for every nanosecond it is running.

5. High availability

In a single server database, if your server goes down, your organization goes down. A data cluster consisting of several servers working together gives you multiple levels of backup. For e-commerce, it is similar to installing extra cash registers to handle a growing line of people. A distributed database also has assignment failover so when a node performing a specific task goes down, that task is automatically transferred to a working node. Writing a transaction to a local disk is a lot faster than having to shuffle it across a distributed network. A good database takes that into consideration and creates innovative solutions to maintain your performance along with guaranteeing always on availability.

SEE ALSO: Don’t press that button: Operational challenges of relational databases in the cloud

In conclusion

Considering these five challenges and solutions is a helpful guide to choosing the right database your company needs to succeed. With databases being the core software resource your business depends on, it is a decision you need to get right the first time around.


Oren Eini

Oren Eini, CEO and founder of Hibernating Rhinos, has more than 20 years of experience in the development world with a strong focus on the Microsoft and .NET ecosystem. Recognized as one of Microsoft’s Most Valuable Professionals since 2007, Oren is also the author of “DSLs in Boo: Domain Specific Languages in .NET.” He frequently speaks at industry conferences such as DevTeach, JAOO, QCon, Oredev, NDC, Yow! and Progressive.NET. An avid blogger, you can also find him under his pseudonym as Ayende Rahien.

Inline Feedbacks
View all comments