Your customers don't want to wait for data - both customers and staff expect instantaneous information.
Fortunately, a rich combination of technologies now exist which can make that a reality.
We call the combination of traditional relational systems, distributed processing, and NoSQL solutions Real Time Information.
Real world examples of real time information include applications like Google Search (indexing hundreds of terabytes of data), Google Analytics, Facebook, Twitter, LinkedIn, and more. These applications are able to create the illusion of instantaneous responses on data sets involving billions of records.
Typically, real time information challenges are described by the volume of data and/or the (apparent) requirement for batch processing. Typical hammers (SQL and traditional BI tools) often are unable to deal with the processing requirements without a combination of import/export and batch processing.
Not Just SQL
In addition to traditional SQL packages, an emerging group of NoSQL packages provides radically different performance characteristics and capabilities. Some of these packages include:
We have found that the best solutions often involve combining these technologies in interesting ways, often in conjunction with existing data storage solutions. For example, combining Neo4j with MongoDB provides a fast, highly performant solution for managing both documents and relationships in real-time - with billions of documents and relationships.
Dev9 has helped many clients implement real time solutions, augmenting or replacing slow or cost-ineffective packages. Our model is straight-forward:
- Identify goals. In particular, a few specific scenarios that are currently impossible to achieve given the existing technology.
- Conduct a to-scale proof-of-concept. This includes both generating and/or importing test data at scale (typically hundreds of millions, billions or more records) as described by the goals, and then conducting load testing to ensure the technology is a good fit.
- Move to production. This includes working with your development and operations team[s] to provide needed hands on training.
For more information about our experience with Real Time Information, including Big Data and NoSQL packages, drop us a line.