Comment on – Big data for Smart cities – Use of real time Big Data algorithms in smart cities

Pankaj Berde  sent an excellent comment below on - Big data for Smart cities – How do we go from Open Data to Big Data for Smart cities specifically on the use of real time Big Data algorithms in smart cities

Very well written white paper. One comment, HOP has fallen behind from its initial promise. The code is dormant for almost 2 years with very little interest. Two other initiatives are very prime for realtime/online/stream processing of big

 data. Twitter open-sourced Storm (http://storm-project.net/) which is nicknamed as realtime Hadoop by many. Also Berkeley Spark (http://www.spark-project.org/) is gaining popularity for being a fast in-memory alternative to Hadoop. and its streaming version (http://www.cs.berkeley.edu/~matei/papers/2012/hotcloud_spark_streaming.pdf) might be another interesting technology for realtime analytics.


Another line of thought for realtime big data analytics is shift towards machine learning models . The NLP and Genome algorithms fit well for smart city big data insights. Mahout is another tool in this area. Still lot more needs to be done in moving forward from traditional data warehouse analogy of collect, mine, analyze, detect, predict insights.
A use case comes to mind is detecting crime/terrorist activities before it strikes using realtime machine learning techniques. The DW analogy to get insights will be too late in generating any meaningful insight.