Working with language data is always challenging, especially capturing the nuances of individual responses within very large datasets. I have been working with language data my whole career and have recently begun creating technology that can effectively handle these problems. The availability of large public datasets has made some truly remarkable things possible and it is an exciting and fast-paced field to be working in.
The business case for cognitive technology is obvious: any large enterprise has growing amounts of unstructured language data pouring in from areas including market surveys, help desk inquiries, complaints, call centers, frontline employee feedback, online reviews, and social media. If we can solve some of the problems with extracting insights from language data at scale, businesses can better understand their customers, improve customer experience and bring better products to market.
I created the cognitive technology that allows my company’s Automated Neural Intelligence Engine (ANIE) to understand language. Building ANIE constantly humbles me. There have been so many instances where the intelligence of the system surprises me with insight that I had not expected. For example, ANIE understands over two dozen common qualities of food including taste, texture, and value. She began to understand these concepts on her own, without any intervention from me or my team. Right now, she understands the concepts well enough to group the themes together into coherent concepts thanks to our unique unsupervised learning methodology. Here are just a few of the results that ANIE has given us.