For AI success, developers should collaborate more efficiently with data scientists & engineers
The future is bright for AI. Whether data science can change the world or not, that remains to be seen. One thing is sure though: developers should collaborate more efficiently with data scientists & engineers. We talked to David Wyatt, Vice President EMEA at Databricks about the challenges when approaching AI initiatives, the role of Spark in this field and the next step for data engineering and data science teams.
JAXenter: How involved are the public cloud vendors in driving ML / AI, compared with streaming and modelling?
David Wyatt: Cloud vendors are increasingly involved as they see ML/AI workloads as a valuable driver of highly scalable resource/compute consumption. Streaming is also a focus, but it’s one of many ways to feed data into an ML/AI model.
The explosion of available ML frameworks and tools will drive cloud vendors to offer integrations so that data scientists can leverage all of these tools on cloud infrastructure.
JAXenter: Are organizations working well around AI and ML, or are they facing real problems today?
David Wyatt: Organizations are facing significant challenges when approaching AI initiatives. The primary challenge they are facing revolve around the divide between data and AI. If you look at it from a macro-level, only a handful of companies have been successful with AI. We call it the “1% problem”. The other “99%” are struggling with data and AI silos.
Databricks recently commissioned a survey of enterprise users pursuing AI initiatives and came away with some interesting data points around the adoption and challenges of leveraging AI.
- 96% cite that data silos are the #1 challenge to AI success
- There is also a major disconnect between data science and engineering with 80% of companies citing collaboration being a major blocker
- Also, companies are using an average of 7 different ML/DL frameworks and tools which adds complexity to the process
- Finally, it’s taking on average 7 months for companies to bring AI projects to production.
JAXenter: What steps can developers take around this, and what can other teams do?
David Wyatt: Developers can find ways to collaborate more efficiently with data scientists and engineers. Bridging that gap will improve overall productivity – accelerating innovation while reducing operational costs.
Adopt a unified approach where not only the teams involved in building AI applications are working more collaboratively, but you also unify the entire workflow and relevant tools from data ingest and preparation to model training and deployment.
If you look at it from a macro-level, only a handful of companies have been successful with AI.
JAXenter: How do you see Spark developing within this?
David Wyatt: Spark is the first unified analytics engine that introduces Big Data processing at massive scale and combines that processing with a platform for Machine Learning. Spark has become wildly popular over the last 5+ years because it uniquely solves these Big Data and AI problems. I think the continued focus on unifying data and AI is what will continue to set Spark apart from other analytics engines.
JAXenter: What’s next for the teams around data engineering and data science?
David Wyatt: The future is really bright for AI. I think data science is going to change the world. The next step for data engineering and data science teams to continue to find more efficient ways to work collaboratively together and to leverage that best available technologies and frameworks to extract valuable insights from data to drive business innovation.