Artificial intelligence could be one of the most significant technological revolutions of the modern world. It will open up endless possibilities for the transformation of companies and businesses. Major companies are already taking full advantage of the benefits of AI. Organisations such as TIME are exploring the new possibilities of generative AI to improve their customer experience and strengthen brand engagement. Wendy's is applying innovative order management processes to the fast food industry. Orange is exploring next-generation contact centres, and Priceline is using AI capabilities in BigQuery alongside its own algorithms. This is helping to deliver unique services and product recommendations to consumers. New capabilities expand opportunities for innovation.

Innovative implementations

Data plays a central role in artificial intelligence, which is why we're introducing new innovations to our cloud-based data and AI services to help businesses make the most of their data.

• BigQuery Studio is an interface that helps unify your data and workflows. It delivers predictive analytics that simplify end-to-end data processes. Additional features provide a convenient platform for processing information, including advanced support for unstructured data, cloud analytics, and secure data sharing and management.
• Bring artificial intelligence to your data in BigQuery by integrating with core Vertex AI models. New innovations help you confidently implement generative AI on your business data. You can deploy models in real-time and use vectors.
• Data teams are more productive with the Duet AI preview on Google Cloud. It allows you to rethink the way you work with data in collaboration with products like Looker, BigQuery and Dataplex.

These innovative processes enable organisations to harness the power of data and artificial intelligence to achieve business goals, including personalising customer interactions, improving supply chain efficiency, reducing operating costs and increasing profitability.

Connect end-to-end workflows and data with BigQuery

Data teams typically use a variety of tools to manage data and machine learning. This can lead to a loss of productivity. To simplify working with data and unify different aspects of data management, Google is introducing an innovative tool - BigQuery Studio. This tool provides customers with a single interface for analysing data in the Google Cloud environment. Now you can combine data engineering, analytics and predictive analytics in one place, simplifying the work of your teams. This eliminates the need to switch between different tools.

"Shopify has invested significant resources in building a team with diverse capabilities to stay at the forefront of data science and engineering. In the early stages of testing BigQuery Studio, we were impressed by Google's ability to combine different tools for different categories of users into a single, simplified interface. This allows us to break down internal team barriers without sacrificing the scalability we expect from BigQuery. - Zach Roberts, Data Engineering Manager, Shopify.

BigQuery Studio allows you to work with SQL, Python, Spark and other languages. This approach greatly simplifies petabyte-scale analytics without adding infrastructure management costs. Laptops and desktops are considered the most convenient tool for writing and editing Python code, so Google has integrated BigQuery Studio with Colab Enterprise. This new offering provides support for Google Cloud enterprise security and regulatory compliance. Colab Enterprise is a modification of the popular Colab scientific laptop developed by Google Research.

New capabilities and innovations

Google is expanding its partnerships with companies like Hex, Deepnote and Jupyter to provide more flexibility in data science. DataFrame in BigQuery provides teams with a data structure to access their favourite notebooks with large amounts of data that typically exceed memory limits.

Distributed and unstructured data presents additional challenges for teams working with it. A lot of data is contained in documents, video files and audio recordings that can be used with generative artificial intelligence. Google is creating new ways to combine your structured business data with unstructured data, providing secure access to it without the need to move it around.

These innovations include:

1. Open source code and improved format support introduced in Hudi and Delta Lake in BigLake to break down siloed data warehouses. Since customers began using BigLake to combine data lake workloads and cloud storage, the size of data lake workloads has grown 27-fold to hundreds of petabytes. These improvements have led to an acceleration of Apache Iceberg performance, enabling continuous data optimisation.
2. BigQuery Omni's materialised insights and cross-cloud joins enable data training and analytics improvements without moving data. Organisations now have the ability to combine data from multiple clouds into a single repository. And Spark's integration with Google Distributed Cloud increases the ability to process analytic queries locally to support data residency requirements.
3. Management innovations in Dataplex allow users to more effectively manage data, its quality and metadata. This helps to understand what data should be analysed and even train machine learning models. This increases the accuracy of results.
4. New privacy-focused connections include BigQuery's empty data rooms and Ads Data Hub. These tools will help you get up-to-date information about the effectiveness of your advertising campaigns on Google and YouTube.

Using artificial intelligence to manage, create and scale data

The importance of data science and artificial intelligence continues to be at the forefront of our customers' minds. Many teams are using BigQuery ML as a key tool to start building their ML models. They are also using their own analytical warehouses and data lakes to build machine learning models. Over the past two years of using BigQuery ML, the number of queries has increased by more than 250%. This year, millions of predictive and learning queries have been run on BigQuery ML.

Google provides access to the underlying Vertex AI models, including PaLM 2, directly from BigQuery. This improves the understanding of your data using generative artificial intelligence, simplifying the process and allowing data teams to easily scale from simple SQL queries to large language models.

With the new modelling functionality in BigQuery, customers can now perform model inference in a variety of formats including TensorFlow, ONNX and XGBoost. With the new real-time inference capabilities, you can detect patterns and automatically generate alerts. This allows you to get up-to-the-minute information and respond quickly.

The Faraday platform, which specialises in predicting user behaviour, previously had to build data pipelines and combine disparate sets of information. Now they can not only simplify the analysis of consumer sentiment, but also integrate this sentiment with additional data from primary sources and use it in LLMs (language models with deep learning capabilities). This is necessary to create customised content within BigQuery.

Google is implementing semantic and vector search in BigQuery to adapt the model. Vector and text embeddings allow unstructured data, such as photos or text, to be efficiently searched and analysed. This innovation enables a new generation of AI applications to more efficiently access unstructured data and provide context for further analysis using language models. With the Vertex AI Feature Store, users can automatically synchronise vector embeds with BigQuery.

The ability to obtain accurate data is critical for developing and training new AI models. This is especially true for specialised models used in industries such as financial services, retail and manufacturing. Google provides access to a variety of data sets in BigQuery from leading data providers such as CoreLogic, Dun & Bradstreet and TransUnion. Through expanded partnerships, consumers can access thousands of datasets from hundreds of providers, including Acxiom, Bloomberg, Equifax, Nielsen and Zoominfo. The availability of these datasets positions Google Cloud as the leading service for businesses to build and train new AI models. With Google Cloud, customers can leverage a wide range of data to improve their solutions and applications.

Improve analytics team productivity with Duet AI

To simplify daily tasks and increase the productivity of analytics teams, Google has launched a preview of Duet AI, a persistent generative collaborator based on artificial intelligence. This tool is already used in several products including Looker, BigQuery and Dataplex. These updates, based on Google's latest foundational models, help data teams clean and prepare data for analysis, answer questions and predict trends.

To provide information in plain language to ordinary consumers, Google has introduced Duet AI in Looker. This tool allows you to build simple conversational queries that allow you to refine the results in the form of visualisations and reports, and get answers to them. Duet AI automatically creates presentations with intelligent summaries, formulas and visual support. It can also generate code using LookML with an understanding of user intent. This makes data analysis and presentation more accessible and effective for a wide range of users, regardless of their technical skills.

With natural language search and generative artificial intelligence, you can 'talk' to your business data in the same way you 'ask questions' to Google. All of this is implemented in Duet AI in Looker. This is the equivalent of having an expert data analyst at every employee's fingertips. This makes analysing and interacting with data more intuitive and accessible, driving productivity and informed decision-making at all levels of your organisation.

Duet AI in BigQuery is an integrated collaboration tool available directly from the BigQuery interface. It provides support for writing SQL queries and Python code. This enables teams to achieve better results and focus on analysis. Duet AI can automatically suggest code in real time, generate complete functions and code blocks, and recommend bug fixes. Duet AI in Dataplex also enables natural language metadata search to provide visibility into your machine learning assets and datasets.

BigQuery's AI Duet provides awareness and capabilities across Google Cloud's integrated data platforms. We see this as an architectural advantage that eliminates the need to build, train, deploy and manage your own specialised AI models. This simplifies the workflow for data teams, enabling them to gain valuable insights faster with the intuitive and context-aware Duet AI tool.

Aritzia

Vice President of Data Engineering

Conclusions

Simplicity and scale built for the age of artificial intelligence. Google has changed the way the world thinks about information. Using cloud technologies and artificial intelligence, Google brings new levels of scalability, security and intelligence to your business data.