Creating A Tactical Architecture Using Google Cloud Big Data Products

With the growing norm of remote work and collaboration, the ability to provide uninterrupted access to data and analytics tools is a serious concern. In this blog, John Loughlin explains how using Google Cloud’s Big Data Products can ensure teams across the world have unfettered access to the insights needed to tackle the challenges their teams are facing today.

The COVID-19 pandemic is posing a host of challenges for businesses and entities trying to adapt to the new norm of remote working and collaboration. At a time when leaders need rapid insights, having access to the latest analytics is now more important than ever before. Unfortunately, many of their analytical systems are struggling due to overloaded networks or employees not being able to access the on-premise tools they have relied on in the past. 

Data processing is a complex endeavor, even in normal times. Teams rely on these complex processes to convert transactional data into dashboards and reports to help them effectively operate and optimize their business and organizations. At a time where every enterprise needs to show flexibility, losing access to these analytical systems is crippling.

At Cloudreach we have been receiving hundreds of inquiries from leaders across the public and private sectors trying to keep their critical business systems functioning while they adjust to this new normal. Issues such as:

  • Overloaded network preventing data pipelines from gathering data from source systems
  • Analysts unable to access data tools running on local desktops
  • Data processing servers being repurposed or shared with other critical IT functions
  • Performance issues due to increased network latency

This is a challenging situation. The expectation that anyone should be able to somehow navigate all this change and do it while flying blind is unrealistic. Further, lack of visibility presents real risks. Competitors may not be so ill-equipped or may be more flexible and better able to adapt to these changes.  A lack of insight puts your organization at a competitive disadvantage. In the public sector, there’s a pressing need to react and respond to quickly evolving circumstances.

Basic Data Processing Architecture

So, what can be done now to provide better insight into current activity and the context in which you're operating?  

Google is renowned as the industry leader for data and security, demonstrated by its consumer products. So we suggest taking your available data and using the tools provided by Google Cloud to gain the necessary visibility and make better-informed business decisions.

Using GCP services such as BigQuery and DataPrep make it possible to create a basic data processing architecture in a matter of hours. There are no servers to install and you only pay for what you use. Prior experience with Google Cloud is always helpful but we are finding that most data professionals are able to pick up the skills rather quickly. You’ll also get free usage, up to monthly limits for select Products, including those we feature. 

Let’s review a basic architecture we have been recommending to our customers:

Step 1: Load data into Google Cloud Storage

After gathering the relevant data, load it on to Google Cloud Storage. Google’s platform allows for the creation of a single, shared source of the data to work allowing for greater convenience in access as well as greater security over data access.

Step 2: Use Cloud Dataprep to transform data

Google’s DataPrep is a cloud data service used to explore, clean, and prepare data for analysis and machine learning. DataPrep is serverless and places no infrastructure burden on you while allowing distributed teams to work together on preparing the data.

Step 3: Use BigQuery to analyze data using SQL

Analyze the data by loading it into Google BigQuery, an enterprise data warehouse that runs on Google’s infrastructure. Again, no infrastructure burden so you can spend your time analyzing your data. BigQuery supports SQL, a well-known industry standard query tool, and allows you to run queries from the console, a web UI, a command line or a REST API.

Step 4: Visualize data using Looker or Data Studio

Once analyzed, share the results by creating dashboards and reports allowing you to explore your data and create coherent stories with it using Google’s Looker or Data Studio. Here your exploration can be entirely graphical, making it usable and useful to the broadest possible range of people in your organization.

Once your foundational setup has been implemented, Google provides access to a large repository of high-quality public data sets and commercial ones too. Through bringing together these data sources with your own, this can bring fresh perspectives to strengthen your insights, as well as the opportunity for you to collaborate with your vendors and partners in new ways. As a timely illustration, Microsoft Research and Google Cloud are collaborating on a public dataset to help researchers combat the COVID-19 novel coronavirus.

These are challenging times and having access to the latest analytics is critical for leaders across all sectors. What is clear are the simple steps available to extend your data analytics capabilities, to meet some of the challenges presented by the current environment.

For more information about our Google Cloud Practice, click here. If your business needs help overcoming any business continuity challenges, you can book a free session with one of our Cloud Strategist to discuss how we can help you dynamically respond to your changing world.