Sticky post

Analyzing logs in real time using Fluentd and BigQuery

This tutorial shows how to log browser traffic and analyze it in real time. This is useful when you have a significant amount of logging from various sources and you want to debug issues or generate up-to-date statistics from the logs. The tutorial describes how to send log information generated by an NGINX web server to BigQuery using Fluentd, and then use BigQuery to analyze the log information. It assumes that you have basic familiarity with Google Cloud Platform (GCP), Linux command lines, application log collection, and log analysis. Introduction Logs are a powerful tool for providing a view into how large-scale … Continue reading Analyzing logs in real time using Fluentd and BigQuery

How to use Cloud Storage and Cloud SQL

In this post, you create a Cloud Storage bucket and place an image in it. You’ll also configure an application running in Compute Engine to use a database managed by Cloud SQL. For this lab, you will configure a web server with PHP, a web development environment that is the basis for popular blogging software. Outside this lab, you will use analogous techniques to configure these packages. You also configure the web server to reference the image in the Cloud Storage bucket. Objectives In this lab, you learn how to perform the following tasks: Create a Cloud Storage bucket and … Continue reading How to use Cloud Storage and Cloud SQL

Sticky post

Use Of Cloud IoT Core

Use IoT Core to create a registry Use IoT Core to create a device Use Stackdriver Logging to view device logs Enable APIs In this section, you check that all the APIs you will use in this lab are enabled. In the GCP Console, on the Navigation menu (), click APIs & Services. Scroll down and confirm that your APIs are enabled. Cloud IoT API Cloud Pub/Sub API Container Registry API If an API is disabled, click Enable APIs and services at the top, search for the API by name, and enable it for your project. Make sure you are in the correct Qwiklabs project. … Continue reading Use Of Cloud IoT Core

Sticky post

Query GitHub data using BigQuery

BigQuery is Google’s fully managed, NoOps, low cost analytics database. With BigQuery you can query terabytes of data without needing a database administrator or any infrastructure to manage. BigQuery uses familiar SQL and a pay-only-for-what-you-use charging model. BigQuery allows you to focus on analyzing data to find meaningful insights. In this post we’ll see how to query the GitHub public dataset to grab hands on experience with it. Sign-in to Google Cloud Platform console (console.cloud.google.com) and navigate to BigQuery. You can also open the BigQuery web UI directly by entering the following URL in your browser. Accept the terms of service. … Continue reading Query GitHub data using BigQuery

Sticky post

Recommend Products using ML with Cloud SQL and Dataproc

As our goal is to provide demo that is why we are using the Cloud SQL or else yo can use spanner for horizontal scaling. our goal is to Create Cloud SQL instance Create database tables by importing .sql files from Cloud Storage Populate the tables by importing .csv files from Cloud Storage Allow access to Cloud SQL Explore the rentals data using SQL statements from CloudShell  the GCP console opens in this tab.Note: You can view the menu with a list of GCP Products and Services by clicking the Navigation menu at the top-left, next to “Google Cloud Platform”.  you populate rentals … Continue reading Recommend Products using ML with Cloud SQL and Dataproc

Sticky post

Spark Cluster Overview

Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming. Security in Spark is OFF by default. This could mean you are vulnerable to attack by default. Spark uses Hadoop’s client libraries for HDFS and YARN.  Users can also download a “Hadoop free” binary and run Spark with any Hadoop version by augmenting Spark’s classpath. Scala and Java users can … Continue reading Spark Cluster Overview

Sticky post

Bayes Classification with Cloud Datalab, Spark, and Pig on Google Cloud

Note: If you are really following with post this job can take upto 1:30 hours to finish and if you stuck in a typo it will increase your resistance power In this post you will learn how to deploy a … Continue reading Bayes Classification with Cloud Datalab, Spark, and Pig on Google Cloud

Sticky post

Building an IoT Analytics Pipeline on Google Cloud Platform step by step

let’s start with the definition of IoT: The term Internet of Things (IoT) refers to the interconnection of physical devices with the global Internet. These devices are equipped with sensors and networking hardware, and each is globally identifiable. Taken together, these capabilities … Continue reading Building an IoT Analytics Pipeline on Google Cloud Platform step by step