Tag: gcp

Data EngineerLeave a comment
Sticky post

Analyzing logs in real time using Fluentd and BigQuery

SATYAJIT MAITRA

This tutorial shows how to log browser traffic and analyze it in real time. This is useful when you have a significant amount of logging from various sources and you want to debug issues or generate up-to-date statistics from the logs. The tutorial describes how to send log information generated by an NGINX web server to BigQuery using Fluentd, and then use BigQuery to analyze the log information. It assumes that you have basic familiarity with Google Cloud Platform (GCP), Linux command lines, application log collection, and log analysis. Introduction Logs are a powerful tool for providing a view into how large-scale … Continue reading Analyzing logs in real time using Fluentd and BigQuery

cloud, Data Engineer1 Comment
Sticky post

Use Of Cloud IoT Core

SATYAJIT MAITRA

Use IoT Core to create a registry Use IoT Core to create a device Use Stackdriver Logging to view device logs Enable APIs In this section, you check that all the APIs you will use in this lab are enabled. In the GCP Console, on the Navigation menu (), click APIs & Services. Scroll down and confirm that your APIs are enabled. Cloud IoT API Cloud Pub/Sub API Container Registry API If an API is disabled, click Enable APIs and services at the top, search for the API by name, and enable it for your project. Make sure you are in the correct Qwiklabs project. … Continue reading Use Of Cloud IoT Core

cloudLeave a comment
Sticky post

Using Pubsub to publish messages

SATYAJIT MAITRA

Google Cloud Pub/Sub is a messaging service for exchanging event data among applications and services. A producer of data publishes messages to a Cloud Pub/Sub topic. A consumer creates a subscription to that topic. Subscribers either pull messages from a subscription or are configured as webhooks for push subscriptions. Every subscriber must acknowledge each message within a configurable window of time. the GCP console opens in this tab.Note: You can view the menu with a list of GCP Products and Services by clicking the Navigation menu at the top-left, next to “Google Cloud Platform”.  The Google Cloud Shell Activate Google Cloud Shell Google … Continue reading Using Pubsub to publish messages

cloudLeave a comment

IOT in GCP

SATYAJIT MAITRA

Security is critical when deploying and managing an IoT network. Cloud IoT Core has several security features to protect your IoT network. Devices are authenticated individually. Which means if there is an attack on your IoT network it is limited to one device and not the whole fleet. There are four public key formats available for devices: RS256 and RSA256_X509, and ES256 and ES256_X509. You specify the key format when creating the device. You can also define an expiration time for each device credential (public key). After it expires, the key is ignored but not automatically deleted. If you don’t … Continue reading IOT in GCP

cloudLeave a comment
Sticky post

Cloud IOT Core

SATYAJIT MAITRA

Cloud IoT Core is a fully managed service that allows you to easily and securely connect, manage, and ingest data from millions of globally dispersed devices. Cloud IoT Core, in combination with other services on Google Cloud platform, provides a complete solution for collecting, processing, analyzing, and visualizing IoT data in real time to support improved operational efficiency. You will transmit telemetry messages from a device and the device will respond to configuration changes from a server based on real-time data. The devices in this system publish temperature data to their telemetry feeds, and a server consumes the telemetry data … Continue reading Cloud IOT Core

big data, Data Engineer, google cloud, machine learning2 Comments
Sticky post

Recommend Products using ML with Cloud SQL and Dataproc

SATYAJIT MAITRA

As our goal is to provide demo that is why we are using the Cloud SQL or else yo can use spanner for horizontal scaling. our goal is to Create Cloud SQL instance Create database tables by importing .sql files from Cloud Storage Populate the tables by importing .csv files from Cloud Storage Allow access to Cloud SQL Explore the rentals data using SQL statements from CloudShell  the GCP console opens in this tab.Note: You can view the menu with a list of GCP Products and Services by clicking the Navigation menu at the top-left, next to “Google Cloud Platform”.  you populate rentals … Continue reading Recommend Products using ML with Cloud SQL and Dataproc

cloud, google cloud, machine learningLeave a comment
Sticky post

Build simple Apps that can convert text-to-speech and speech-to-text but in c#

SATYAJIT MAITRA

As a developer back in 2017 I always wonder it will be nice to write Machine learning code in c# .Net framework to show my manager that i know enough to become Team Lead but past is past and i left that productive company most of the company manager’s in the world are same full with dull insights as they tries to bring people down and demotivate them from their goal as they didn’t get their anyways the other day i was searching memes in the internet and all of a sudden one of the website gives me two HD … Continue reading Build simple Apps that can convert text-to-speech and speech-to-text but in c#

big data, cloud, Data Engineer, data science, google cloud, machine learningLeave a comment
Sticky post

Bayes Classification with Cloud Datalab, Spark, and Pig on Google Cloud

SATYAJIT MAITRA

Note: If you are really following with post this job can take upto 1:30 hours to finish and if you stuck in a typo it will increase your resistance power In this post you will learn how to deploy a … Continue reading Bayes Classification with Cloud Datalab, Spark, and Pig on Google Cloud

cloud, google cloudLeave a comment

How To Set up a Development Environment in Python in Google Cloud

SATYAJIT MAITRA

In this blog, you set up a Python development environment on Google Cloud Platform, using Google Compute Engine to create a virtual machine (VM) and installing software libraries for software development. You perform the following tasks: Provision a Google Compute Engine instance. Connect to the instance using SSH. Install a Python library on the instance. Verify the software installation. Compute Engine is just one resource provided on Google Cloud Platform. Google Cloud Platform Google Cloud Platform (GCP) consists of a set of physical assets, such as computers and hard disk drives, and virtual resources, such as virtual machines (VMs), that … Continue reading How To Set up a Development Environment in Python in Google Cloud

cloud, data science, google cloud, machine learningLeave a comment
Sticky post

Analyzing Financial Time Series Using BigQuery and Cloud Datalab

SATYAJIT MAITRA

This solution illustrates the power and utility of BigQuery and Cloud Datalab as tools for quantitative analysis. The solution provides an introduction (this document) and gets you set up to run a notebook-based Cloud Datalab tutorial. If you’re a quantitative analyst, you use a … Continue reading Analyzing Financial Time Series Using BigQuery and Cloud Datalab