insights from E-Commerce retail data set

December 7, 2019 SATYAJIT MAITRA

We are using Bigquery as our data warehouse solution and using standard SQL as query language . For dataset we use Google’s Google Analytics logs of an merchants website. You need to enable your bigquery account which has a daily limit and there after it is cost effective. Click Navigation menu > BigQuery. Click Done. BigQuery public datasets are not displayed by default in the BigQuery web UI. To open the public datasets project, open https://console.cloud.google.com/bigquery?p=data-to-insights&page=ecommerce in a new browser window. In the left pane, in the Resource section, click data-to-insights. In the right pane, click Pin Project. Explore ecommerce data Problem : Your data analyst team exported … Continue reading insights from E-Commerce retail data set

big data, cloud, data science, UncategorizedLeave a comment

BigQuery ML(move your model towards data and not data towards model)

November 24, 2019 SATYAJIT MAITRA

Overview BigQuery ML enables users to create and execute machine learning models in BigQuery using standard SQL queries. BigQuery ML democratizes machine learning by enabling SQL practitioners to build models using existing SQL tools and skills. BigQuery ML increases development speed by eliminating the need to move data. BigQuery ML functionality is available by using: The BigQuery web UI The bq command-line tool The BigQuery REST API An external tool such as a Jupyter notebook or business intelligence platform Data Analyst is a Machine learning engineer now? Machine learning on large data sets requires extensive programming and knowledge of ML frameworks. These … Continue reading BigQuery ML(move your model towards data and not data towards model)

big dataLeave a comment

.Net framework and Apache spark

November 21, 2019 SATYAJIT MAITRA

Why choose .NET for Apache Spark? .NET for Apache Spark empowers developers with .NET experience or code bases to participate in the world of big data analytics. .NET for Apache Spark provides high performance APIs for using Spark from C# and F#. With C# and F#, you can access: DataFrame and SparkSQL for working with structured data. Spark Structured Streaming for working with streaming data. Spark SQL for writing queries with SQL syntax. Machine learning integration for faster training and prediction (that is, use .NET for Apache Spark alongside ML.NET). .NET for Apache Spark is compliant with .NET Standard, a formal … Continue reading .Net framework and Apache spark

Data EngineerLeave a comment

Analyzing logs in real time using Fluentd and BigQuery

November 20, 2019 SATYAJIT MAITRA

This tutorial shows how to log browser traffic and analyze it in real time. This is useful when you have a significant amount of logging from various sources and you want to debug issues or generate up-to-date statistics from the logs. The tutorial describes how to send log information generated by an NGINX web server to BigQuery using Fluentd, and then use BigQuery to analyze the log information. It assumes that you have basic familiarity with Google Cloud Platform (GCP), Linux command lines, application log collection, and log analysis. Introduction Logs are a powerful tool for providing a view into how large-scale … Continue reading Analyzing logs in real time using Fluentd and BigQuery

quantumLeave a comment

What is quantum computing?

November 20, 2019 SATYAJIT MAITRA

There are some problems so difficult, so incredibly vast, that even if every supercomputer in the world worked on the problem, it would still take longer than the lifetime of the universe to solve. Quantum computers hold the promise to solve some of our planet’s biggest challenges – in environment, agriculture, health, energy, climate, materials science, and problems we’ve not yet even imagined. The impact of quantum computers will be far-reaching and have as great an impact as the creation of the transistor in 1947, which paved the way for today’s digital economy. Quantum computing harnesses the unique behavior of … Continue reading What is quantum computing?

cloud, Data EngineerLeave a comment

How to use Cloud Storage and Cloud SQL

November 20, 2019 SATYAJIT MAITRA

In this post, you create a Cloud Storage bucket and place an image in it. You’ll also configure an application running in Compute Engine to use a database managed by Cloud SQL. For this lab, you will configure a web server with PHP, a web development environment that is the basis for popular blogging software. Outside this lab, you will use analogous techniques to configure these packages. You also configure the web server to reference the image in the Cloud Storage bucket. Objectives In this lab, you learn how to perform the following tasks: Create a Cloud Storage bucket and … Continue reading How to use Cloud Storage and Cloud SQL

cloudLeave a comment

Using Pubsub to publish messages

November 15, 2019 SATYAJIT MAITRA

Google Cloud Pub/Sub is a messaging service for exchanging event data among applications and services. A producer of data publishes messages to a Cloud Pub/Sub topic. A consumer creates a subscription to that topic. Subscribers either pull messages from a subscription or are configured as webhooks for push subscriptions. Every subscriber must acknowledge each message within a configurable window of time. the GCP console opens in this tab.Note: You can view the menu with a list of GCP Products and Services by clicking the Navigation menu at the top-left, next to “Google Cloud Platform”. The Google Cloud Shell Activate Google Cloud Shell Google … Continue reading Using Pubsub to publish messages

cloudLeave a comment

IOT Sensors and connections

November 15, 2019 SATYAJIT MAITRA

A sensor is a module that observes changes in its environment and sends information about these changes to a device. Devices collect data from sensors and send it to the cloud. Devices can be very small and have very few resources in terms of compute, storage, and so on. They might be able to communicate only through networks that cannot reach a cloud platform directly, such as over Bluetooth Low Energy (BLE). Standard devices are more likely to resemble small computers and may have the ability to store, process, and analyze data before sending it to the cloud. There are … Continue reading IOT Sensors and connections

cloudLeave a comment

IOT in GCP

November 15, 2019 SATYAJIT MAITRA

Security is critical when deploying and managing an IoT network. Cloud IoT Core has several security features to protect your IoT network. Devices are authenticated individually. Which means if there is an attack on your IoT network it is limited to one device and not the whole fleet. There are four public key formats available for devices: RS256 and RSA256_X509, and ES256 and ES256_X509. You specify the key format when creating the device. You can also define an expiration time for each device credential (public key). After it expires, the key is ignored but not automatically deleted. If you don’t … Continue reading IOT in GCP

cloudLeave a comment

Cloud IOT Core

November 9, 2019 SATYAJIT MAITRA

Cloud IoT Core is a fully managed service that allows you to easily and securely connect, manage, and ingest data from millions of globally dispersed devices. Cloud IoT Core, in combination with other services on Google Cloud platform, provides a complete solution for collecting, processing, analyzing, and visualizing IoT data in real time to support improved operational efficiency. You will transmit telemetry messages from a device and the device will respond to configuration changes from a server based on real-time data. The devices in this system publish temperature data to their telemetry feeds, and a server consumes the telemetry data … Continue reading Cloud IOT Core

big data, Data EngineerLeave a comment

Query GitHub data using BigQuery

November 9, 2019 SATYAJIT MAITRA

BigQuery is Google’s fully managed, NoOps, low cost analytics database. With BigQuery you can query terabytes of data without needing a database administrator or any infrastructure to manage. BigQuery uses familiar SQL and a pay-only-for-what-you-use charging model. BigQuery allows you to focus on analyzing data to find meaningful insights. In this post we’ll see how to query the GitHub public dataset to grab hands on experience with it. Sign-in to Google Cloud Platform console (console.cloud.google.com) and navigate to BigQuery. You can also open the BigQuery web UI directly by entering the following URL in your browser. Accept the terms of service. … Continue reading Query GitHub data using BigQuery

big data, Data Engineer, google cloud, machine learningLeave a comment

Recommend Products using ML with Cloud SQL and Dataproc

November 9, 2019 SATYAJIT MAITRA

As our goal is to provide demo that is why we are using the Cloud SQL or else yo can use spanner for horizontal scaling. our goal is to Create Cloud SQL instance Create database tables by importing .sql files from Cloud Storage Populate the tables by importing .csv files from Cloud Storage Allow access to Cloud SQL Explore the rentals data using SQL statements from CloudShell the GCP console opens in this tab.Note: You can view the menu with a list of GCP Products and Services by clicking the Navigation menu at the top-left, next to “Google Cloud Platform”. you populate rentals … Continue reading Recommend Products using ML with Cloud SQL and Dataproc

machine learningLeave a comment

Dimensionality reduction using sklearn a way of reducing burden

November 8, 2019 SATYAJIT MAITRA

Principal component analysis (PCA): PCA is used to decompose a multivariate dataset in a set of successive orthogonal components that explain a maximum amount of the variance. In scikit-learn, PCA is implemented as a transformer object that learns n components in its fit method, and can be used on new data to project it on these components. PCA centers but does not scale the input data for each feature before applying the SVD. The optional parameter parameter whiten=True makes it possible to project the data onto the singular space while scaling each component to unit variance. The PCA object also provides a probabilistic interpretation of the PCA that can give a likelihood … Continue reading Dimensionality reduction using sklearn a way of reducing burden

machine learningLeave a comment

Machine Learning crash course (Tensorflow Examples)

October 21, 2019 SATYAJIT MAITRA

machine learning comes with the learning pattern which is supervised learning at a first glance .so here is a brief about it terms used here are : the very first thing needs to keep in mind is framing your machine learning model/projects means what you want to achieve out of the data. example may contains as follows: A regression model predicts continuous values. For example, regression models make predictions that answer questions like the following: What is the value of a house in California? What is the probability that a user will click on this ad? A classification model predicts discrete values. For example, … Continue reading Machine Learning crash course (Tensorflow Examples)

Data EngineerLeave a comment

Spark Cluster Overview

September 11, 2019 SATYAJIT MAITRA

Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming. Security in Spark is OFF by default. This could mean you are vulnerable to attack by default. Spark uses Hadoop’s client libraries for HDFS and YARN. Users can also download a “Hadoop free” binary and run Spark with any Hadoop version by augmenting Spark’s classpath. Scala and Java users can … Continue reading Spark Cluster Overview

big data, cloud, data science, google cloud, machine learningLeave a comment

Be different build a machine learning model with some extra line in your SQL query and grab attention

September 6, 2019 SATYAJIT MAITRA

By the introduction you probably get it and yes we are talking about Biguery ML . BigQuery ML enables users to create and execute machine learning models in BigQuery using standard SQL queries. BigQuery ML democratizes machine learning by enabling SQL practitioners to build models using existing SQL tools and skills. BigQuery ML increases development speed by eliminating the need to move data. SEND FEEDBACK BigQuery ML Documentation Introduction to BigQuery ML Overview BigQuery ML enables users to create and execute machine learning models in BigQuery using standard SQL queries. BigQuery ML democratizes machine learning by enabling SQL practitioners to … Continue reading Be different build a machine learning model with some extra line in your SQL query and grab attention

cloud, google cloud, machine learningLeave a comment

Build simple Apps that can convert text-to-speech and speech-to-text but in c#

August 30, 2019August 31, 2019 SATYAJIT MAITRA

As a developer back in 2017 I always wonder it will be nice to write Machine learning code in c# .Net framework to show my manager that i know enough to become Team Lead but past is past and i left that productive company most of the company manager’s in the world are same full with dull insights as they tries to bring people down and demotivate them from their goal as they didn’t get their anyways the other day i was searching memes in the internet and all of a sudden one of the website gives me two HD … Continue reading Build simple Apps that can convert text-to-speech and speech-to-text but in c#

cloud, data scienceLeave a comment

Become A Marketing Expert By using Google Cloud products learn the Art of Asking with a Browser

August 12, 2019 SATYAJIT MAITRA

First thing first We will discuss what is Bigquery and why we are choosing Bigquery….. BigQuery is Google’s fully managed, NoOps, low cost analytics database. With BigQuery you can query terabytes and terabytes of data without having any infrastructure to … Continue reading Become A Marketing Expert By using Google Cloud products learn the Art of Asking with a Browser

big data, cloud, Data Engineer, data science, google cloud, machine learningLeave a comment

Bayes Classification with Cloud Datalab, Spark, and Pig on Google Cloud

August 10, 2019August 10, 2019 SATYAJIT MAITRA

Note: If you are really following with post this job can take upto 1:30 hours to finish and if you stuck in a typo it will increase your resistance power In this post you will learn how to deploy a … Continue reading Bayes Classification with Cloud Datalab, Spark, and Pig on Google Cloud

big data, cloud, Data EngineerLeave a comment

Building an IoT Analytics Pipeline on Google Cloud Platform step by step

July 27, 2019 SATYAJIT MAITRA

let’s start with the definition of IoT: The term Internet of Things (IoT) refers to the interconnection of physical devices with the global Internet. These devices are equipped with sensors and networking hardware, and each is globally identifiable. Taken together, these capabilities … Continue reading Building an IoT Analytics Pipeline on Google Cloud Platform step by step

deep learning, google cloud, machine learningLeave a comment

Cloud ML Engine Your Friend on cloud

July 26, 2019 SATYAJIT MAITRA

What we are doing here. Theory of Not relativity but cloud ml engine a bit of tensorflow(not stack overflow) and hands on in Create a TensorFlow training application and validate it locally. Run your training job on a single worker instance in the cloud. Run your training job as a distributed training job in the cloud. Optimize your hyperparameters by using hyperparameter tuning. Deploy a model to support prediction. Request an online prediction and see the response. Request a batch prediction. What We are building here: a wide and deep model for predicting income category based on United States Census … Continue reading Cloud ML Engine Your Friend on cloud

big data, data scienceLeave a comment

Visualizing BigQuery data in a Jupyter notebook with SQL

July 24, 2019 SATYAJIT MAITRA

BigQuery is a petabyte-scale analytics data warehouse that you can use to run SQL queries over vast amounts of data in near realtime. Data visualization tools can help you make sense of your BigQuery data and help you analyze the data interactively. You can use visualization tools to help you identify trends, respond to them, and make predictions using your data. In this tutorial, you use the BigQuery Python client library and Pandas in a Jupyter notebook to visualize data in the BigQuery natality sample table. SEND FEEDBACK BigQuery Visualizing BigQuery data in a Jupyter notebook Contents Objectives Costs Before you begin Setting … Continue reading Visualizing BigQuery data in a Jupyter notebook with SQL

cloud, data science, google cloud, machine learningLeave a comment

Analyzing Financial Time Series Using BigQuery and Cloud Datalab

July 24, 2019July 24, 2019 SATYAJIT MAITRA

This solution illustrates the power and utility of BigQuery and Cloud Datalab as tools for quantitative analysis. The solution provides an introduction (this document) and gets you set up to run a notebook-based Cloud Datalab tutorial. If you’re a quantitative analyst, you use a … Continue reading Analyzing Financial Time Series Using BigQuery and Cloud Datalab

big data, data science, machine learningLeave a comment

detecting fraud with decision tree and spark

July 24, 2019July 24, 2019 SATYAJIT MAITRA

Apache Spark™ is a unified analytics engine for large-scale data processing. Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It … Continue reading detecting fraud with decision tree and spark

r programming, statisticsLeave a comment

A/B testing

July 22, 2019July 22, 2019 SATYAJIT MAITRA

The A/B test (also known as a randomised controlled trial, or RCT, in the other sciences) is a powerful tool for product development. some motivations: With the rise of digital marketing led by tools including Google Analytics, Google Adwords, and Facebook Ads, a key competitive advantage for businesses is using A/B testing to determine effects of digital marketing efforts. Why? In short, small changes can have big effects. This is why A/B testing is a huge benefit. A/B Testing enables us to determine whether changes in landing pages, popup forms, article titles, and other digital marketing decisions improve conversion rates … Continue reading A/B testing

mathematics, statisticsLeave a comment

Binomial Random Variables: Introduction

July 21, 2019 SATYAJIT MAITRA

Binomial Random Variables So far, in our discussion about discrete random variables, we have been introduced to: The probability distribution, which tells us which values a variable takes, and how often it takes them. The mean of the random variable, which tells us the long-run average value that the random variable takes. The standard deviation of the random variable, which tells us a typical (or long-run average) distance between the mean of the random variable and the values it takes. We will now introduce a special class of discrete random variables that are very common, because as you’ll see, they … Continue reading Binomial Random Variables: Introduction

data science, statisticsLeave a comment

How To Distribute Sample

July 18, 2019 SATYAJIT MAITRA

Sampling Distributions Introduction Already on several occasions we have pointed out the important distinction between a population and a sample. In Exploratory Data Analysis, we learned to summarize and display values of a variable for a sample, such as displaying the blood types of 100 randomly chosen U.S. adults using a pie chart, or displaying the heights of 150 males using a histogram and supplementing it with the sample mean (X¯) and sample standard deviation (S). In our study of Probability and Random Variables, we discussed the long-run behavior of a variable, considering the population of all possible values taken by that variable. For example, we … Continue reading How To Distribute Sample

cloud, machine learningLeave a comment

TensorFlow Machine Learning on the Amazon Deep Learning AMI

July 18, 2019 SATYAJIT MAITRA

TensorFlow is a popular framework used for machine learning. The Amazon Deep Learning AMI comes bundled with everything you need to start using TensorFlow from development through to production. In this post, you will develop, visualize, serve, and consume a TensorFlow machine learning model using the Amazon Deep Learning AMI. Objectives Upon completion of this post you will be able to: Create machine learning models in TensorFlow Visualize TensorFlow graphs and the learning process in TensorBoard Serve trained TensorFlow models with TensorFlow Serving Create clients that consume served TensorFlow models, all with the Amazon Deep Learning AMI Prerequisites You should be familiar … Continue reading TensorFlow Machine Learning on the Amazon Deep Learning AMI

mathematics, statistics, UncategorizedLeave a comment

Probability A short story

July 17, 2019July 17, 2019 SATYAJIT MAITRA

Sample Spaces As we saw in the previous section, probability questions arise when we are faced with a situation that involves uncertainty. Such a situation is called a random experiment, an experiment that produces an outcome that cannot be predicted in advance (hence the uncertainty). Here are a few examples of random experiments: Toss a coin once and record whether you get heads (H) or tails (T). The possible outcomes that this random experiment can produce are: {H, T}. Toss a coin twice. The possible outcomes that this random experiment can produce are: {HH, HT, TH, TT}. Toss a coin 3 … Continue reading Probability A short story

mathematics, r programming, statisticsLeave a comment

Causation and Lurking Variables With simpson’s paradox

July 15, 2019 SATYAJIT MAITRA

The one and only principle rule in statistics is Principle:Association does not imply causation! The scatterplot below illustrates how the number of firefighters sent to fires (X) is related to the amount of damage caused by fires (Y) in a certain city. The scatterplot clearly displays a fairly strong (slightly curved) positive relationship between the two variables. Would it, then, be reasonable to conclude that sending more firefighters to a fire causes more damage, or that the city should send fewer firefighters to a fire, in order to decrease the amount of damage done by the fire? Of course not! So what is going … Continue reading Causation and Lurking Variables With simpson’s paradox

data science, storytellingLeave a comment

Data Raconteur Suresh Convince his Wife Reshmi that chennai is becoming dry ……

July 11, 2019 SATYAJIT MAITRA

Hi this is suresh working as a software engineer in a HUGE MNC for past 7 years based on chennai but recently i used to work from home for the same company but my wife is much scared of it as she thinks my job is lost and we have to starved to death but how can I told you that we are gonna die but not due to starvation but due to thrust so I decided to tell her the story of my city which is becoming dry with data as now even she goes to grocery shopping she … Continue reading Data Raconteur Suresh Convince his Wife Reshmi that chennai is becoming dry ……

command line utils, computer vision, machine learningLeave a comment

Drawing With OpenCV

July 11, 2019 SATYAJIT MAITRA

We have discussed that an image is nothing but a logical array with number in each pixel representing the intensity of light in each pixel . And we can think it as numpy array and to work with it we … Continue reading Drawing With OpenCV

data science, r programming, statisticsLeave a comment

Scales Of Measurement:

July 10, 2019 SATYAJIT MAITRA

The four different scales of measurement, from least to most precise, are Nominal Ordinal Interval Ratio Nominal: The nominal scale of measurement is a qualitative measure that uses discrete categories to describe a characteristic of the research participants. For each participant, the researcher determines the presence, absence, and type of the attribute. Nominal scales of measurement may have two categories, such as citizen status (citizen/non-citizen), or they can have more than two categories, like religious affiliation (e.g., Agnostic, Buddhist, Jewish, Muslim) or marital status (e.g., divorced, married, single). Often, as described here, the categories have names; however, researchers code them with numbers … Continue reading Scales Of Measurement:

mathematics, r programming, statisticsLeave a comment

The Big Picture of Statistics

July 10, 2019July 10, 2019 SATYAJIT MAITRA

this post is inspired by stanford.edu The process of statistics starts when we identify what group we want to study or learn something about. We call this group the population. Note that the word populationhere (and in the entire course) does not … Continue reading The Big Picture of Statistics

big data, machine learningLeave a comment

Machine Learning Applications for Big Data(Regression):

June 27, 2019 SATYAJIT MAITRA

Machine Learning: Machine learning is an application of artificial intelligence (AI) that provides systems the ability to automatically learn and improve from experience without being explicitly programmed. Machine learning focuses on the development of computer programs that can access data and use it … Continue reading Machine Learning Applications for Big Data(Regression):