big dataLeave a comment
Sticky post

.Net framework and Apache spark

SATYAJIT MAITRA

Why choose .NET for Apache Spark? .NET for Apache Spark empowers developers with .NET experience or code bases to participate in the world of big data analytics. .NET for Apache Spark provides high performance APIs for using Spark from C# and F#. With C# and F#, you can access: DataFrame and SparkSQL for working with structured data. Spark Structured Streaming for working with streaming data. Spark SQL for writing queries with SQL syntax. Machine learning integration for faster training and prediction (that is, use .NET for Apache Spark alongside ML.NET). .NET for Apache Spark is compliant with .NET Standard, a formal … Continue reading .Net framework and Apache spark

Data EngineerLeave a comment
Sticky post

Analyzing logs in real time using Fluentd and BigQuery

SATYAJIT MAITRA

This tutorial shows how to log browser traffic and analyze it in real time. This is useful when you have a significant amount of logging from various sources and you want to debug issues or generate up-to-date statistics from the logs. The tutorial describes how to send log information generated by an NGINX web server to BigQuery using Fluentd, and then use BigQuery to analyze the log information. It assumes that you have basic familiarity with Google Cloud Platform (GCP), Linux command lines, application log collection, and log analysis. Build dataprep skills real time with me here. Introduction Logs are a powerful … Continue reading Analyzing logs in real time using Fluentd and BigQuery

quantumLeave a comment

What is quantum computing?

SATYAJIT MAITRA

There are some problems so difficult, so incredibly vast, that even if every supercomputer in the world worked on the problem, it would still take longer than the lifetime of the universe to solve. Quantum computers hold the promise to solve some of our planet’s biggest challenges – in environment, agriculture, health, energy, climate, materials science, and problems we’ve not yet even imagined. The impact of quantum computers will be far-reaching and have as great an impact as the creation of the transistor in 1947, which paved the way for today’s digital economy. Quantum computing harnesses the unique behavior of … Continue reading What is quantum computing?

cloud, Data EngineerLeave a comment

How to use Cloud Storage and Cloud SQL

SATYAJIT MAITRA

In this post, you create a Cloud Storage bucket and place an image in it. You’ll also configure an application running in Compute Engine to use a database managed by Cloud SQL. For this lab, you will configure a web server with PHP, a web development environment that is the basis for popular blogging software. Outside this lab, you will use analogous techniques to configure these packages. You also configure the web server to reference the image in the Cloud Storage bucket. Objectives In this lab, you learn how to perform the following tasks: Create a Cloud Storage bucket and … Continue reading How to use Cloud Storage and Cloud SQL

cloud, Data Engineer1 Comment
Sticky post

Use Of Cloud IoT Core

SATYAJIT MAITRA

Use IoT Core to create a registry Use IoT Core to create a device Use Stackdriver Logging to view device logs Enable APIs In this section, you check that all the APIs you will use in this lab are enabled. In the GCP Console, on the Navigation menu (), click APIs & Services. Scroll down and confirm that your APIs are enabled. Cloud IoT API Cloud Pub/Sub API Container Registry API If an API is disabled, click Enable APIs and services at the top, search for the API by name, and enable it for your project. Make sure you are in the correct Qwiklabs project. … Continue reading Use Of Cloud IoT Core

cloudLeave a comment
Sticky post

Using Pubsub to publish messages

SATYAJIT MAITRA

Google Cloud Pub/Sub is a messaging service for exchanging event data among applications and services. A producer of data publishes messages to a Cloud Pub/Sub topic. A consumer creates a subscription to that topic. Subscribers either pull messages from a subscription or are configured as webhooks for push subscriptions. Every subscriber must acknowledge each message within a configurable window of time. the GCP console opens in this tab.Note: You can view the menu with a list of GCP Products and Services by clicking the Navigation menu at the top-left, next to “Google Cloud Platform”.  The Google Cloud Shell Activate Google Cloud Shell Google … Continue reading Using Pubsub to publish messages

cloudLeave a comment
Sticky post

IOT Sensors and connections

SATYAJIT MAITRA

A sensor is a module that observes changes in its environment and sends information about these changes to a device. Devices collect data from sensors and send it to the cloud. Devices can be very small and have very few resources in terms of compute, storage, and so on. They might be able to communicate only through networks that cannot reach a cloud platform directly, such as over Bluetooth Low Energy (BLE). Standard devices are more likely to resemble small computers and may have the ability to store, process, and analyze data before sending it to the cloud. There are … Continue reading IOT Sensors and connections

cloudLeave a comment

IOT in GCP

SATYAJIT MAITRA

Security is critical when deploying and managing an IoT network. Cloud IoT Core has several security features to protect your IoT network. Devices are authenticated individually. Which means if there is an attack on your IoT network it is limited to one device and not the whole fleet. There are four public key formats available for devices: RS256 and RSA256_X509, and ES256 and ES256_X509. You specify the key format when creating the device. You can also define an expiration time for each device credential (public key). After it expires, the key is ignored but not automatically deleted. If you don’t … Continue reading IOT in GCP

cloudLeave a comment
Sticky post

Cloud IOT Core

SATYAJIT MAITRA

Cloud IoT Core is a fully managed service that allows you to easily and securely connect, manage, and ingest data from millions of globally dispersed devices. Cloud IoT Core, in combination with other services on Google Cloud platform, provides a complete solution for collecting, processing, analyzing, and visualizing IoT data in real time to support improved operational efficiency. You will transmit telemetry messages from a device and the device will respond to configuration changes from a server based on real-time data. The devices in this system publish temperature data to their telemetry feeds, and a server consumes the telemetry data … Continue reading Cloud IOT Core

big data, Data EngineerLeave a comment
Sticky post

Query GitHub data using BigQuery

SATYAJIT MAITRA

BigQuery is Google’s fully managed, NoOps, low cost analytics database. With BigQuery you can query terabytes of data without needing a database administrator or any infrastructure to manage. BigQuery uses familiar SQL and a pay-only-for-what-you-use charging model. BigQuery allows you to focus on analyzing data to find meaningful insights. In this post we’ll see how to query the GitHub public dataset to grab hands on experience with it. Sign-in to Google Cloud Platform console (console.cloud.google.com) and navigate to BigQuery. You can also open the BigQuery web UI directly by entering the following URL in your browser. Accept the terms of service. … Continue reading Query GitHub data using BigQuery