Understand The Service Azure Provides for Data Engineers

Even the most experienced data engineer can feel overwhelmed by the range of data platform technologies in Microsoft Azure. In diverse scenarios and industries, data engineers must solve complex data problems to provide business value through data. By understanding the data types and capabilities of the data platform technologies, a data engineer can pick the…

Using Pubsub to publish messages

Google Cloud Pub/Sub is a messaging service for exchanging event data among applications and services. A producer of data publishes messages to a Cloud Pub/Sub topic. A consumer creates a subscription to that topic. Subscribers either pull messages from a subscription or are configured as webhooks for push subscriptions. Every subscriber must acknowledge each message…

Building an IoT Analytics Pipeline on Google Cloud Platform step by step

let’s start with the definition of IoT: The term Internet of Things (IoT) refers to the interconnection of physical devices with the global Internet. These devices are equipped with sensors and networking hardware, and each is globally identifiable. Taken together, these capabilities afford rich data about items in the physical world. The internet of things, or IoT, is…

Big Data Essentials(unix command,DFS):

from internet search you will have a result like this for Big data definition: extremely large data sets that may be analysed computationally to reveal patterns, trends, and associations, especially relating to human behaviour and interactions.”much IT investment is going towards managing and maintaining big data” . let’s understand it first: every time you search…

How To Do Business Reporting in Hive Using HDP

The Apache Hive ™ data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storage using SQL. Structure can be projected onto data already in storage. A command line tool and JDBC driver are provided to connect users to Hive. Now as you don’t want to manage Hadoop cluster by yourself you…