Cloud ML Engine Your Friend on cloud

What we are doing here. Theory of Not relativity but cloud ml engine a bit of tensorflow(not stack overflow) and hands on in

Create a TensorFlow training application and validate it locally.
Run your training job on a single worker instance in the cloud.
Run your training job as a distributed training job in the cloud.
Optimize your hyperparameters by using hyperparameter tuning.
Deploy a model to support prediction.
Request an online prediction and see the response.
Request a batch prediction.

What We are building here:

a wide and deep model for predicting income category based on United States Census Income Dataset. The two income categories (also known as labels) are:

>50K — Greater than 50,000 dollars
<=50K — Less than or equal to 50,000 dollars

Wide and deep models use deep neural nets (DNNs) to learn high-level abstractions about complex features or interactions between such features. These models then combine the outputs from the DNN with a linear regression performed on simpler features. This provides a balance between power and speed that is effective on many structured data problems.

The sample defines the model using TensorFlow’s prebuilt DNNCombinedLinearClassifier class. The sample defines the data transformations particular to the census dataset, then assigns these (potentially) transformed features to either the DNN or the linear portion of the model.

first log in into your account :

first Activate Google Cloud Shell. what is Google cloud shell? have answer write on comment box or take a look………

tag line says Manage your infrastructure and applications from the command-line in any browser.

Google Cloud Shell provides you with command-line access to your cloud resources directly from your browser. You can easily manage your projects and resources without having to install the Google Cloud SDK or other tools on your system. With Cloud Shell, the Cloud SDK gcloud command-line tool and other utilities you need are always available, up to date and fully authenticated when you need them.

why should you use it:

Secure and Fully Authenticated by default
Your Favorite Tools Pre-installed and Up-to-Date
Developer ready
5GB of Persistent Disk Storage

first we will install tensorflow :

pip install –user –upgrade tensorflow

we are using pip command to install tensorflow in our shell which is a mini vm remember. to verify it has installed successfully type

python -c “import tensorflow as tf; print(‘TensorFlow version {} is installed.’.format(tf.VERSION))”

on console . We are telling run one python file with code script given as follows.

download the following repository on your shell git clone https://github.com/GoogleCloudPlatform/cloudml-samples.git

this repository contains all the code we required to run this whatever you call it a tutorial timepass google cloud hands on etc.

first we will Develop and validate your training application locally. Local environments provide an efficient development and validation workflow so that you can iterate quickly. You also won’t incur charges for cloud resources when debugging your application locally.

The relevant data files, adult.data and adult.test, are hosted in a public Google Cloud Storage bucket.

You can read the files directly from Cloud Storage or copy them to your local environment. For this lab you will download the samples for local training, and later upload them to your own Cloud Storage bucket for cloud training.

Run the following command to download the data to a local file directory and set variables that point to the downloaded data files

mkdir data

gsutil -m cp gs://cloud-samples-data/ml-engine/census/data/* data/

mkdir makes a data directory and gsutil is GCP’S utility tool box cp command to copy files from given public storage set training and validating data path

export TRAIN_DATA=$(pwd)/data/adult.data.csv

export EVAL_DATA=$(pwd)/data/adult.test.csv

export command will save the variables value in your local environment so next time when you type TRAIN_DATA IN your shell you will receive the write hand side .

to run with same version of tensorflow in which this lab was prepared you must type following why you will need this to avoid any uncertainty that makes us uneducated.

pip install –user -r ../requirements.txt

A local training job loads your Python training program and starts a training process in an environment that’s similar to that of a live Cloud ML Engine cloud training job.

export MODEL_DIR=output #ensures our output dir

Run this training job locally by running the following command:

gcloud ai-platform local train \ –module-name trainer.task \ –package-path trainer/ \ –job-dir $MODEL_DIR \ — \ –train-files $TRAIN_DATA \ –eval-files $EVAL_DATA \ –train-steps 1000 \ –eval-steps 100

we are training locally we provides our path of model provides training data test data batch size .

so you are training a classification model here to predict something (NI) now from the command you understood that your model is in trainer dir and output means log and etc will be saved on output dir let’s understand the code first:

we are using a wide and deep neural net for the prediction in the input.py file we will have multiprocessing and tensorflow imported now multiprocessing is just like threading module in python supports or offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using sub processes instead of threads we are using a wide and deep neural net for the prediction in the input.py file we will have multiprocessing and tensorflow imported now multiprocessing is just like threading module in python supports .

there is a constant folder in the package which contains the column names the default values to the columns and the levels do check it. we also have featurizer file that ensures handling of categorical and numerical features now if you look at the code you will find for occupation and native_country we have used hashing to bucket them but why we will do so if unknown values are there or too much category are presented but how do we know that remember EDA which is not included in the file but is also part of your project.

in input first we convert rank 1 tensor into rank 2 tensor ,then we are making a input_fn for making feature and label automation as you know to deploy a model you need two files model.py and task.py in GCP that includes in trainer.

Inspect the summary logs using Tensorboard

To see the evaluation results, you can use the visualization tool called TensorBoard. With TensorBoard, you can visualize your TensorFlow graph, plot quantitative metrics about the execution of your graph, and show additional data like images that pass through the graph. Tensorboard is available as part of the TensorFlow installation.

Follow the steps below to launch TensorBoard and point it at the summary logs produced during training, both during and after execution.

tensorboard –logdir=$MODEL_DIR –port=8080

Click on Accuracy to see graphical representations of how accuracy changes as your job progresses.

Type CTRL+C in Cloud Shell to shut down TensorBoard.

The output/export/census directory holds the model exported as a result of running training locally. List that directory to see the generated timestamp subdirectory.

You should receive a similar output:

1547669983

Copy the timestamp that was generated.

Run the following command in Cloud Shell, replacing <timestamp> with your timestamp:

gcloud ai-platform local predict \
--model-dir output/export/census/<timestamp> \
--json-instances ../test.json

You should see a result that looks something like the following:

CLASS_IDS  CLASSES  LOGISTIC              LOGITS                PROBABILITIES
[0]        [u'0']   [0.0583675391972065]  [-2.780855178833008]  [0.9416324496269226, 0.0583675354719162]

Where class 0 means income <= 50k and class 1 means income >50k.

Use your trained model for prediction

Once you’ve trained your TensorFlow model, you can use it for prediction on new data. In this case, you’ve trained a census model to predict income category given some information about a person.

For this section we’ll use a predefined prediction request file, test.json, included in the GitHub repository in the census directory.

till now

tensorflow champ
directory understanding
local deployer
predictor of trained model

Run your training job in the cloud

Set up a Google Cloud Storage bucket

PROJECT_ID=$(gcloud config list project –format “value(core.project)”) BUCKET_NAME=${PROJECT_ID}-mlengine/(yourwish)

echo $BUCKET_NAME

REGION=us-central1

create a bucket

gsutil mb -l $REGION gs://$BUCKET_NAME

change locations of every file you used and set according to cloud storage

With a validated training job that runs in both single-instance and distributed mode, you’re now ready to run a training job in the cloud. You’ll start by requesting a single-instance training job.

Use the default BASIC scale tier to run a single-instance training job. The initial job request can take a few minutes to start, but subsequent jobs run more quickly. This enables quick iteration as you develop and validate your training job.

Select a name for the initial training run that distinguishes it from any subsequent training runs. For example, you can append a number to represent the iteration:

JOB_NAME=census_single_1

Specify a directory for output generated by Cloud ML Engine by setting an OUTPUT_PATH variable to include when requesting training and prediction jobs. The OUTPUT_PATH represents the fully qualified Cloud Storage location for model checkpoints, summaries, and exports. You can use the BUCKET_NAME variable you defined in a previous step. It’s a good practice to use the job name as the output directory:

submit the job

gcloud ai-platform jobs submit training $JOB_NAME \ –job-dir $OUTPUT_PATH \ –runtime-version 1.10 \ –module-name trainer.task \ –package-path trainer/ \ –region $REGION \ — \ –train-files $TRAIN_DATA \ –eval-files $EVAL_DATA \ –train-steps 1000 \ –eval-steps 100 \ –verbosity DEBUG

monitor the progress

gcloud ai-platform jobs stream-logs $JOB_NAME

It may take several minutes to deploy your trained model. When done, you can see a list of your models using the models list command:

gcloud ai-platform models list

congratulations you have done everything. kudos. now watch youtube relax and do 5 min exercise to minimise your gig feelings.