Prepare data for ml apis on google cloud

Module 1: Data Ingestion & Storage

1.1 Cloud Storage Integration

Cloud Storage Integration in the context of preparing data for ML APIs on Google Cloud refers to connecting your data residing in Google Cloud Storage (GCS) buckets to those ML APIs. It’s about enabling the ML APIs to directly access and process the data you have stored in GCS. This avoids the need to manually download, transfer, or re-upload data every time you want to use an ML API.

Here’s a breakdown with examples:

• Data Location: Your datasets (images, text documents, audio files, video files, tabular data in CSV or JSON format, etc.) are stored in Google Cloud Storage buckets.

• ML API Access: You need to give the ML API (like Vision API, Natural Language API, Speech-to-Text API, or AutoML) permission to read the data from your GCS bucket. This is usually done through service accounts and granting appropriate roles (e.g., Storage Object Viewer).

• Specifying Input: When you make a request to the ML API, you specify the GCS URI (Uniform Resource Identifier) of the data file or the directory containing the data. This URI tells the API exactly where to find the data to process.

• Example: Vision API

  • You have images of cats in a GCS bucket named my-cats-bucket and an image named fluffy.jpg within it.
  • The GCS URI for this image would be gs://my-cats-bucket/fluffy.jpg.
  • When calling the Vision API’s detectLabels method, you would include this URI in your request, telling the API to analyze the image located in GCS.

• Example: Natural Language API

  • You have a text document stored in GCS named article.txt in the bucket my-text-bucket.
  • The GCS URI is gs://my-text-bucket/article.txt.
  • To analyze sentiment using the Natural Language API, you would provide this URI as the input document.

• Example: AutoML Training

  • You have a CSV file with training data for your custom ML model stored in a GCS bucket named my-training-data.
  • You tell AutoML during the training process the GCS path to this CSV file. AutoML then directly reads the training data from GCS to train your model.

In summary, Cloud Storage Integration simplifies the workflow by allowing ML APIs to directly access and use data stored in GCS without requiring data movement, making it more efficient and scalable. The core component is providing the correct GCS URI to the ML API.

1.2 BigQuery Data Loading

1.3 Dataflow for Batch Ingestion

1.4 Pub/Sub for Real-time Ingestion

Module 2: Learn How to Clean Data

2.1 Handling Missing Values

2.2 Removing Duplicate Data

2.3 Correcting Data Inconsistencies

2.4 Outlier Detection and Removal

2.5 Data Type Conversion

Module 3: Data Transformation Techniques

3.1 Feature Scaling (Normalization/Standardization)

3.2 Feature Encoding (One-Hot Encoding/Label Encoding)

3.3 Feature Engineering

3.4 Text Data Processing (Tokenization/Stemming)

Module 4: Run Pipelines

4.1 Orchestrating Workflows with Cloud Composer

4.2 Building Data Pipelines with Dataflow

4.3 Scheduling Tasks with Cloud Scheduler

4.4 Monitoring Pipeline Execution

Module 5: Transform Data for Use with Google’s ML APIs

5.1 Formatting Data for Vision API

5.2 Formatting Data for Natural Language API

5.3 Formatting Data for Translation API

5.4 Formatting Data for Video Intelligence API

5.5 Choosing Appropriate Data Types for APIs

Module 6: Data Validation and Quality Checks

6.1 Implementing Data Validation Rules

6.2 Using Dataflow for Data Quality Assessment

6.3 Monitoring Data Quality Metrics

Module 7: Feature Store Concepts and Implementation

7.1 Designing a Feature Store for ML APIs

7.2 Storing and Retrieving Features

7.3 Feature Store Optimization

Module 8: Security and Access Control

8.1 IAM Roles and Permissions for Data Access

8.2 Data Encryption

8.3 Auditing Data Access