Use AutoML to quickly build deep learning models

Deep learning project combat: use AutoML to quickly build deep learning models.

In today's era of data explosion, deep learning has become a powerful tool for solving complex problems.

However, building an efficient and accurate deep learning model often requires a lot of expertise and time investment.

From choosing a neural network architecture to adjusting hyperparameters, each step can become a bottleneck for performance improvement.

Fortunately, with the development of AutoML (Automatic Machine Learning) technology, we can now automate this tedious process, leaving the machine to find the best model architecture and parameter configuration by itself.

This article will show how to use the AutoML tool — Google's AutoML Vision — to quickly build and train an image classification model through a practical case, the whole process is simple and efficient.

\n#

I. Introduction.

Imagine that you are a product manager of an e-commerce platform. Faced with a large number of product pictures, you urgently need an intelligent system that can automatically identify product types, so as to better organize products and optimize the search experience.

The traditional method is to manually design feature extractors and classifiers, which is not only time-consuming, but also requires extremely high professional knowledge.

Now, with AutoML technology, it's all made easier.

\n#

II. Environmental preparation.

First, make sure that Python and the necessary libraries are installed in your development environment, such as google-cloud-automlSumtensorflow。

If not installed, you can install it with the following command:

pip install google-cloud-automl tensorflow

\n#

III. Data set preparation.

Suppose we have collected a set of labeled product image datasets, including product images of various categories and their corresponding labels.

We divided these images into a training set and a validation set for subsequent evaluation of model performance.

import os
import shutil
from sklearn.model_selection import train_test_split

# 假设所有图片存放在'data/raw_images'目录下
src_dir = 'data/raw_images'
train_dir, val_dir = 'data/train', 'data/val'
os.makedirs(train_dir, exist_ok=True)
os.makedirs(val_dir, exist_ok=True)

# 分割数据集
train_files, val_files = train_test_split(os.listdir(src_dir), test_size=0.2, random_state=42)

for file in train_files:
    shutil.copy(os.path.join(src_dir, file), os.path.join(train_dir, file))

for file in val_files:
    shutil.copy(os.path.join(src_dir, file), os.path.join(val_dir, file))

\n#

IV. Create an AutoML project.

Next, we need to create an AutoML Vision project on Google Cloud and upload the training data to Google Cloud Storage (GCS).

from google.cloud import automl_v1beta4 as automl
from google.oauth2 import service_account

# 设置凭证文件路径
credentials_path = 'path/to/your/service-account-file.json'
credentials = service_account.Credentials.from_service_account_file(credentials_path)

# 初始化AutoML客户端
client = automl.AutoMlClient(credentials=credentials)

# 创建项目
project_id = 'YOUR_PROJECT_ID'
location = 'us-central1'  # 根据实际需求选择区域
parent = f'projects/{project_id}/locations/{location}'
display_name = 'product_image_classifier'

response = client.create_dataset(request={
    'parent': parent,
    'dataset': {
        'display_name': display_name,
        'image_classification_dataset_metadata': {
            'classification_type': automl.proto.ImageClassificationType.MULTICLASS
        }
    }
})
print('Dataset created:', response.name)

\n#

V. Upload data to GCS.

To simplify the presentation, assume here that you already have a GCS bucket ready.

You can use the gsutil tool to upload local data to GCS.

gsutil -m cp -r data/* gs://your-bucket-name/path/to/destination/

\n#

VI. Import data into the AutoML project.

import time

def import_data(dataset_id, gcs_uri):
    dataset_full_id = f'projects/{project_id}/locations/{location}/datasets/{dataset_id}'
    gcs_source = {'input_uri': gcs_uri}
    output_config = {'destination': {'output_uri_prefix': f'gs://{project_id}-automl/{dataset_id}/'}}
    import_request = automl.InputConfig(name='train', type_=automl.proto.InputConfig.DataSource.GCS_SOURCE, gcs_source=gcs_source)
    response = client.import_data(name=dataset_full_id, input_config=import_request)
    operation = response.operation
    print(f'Importing data: {operation.name}')
    return operation.name

gcs_uri = f'gs://your-bucket-name/path/to/destination/*'
operation_name = import_data(display_name, gcs_uri)
while True:
    result = client.get_operation(name=operation_name)
    if result.done:
        break
    time.sleep(10)
print('Data import completed.')

\n#

VII. Train the model.

Once the data is successfully imported, we can start training the model.

def train_model(dataset_id):
    model_display_name = 'product_image_classifier_model'
    model_full_id = f'projects/{project_id}/locations/{location}/models/{model_display_name}'
    response = client.create_model(request={
        'parent': parent,
        'model': {
            'display_name': model_display_name,
            'dataset_id': dataset_id,
            'image_object_detection_model_metadata': {}
        }
    })
    model = response.name
    print('Model training started:', model)
    return model

model = train_model(display_name)

\n#

VIII. Evaluation and deployment model.

After training, we can evaluate the performance of the model and deploy it as an API service.

def evaluate_model(model_id):
    evaluation = client.deploy_model(name=model_id)
    print('Model deployed:', evaluation)
    return evaluation.name

evaluated_model = evaluate_model(model)

\n#

IX. Summary and Outlook.

Through the above steps, we use AutoML Vision to successfully build a product image classification model. From data preparation to model deployment, the whole process does not require in-depth and complex neural network architecture design and parameter tuning, which greatly saves time and labor costs.

AutoML not only lowers the application threshold of deep learning, but also enables non-professionals to enjoy the convenience brought by AI.

In the future, with the continuous advancement of technology, AutoML will realize its potential in more fields and promote the intelligent transformation of all walks of life.