This is Part 3 of my Machine Learning in iOS tutorials, please check Part 1 and Part 2 first.

Machine learning on iOS became trendy when Apple introduced CoreML at WWDC 2017. It can be intimidating at first with all the new concepts, frameworks and models, so we started with cloud services to learn basic machine learning, dataset preparation and training in IBM Watson and CoreML (Part 1) and Azure Custom Vision and CoreML (Part2).

Using cloud services is convenient. However, it costs money and has certain limitation on how we want to customise our model training. And not to mention the privacy issue with all of the user’s images being uploaded to someone’s servers.

Today we try a less easy approach: using a framework to train model locally. There are many frameworks like TensorFlow, Keras, Scikit learn, Caffe, etc. We will work with Turi Create because it’s relatively easy to set up and it can export CoreML compatible model on the fly. Also, Turi Create gives us better understanding of some machine learning tasks and concepts, those will be the foundation for us to use other advanced frameworks.

In this tutorial we will build another Avengers superheroes classification using Turi Create. Let’s dive in!

Turi Create

Turi was initially a machine learning startup in Seattle, known for its product GraphLab that simplifies common machine learning tasks. In 2016 Apple acquired Turi and later in December 2017 made Turi Create as an open source project.

Turi Create simplifies the development of custom machine learning models. You don’t have to be a machine learning expert to add recommendations, object detection, image classification, image similarity or activity classification to your app.

As I mentioned before, the framework is simple to use, visual, flexible and exportable to CoreML compatible models. So what you trained can be used right away in iOS, macOS, tvOS and watchOS apps without any extra conversion.

Turi Create focuses on app domain, rather than model domain. It means that we don’t need to worry about constructing model architecture and connecting layers, instead we deal with high level APIs for the tasks at hands like loading images, training and evaluating. It also means that we don’t need to be machine learning experts to use it. For now the supported tasks are:

• Recommender systems
• Image classification
• Image similarity
• Object detection
• Activity classification
• Text classifier

In this tutorial, we use Image classification task to train custom model for our superheroes dataset. I have also put lots of hyperlinks to source code and reference in this post for you to explore further. Let’s take the first step now.

Given an image, the goal of an image classifier is to assign it to one of a pre-determined number of labels.

Step 1: Python

Like many other machine learning frameworks, Turi Create is written in Python and we need to write some Python code to call its APIs, so some basic understanding of Python is required. There are many tutorials online.

According to system requirements, Python 2.7, 3.5, 3.6 are supported, this is confirmed in this issue as from Turi Create 4.1. I’m using a MacBook with macOS High Sierra so I have Python 2.7 by default. Although there’s movement for Python 3, let’s use the system Python 2.7 for now.

You can check Python version and its executable by running the following commands in terminal:

If for some reasons you don’t have Python installed, you can install it here. Python comes with pip, which is a package management system used to install and manage software packages written in Python. We need pip to install turicreate, run the following command:

pip install turicreate

Turi Create recommends using virtualenv or Anaconda to create isolated Python environment. You are free to create virtual environment, but in this post we just merely execute Python scripts for simplicity.

Step 2: Data set

We use the same data set from Machine Learning in iOS: IBM Watson and CoreML post. You can collect your own dataset or use ones in this GitHub repo. Eventually, we train with images of 4 superheroes: Ironman, Captain America, Spiderman and Thor. Images for each superhero lie in their own folder, the name of the folder can be seen as a label or a tag.

Step 3: Load data and explore

In the same directory of dataset , create a file called turi.python and type the following. You can use editors like PyCharm to easily edit your Python code. Type the following:

Firstly, we need to import turicreate as tc . The import statement with as is to define tc as a local namespace in this file for the turicreate module.

In #1 As we said earlier, Turi Create has many modules for working with different kinds of tasks, they are tc.object_detector, tc.sentence_classifier, tc.recommender, ... but here we need to use tc.image_analysis because we’re working with image classification.

The load_images is used to loads images from a directory. JPEG and PNG images are supported. The with_path parameter is to indicates whether a path column is added to the SFrame. If with_path is set to True, the returned SFrame contains a path column, which holds a path string for each image object. As of this line, the data object contains a table with many records for each image, with 2 columns: the loaded image data and a path.

In #2, we already have images as our features/input variable, but we need a class/label to classify. Here we use os module to get the folder name, as the name of each folder is the class name. For example, the path might be /Users/khoa/XcodeProject2/Avengers/dataset/captain_america/steve-roger-america.jpg, but the hero_name should be mapped to captain_america.

The apply function is used with a lambda to create another column called hero_name based on the path, it means that each record has a new property called hero_name.

In #3 We save our data as .SFrame object, which is a data structured known by Turi Create. SFrame has a tabular data structure that knows how to load from many common file formats like CSV and has many handy data manipulation methods. You can read more about it in Logical Filter section.

Turi Create is kind enough to provide with a macOS app that shows our data object in a user-friendly interface, so in #4 all we need is to call explore.

So go to terminal to execute our script:

python turi.python

After a short while with some “Unsupported image format” (Don’t worry, it’s because of some malformed images that I downloaded from Google), you will get a saved turi.sframe folder on disk and the app Turi Create Visualization appears. Here you can see we have a table with many records and 3 columns path, image and hero_name

Step 4: Train

In this step, we will use the saved turi.sframe to create our model. Add a file called turi_train.python and type the following:

We first load the saved turi.sframe into an SFrame object called data . Then we split the dataset into train and test data. It is a good practice to use about 80% or 90% of data for training, and the rest for validating and testing. This way we are sure that our trained model will work.

In #3 we create a classifier model based on the train data, with hero_name as the label for our images. The model is of type ImageClassifier , you can read more about creating function of module image_classifier here. This model is a trained model and you can use it for prediction or classification (as known as prediction with confidence).

The predict method in #4 is for demonstration purpose only. It performs prediction for our test data and returns an SArray object with the labels for each image in test data. Something like below, you can print and check it against your test data.

[‘captain_america’, ‘thor’, ‘captain_america’, ...]

The evaluate method in #5 is very handy, as it evaluates the model by making predictions of target values and comparing these to actual values. The returned value is a dictionary of evaluation results where the key is the name of the evaluation metric (e.g. `accuracy`) and the value is the evaluation score. Here we usually get accuracy between 0.75 to 0.9 , not very bad for our humble dataset.

Lastly in #6 and #7, we save the trained model as Turi compatible turi.model for future use and export to CoreML model. We will soon use this CoreML model in our iOS app.

Run python turi_train.python to execute our script. After a short while, you will get a turi.model folder and also a file TuriCreate.mlmodel .

But wait, the exported .mlmodel is around 94, 2 Mb !! It is a bit heavy for an iOS app, let’s find a way to reduce the size.

Advanced parameters

Dig into the source code

Turi Create exposes high level APIs with some default parameters. In case you want to tweak parameters to your needs, read the official documentation, or better, consult the source code. Here are some links to very important files for image classification

• image_classifier.py: Class definition and utilities for the image classification toolkit. Contains methods create, evaluate, predict, classify
• image_analysis.py: Contains useful methods load_images, resize for images loading and customisation
• sframe.py: This module defines the SFrame class which provides the
  ability to create, access and manipulate a remote scalable dataframe object. It has handy methods load_sframe , explore to load saved .sframe and to explore SFrame data structure in a GUI app.

Transfer learning

Normally the training process takes a lot of time, but Turi Create manages to do it fast. Take a closer look at the console output when you run turi_train.python:

Downloading https://docs-assets.developer.apple.com/turicreate/models/resnet-50-symbol.json
Download completed: /var/tmp/model_cache/resnet-50-symbol.json
Downloading https://docs-assets.developer.apple.com/turicreate/models/resnet-50-0000.params
Download completed: /var/tmp/model_cache/resnet-50–0000.params

It downloaded some Resnet files. May you wonder? The reason is that it uses a technique called Transfer learning, read more about the tech details here.

Learning high-level concepts about data means that deep learning models take data, for instance raw pixel values of an image, and learns abstract ideas like ‘is animal’ or ‘is cat’ about that data

Conceptually, all this means is that you have a composition of simple non-linear functions, forming a complex non-linear function, which can map things as complex as raw pixel values to image category. This is what allows deep learning models to attain such amazing results.

It’s not uncommon for the task you want to solve to be related to something that has already been solved. Take, for example, the task of distinguishing cats from dogs. The famous ImageNet Challenge, for which CNN’s are the state-of-the-art, asks the trained model to categorize input into one of 1000 classes. Shouldn’t features that distinguish between categories like lions and wolves should also be useful for discriminating between cats and dogs?

Turi Create retrain the model based on its 2 pre-trained image classifiers Resnet and Squeezenet. Each differentiates in model architecture and size. Squeezenet supports nearly the same 1000 categories as Resnet, but with much fewer parameters, therefore Squeezenet is much smaller in size and takes less time to retrain. The performance and accuracy are apparently reduced a bit, but not a lot, so we will use Squeezenet for our app.

Squeezenet

Detects the dominant objects present in an image from a set of 1000 categories such as trees, animals, food, vehicles, people, and more. With an overall footprint of only 5 MB, SqueezeNet has a similar level of accuracy as AlexNet but with 50 times fewer parameters.

You can learn more about Squeezenet at its GitHub repo and also the CoreML converted model. For now we want a custom retrain with our Avengers images, let’s revisit create function:

def create(dataset, target, feature = None, model = ‘resnet-50’,
 max_iterations=10, verbose=True, seed=None):

model : string optional Uses a pretrained model to bootstrap an image classifier

— “resnet-50” : Uses a pretrained resnet model.

— “squeezenet_v1.1” : Uses a pretrained squeezenet model. Models are downloaded from the internet if not available locally.

Once downloaded, the models are cached for future use.

Now open turi_train.python and change the model to squeezenet_v1.1 , and run the script again. After a while, you will get a TuriCreate.mlmodel of just 5Mb, how beautiful is that. Take a look again at the console log for more information:

You can see here the number of train images (examples), classes (we have 4 classes for 4 superheroes) and feature columns (which is just the name of each hero). The training runs through 10 iteration, which is the default value for max_iterations parameter in create function.

The maximum number of allowed passes through the data. More passes over the data can result in a more accurately trained model. Consider increasing this (the default value is 10) if the training accuracy is low and the *Grad-Norm* in the display is large.

Advanced usages

For more other features, read here for more information about changing models, using GPUs and deploying to CoreML model.

Using CoreML model in iOS app

Now let’s add the TuriCreate.mlmodel model to our iOS project. If you click on our model, you should see some information about its base classifier, input and output. Xcode should also autogenerate a class TuriCreate for us to call.

We could instantiate a TuriCreateInput object with image of type CVPixelBuffer and call TuriCreate().prediction(input: input) . But for quick demo, let’s just use Vision framework as it plays nicely with CoreML and has the ability to load correct image format and do image resizing before prediction.

Build and run the app to see the model in action. Although the model is small, it predicts pretty well with high accuracy. Kudos to Turi Create 🤘

Where to go from here

Hope you learn something about image classification with Turi Create. Here are some links to learn more about Turi Create and machine learning in general:

• Image classification: Introduction to image classification task and simple demo for cats and dogs labelling.
• Machine learning essentials: learn about other concepts like clustering, graph analysis, regression, nearest neighbours.
• Machine Learning in iOS: Azure Custom Vision and CoreML (Part 2)
• Machine Learning in iOS: IBM Watson and CoreML (Part 1)
• How Turi Create is Disrupting the Machine Learning Landscape.

If you like this post, consider visiting my other articles and apps 🔥