An introduction to the Information Retrieval Project based on Food Image Recognition using Deep Learning Algorithms
Today’s digital world has a plethora of social media platforms where people like to share their experiences, getaways, thoughts and even showcase their creative talents. Facebook, Instagram, Twitter, etc. are very popular social media platforms where people like to explore through different images and follow certain trends. Foodies and Nutrition-seekers are also often seen in such platforms posting a variety of healthy and appetizing food dishes. Some of these catchy-looking delectable food item social media posts may attract people to consume the dish right away but unfortunately may not be accompanied by any information about the dish name or the recipe for the food.
For this reason, our product-based Image Recognition Model project named “Know Your Food” attempts to solve the grievance of such food-lovers. In this project, we create a huge database of thousands of images for about 25 food classes by refining and pre-processing the base Kaggle dataset due to restricted amount of resources to train the data on our neural net models.
Our Dataset
After manual removal of noise from the images for a better model performance, the distinct classes in our modified dataset is as shown below:
Now, let us look at a few of the images we have for these classes…
After splitting this dataset into its respective train and test sets in the approximate ratio of 80:20, the class wise image data distribution (number of images per class) is shown in the bar plot below:
Training Models
For our Food Recognition project, we trained 4 different types of neural net models, they are:
- Inception Model (Test Accuracy:72.02%)
- Resnet-50 (Test Accuracy:28.05%)
- VGG-19 (Test Accuracy:29.15)
- TensorFlow Model Maker library based model (Test Accuracy:82.88%)
In our case, the model created using the TensorFlow Model Maker library outperformed the rest of the three models.
Now, based on this model, we built our product app using Android Studio (language: Java) that predicts the name of the dish using the TensorFlow model maker based model.
Rapport Foodo App (Product of the project)
Our product app for the project first opens up to the page where it instructs you to either click a pic of the image you want the information for or either upload an image from your phone.
After clicking the picture and confirming the picture is taken properly by clicking ‘Ok’ or selecting the picture from the phone gallery, the picture imported into the app is displayed on the current app page along with the name of the dish our model predicts it to be. If the model gives wrong prediction, then the user is allowed to add photo again (suitably a clearer picture) by the click of the ‘Add Photo’ button.
Then , by clicking a button ‘Check Dish Info’, the app takes you to the next page where the dish name is displayed on top and the user is given two options here in the form of two other buttons, ‘Order Dish’ and ‘Get Recipe’ for the dish name displayed.
Upon clicking the ‘Get Recipe’ button, the app opens to another page on phone where it displays the retrieved recipe for the dish name predicted by our model. The recipe of the dish includes ingredients and steps for the dish as shown in the images below.
On the other hand, if the user wants the dish but does not want to cook it then the ‘Order Dish’ button directs the user to the page where we display a list of shops nearby (by the utilization of Foursquare API) based on the type of dish and the user can choose the nearest restaurant where the dish is available. Also, a floating button at the end of the page redirects the user to the linked Zomato App (or Zomato Website if the user does not have the app installed on their phone).
Future Work
In-app food ordering feature can also be added for the app to search the nearby restaurants where the model predicted dish from the captured image is available for a greater user experience. Also, incorporation of a broader spectrum of dishes for a precise food dish prediction with optimum resources to train models for all the types of dishes that we see in social media.
Project Link and Team Info
You can find our Project (models, data pre-processing and app product with Android Studio code) in GitHub. The link is given here.
Our team:
Tanishi Srivastava (GitHub)(LinkedIn)
Prernakaruna (GitHub)(LinkedIn)
