Using Data Science to: Enforce mask policies inside private businesses.
Introduction
As COVID became a normal part of our lives, we saw many businesses adopt policies around masks. But any rule is only as powerful as its enforcement. This has proven to be a difficult one to enforce.
Not only is the topic contentious, but most businesses are still trying to enforce this manually. This requires a huge amount of labor and puts frontline workers in a difficult position when they have to confront a non-compliant customer.
I wondered if there could be a way to automate the task and take the burden off of the front-line workers. So, I set out to see if it could be done.
The Plan
My goal in this project was to build out a proof-of-concept that would be as close to “production ready” as possible. To do this, I would first have to train a model to detect masks. Then, I would have to find a way to deploy this on an Android device. That way a business could theoretically use this with low cost hardware.
I broke my plan down into the following steps.
- Find training data.
- Train a model.
- Optimize and convert the model for mobile.
- Deploy it on an Android device.
1) Find Data
Browsing through Kaggle, I came across this dataset which was exactly what I needed.
It included 3725 Images with masks, and 3828 without masks. Because these numbers were very close, I would be working with “balanced” classes, which makes my job even easier!
Looking through the data, I see that there are images from varied conditions and angles. That’s good as it will help give my model additional flexibility.
The one concern I did identify, was the disproportional number of images with the “surgical” style face mask. That may make it harder for my model to recognize different styles of masks.
2) Train a Model
The next step was to train a model, however, this model was going to have to balance size with accuracy as this would have to be able to run on low-end hardware. Because of this, I opted for MobileNetV2 as it only has 3.4 million parameters but was pre-trained on a massive 1.4 million image dataset.
The transfer learning approach I took here was very similar to that of another model I wrote about here. Basically, I loaded the model, added a new top layer, froze the old parts, and trained the new ones.
Then I unfroze additional layers deeper into the model and trained a bit more.
When it was done, the model could identify masks with +98% accuracy on images it had not seen before.
Before moving on, I tested it locally using a screen grab to make sure it was working on live data.
3) Convert and Optimize
With my model trained, I now had to convert and optimize it. Basically, this step is designed to reduce the size of the model and put it in a format that will work on a mobile device. To do this I used a tool called TensorFlow Lite, designed for this specific application.
TensorFlow Lite uses some pretty creative tricks, such as changing the datatype of some of the stored values. This has the effect of reducing the number of decimal places stored in each parameter. That cuts the model size in half, but does not significantly impact performance.
Luckily this is also pretty easy to do because the API does most of the work for us. So with 4 lines of code, this step was complete and I was ready to move on to deployment.
4) Deploy on Android
Now it was time to get the model up and running on my phone so I could test it out. There was only one problem: I have 0 experience building Android Apps. I figured that between YouTube and the TensorFlow documentation it wouldn’t be that hard to figure out. I was wrong.
After a week of debugging Android Studio, my model, and my phone, I finally got it up and running… barely. In its current form, it’s pretty slow and tends to crash, and may only work on my version of Android. But for a proof of concept it’s good enough.
Overall, when it worked, it was typically correct. However, it was easily fooled by covering my face with something other than a mask (my shirt, my hand, etc). So before true deployment, the training data would need also need to include examples where the face was obstructed by something other than a mask.
Next Steps
While the app was (somewhat) working, my goal with this project was to get this as close to production ready as possible. However, given my limited experience with hardware, I did not think I could go much further than this on my own. Instead I brainstormed through some potential use cases for this model, based on the technology available at various businesses.
Low Tech: Entrance Mask Alarm: This app could be hooked up to a low-cost Android tablet which can be mounted near the entrance of any private business. Anytime someone attempts to enter without a mask, it can set off some alert, and even potentially store the image of the individual in case they become irate or refuse to leave.
Mid Tech: Entrance Turn-styles: To take this a step further, a business can mount these devices to a turn-style or gate system to scan faces of individuals as they approach. If they are compliant the gate will open without issue. If not, it can stay shut and issue some sort of alert. In fact, some businesses such as Walmart already use gate systems in many of their stores, so launching a program like this could leverage existing hardware and be stood up relatively quickly.
High Tech: Perpetual Monitoring: The two use cases above would be good solutions at the entrance of the businesses, but that does not stop shoppers from taking off their masks once they are inside the store. This is a more difficult problem to solve. However, some businesses such as “Amazon Go” employ cameras all throughout their stores. In these cases, an app like this could be deployed to continually monitor mask compliance of both customers and employees for the whole duration of the shopping experience.
Conclusion
The question I set out to answer with this project was: Can I use Data Science to enforce mask usage within private businesses? The answer appears to be yes, although a true working version of this will require a model + an app + hardware, which is beyond my current skill set.
However, even this may not “fix” the whole problem. The reality is that this is a contentious topic in the United States. While an app like this may act as a buffer and help suppress the rate of non-compliance, there will likely be a subset of people who will still refuse to comply when confronted.
However, if we only deploy applications that solve every edge case, nothing will ever get built. So I still feel that an app like this has value in our current world, so if you want to pick up where I left off, the code is available here.
