Classifying simple color matching outfits with the help of Fuzzy Logic

Hands-on Tutorials

Follow along as we create an intelligent assistant that will help us decide if an outfit has matching colors

---

In the following article we will go through the complete development process of a Proof of Concept project that will help us classify simple color matching outfits composed of shirts, bottoms and shoes. This article will be divided in the following sections:

• Understanding the issue (origin of the problem)
• Quick recap of our main tool ‘Fuzzy Logic’
• Overview of the HSV color representation
• Building the fuzzy logic model
• Adding outfit rules to filter matching outfits
• Final results and conclusions

---

Understanding the issue

One day at college, before Covid had arrived in our country, some colleagues and I were waiting for the teacher to begin the lecture when the topic of outfits popped up. At some point in the conversation we asked our colorblind friend how he chose the clothes to wear each day. He went on to explain the fact that he used to rely on his family to tell him whether his clothes would match or not and ever since he moved out and started living on his own, he could no longer ask for their help. His current method required him to guess or remember good outfits by identifying the brand or patterns in the garment, which sounded very counterproductive.

I couldn’t imagine the concept of not seeing the complete color spectrum and its implications on my day to day life, and this led me to think about the relationship between colors and fashion. Later that day I decided to contact Valeria, a friend who is currently studying fashion and design, with the hope of gaining some knowledge and finding a solution to my friend’s issue. I asked her about the process she followed when choosing a color scheme for an outfit and after a long productive talk I found myself thinking, ‘no wonder there isn’t a current solution, it’s all more complicated and subjective than I thought!’. I tried to narrow down the guidelines, and these were the rules that popped up more frequently:

• Neutral colors (white, black, gray, navy blue, etc.) can fit any outfit.
• There are warm (red) and cool (blue) colors, these can be used in different combinations to match an outfit.
• There are different tones for each color, which should also be taken into account when matching clothes.

At that time I was taking an introductory course to Machine Learning. I naively thought that there was most likely an easy way to train a model to classify matching clothes, and that the model would infer these rules from all of the data feeded into it. The initial idea was relatively simple and straightforward:

• _- Gather data of matching and non-matching outfits and pre-process it
• Train a neural net with the data
• Check the results and make adjustments
• Use the model to classify any combination of three colors as matching/non-matching_

The plan sounded achievable, but as soon as I started designing a more concrete solution, I found out that there were some major details I didn’t take into consideration.

The first obstacle I encountered had to do with color, a topic which I will get into shortly. The second obstacle was related to training the model and gathering the vast amount of data required to make it work. There were plenty of databases on fashion, but none were focused on color. I thought about gathering the data myself but it also presented some time consuming issues.

It all started to crumble and seem like a project that was too complicated to do, so with time I lost interest and added it to the list of ‘interesting but currently infeasible’ projects. It was until I had a particular lecture in my Computer Intelligence course that I regained motivation to continue the project, this time with a new concept in mind: Fuzzy Logic.

---

Fuzzy Logic recap

To keep this article short, I will summarize my understanding of Fuzzy Logic in simple terms, just enough to get you through the rest of this project. I am by no means an expert on the topic as I just learned about it myself, so if you want a more in depth explanation of Fuzzy Logic I suggest you take a look at https://www.guru99.com/what-is-fuzzy-logic.html or Computerphile’s video on the topic.

We first need to go back to ‘Classic Logic’. Classic logic relies on discrete, well defined values like ‘True’ or ‘False’. There is no confusion between each of these values and they can completely describe a state or result, such as if a heater is ‘on’ or ‘off’, there are no in between values like ‘somewhat on’. Fuzzy logic on the other hand, can use ‘in between’ values that aren’t strictly well defined. Let’s go over one of the most popular examples: temperature.

Let’s say you want to create a logical system that turns off the heater for a while when the room is hot, the problem being, how do you define ‘hot’? There is no clear boundary between hot, warm and cold; it seems to transition in a continuous manner and at one point subjectively becomes warm or cold. Fuzzy logic allows you to define sets of variables without clear boundaries such as [‘hot’, ‘warm’, ‘cold’], and use them along with a set of rules to determine different outcomes. An example of a rule could be: if the room is ‘hot’, turn the heater off for ‘a few’ minutes. Keep in mind that you need to specify these fuzzy sets along with their membership function, which help the model calculate the degree to which a certain value belongs to a subset (fuzzy value). Let’s look at an example:

This is a graphical representation of the ‘Temperature’ membership function. It explains the fuzzy set [‘cold’, ‘warm’, ‘cold’] associated with the Temperature variable. Given a crisp value, which can be thought of as the measurable value of the variable (in this case the temperature in degrees Celsius or Fahrenheit), the function will determine the extent to which each fuzzy element describes the value. In this example we see the pivot being placed in such a way that it intersects with both ‘cold’ and ‘warm’. We can infer that the temperature is at a place where it’s not completely cold, but it isn’t too warm to be considered fully warm. Given that it is more accurate to describe it as cold, it will give the favourable match to ‘cold’ as its fuzzy value. In this example we are looking at trapezoidal membership functions, but you can use different functions to map each variable including: Triangular, Gaussian, Bell, etc.

Once we have fuzzified our temperature’s crisp value, we can use a set of rules described in an intuitive way to specify the expected outputs, for example:

• If the temperature is ‘hot’ turn off the heater for ‘some’ minutes
• If the temperature is ‘warm’ turn off the heater for ‘a few’ minutes

These rules will then be used by an inference machine to calculate an output. Finally, we can defuzzify the resulting output by mapping it to a membership function that describes the amount of minutes [‘some’, ‘a few’, …]. This defuzzified value will give us the numerical result indicating the precise amount of minutes we should turn the heater off for.

Understanding digital color representations

Given the fact that our assistant will be living in the digital space, colors as we know them will need to be represented in such a way that computers can process them. There are several numerical representations for color, the most known one being RGB (RedGreenBlue).

Naturally, we could start looking into using RGB as crisp values for the Fuzzy Logic model but the more we learn about RGB, the less optimistic we can become about using it. RGB allows for more than 16 million different colors, and there’s no intuitive transition/pattern between one color and the other besides the idea that one may have more ‘red’ and less ‘blue’.

This may feel like a huge setback, but if you’ve ever used a photo editing tool like Photoshop and tried to pick a new color you may recall seeing plenty of different options to specify it. Within these options we can see RGB, Hexadecimal, CYM, and HSV. If we do some investigation on all of these options and read about the differences between them, we can observe that HSV has some useful characteristics that are worth exploring.

HSV stands for Hue, Saturation and Value. Each component has a specific function in the representation and a range of possible values:

• Hue: represents the perceived color without taking into account shades. Values go from [0, 360]
• Saturation: indicates how colorful the color is. The less saturation, the more faded and grayish a color will be. Values go from [0, 100]
• Value: indicates the brightness of the color, or amount of light emitted. The lower the value, the darker the color. Values go from [0, 100]

The biggest pros of the HSV representation are:

• Color is separated from tones/intensity
• Hue component less sensitive to light changes

Here’s an example where we have the same color under different lighting:

RGB (50, 132, 219) HSV (211, 77, 86)

RGB (30, 80, 133) HSV (211, 77, 52)

As you can see, the RGB representation gave us different values in each dimension for both elements, while HSV retained the hue component and only had an adjustment to the value component.

This is useful because we can separate the hue to determine if a color has a ‘cool’ or ‘warm’ temperature without taking into account its shade/tone, and then use saturation and value to describe the tone. We can also determine if a color is ‘neutral’ by inspecting both the saturation and value at the same time. Low saturation results in a faded color that can be close to gray. Low value can result in dark (black) colors, and high value can result in very bright (white) colors. All of these can be categorized as ‘neutral’ colors that can be used as wildcards when matching outfits.

Improved project plan

Now that we have a better understanding of the data and the problem, we can elaborate a new plan to tackle the project in two main steps:

1. Create a fuzzy logic model that, given a color in HSV format, can determine if the color is ‘neutral’, ‘warm’, or ‘cool’, along with an indication of its tone as ‘dark’ or ‘bright’. i.e. given a color in HSV, return a description along the lines of ‘dark warm’, or ‘bright cool’
2. Use the model to obtain the description of each clothing item, and then use fashion rules to determine if the outfit matches. Ex. winter outfits should contain all ‘dark’ colors preferably with a ‘cool’ hue.

Fuzzy logic model

The first step revolves around creating the Fuzzy Logic models that will give us the description of the colors. The goal is to input a color in HSV format and have a resulting descriptive text of the color.

If we take a look at the notes gathered during the talk with Valeria, we will see that these words for color descriptions keep showing up: ‘neutral’, ‘warm’, ‘cool’, ‘dark’, ‘bright’, so it may be useful to find of a way of transforming the colors from HSV to something similar to these. These words were used mostly to indicate good matches of an outfit, ex. two ‘dark cool’ colors and a ‘bright cool’ color, or two ‘neutral’ colors and a ‘bright warm’ color.

HSV separates hue from the other components such that you can get the same hue for a color even if they have different tones. This will help us categorize what can be described as the temperature of the color (‘warm’ or ‘cool’).

This is the membership function for the ‘Temperature’ variable, which is composed of the fuzzy set [‘WARM’, ‘COOL’] and the crisp values range from [0, 360]. As seen in the color wheel, the hue transitions between cool, warm and back to cool again given its circular representation. There are two transition colors (green and brown) where the concept of ‘warm’ and ‘cool’ gets very subjective.

These membership functions are made somewhat more straightforward. The first graph describes the membership function for the ‘Value’ of the color, and it can help us identify the brightness of the color all the way from ‘black’ to ‘very bright’. The second one identifies the membership for the ‘Saturation’ function that can describe if the color is completely ‘gray’, ‘faded’, or ‘saturated’ to different degrees based on the saturation value. These are all gaussian functions that are evenly split for simplicity, but it doesn’t need to be this way.

The mix of both of these components can help us identify both different tones (‘dark’ or ‘bright’) and ‘neutral’ colors, given that low saturation results in grayer colors and low or high value results in black or white colors respectively.

We need to add the rules which will guide the calculation for the tone variable. We can start by having some straightforward rules such as:

• _1. Colors with saturation identified as ‘gray’ or ‘very_faded’ will be counted as Neutral
  2. Colors with ‘black’ value will be counted as Neutral
  3. Colors that are ‘saturated’ and ‘very_dark’ will be counted as Dark
  4. Colors that are ‘faded’ and ‘dark’ will be counted as Dark
  5. Colors that are ‘very_saturated’ and ‘dark’ will be counted as Dark
  6. Colors that are ‘faded’ and ‘very_light’ will be counted as Bright
  7. Colors that are ‘saturated’ and ‘light’ will be counted as Bright
  8. Colors that are ‘very_saturated’ and ‘verylight’ will be counted as Bright

If we were to map all the different combinations of Value vs Saturation, we would get the table demonstrated below. The red cells indicate resulting values which have been specifically assigned in the rule set (ej. ‘saturated’ and ‘very_dark’ result in a ‘Dark’ tone). The white cells are not explicitly programmed, but the model will infer them based on the previous rules. If these results are not working properly we can always alter the rule set to our liking.

With these rules ready, we only need to create the membership function for the Tone variable so that we can map the result of the inference machine to its corresponding category.

This is the graphical representation of the membership function for the Tone variable. We can use a variety of functions that will help us map the results from the inference machine to its corresponding fuzzy value. The intuition behind these functions starts with the idea that a tone can transition between Dark and Cool, yet we still have to take into account Neutral colors. Neutral colors could be considered as those who are dark enough or faded enough that there’s no perceivable hue, that is why we see the mapping of Neutral colors close to Dark colors. Usually, we would proceed to defuzzify the result of the inference machine to get a crisp value, but for this particular project we are interested in the descriptive fuzzy values like ‘Bright’.

The diagram below shows the stages through which data will be modified and transformed between crisp and fuzzy values, starting from a numerical color representation in HSV format, and resulting in a descriptive text for the color.

After having these actions completed, the next step will be feeding the HSV values to their respective fuzzifier, and combine the results to get a description composed of (TONE, TEMP). Here are some test results:

Nice! The process seems to be working. There are some cases where we could describe the color a bit differently, but it can also be subjective. Luckily, since this is a fuzzy logic model guided by rules we can adjust the rules to our liking in no time. Now that we have the descriptions available, we can proceed to the final piece of the puzzle. *Neutral colors contain a temperature description because the hue is still being calculated in the process separately, but it can be ignored.

It is important to reiterate that we could add some complexity to the model to make it more accurate, such as incrementing the tone set for precision, or adding special hard coded ‘Neutral’ cases like brown or beige colors that could be hard to calculate. We could also add some bias so that cool colors have a ‘dark’ tone bias, and warm colors have a ‘bright’ tone bias. For the sake of finishing this article, we will continue with our current results.

Adding matching outfit rules

There are different outfit color schemes we can follow, each with their own set of rules. The most common ones we can find are the following (in most of these ‘neutral’ colors can be used as wildcards since they basically match anywhere):

• _- Neutral: composed of only neutral colors
• Basic (safe-bet): no high contrast between any colors, no more than one bright warm color
• Contrast: two dark colors and one bright color, can mix warm and cool colors.
• Analogous: colors with same temperature under different tones
• Winter: mostly dark cool colors
• Summer: mostly warm colors and some bright colors for contrast_

We can create different rule sets for the alternate outfit color schemes and pass in the outfit color descriptions to a function that will validate the rules and decide if the outfit matches the particular color scheme.

With these rules ready, we can either look for a specific outfit match, or run the colors through all of the schemes and check for at least one matching outcome.

Results

To try and test the algorithm, we can do a quick Google image search for matching and mismatching outfits. It’s easy enough to find matching examples as there are plenty of guides on matching clothes. We may have some trouble finding mismatched outfits since those aren’t so publicly available.

Works as expected given that all colors are neutral. The ‘Analogous’ classification is technically correct, but a more accurate description would me ‘Monochromatic’, given that the colors fall under the same hue but have different tones.

Works as expected.

Works as expected since this outfit is composed of mostly dark and cool colors.

Works as expected. We can observe a potential problem in this picture. We know that the shoes are white, but the backdrop of the photo is giving some blue hue to them. The classification is still accurate, but more tests should be made to see any similar issues.

Works as expected. This outfit was found while searching for ‘mismatching outfits’.

Works as expected. This looks like a colorful but risky outfit, so it isn’t being classified as ‘Basic’.

Final thoughts

Overall, we can find that these results are good enough for a proof of concept project but can be further tweaked to increase accuracy. There are some tricky colors such as purple, brown and green that should be investigated more thoroughly and possibly added as special cases.

If this project was to be put into production as a product targeted towards the colorblind, it would most likely only consider safer outfits such as the Basic, Neutral, Winter styles, and avoid some risky outfits with high contrast like Analogous, and Summer. Then again, some of these risky outfits can also come out as fashionable depending on the context.

It is also important to mention that it isn’t too hard to mismatch a simple outfit, and that the mismatching probability increases when more items are added to the outfit like jackets, belts, socks, etc.. Additionally, there are some other details that can be taken into consideration when choosing an outfit:

• _- Textures/patterns may also have a big impact on the outfit
• Formality/elegance of an item should also be taken into account ex. polo shirt (formal) vs sweatpants (informal)
• Skin color may have an impact in the overall look on an outfit_

As for the fuzzy logic model, we can always increase it’s complexity to get more precise descriptions, as well as dig deeper into the different membership functions that can be used with the fuzzy variables to get better results.

The ideal final product would implement this solution along with computer vision to identify the garments with their respective color. Additionally this could be developed as an AR app for any upcoming AR glasses to simulate a real time clothing assistant that could help the color blind or anyone who would like fashion advice.

---

Any feedback or constructive criticism is welcomed in the comments

Github Repo: https://github.com/FCARRILLOM/ClassifyingColorMatchingOutfits

Special thanks to: Valeria, for sharing her fashion knowledge. (find her fashion profile on Instagram @wtf.isfashion) Juan Diego, for sharing his experience with colorblindness.