Setting Up

You can follow along this tutorial by running this notebook on Google Colab.

🚧 Google Colab Free Tier Running this tutorial on Google Colab is possible but may take a while to complete due to the low computing resources provided in the free tier. We recommend running this tutorial on your local machine, Google Colab Pro, or equivalent.

If you’re running this tutorial on your local machine, install fastdup with:

pip install fastdup

To verify the installation, run:

import fastdup
fastdup.__version__

This tutorial runs on version 0.906. For a detailed list of installation options and supported platforms, see our installation guide.

Download Dataset

For the purpose of demonstration, we will be using the food-101 dataset which consists of 101 food classes with 1,000 images per class. Download and extract the dataset by running:

wget http://data.vision.ee.ethz.ch/cvl/food-101.tar.gz
tar xzf food-101.tar.gz

Once done, you should have a food-101/images folder which contains the images.

📘 Why this dataset? We use the food-101 dataset in this tutorial because of the general availability of the dataset to the public. Bear in mind this is a highly curated dataset, and we may not find as many issues compared to a non-curated dataset. Feel free to swap out this dataset for your own!

Run fastdup

With the folder set in place, let’s run fastdup:

import fastdup  
fd = fastdup.create(work_dir="fastdup_food101_work_dir/",
                    input_dir="food-101/images/")
fd.run(ccthreshold=0.9) 

📘 Parameters

work_dir - Path to store the artifacts generated from the run.

input_dir - Path to the images.

ccthreshold - The cluster threshold parameter. Controls the minimal distance for clustering. Defaults to 0.96. Best value of ccthreshold varies depending on use case and data.

A higher threshold clusters images that are highly similar resulting in fewer images in a cluster.

A lower threshold clusters less similar images together. Clusters have more diversity and a larger possible difference between images in the cluster.

👍 Reduce run time on free tier of Google Colab If you’re running this tutorial on the free tier of Google Colab, we recommend running the analysis on a subset of the dataset instead of the entire dataset. This is done to reduce the waiting time for the run to complete. You can specify the number of images to run on by specifying the num_images argument in fd.run. For example, fd.run(num_images=40000) runs only on 40,000 images in the dataset.

Once the run completes, you can get a summary of the run with:

fd.summary()

which outputs:

########################################################################################

Dataset Analysis Summary: 

    Dataset contains 40000 images
    Valid images are 100.00% (40,000) of the data, invalid are 0.00% (0) of the data
    Similarity:  1.26% (504) belong to 17 similarity clusters (components).
    98.74% (39,496) images do not belong to any similarity cluster.
    Largest cluster has 30 (0.07%) images.
    For a detailed analysis, use `.connected_components()`
(similarity threshold used is 0.9, connected component threshold used is 0.9).

    Outliers: 6.02% (2,409) of images are possible outliers, and fall in the bottom 5.00% of similarity values.
    For a detailed list of outliers, use `.outliers()`.
['Dataset contains 40000 images',
 'Valid images are 100.00% (40,000) of the data, invalid are 0.00% (0) of the data',
 'Similarity:  1.26% (504) belong to 17 similarity clusters (components).',
 '98.74% (39,496) images do not belong to any similarity cluster.',
 'Largest cluster has 30 (0.07%) images.',
 'For a detailed analysis, use `.connected_components()`\n(similarity threshold used is 0.9, connected component threshold used is 0.9).\n',
 'Outliers: 6.02% (2,409) of images are possible outliers, and fall in the bottom 5.00% of similarity values.',
 'For a detailed list of outliers, use `.outliers()`.']

Broken Images

Similar to the previous tutorial, let’s start with low-hanging fruit of finding corrupted images:

fd.invalid_instances()

which outputs:

img_filename	fastdup_id	error_code	is_valid
Abyssinian_34.jpg	135	ERROR_ZERO_SIZE_FILE	False
Egyptian_Mau_139.jpg	2240	ERROR_ZERO_SIZE_FILE	False
Egyptian_Mau_145.jpg	2247	ERROR_ZERO_SIZE_FILE	False

📘 No broken images! The output shows no broken images. So we are good to go here.

However, if there are broken images present (like in the previous tutorial), you’d see something like the following:

List of Broken Images

To get a list of broken images run:

broken_images = fd.invalid_instances()
list_of_broken_images = broken_images['img_filename'].to_list()
list_of_broken_images

Since we did not have any broke images the output of the above code is:

[]

If fastdup encounters broken images, the output of the above snippet would look something like:

['Abyssinian_34.jpg',
 'Egyptian_Mau_139.jpg',
 'Egyptian_Mau_145.jpg']

👍 Tips You can store these output list somewhere to take further action on. You might want to move the files, delete it, or relabel them.

Duplicates

Let’s visualize duplicate image pairs with:

fd.vis.duplicates_gallery(num_images=5)

which outputs: Duplicate Image Pairs

👍 Tips

Setting num_images=5 shows a gallery of 5 duplicate pairs. Change this value to view more/less.

Running fd.vis.duplicates_gallery also saves the resulting duplicates.html file into fastdup_food101_work_dir/gallery/.

Distance of 1.0 indicates that the image is an exact copy.

Image Clusters

You can visualize image clusters with:

fd.vis.component_gallery(num_images=5)

which outputs:

List of Duplicates

Now let’s single out all duplicates and near-duplicates by running using the connected components function:

connected_components_df , _ = fd.connected_components()
connected_components_df.head()

which outputs:

	fastdup_id	component_id	img_filename	error_code	is_valid
0	0	0	apple_pie/1005649.jpg	VALID	True
1	1	1	apple_pie/1011328.jpg	VALID	True
2	2	2	apple_pie/101251.jpg	VALID	True
3	3	3	apple_pie/1014775.jpg	VALID	True
4	4	4	apple_pie/1026328.jpg	VALID	True

Let’s now write a utility function to get the clusters:

# a function to group connected components
def get_clusters(df, sort_by='count', min_count=2, ascending=False):
    # columns to aggregate
    agg_dict = {'img_filename': list, 'mean_distance': max, 'count': len}

    if 'label' in df.columns:
        agg_dict['label'] = list
    
    # filter by count
    df = df[df['count'] >= min_count]
    
    # group and aggregate columns
    grouped_df = df.groupby('component_id').agg(agg_dict)
    
    # sort
    grouped_df = grouped_df.sort_values(by=[sort_by], ascending=ascending)
    return grouped_df

And run it:

clusters_df = get_clusters(connected_components_df)
clusters_df.head()

	img_filename	mean_distance	count
component_id
23830	[clam_chowder/1072684.jpg, clam_chowder/1113834.jpg, clam_chowder/1322415.jpg, clam_chowder/1437241.jpg, clam_chowder/2113399.jpg, clam_chowder/2140703.jpg, clam_chowder/2248997.jpg, clam_chowder/2361787.jpg, clam_chowder/2398168.jpg, clam_chowder/2542800.jpg, clam_chowder/2685745.jpg, clam_chowder/2770581.jpg, clam_chowder/3914755.jpg, clam_chowder/546975.jpg, clam_chowder/75800.jpg, clam_chowder/854517.jpg]	0.9163	16
31637	[dumplings/1045500.jpg, dumplings/140004.jpg, dumplings/1630799.jpg, dumplings/1695231.jpg, dumplings/1848359.jpg, dumplings/1872410.jpg, dumplings/1918394.jpg, dumplings/2524385.jpg, dumplings/3683752.jpg, dumplings/3739057.jpg, dumplings/3781725.jpg, dumplings/468796.jpg]	0.9302	12
31767	[dumplings/1450685.jpg, dumplings/1564985.jpg, dumplings/2500721.jpg, dumplings/2600333.jpg, dumplings/2606645.jpg, dumplings/2675187.jpg, dumplings/3030550.jpg, dumplings/3242297.jpg, dumplings/3532122.jpg, dumplings/625116.jpg]	0.9127	10
31760	[dumplings/1433645.jpg, dumplings/1813271.jpg, dumplings/1881086.jpg, dumplings/1998135.jpg, dumplings/2229749.jpg, dumplings/2561548.jpg, dumplings/2750447.jpg, dumplings/3363745.jpg, dumplings/834049.jpg]	0.9119	9
31699	[dumplings/1228546.jpg, dumplings/1270308.jpg, dumplings/231028.jpg, dumplings/2373653.jpg, dumplings/2571523.jpg, dumplings/263589.jpg, dumplings/2909040.jpg, dumplings/2950605.jpg, dumplings/3191742.jpg]	0.9180	9

The above shows the component (clusters) with the highest duplicates/near-duplicates. Now let’s keep one image from each cluster and remove the rest:

# First sample from each cluster that is kept
cluster_images_to_keep = []
list_of_duplicates = []

for cluster_file_list in clusters_df.img_filename:
    # keep first file, discard rest
    keep = cluster_file_list[0]
    discard = cluster_file_list[1:]
    
    cluster_images_to_keep.append(keep)
    list_of_duplicates.extend(discard)

print(f"Found {len(set(list_of_duplicates))} highly similar images to discard")

outputs:

Found 610 highly similar images to discard

Inspecting list_of_duplicates:

list_of_duplicates

outputs:

['clam_chowder/1113834.jpg',
 'clam_chowder/1322415.jpg',
 'clam_chowder/1437241.jpg',
 'clam_chowder/2113399.jpg',
 'clam_chowder/2140703.jpg',
 'clam_chowder/2248997.jpg',
 'clam_chowder/2361787.jpg',
 'clam_chowder/2398168.jpg',
 'clam_chowder/2542800.jpg',
 'clam_chowder/2685745.jpg',
 'clam_chowder/2770581.jpg',
 'clam_chowder/3914755.jpg',
 'clam_chowder/546975.jpg',
 'clam_chowder/75800.jpg',
 'clam_chowder/854517.jpg',
 'dumplings/140004.jpg',
 'dumplings/1630799.jpg',
 'dumplings/1695231.jpg',
 'dumplings/1848359.jpg',
 'dumplings/1872410.jpg',
 'dumplings/1918394.jpg',
 'dumplings/2524385.jpg',
 'dumplings/3683752.jpg',
 'dumplings/3739057.jpg',
 'dumplings/3781725.jpg',
 'dumplings/468796.jpg',
 'dumplings/1564985.jpg',
 'dumplings/2500721.jpg',
 'dumplings/2600333.jpg',
 'dumplings/2606645.jpg',
 'dumplings/2675187.jpg',
 'dumplings/3030550.jpg',
 'dumplings/3242297.jpg',
 'dumplings/3532122.jpg',
 'dumplings/625116.jpg',
 'dumplings/1813271.jpg',
 'dumplings/1881086.jpg',
 'dumplings/1998135.jpg',
 'dumplings/2229749.jpg',
 'dumplings/2561548.jpg',
 'dumplings/2750447.jpg',
 'dumplings/3363745.jpg',
 'dumplings/834049.jpg',
 'dumplings/1270308.jpg',
 'dumplings/231028.jpg',
 'dumplings/2373653.jpg',
 'dumplings/2571523.jpg',
 'dumplings/263589.jpg',
 'dumplings/2909040.jpg',
 'dumplings/2950605.jpg',
 'dumplings/3191742.jpg',
 'dumplings/1276808.jpg',
 'dumplings/1308246.jpg',
 'dumplings/1598923.jpg',
 'dumplings/2546897.jpg',
 'dumplings/2630977.jpg',
 'dumplings/263764.jpg',
 'dumplings/3412861.jpg',
 'dumplings/646942.jpg',
 'edamame/1714523.jpg',
 'edamame/2204418.jpg',
 'edamame/2483789.jpg',
 'edamame/2670224.jpg',
 'edamame/3432193.jpg',
 'edamame/3666348.jpg',
 'edamame/3788141.jpg',
 'edamame/825581.jpg',
 'clam_chowder/1945594.jpg',
 'clam_chowder/2508514.jpg',
 'clam_chowder/2673628.jpg',
 'clam_chowder/2789238.jpg',
 'clam_chowder/3549975.jpg',
 'clam_chowder/758162.jpg',
 'clam_chowder/903815.jpg',
 'clam_chowder/2509774.jpg',
 'clam_chowder/2676197.jpg',
 'clam_chowder/2992252.jpg',
 'clam_chowder/3264840.jpg',
 'clam_chowder/3395059.jpg',
 'clam_chowder/906900.jpg',
 'clam_chowder/967946.jpg',
 'clam_chowder/1511884.jpg',
 'clam_chowder/2426821.jpg',
 'clam_chowder/2476027.jpg',
 'clam_chowder/2869301.jpg',
 'clam_chowder/2897057.jpg',
 'clam_chowder/3291238.jpg',
 'clam_chowder/3830343.jpg',
 'dumplings/1563646.jpg',
 'dumplings/1897260.jpg',
 'dumplings/2444294.jpg',
 'dumplings/2951551.jpg',
 'dumplings/3101737.jpg',
 'dumplings/310672.jpg',
 'dumplings/3279575.jpg',
 'creme_brulee/1207812.jpg',
 'creme_brulee/1742194.jpg',
 'creme_brulee/1816938.jpg',
 'creme_brulee/312639.jpg',
 'creme_brulee/480234.jpg',
 'creme_brulee/59534.jpg',
 'club_sandwich/1318118.jpg',
 'club_sandwich/1775789.jpg',
 'club_sandwich/1886101.jpg',
 'club_sandwich/2778614.jpg',
 'club_sandwich/3106065.jpg',
 'club_sandwich/588478.jpg',
 'edamame/1086703.jpg',
 'edamame/2040753.jpg',
 'edamame/2390868.jpg',
 'edamame/3325153.jpg',
 'edamame/3520889.jpg',
 'edamame/677508.jpg',
 'club_sandwich/1840706.jpg',
 'club_sandwich/2272423.jpg',
 'club_sandwich/3526250.jpg',
 'club_sandwich/3646665.jpg',
 'club_sandwich/3664710.jpg',
 'dumplings/2380724.jpg',
 'dumplings/2707946.jpg',
 'dumplings/587831.jpg',
 'dumplings/876327.jpg',
 'dumplings/937912.jpg',
 'dumplings/1545564.jpg',
 'dumplings/1848509.jpg',
 'dumplings/3359158.jpg',
 'dumplings/3619519.jpg',
 'dumplings/3686831.jpg',
 'beignets/2399174.jpg',
 'beignets/2683786.jpg',
 'beignets/3520470.jpg',
 'beignets/595743.jpg',
 'beignets/832877.jpg',
 'edamame/2778957.jpg',
 'edamame/3519994.jpg',
 'edamame/3546677.jpg',
 'edamame/579614.jpg',
 'edamame/846598.jpg',
 'clam_chowder/3137773.jpg',
 'clam_chowder/686716.jpg',
 'clam_chowder/762499.jpg',
 'clam_chowder/777422.jpg',
 'clam_chowder/804904.jpg',
 'clam_chowder/3073323.jpg',
 'clam_chowder/3142771.jpg',
 'clam_chowder/3228022.jpg',
 'clam_chowder/513498.jpg',
 'bruschetta/2018603.jpg',
 'bruschetta/2229245.jpg',
 'bruschetta/261311.jpg',
 'bruschetta/3743680.jpg',
 'breakfast_burrito/1058434.jpg',
 'caesar_salad/2599756.jpg',
 'caesar_salad/520391.jpg',
 'eggs_benedict/2066348.jpg',
 'clam_chowder/2894611.jpg',
 'clam_chowder/596255.jpg',
 'clam_chowder/907742.jpg',
 'clam_chowder/947484.jpg',
 'edamame/2588718.jpg',
 'edamame/2847124.jpg',
 'edamame/3558096.jpg',
 'edamame/601042.jpg',
 'bruschetta/243736.jpg',
 'bruschetta/3805917.jpg',
 'bruschetta/3836578.jpg',
 'bruschetta/3896592.jpg',
 'clam_chowder/2396225.jpg',
 'clam_chowder/2603953.jpg',
 'clam_chowder/3689947.jpg',
 'clam_chowder/655847.jpg',
 'club_sandwich/1413794.jpg',
 'club_sandwich/1811271.jpg',
 'club_sandwich/2163422.jpg',
 'club_sandwich/3543955.jpg',
 'dumplings/1370046.jpg',
 'dumplings/1510091.jpg',
 'dumplings/2531851.jpg',
 'edamame/3185310.jpg',
 'edamame/3569901.jpg',
 'edamame/965396.jpg',
 'edamame/2975349.jpg',
 'edamame/3152528.jpg',
 'edamame/3301986.jpg',
 'bibimbap/1615665.jpg',
 'bibimbap/2795629.jpg',
 'bibimbap/964368.jpg',
 'bibimbap/2519286.jpg',
 'bibimbap/2572183.jpg',
 'bibimbap/3003579.jpg',
 'caesar_salad/3673948.jpg',
 'caesar_salad/620905.jpg',
 'caesar_salad/709638.jpg',
 'dumplings/2975772.jpg',
 'dumplings/3888349.jpg',
 'dumplings/599168.jpg',
 'bibimbap/2534963.jpg',
 'bibimbap/3571528.jpg',
 'bibimbap/3627919.jpg',
 'edamame/3213278.jpg',
 'edamame/3634423.jpg',
 'edamame/3920329.jpg',
 'dumplings/322034.jpg',
 'dumplings/3428971.jpg',
 'dumplings/432.jpg',
 'chicken_quesadilla/3004094.jpg',
 'chicken_quesadilla/3779974.jpg',
 'dumplings/2736144.jpg',
 'dumplings/3430692.jpg',
 'caesar_salad/3402604.jpg',
 'caesar_salad/3703325.jpg',
 'clam_chowder/1942294.jpg',
 'clam_chowder/2027156.jpg',
 'breakfast_burrito/662423.jpg',
 'breakfast_burrito/662424.jpg',
 'churros/3303373.jpg',
 'churros/3303522.jpg',
 'chicken_curry/2701143.jpg',
 'chicken_curry/882723.jpg',
 'clam_chowder/1063260.jpg',
 'clam_chowder/3024138.jpg',
 'bibimbap/2041700.jpg',
 'bibimbap/2346855.jpg',
 'edamame/1622192.jpg',
 'edamame/561133.jpg',
 'beignets/1728932.jpg',
 'beignets/1751352.jpg',
 'dumplings/2770853.jpg',
 'dumplings/625233.jpg',
 'chocolate_cake/51717.jpg',
 'chocolate_cake/55122.jpg',
 'bruschetta/1890619.jpg',
 'bruschetta/3462434.jpg',
 'edamame/1620027.jpg',
 'edamame/2916151.jpg',
 'dumplings/521153.jpg',
 'dumplings/882708.jpg',
 'edamame/1144040.jpg',
 'edamame/1225330.jpg',
 'edamame/684483.jpg',
 'edamame/952423.jpg',
 'crab_cakes/2780621.jpg',
 'crab_cakes/2780623.jpg',
 'edamame/3253578.jpg',
 'edamame/3620419.jpg',
 'croque_madame/3163125.jpg',
 'croque_madame/3865436.jpg',
 'dumplings/2182931.jpg',
 'dumplings/3458910.jpg',
 'dumplings/1146384.jpg',
 'dumplings/2108794.jpg',
 'creme_brulee/2418653.jpg',
 'creme_brulee/3684311.jpg',
 'dumplings/3537145.jpg',
 'dumplings/808822.jpg',
 'dumplings/231024.jpg',
 'dumplings/35818.jpg',
 'croque_madame/3224280.jpg',
 'croque_madame/3288700.jpg',
 'croque_madame/2598646.jpg',
 'croque_madame/3036159.jpg',
 'falafel/3370784.jpg',
 'falafel/438562.jpg',
 'dumplings/1557735.jpg',
 'dumplings/3635848.jpg',
 'escargots/637187.jpg',
 'escargots/637188.jpg',
 'croque_madame/157692.jpg',
 'croque_madame/290729.jpg',
 'ceviche/2796501.jpg',
 'ceviche/895716.jpg',
 'donuts/1774835.jpg',
 'donuts/2563686.jpg',
 'edamame/3243030.jpg',
 'edamame/3313851.jpg',
 'chicken_quesadilla/535532.jpg',
 'chicken_quesadilla/535546.jpg',
 'eggs_benedict/1972975.jpg',
 'eggs_benedict/2528340.jpg',
 'dumplings/3424747.jpg',
 'dumplings/55070.jpg',
 'edamame/3028728.jpg',
 'edamame/3112981.jpg',
 'eggs_benedict/3225684.jpg',
 'eggs_benedict/535020.jpg',
 'beef_tartare/1361899.jpg',
 'beef_tartare/3437886.jpg',
 'clam_chowder/2862215.jpg',
 'clam_chowder/795839.jpg',
 'croque_madame/3776229.jpg',
 'croque_madame/3873257.jpg',
 'clam_chowder/2641960.jpg',
 'clam_chowder/3289212.jpg',
 'donuts/2117632.jpg',
 'deviled_eggs/3281495.jpg',
 'donuts/3089074.jpg',
 'deviled_eggs/3902179.jpg',
 'deviled_eggs/3058137.jpg',
 'deviled_eggs/584369.jpg',
 'donuts/3124075.jpg',
 'deviled_eggs/3246571.jpg',
 'donuts/1954438.jpg',
 'deviled_eggs/3806337.jpg',
 'deviled_eggs/2671994.jpg',
 'apple_pie/1469191.jpg',
 'deviled_eggs/3491525.jpg',
 'croque_madame/2555777.jpg',
 'croque_madame/1870619.jpg',
 'croque_madame/3322423.jpg',
 'croque_madame/2269229.jpg',
 'croque_madame/1306940.jpg',
 'croque_madame/1497073.jpg',
 'creme_brulee/3245776.jpg',
 'creme_brulee/3155386.jpg',
 'creme_brulee/3487185.jpg',
 'creme_brulee/3054304.jpg',
 'creme_brulee/332369.jpg',
 'creme_brulee/2610691.jpg',
 'creme_brulee/2680133.jpg',
 'creme_brulee/2262132.jpg',
 'creme_brulee/2602002.jpg',
 'creme_brulee/2085820.jpg',
 'creme_brulee/2376691.jpg',
 'creme_brulee/722718.jpg',
 'croque_madame/611043.jpg',
 'croque_madame/691718.jpg',
 'deviled_eggs/2178531.jpg',
 'croque_madame/2962203.jpg',
 'deviled_eggs/1923965.jpg',
 'deviled_eggs/1721209.jpg',
 'deviled_eggs/1619934.jpg',
 'deviled_eggs/1568041.jpg',
 'deviled_eggs/1527126.jpg',
 'deviled_eggs/1378330.jpg',
 'donuts/2512789.jpg',
 'deviled_eggs/3021655.jpg',
 'cup_cakes/556378.jpg',
 'cup_cakes/1493261.jpg',
 'cup_cakes/1082593.jpg',
 'croque_madame/914187.jpg',
 'croque_madame/878201.jpg',
 'croque_madame/580678.jpg',
 'croque_madame/880779.jpg',
 'croque_madame/392709.jpg',
 'croque_madame/3414159.jpg',
 'donuts/2499239.jpg',
 'dumplings/2084607.jpg',
 'donuts/861022.jpg',
 'edamame/3840513.jpg',
 'falafel/1206667.jpg',
 'escargots/563386.jpg',
 'escargots/3688869.jpg',
 'escargots/3468449.jpg',
 'escargots/2667969.jpg',
 'escargots/2646994.jpg',
 'escargots/3004581.jpg',
 'escargots/2211156.jpg',
 'escargots/1637284.jpg',
 'escargots/2491502.jpg',
 'eggs_benedict/901333.jpg',
 'eggs_benedict/3238266.jpg',
 'eggs_benedict/3574668.jpg',
 'eggs_benedict/721876.jpg',
 'edamame/979556.jpg',
 'edamame/587222.jpg',
 'edamame/453226.jpg',
 'falafel/295629.jpg',
 'falafel/2505830.jpg',
 'falafel/3086998.jpg',
 'foie_gras/2870358.jpg',
 'foie_gras/459507.jpg',
 'foie_gras/3382988.jpg',
 'foie_gras/3029045.jpg',
 'foie_gras/3105826.jpg',
 'foie_gras/2857159.jpg',
 'foie_gras/2291174.jpg',
 'foie_gras/1721540.jpg',
 'foie_gras/21278.jpg',
 'falafel/3464997.jpg',
 'foie_gras/1051567.jpg',
 'filet_mignon/734006.jpg',
 'filet_mignon/646511.jpg',
 'filet_mignon/1666949.jpg',
 'falafel/3882357.jpg',
 'falafel/3789344.jpg',
 'falafel/3001734.jpg',
 'edamame/3831507.jpg',
 'edamame/667469.jpg',
 'dumplings/2106100.jpg',
 'edamame/2900759.jpg',
 'edamame/1659005.jpg',
 'dumplings/955413.jpg',
 'dumplings/774604.jpg',
 'dumplings/6201.jpg',
 'dumplings/663266.jpg',
 'dumplings/2545565.jpg',
 'dumplings/633367.jpg',
 'dumplings/28220.jpg',
 'dumplings/856176.jpg',
 'dumplings/267852.jpg',
 'dumplings/2800182.jpg',
 'dumplings/3554779.jpg',
 'dumplings/180290.jpg',
 'dumplings/231026.jpg',
 'dumplings/3153246.jpg',
 'dumplings/2932420.jpg',
 'dumplings/2942258.jpg',
 'edamame/1346107.jpg',
 'edamame/488373.jpg',
 'edamame/2977649.jpg',
 'edamame/2473555.jpg',
 'edamame/2803276.jpg',
 'edamame/3041151.jpg',
 'edamame/2708664.jpg',
 'edamame/2230705.jpg',
 'edamame/2545734.jpg',
 'edamame/3119358.jpg',
 'edamame/336171.jpg',
 'edamame/2574083.jpg',
 'edamame/2558511.jpg',
 'edamame/804283.jpg',
 'edamame/1969958.jpg',
 'edamame/864875.jpg',
 'edamame/2499082.jpg',
 'edamame/2302171.jpg',
 'edamame/1821106.jpg',
 'edamame/2157980.jpg',
 'creme_brulee/1888025.jpg',
 'clam_chowder/390727.jpg',
 'crab_cakes/2194081.jpg',
 'bibimbap/1809239.jpg',
 'bibimbap/1792799.jpg',
 'bibimbap/628343.jpg',
 'bibimbap/892182.jpg',
 'beignets/727595.jpg',
 'beignets/3573964.jpg',
 'beignets/492391.jpg',
 'beignets/2706264.jpg',
 'donuts/708597.jpg',
 'beignets/518797.jpg',
 'beignets/2004832.jpg',
 'beignets/1997437.jpg',
 'beignets/2735628.jpg',
 'beignets/935415.jpg',
 'beignets/1428238.jpg',
 'beignets/3873758.jpg',
 'beet_salad/3268468.jpg',
 'beet_salad/2671983.jpg',
 'beet_salad/374126.jpg',
 'beet_salad/1855829.jpg',
 'bibimbap/2499871.jpg',
 'bibimbap/574280.jpg',
 'bruschetta/3838937.jpg',
 'bibimbap/913532.jpg',
 'bruschetta/619290.jpg',
 'bruschetta/3696492.jpg',
 'bruschetta/3711344.jpg',
 'bruschetta/2161394.jpg',
 'caprese_salad/2730842.jpg',
 'breakfast_burrito/931734.jpg',
 'breakfast_burrito/491065.jpg',
 'ceviche/1205283.jpg',
 'bruschetta/711623.jpg',
 'bread_pudding/502700.jpg',
 'bibimbap/3884378.jpg',
 'bibimbap/3611974.jpg',
 'bibimbap/890594.jpg',
 'bibimbap/3670923.jpg',
 'bibimbap/495544.jpg',
 'bibimbap/2988372.jpg',
 'bibimbap/3096950.jpg',
 'bibimbap/3837493.jpg',
 'bibimbap/2399561.jpg',
 'beet_salad/1404312.jpg',
 'beef_tartare/50036.jpg',
 'beef_tartare/3646367.jpg',
 'beef_tartare/97478.jpg',
 'baklava/3518558.jpg',
 'baklava/3158786.jpg',
 'baklava/2209150.jpg',
 'baklava/2015716.jpg',
 'baklava/2186251.jpg',
 'baklava/1458610.jpg',
 'baklava/1150170.jpg',
 'baby_back_ribs/620997.jpg',
 'baby_back_ribs/3265047.jpg',
 'baby_back_ribs/3620137.jpg',
 'baby_back_ribs/3142431.jpg',
 'baby_back_ribs/3125728.jpg',
 'filet_mignon/2427308.jpg',
 'baby_back_ribs/801284.jpg',
 'baby_back_ribs/2306066.jpg',
 'baby_back_ribs/2129884.jpg',
 'apple_pie/839845.jpg',
 'apple_pie/3670966.jpg',
 'apple_pie/3324492.jpg',
 'beef_carpaccio/885771.jpg',
 'beef_carpaccio/679379.jpg',
 'beef_carpaccio/2290534.jpg',
 'beet_salad/686615.jpg',
 'beef_tartare/3191961.jpg',
 'beef_tartare/3185389.jpg',
 'beef_tartare/2561385.jpg',
 'beef_tartare/2426755.jpg',
 'beef_tartare/2030974.jpg',
 'beef_tartare/1562966.jpg',
 'beef_tartare/3722200.jpg',
 'beef_tartare/2038606.jpg',
 'beef_carpaccio/721638.jpg',
 'beef_carpaccio/2434359.jpg',
 'beef_carpaccio/3323355.jpg',
 'beef_carpaccio/3252686.jpg',
 'beef_carpaccio/3289048.jpg',
 'beef_carpaccio/3394009.jpg',
 'beef_carpaccio/2035002.jpg',
 'beef_carpaccio/1926900.jpg',
 'beef_carpaccio/1801501.jpg',
 'beef_carpaccio/2907748.jpg',
 'bruschetta/3387732.jpg',
 'bruschetta/3790099.jpg',
 'crab_cakes/814716.jpg',
 'clam_chowder/2385341.jpg',
 'clam_chowder/2742139.jpg',
 'clam_chowder/2754706.jpg',
 'clam_chowder/2148133.jpg',
 'clam_chowder/3126055.jpg',
 'clam_chowder/3045182.jpg',
 'clam_chowder/3457812.jpg',
 'clam_chowder/780765.jpg',
 'clam_chowder/3588064.jpg',
 'clam_chowder/1783836.jpg',
 'deviled_eggs/1999024.jpg',
 'clam_chowder/2762472.jpg',
 'clam_chowder/2553830.jpg',
 'clam_chowder/1871262.jpg',
 'clam_chowder/3572725.jpg',
 'churros/644700.jpg',
 'churros/2617186.jpg',
 'churros/1683636.jpg',
 'chocolate_mousse/2048018.jpg',
 'chocolate_mousse/2673864.jpg',
 'clam_chowder/2503659.jpg',
 'clam_chowder/3031443.jpg',
 'caesar_salad/728727.jpg',
 'clam_chowder/924933.jpg',
 'crab_cakes/20845.jpg',
 'crab_cakes/1460400.jpg',
 'crab_cakes/2885408.jpg',
 'fish_and_chips/359280.jpg',
 'club_sandwich/3845195.jpg',
 'club_sandwich/3550782.jpg',
 'club_sandwich/1377451.jpg',
 'club_sandwich/2856571.jpg',
 'club_sandwich/1380208.jpg',
 'club_sandwich/736461.jpg',
 'clam_chowder/948137.jpg',
 'clam_chowder/9768.jpg',
 'clam_chowder/511201.jpg',
 'apple_pie/1487150.jpg',
 'clam_chowder/3322877.jpg',
 'clam_chowder/3508063.jpg',
 'clam_chowder/3307340.jpg',
 'clam_chowder/3115414.jpg',
 'clam_chowder/2961270.jpg',
 'chicken_wings/834669.jpg',
 'chicken_wings/811798.jpg',
 'chicken_wings/3108137.jpg',
 'chicken_quesadilla/2223295.jpg',
 'cannoli/3846450.jpg',
 'cannoli/2295498.jpg',
 'cannoli/1982944.jpg',
 'cannoli/1357678.jpg',
 'eggs_benedict/1010197.jpg',
 'caesar_salad/3791298.jpg',
 'caesar_salad/3627251.jpg',
 'caesar_salad/3325086.jpg',
 'caesar_salad/3381505.jpg',
 'eggs_benedict/2748311.jpg',
 'caesar_salad/2707518.jpg',
 'caesar_salad/2683955.jpg',
 'caesar_salad/2319739.jpg',
 'caesar_salad/3912473.jpg',
 'caesar_salad/3315261.jpg',
 'caesar_salad/3479395.jpg',
 'chicken_quesadilla/2025030.jpg',
 'caesar_salad/3637443.jpg',
 'caesar_salad/2874871.jpg',
 'caprese_salad/992553.jpg',
 'caprese_salad/1473449.jpg',
 'caprese_salad/1411082.jpg',
 'ceviche/2523261.jpg',
 'chicken_quesadilla/1590716.jpg',
 'chicken_curry/70091.jpg',
 'chicken_curry/3496679.jpg',
 'chicken_curry/2617143.jpg',
 'cheese_plate/618425.jpg',
 'cheese_plate/3545251.jpg',
 'cheese_plate/3026695.jpg',
 'eggs_benedict/158871.jpg',
 'ceviche/172529.jpg',
 'caprese_salad/3753434.jpg',
 'carrot_cake/527702.jpg',
 'carrot_cake/3768473.jpg',
 'carrot_cake/3512754.jpg',
 'carrot_cake/3374621.jpg',
 'carrot_cake/3889387.jpg',
 'caprese_salad/763201.jpg',
 'caprese_salad/3289013.jpg',
 'caprese_salad/87213.jpg',
 'foie_gras/79314.jpg']

Additional reading: remove_duplicates

Outliers

Now, visualize the outliers with:

fd.vis.outliers_gallery(num_images=5)

which outputs: Outliers

👍 Tips

Lower Distance value indicates the image is different than others in the dataset. Hence, the lower the Distance value, the higher the chances of outliers.

Since we know that the label of the image is by the name of the parent folder, we can already spot a couple of outliers from this gallery. For example, take the last image in the gallery, the label is given as dumplings but the image does not contain any dumplings.

List of Outliers

Let’s first get the outliers DataFrame:

outlier_df = fd.outliers()
outlier_df.head()

Which outputs:

	index	outlier	nearest	distance	img_filename_outlier	error_code_outlier	is_valid_outlier	img_filename_nearest	error_code_nearest	is_valid_nearest
0	3999	9797	27221	0.295020	breakfast_burrito/462294.jpg	VALID	True	creme_brulee/1661605.jpg	VALID	True
1	3997	21410	37470	0.556575	chocolate_cake/2518457.jpg	VALID	True	filet_mignon/2685908.jpg	VALID	True
2	3995	11063	16727	0.563040	caesar_salad/1303023.jpg	VALID	True	cheesecake/358018.jpg	VALID	True
3	3994	21885	2669	0.564055	chocolate_cake/577717.jpg	VALID	True	baklava/3363412.jpg	VALID	True
4	3992	32123	31207	0.578329	dumplings/1339572.jpg	VALID	True	donuts/1750980.jpg	VALID	True

Let’s treat all images with distance<0.68 as outliers:

list_of_outliers = outlier_df[outlier_df.distance < 0.68].img_filename_outlier.tolist()
list_of_outliers

Outputs:

['breakfast_burrito/462294.jpg',  
 'chocolate_cake/2518457.jpg',  
 'caesar_salad/1303023.jpg',  
 'chocolate_cake/577717.jpg',  
 'dumplings/1339572.jpg',  
 'bibimbap/2594394.jpg',  
 'ceviche/2363511.jpg',  
 'churros/2327883.jpg',  
 'chicken_wings/693809.jpg',  
 'foie_gras/3776193.jpg',  
 'chicken_curry/2523126.jpg',  
 'churros/1440917.jpg',  
 'creme_brulee/1661605.jpg',  
 'apple_pie/484038.jpg',  
 'foie_gras/33258.jpg',  
 'cheesecake/2160930.jpg',  
 'cheesecake/1955517.jpg',  
 'chicken_curry/789540.jpg',  
 'cup_cakes/451074.jpg',  
 'cup_cakes/1005580.jpg',  
 'bread_pudding/1375816.jpg',  
 'chocolate_mousse/2177988.jpg',  
 'bruschetta/1883187.jpg',  
 'chocolate_cake/3600589.jpg',  
 'apple_pie/236966.jpg',  
 'caprese_salad/2719211.jpg',  
 'bibimbap/3230839.jpg',  
 'apple_pie/2008772.jpg',  
 'edamame/2979095.jpg',  
 'fish_and_chips/1566646.jpg',  
 'cup_cakes/601989.jpg',  
 'filet_mignon/2685908.jpg',  
 'baklava/3236360.jpg',  
 'baby_back_ribs/1676135.jpg',  
 'cup_cakes/2590269.jpg',  
 'chocolate_cake/2814515.jpg',  
 'churros/1972000.jpg',  
 'clam_chowder/759125.jpg',  
 'falafel/2585154.jpg',  
 'cup_cakes/630654.jpg',  
 'baklava/1553505.jpg',  
 'chocolate_cake/1749296.jpg',  
 'beignets/3506219.jpg',  
 'cheesecake/811556.jpg',  
 'chocolate_cake/1646662.jpg',  
 'donuts/921183.jpg',  
 'donuts/3316195.jpg',  
 'foie_gras/235773.jpg',  
 'churros/2550886.jpg',  
 'filet_mignon/2685899.jpg',  
 'chocolate_cake/2479257.jpg',  
 'beet_salad/1456898.jpg',  
 'cheesecake/2465886.jpg',  
 'churros/1658982.jpg',  
 'creme_brulee/107007.jpg',  
 'churros/3690003.jpg',  
 'chocolate_cake/1244445.jpg',  
 'apple_pie/755031.jpg',  
 'deviled_eggs/2854885.jpg',  
 'cannoli/3300725.jpg',  
 'churros/3169818.jpg',  
 'donuts/794976.jpg',  
 'cannoli/1070382.jpg',  
 'beet_salad/1643533.jpg',  
 'chocolate_mousse/2048999.jpg',  
 'churros/2741606.jpg',  
 'beignets/726875.jpg',  
 'chocolate_mousse/2287892.jpg',  
 'filet_mignon/3030737.jpg',  
 'fish_and_chips/876010.jpg',  
 'churros/1944265.jpg',  
 'cheese_plate/3119696.jpg',  
 'donuts/456541.jpg',  
 'churros/962826.jpg',  
 'churros/679673.jpg',  
 'donuts/1452592.jpg',  
 'donuts/3347684.jpg',  
 'baklava/3278527.jpg',  
 'bread_pudding/2585974.jpg',  
 'beef_tartare/913291.jpg',  
 'creme_brulee/1138671.jpg',  
 'chocolate_mousse/3604313.jpg',  
 'chocolate_mousse/1320051.jpg',  
 'chocolate_cake/985141.jpg',  
 'chocolate_cake/51412.jpg',  
 'cheesecake/2617496.jpg',  
 'club_sandwich/1127992.jpg',  
 'escargots/3406878.jpg',  
 'carrot_cake/580925.jpg',  
 'chocolate_cake/2174801.jpg',  
 'chicken_curry/889805.jpg',  
 'chocolate_cake/2067510.jpg',  
 'creme_brulee/202057.jpg',  
 'caprese_salad/2298180.jpg',  
 'chocolate_mousse/2688431.jpg',  
 'chocolate_mousse/2616372.jpg',  
 'chocolate_cake/771009.jpg',  
 'churros/1995090.jpg',  
 'breakfast_burrito/1229548.jpg',  
 'donuts/1167771.jpg',  
 'baby_back_ribs/2083106.jpg',  
 'bibimbap/2011447.jpg',  
 'churros/1977745.jpg',  
 'churros/3447996.jpg',  
 'chocolate_cake/3169533.jpg',  
 'donuts/1828646.jpg',  
 'baklava/1452465.jpg',  
 'chocolate_cake/2280321.jpg',  
 'beignets/3568316.jpg',  
 'beef_tartare/1054197.jpg',  
 'cup_cakes/3691610.jpg']

Dark, Bright, Blurry Images

Similar to the previous tutorial, we can visualize the dark, bright, and blurry images with:

fd.vis.stats_gallery(metric='dark')
fd.vis.stats_gallery(metric='bright')
fd.vis.stats_gallery(metric='blur')

List of Dark Images

Get a DataFrame of dark images:

stats_df = fd.img_stats()

If an image has amean<13 then we conclude it’s a dark image:

dark_images = stats_df[stats_df['mean'] < 13]  
dark_images

Outputs:

	fastdup_id	img_w	img_h	unique	blur	mean	min	max	stdv	file_size	contrast	img_filename	error_code	is_valid
3090	3090	512	306	0	535.7338	5.5205	0.0	255.0	15.3110	27433	1.0	beef_carpaccio/1259270.jpg	VALID	True
9797	9797	511	512	0	9.0875	1.8431	0.0	30.0	1.0524	8693	1.0	breakfast_burrito/462294.jpg	VALID	True

To get a list of the dark images:

list_of_dark_images = dark_images['img_filename'].to_list()
list_of_dark_images

Which outputs:

['beef_carpaccio/1259270.jpg', 'breakfast_burrito/462294.jpg']

List of Bright Images

Similar to the above, let’s set that if mean>220.5 we will conclude it’s a bright image:

bright_images = stats_df[stats_df['mean'] > 220.5]
bright_images.head()

	fastdup_id	img_w	img_h	blur	mean	max	stdv	file_size	contrast	img_filename	error_code	is_valid
81	81	512	512	538.6821	225.8266	255.0	32.2799	32229	1.0	apple_pie/1289014.jpg	VALID	True
436	436	512	512	1245.6737	220.9703	255.0	40.3034	40344	1.0	apple_pie/2601590.jpg	VALID	True
589	589	512	512	1468.0642	227.5742	255.0	41.6247	50437	1.0	apple_pie/2997124.jpg	VALID	True
933	933	512	512	554.9135	232.6887	255.0	41.5226	41395	1.0	apple_pie/817552.jpg	VALID	True
1115	1115	512	512	1219.0579	230.7839	255.0	32.7307	52154	1.0	baby_back_ribs/1395570.jpg	VALID	True

Get a list of bright images:

list_of_bright_images = bright_images['img_filename'].to_list()
list_of_bright_images

Outputs:

['apple_pie/1289014.jpg',
 'apple_pie/2601590.jpg',
 'apple_pie/2997124.jpg',
 'apple_pie/817552.jpg',
 'baby_back_ribs/1395570.jpg',
 'baby_back_ribs/3841100.jpg',
 'baklava/1542333.jpg',
 'baklava/2229944.jpg',
 'baklava/2663954.jpg',
 'beef_carpaccio/1364391.jpg',
 'beef_carpaccio/1713850.jpg',
 'beef_carpaccio/1990775.jpg',
 'beef_carpaccio/3169022.jpg',
 'beef_carpaccio/872076.jpg',
 'beef_tartare/1282738.jpg',
 'beef_tartare/1720794.jpg',
 'beef_tartare/3603995.jpg',
 'beef_tartare/717367.jpg',
 'beignets/1688450.jpg',
 'beignets/3723694.jpg',
 'beignets/529117.jpg',
 'bread_pudding/1256062.jpg',
 'bread_pudding/3660360.jpg',
 'bread_pudding/3716756.jpg',
 'breakfast_burrito/2840993.jpg',
 'breakfast_burrito/3635548.jpg',
 'bruschetta/1346725.jpg',
 'bruschetta/2275519.jpg',
 'bruschetta/3269901.jpg',
 'bruschetta/770721.jpg',
 'caesar_salad/2039808.jpg',
 'caesar_salad/2761224.jpg',
 'cannoli/1237436.jpg',
 'cannoli/1793781.jpg',
 'cannoli/2799600.jpg',
 'cannoli/2821147.jpg',
 'cannoli/421018.jpg',
 'caprese_salad/2126956.jpg',
 'carrot_cake/1932607.jpg',
 'cheesecake/1325649.jpg',
 'cheesecake/2572821.jpg',
 'cheese_plate/1842697.jpg',
 'cheese_plate/3159443.jpg',
 'chicken_curry/2051444.jpg',
 'chicken_curry/2590404.jpg',
 'chicken_curry/3144187.jpg',
 'chicken_curry/3679727.jpg',
 'chicken_quesadilla/2786630.jpg',
 'chicken_quesadilla/3362240.jpg',
 'chicken_wings/2693829.jpg',
 'chocolate_cake/2584547.jpg',
 'churros/1572415.jpg',
 'churros/2151645.jpg',
 'churros/706007.jpg',
 'clam_chowder/1455612.jpg',
 'clam_chowder/172055.jpg',
 'clam_chowder/2009191.jpg',
 'clam_chowder/2054906.jpg',
 'clam_chowder/673650.jpg',
 'crab_cakes/445057.jpg',
 'crab_cakes/761280.jpg',
 'creme_brulee/1306834.jpg',
 'creme_brulee/1658062.jpg',
 'creme_brulee/2318273.jpg',
 'creme_brulee/2506003.jpg',
 'creme_brulee/3900789.jpg',
 'creme_brulee/392008.jpg',
 'creme_brulee/730057.jpg',
 'creme_brulee/849220.jpg',
 'croque_madame/3079934.jpg',
 'croque_madame/3484037.jpg',
 'deviled_eggs/1276764.jpg',
 'deviled_eggs/2218705.jpg',
 'deviled_eggs/50398.jpg',
 'donuts/2036733.jpg',
 'dumplings/1141514.jpg',
 'dumplings/1483996.jpg',
 'dumplings/2174768.jpg',
 'eggs_benedict/2492820.jpg',
 'falafel/2437617.jpg',
 'falafel/366728.jpg',
 'filet_mignon/103497.jpg',
 'filet_mignon/1841480.jpg',
 'foie_gras/1044237.jpg',
 'foie_gras/139942.jpg',
 'foie_gras/2721736.jpg',
 'foie_gras/302051.jpg',
 'foie_gras/3267247.jpg',
 'foie_gras/35694.jpg',
 'foie_gras/3917886.jpg',
 'foie_gras/583722.jpg',
 'foie_gras/71445.jpg',
 'foie_gras/71461.jpg']

List of Blurry Images

Similarly with blur images

blurry_images = stats_df[stats_df['blur'] < 50]
blurry_images.head()

	fastdup_id	img_w	img_h	blur	mean	min	max	stdv	file_size	contrast	img_filename	error_code	is_valid
2123	2123	512	512	41.3781	116.2239	2.0	198.0	30.7362	25479	0.9800	baklava/1413667.jpg	VALID	True
2768	2768	384	512	45.5609	102.8172	40.0	226.0	37.5435	18740	0.6992	baklava/3681797.jpg	VALID	True
2829	2829	512	512	38.8840	214.0211	71.0	255.0	25.0954	23869	0.5644	baklava/3877397.jpg	VALID	True
2918	2918	512	512	47.2227	138.1602	0.0	255.0	24.5464	22279	1.0000	baklava/683225.jpg	VALID	True
6924	6924	384	512	45.9959	75.9253	0.0	176.0	23.9812	19018	1.0000	beignets/726875.jpg	VALID	True

Get list of blurry images

list_of_blurry_images = blurry_images['img_filename'].to_list()
list_of_blurry_images

Outputs:

['baklava/1413667.jpg',
 'baklava/3681797.jpg',
 'baklava/3877397.jpg',
 'baklava/683225.jpg',
 'beignets/726875.jpg',
 'bread_pudding/444890.jpg',
 'breakfast_burrito/462294.jpg',
 'carrot_cake/345630.jpg',
 'chocolate_mousse/1653769.jpg',
 'clam_chowder/1472641.jpg',
 'clam_chowder/2250407.jpg',
 'clam_chowder/908590.jpg',
 'dumplings/2174768.jpg']

Summary

In this tutorial we found 825 images with potential issues.

print(f"Broken: {len(list_of_broken_images)}")
print(f"Duplicates: {len(list_of_duplicates)}")
print(f"Outliers: {len(list_of_outliers)}")
print(f"Dark: {len(list_of_dark_images)}")
print(f"Bright: {len(list_of_bright_images)}")
print(f"Blurry: {len(list_of_blurry_images)}")

problem_images = list_of_duplicates + list_of_broken_images + list_of_outliers + list_of_dark_images + list_of_bright_images + list_of_blurry_images

print(f"Total unique images: {len(set(problem_images))}")

Outputs:

Broken: 0
Duplicates: 610
Outliers: 111
Dark: 2
Bright: 93
Blurry: 13
Total unique images: 825

In this tutorial, we’ve seen how to use fastdup to analyze an image dataset for potential problems such as broken image, duplicate, outliers and dark/bright/blurry image. For each problem, we got a list of file names for further action. Depending on use cases, you might choose to delete the images, relabel them, or simply move the images elsewhere.

👍 TLDR In this tutorial we learned how to:

Find various dataset issues with fastdup.

Collect problematic images for further action.

Introduction

Quick Start

Explore & Search

Collab & Downstream

Models & Enrichment

Advanced Creation & Management

Integrations

Troubleshooting

Cleaning Image Dataset

Setting Up

Download Dataset

Run fastdup

Broken Images

List of Broken Images

Duplicates

Image Clusters

List of Duplicates

Outliers

List of Outliers

Dark, Bright, Blurry Images

List of Dark Images

List of Bright Images

List of Blurry Images

Summary

​Setting Up

​Download Dataset

​Run fastdup

​Broken Images

​List of Broken Images

​Duplicates

​Image Clusters

​List of Duplicates

​Outliers

​List of Outliers

​Dark, Bright, Blurry Images

​List of Dark Images

​List of Bright Images

​List of Blurry Images

​Summary

Setting Up

Download Dataset

Run fastdup

Broken Images

List of Broken Images

Duplicates

Image Clusters

List of Duplicates

Outliers

List of Outliers

Dark, Bright, Blurry Images

List of Dark Images

List of Bright Images

List of Blurry Images

Summary