Modern art challenges face detection (2023)

Table of Contents
Article preview Pattern Recognition Letters Abstract Introduction Section snippets Face detection Fine arts Modern art MAFD-150 dataset MAFD-150 face detection baselines Consensus and context-aware visual question answering Conformal prediction Discussion Conclusions References (33) Comput. Secur. Unconstrained face recognition: identifying a person of interest from a media collection IEEE Trans. Inf. Forensics Secur. Semi-supervised learning (Chapelle, O. etal., eds.; 2006)[book reviews] IEEE Trans. Neural Netw. Adversarial spam detection using the randomized hough transform-support vector machine Machine Learning and Applications (ICMLA), 2013 12th International Conference on Biometric interoperability across training, enrollment, and testing for face authentication Biometric Measurements and Systems for Security and Medical Applications (BIOMS), 2012 IEEE Workshop on Image forgery detection IEEE Signal Process. Mag. Progress in Art Visual turing test for computer vision systems Proc. Natl. Acad. Sci. Art and Illusion: A Study in the Psychology of Pictorial Representation Classifier technology and the illusion of progress Stat.Sci. An introduction to biometric recognition IEEE Trans. Circuits Syst. Video Technol. The megaface benchmark: 1 million faces for recognition at scale Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Forensic science: oxymoron? Science Cited by (3) Understanding and Creating Art with AI: Review and Outlook Intensified Biological Signature Recognition in the Wild: A Case Study Immunity and security using holism, ambient intelligence, triangulation, and stigmergy: Sensitivity analysis confronts fake news and COVID-19 using open set transduction Recommended articles (6) Videos
ScienceDirect

RegisterSign in

ViewPDF

  • Access throughyour institution

Article preview

  • Abstract
  • Introduction
  • Section snippets
  • References (33)
  • Cited by (3)
  • Recommended articles (6)

Pattern Recognition Letters

Volume 126,

1 September 2019

, Pages 3-10

Abstract

There is a widely held belief that computer vision, in general, and face authentication, in particular, are to a large extent solved problems. This paper challenges this belief regarding face authentication using examples from modern art that significantly confound face detection. The challenges are made concrete using a new MAFD-150 dataset (Modern Art Face Detection) composed mostly of modern art examples that cover much diversity in style and artists. MAFD-150 challenges the belief that singleton and crowd face detection is an almost solved problem, and provides baselines and preliminary results that highlight the inadequacy of current expertise and methods to address face detection. In particular, we show that well-known face detection algorithms are only able to achieve an F1 score of less than 35% overall across the new dataset. Additionally, we discuss the performance of the selected face detectors on varying art categories (such as Impressionism, Pop Art, etal.) to show how style and face representation may impact these algorithms. The paper concludes with suggestions on how to advance face processing by leveraging the complementarity between Show-and-Tell-like methods and a context and cooperative driven visual question answering framework using relevance-based triage. The very challenges detailed throughout are then shown to be helpful with developing novel, robust, and secure access protocols that combine text and modern art images using the visual question answering framework.

Introduction

According to Ludwig Wittgenstein the face is the soul of the body. Face processing, part and parcel of computer vision including image analysis and understanding, is all encompassing and includes biometric categorization and recognition; face detection, surveillance, and tracking; and face rendering and synthesis. This paper is primarily about visual arts, in general, and modern art, in particular, challenging current face processing methods on face detection. The motivation for us raising the modern art challenge comes from comments made by Google Alphabet Chairman Eric Schmidt who recently was quoted as saying, “when it comes to artificial intelligence, the most interesting advancements are being made in computer vision - now better than human vision in many instances” [22]. Face recognition performance has been reported to top in the upper 99% accuracy but more recent results on challenging and large face datasets such as MegaFace [13] have been benchmarked at below 75%. The terrain for face processing using modern art is much wider than the one for computer vision as it draws much from art theory, color, and personal artistic style. Furthermore, varying demographics (e.g., age, ethnicity, and gender) affecting face processing, in general, and face aging, in particular, are much more varied in appearance (e.g., caricature and PIE) according to the rendering style used. The overall MAFD challenge considered here deals with style rather than artists.

Advances on face detection and authentication on modern art are required to make progress on the visual Turing test [8] including interoperability across varying artistic styles and creation times. Such advances are predicated on having access to modern art data sets and benchmarks on using them. Towards that end, we make available the first visual arts data set MAFD-150 (Modern Art Face Detection)1 to start benchmark evaluation on face detection using modern art. Images lacking faces are included in the proposed data set so one can estimate the trade-offs between false positive and false negative rates against hit rates. The face detection results we report using state-of-the art algorithms are much lower than those usually reported to suggest that face detection is a solved problem. The paper also speculates on tentative solutions for the modern art challenge. Solutions could involve multi-modal integration and gating of varying information channel sources using crowd sourcing, information retrieval and contents-based image retrieval, recommender systems characteristic of consensus seeking, meta-reasoning using auxiliary information, context-awareness (e.g., grounding and situatedeness [12]) including smart edits and revisions, and last but not least collaborative filtering using relevance and triage.

Face authentication covers identification, verification (if two images belong to the same subject or not), and spatial-temporal surveillance for the purpose of re-authentication. Surveillance, intimately related to open set recognition, involves (a) negative identification (“rejection”) due to the obvious fact that the large majority (almost all) of the subjects screened at security entry points are law abiding people, and (b) correct identification for the those that make up the watch list. One can further distinguish a layered categorization hierarchy with Level 1 and 2 tasked with subject or pedestrian detection (prior to her classification) (e.g., Face in a Crowd), and stratification and binning for diversity (e.g., age, ethnicity, and gender demographics), respectively. Layer 3 involves soft biometrics, while Layer 4 and 5 are most specific to eventually address particular context and authentication using W5+ (Who, Where, What, Why, When, and How). Metrics, performance and biometric protocols cover for the way face processing takes place and its results adjudicated. Here major interest is placed on the ability to process uncontrolled setting including denial (e.g., occlusion) and deception (e.g., impersonation and spoofing), on one side, and varying image quality, on the other side. The modern art challenge is restricted here to only face detection with the possibility to add in the future face verification using the likelihood that the same artist has painted two given faces. Deception and related forgeries can be accounted too using extensions to verification.

The outline for the paper is at follows. Face detection is discussed in Section2, including art forgery (Section2.1) and interoperability (Section2.2). Fine arts, in general, and modern art, in particular, are discussed in Sections3 and 4. The MAFD-150 dataset and face detection baseline results using the dataset are the subject for Sections5 and 6. Consensus and context-aware visual question answering (C2VQA) and conformal prediction including incremental evidence accumulation sketch possible solutions for face detection on modern art in Sections7 and 8. Discussion and conclusions bring closure to the paper in Sections9 and 10.

Section snippets

Face detection

We discuss here challenges to face detection on modern art using digital forensics that cover for art forgeries and interoperability.

Fine arts

Fine or visual arts are all encompassing. They usually employ color and require active participation from the viewer for their appreciation. Visual or fine arts are intimately related to the psychology of representation including the study of perception including optical illusions and hallucinations. The mysterious ways in which shapes and marks can be made to signify and suggest other things beyond themselves have intrigued Gombrich [10] as he “sketched the development of representation from

Modern art

As modern art is most challenging for face authentication it is of primary interest for this paper. Compared to classical art, modern art is different, shocking and avant-garde in nature. It is difficult to decide on a precise onset for its birth but for the purpose of this paper modern art starts with the Impressionists. Modern art is mostly conceptual, cultural, and symbolic in nature rather than iconic. For both developmental and intrinsic performance evaluation purposes, face authentication

MAFD-150 dataset

This section concerns the characteristics of the novel MAFD-150 dataset. Modern art forms the core of the current challenge as it contains representations of faces that are simple for humans to process, and yet difficult for current algorithms. Fig.1 gives examples from the dataset.

As outlined in Section4, the MAFD-150 dataset includes examples from a variety of fine and modern artists (Table1 outlines the broad categories with sample artist names from the dataset). These categories were

MAFD-150 face detection baselines

In this section, we detail the performance of current face detection algorithms on the MAFD-150 dataset. Face detection methods, beginning with the Viola-Jones face detector [30], have been recently surveyed [32] and categorized into two general schemes: rigid templates using boosting or deep learning, and deformable methods using constellation of parts. In our face detection baselines, we score four main methods against the MAFD-150 dataset. All of the methods use pretrained models and none

(Video) Why Modern Art Is So Expensive | So Expensive

Consensus and context-aware visual question answering

We conjecture and propose here the framework of consensus and context-aware (C2) visual question answering (C2VQA) [24] to address the modern art challenges to face detection. C2VQA is particularly suited to address multi-modal information (biometric and forensic) channel integration. C2VQA expands on the interplay between the Visual Turing test, Show and Tell [29], contents-based image retrieval [3] and their offspring using relevance, smart edits, and triage [25]. In particular, C2VQA can

Conformal prediction

The grand challenge for face detection is that of interoperability. Learning, which is about generalization and prediction, plays a fundamental role in facilitating “the balance between internal representations and external regularities” [17]. Regularity can be addressed using randomness deficiency and Kolmogorov complexity, which are intricately related. The larger the randomness deficiency is the more regular and more probable some string (e.g., message) is. Transduction chooses from all the

Discussion

This paper challenges the visual Turing test (VTT) [8] using face detection on modern art. Towards that end we describe the fine arts MAFD-150 as a benchmark dataset of interest for such purposes. The biometric functionality of interest here is about face detection from images that contain one face (head), two or more faces, or no faces at all, with preliminary results on MAFD-150 using state-of-the art face detection software duly reported. Face synthesis and style identification are not

Conclusions

This paper challenges the widely held belief that computer vision and face authentication (including detection) are nearly solved problems. The challenge is illustrated using fine arts, in general, and modern art, in particular. A new dataset MAFD-150 is proposed for such ends including examples that defy their adequate automatic processing and resolution regarding face detection. Performance on the MAFD-150 dataset using current face detectors highlighted gaps and biases with regard to an

References (33)

  • A.S. Toor et al.Visual question authentication protocol (VQAP)

    Comput. Secur.

    (2017)

  • Y. Taigman et al.DeepFace: closing the gap to human-level performance in face verificationProceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (2014)

  • L. Best-Rowden et al.

    Unconstrained face recognition: identifying a person of interest from a media collection

    IEEE Trans. Inf. Forensics Secur.

    (2014)

  • S. Cascone, New rembrandt artwork created via 168,263 ‘painting fragments’, 2016,...
  • O. Chapelle et al.

    Semi-supervised learning (Chapelle, O. etal., eds.; 2006)[book reviews]

    IEEE Trans. Neural Netw.

    (2009)

  • D. Debarr et al.

    Adversarial spam detection using the randomized hough transform-support vector machine

    Machine Learning and Applications (ICMLA), 2013 12th International Conference on

    (2013)

    See Also
    Business Model Canvas: A 9-Step Guide to Analzye Any BusinessThe best canvas print services in 2022AGENTE Named Top Global B2B Company | AGENTESTEM Color Activities: Easy Science Activities to Teach Kids About Color

  • H. El Khiyari et al.

    Biometric interoperability across training, enrollment, and testing for face authentication

    Biometric Measurements and Systems for Security and Medical Applications (BIOMS), 2012 IEEE Workshop on

    (2012)

  • H. Farid

    Image forgery detection

    IEEE Signal Process. Mag.

    (2009)

  • S. Gablik

    Progress in Art

    (1977)

  • D. Geman et al.

    Visual turing test for computer vision systems

    Proc. Natl. Acad. Sci.

    (2015)

  • K. Goldberg, The robot-human alliance, 2017,...
  • E.H. Gombrich et al.

    Art and Illusion: A Study in the Psychology of Pictorial Representation

    (1977)

  • D.J. Hand

    Classifier technology and the illusion of progress

    Stat.Sci.

    (2006)

  • A.K. Jain et al.

    An introduction to biometric recognition

    IEEE Trans. Circuits Syst. Video Technol.

    (2004)

  • I. Kemelmacher-Shlizerman et al.

    The megaface benchmark: 1 million faces for recognition at scale

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (2016)

    (Video) Under the Cover: Wolfgang Tillmans

  • D. Kennedy

    Forensic science: oxymoron?

    Science

    (2003)

    • Cited by (3)

    Recommended articles (6)

    • We propose a novel deep multibranch and multitask neural network for artist, style, and genre painting categorization. The multibranch approach allows us to exploit at the same time the coarse layout of the painting and the fine-grained structures by using painting crops at different resolutions that are wisely extracted using a Spatial Transformer Network trained to identify the most discriminative subregions of paintings. The effectiveness of the proposed network is proved in experiments that are performed on a new dataset originally sourced from wikiart.org and hosted by Kaggle, and made suitable for artist, style and genre multitask learning. The dataset here proposed and made available for research is named MultitaskPainting100k, and is composed by 100K paintings, 1508 artists, 125 styles and 41 genres annotated by human experts. Among the different variants of the proposed network, the best method achieves accuracy levels of 56.5%, 57.2%, and 63.6% on the MultitaskPainting100k dataset for the tasks of artist, style and genre prediction respectively.

    • Research article

      Random Slope method for generation of cancelable biometric features

      Pattern Recognition Letters, Volume 126, 2019, pp. 31-40

      (Video) Sophie Ellis-Bextor - Murder On The Dancefloor

      Cancelable biometric templates are transformed versions of original biometric templates used for authentication purposes. The transformation functions should fulfill the important template protection criteria of diversity, revocability, non-invertibility, and performance. Although there exists a number of transformation techniques, yet many of these techniques fail to meet security and privacy requirements in the stolen token scenario and tend to become invertible with degraded performance. The work proposes a novel cancelable biometric technique named as ‘Random Slope’ method for generating secure, revocable, and non-invertible templates. Two approaches (RS-V1 and RS-V2) developed using the proposed concept not only fulfill the important cancelability criteria, but also provide dimensionality reduction upto 75%. The performances of the proposed approaches are experimentally verified for various biometric modalities such as visible and thermal face, palmprint, palmvein, and fingervein. As compared to some state of the art template transformation schemes, the proposed RS-V1 and RS-V2 approaches establish their reliability and effectiveness by performing better than these existing techniques with significant reduction in dimensions.

    • Research article

      Editorial

      Pattern Recognition Letters, Volume 127, 2019, pp. 1-2

    • Research article

      Deceiving faces: When plastic surgery challenges face recognition

      Image and Vision Computing, Volume 54, 2016, pp. 71-82

      An exponential growth of the number of plastic surgery treatments specific to face (from the minimally-invasive ones to the real surgical procedures) has characterized the last two decades and it seems likely that this phenomenon, that has social and cultural meanings and implications, could spread even further in the next years as the average cost of these treatments is lowering and the wish for “beautification” is becoming part of the global esthetics sense. For these reasons, face recognition as an established research topic has a new major challenge: delivering methods capable of high recognition accuracy even in case probe and gallery differ by a surgical alteration of face shape. To this aim is of fundamental importance understanding the range and the extent of the modification produced by the various types of treatments or by a combination of them. We present a survey of the state of the art on this topic, starting by an analysis of the diffusion of the facial plastic surgery and describing the key aspects of each of the most statistically relevant treatments available, resumed by a synthetic table. The paper includes a brief description of all the approaches proposed in the field so far to the best of authors' knowledge and a comparison of the performance reported by the existing methods when applied to the most referenced plastic surgery face dataset to date. A critical discussion of the results achieved so far and an insight about the challenges that still have to be addressed concludes this work.

    • Research article

      Synthesis, crystal structure, DFT analysis and fungicidal activity of a novel series O-substituted trifluoroatrolactamide derivatives

      Journal of Molecular Structure, Volume 1128, 2017, pp. 507-512

      A series of O-substituted trifluoroatrolactamide derivatives has been synthesized and fully characterized by 1H NMR, 13C NMR, 19F NMR, HRMS and X-ray diffraction analyses. The fungicidal activity of these compounds was evaluated and the results showed that some of them exhibited potent invitro fungicidal activity against Erysiphe graminis and Pyricularia oryzae. Their structure-property relationships were investigated using density functional theory calculations. The X-ray crystal structure of one of these compounds adopted a monoclinic space group with the following unit cell parameters: a=24.285 (13) Å, b=9.006 (5) Å, c=9.794 (5) Å, β=92.110 (9)º, V=2140.6 (19) Å3 and Z=4. A comparison of these experimental results with the theoretical values revealed that there was good agreement between the two sets of data. The subsequent biological evaluation of these compounds showed that some of them exhibited potent invitro fungicidal activity against Erysiphe graminis and Pyricularia oryzae.

    • Research article

      Modified optical and magnetic properties at room-temperature across lead-free morphotropic phase boundary in (1-x)BiTi3/8Fe2/8Mg3/8O3–xCaTiO3

      Ceramics International, Volume 43, Issue 8, 2017, pp. 6453-6459

      Perovskite compounds of the formula (1-x)BiTi3/8Fe2/8Mg3/8O3–xCaTiO3 ((1-x)BMFT-xCTO), a class of highly potential ferroelectric photovoltaic materials with a morphotropic phase boundary (MPB), have been synthesized by solid-state reaction and exhibit simultaneously visible-light response and ferromagnetic order at room-temperature (RT). The effects of structural phases and lattice distortions on electron transitions and orbital orderings are systematically investigated. The crystal lattice displays constriction with increasing x and Raman peaks show non-liner shift due to Bi3+ and Fe3+/Mg2+ replaced by Ca2+ and Ti4+ randomly, respectively. Microstructural characterizations illustrate the rhombohedral-orthorhombic MPB occurrence during the crystal growth. Optical spectroscopy analysis shows band-gaps of our samples are about 2.2eV with anomalies at the point of phase transition. Furthermore, the (1-x)BMFT-xCTO displays RT ferromagnetism when x<0.15, which can be explained by an F center theory. These results reveal rich physical phenomena and open an avenue to design promising perovskites for solar-energy conversion devices and multiferroic applications.

      See Also
      Why is framing a picture so expensive?Best online photo printing services 2022 - Which?23 luxury bathrooms – to create a hotel feel at homeThe Ultimate Farmhouse Bedroom Decor Ideas
      (Video) How This Guy Uses A.I. to Create Art | Obsessed | WIRED
    View full text

    © 2018 Elsevier B.V. All rights reserved.

    Videos

    1. 12 BEST TESTS TO REVEAL YOUR PERSONALITY TYPE
    (5-Minute Crafts)
    2. Are BILLY'S BOOTS in TOMBSTONE??
    (All Things Billy the Kid)
    3. People really came for Meghan Markle this week, Royal family continues its downward spiral to hell.
    (Sussex Squad)
    4. Artificial intelligence in healthcare: opportunities and challenges | Navid Toosi Saidy | TEDxQUT
    (TEDx Talks)
    5. Designing Tomorrow’s Met: Frida Escobedo
    (The Met)
    6. Why are there so few female artists?
    (nationalgalleries)
    Top Articles
    Latest Posts
    Article information

    Author: Greg O'Connell

    Last Updated: 01/02/2023

    Views: 5629

    Rating: 4.1 / 5 (42 voted)

    Reviews: 89% of readers found this page helpful

    Author information

    Name: Greg O'Connell

    Birthday: 1992-01-10

    Address: Suite 517 2436 Jefferey Pass, Shanitaside, UT 27519

    Phone: +2614651609714

    Job: Education Developer

    Hobby: Cooking, Gambling, Pottery, Shooting, Baseball, Singing, Snowboarding

    Introduction: My name is Greg O'Connell, I am a delightful, colorful, talented, kind, lively, modern, tender person who loves writing and wants to share my knowledge and understanding with you.