Jingyan Wang

I am a research assistant professor at the Toyota Technological Institute at Chicago. My research interests lie in statistical machine learning, particularly in applications to high-stakes decision-making and evaluation problems such as admissions, hiring, grading, and peer review. I draw inspirations from psychology to build human behavioral models, develop algorithms with theoretical guarantees, conduct crowdsourcing experiments, and implement policy changes that make real-world impacts.

Previously, I was a President’s postdoctoral fellow in the School of Industrial and Systems Engineering (ISyE) and the Algorithms and Randomness Center (ARC) at Georgia Institute of Technology, working with Ashwin Pananjady and Juba Ziani. I received my Ph.D. in the School of Computer Science at Carnegie Mellon University, advised by Nihar Shah. I received my B.S. in Electrical Engineering and Computer Sciences from UC Berkeley.

My CV is available here (updated 02/2024).

Email: jingyanw [at] ttic.edu

News

Feb 2024: Presenting at the ITA workshop in San Diego.
Feb 2024: Presenting my research at the CSIP Seminar in the ECE Department at Georgia Tech.
Jan 2024: Excited to participate in the Dagstuhl Seminar and give an overview of my work on various bias mitigation algorithms.
Dec 2023: Presenting at the Network Lunch at SLMath.
Oct 2023: Visiting SLMath as a research member for the Algorithms, Fairness, and Equity program in Berkeley during Oct - Dec 2023. Happy to chat more about research!
Oct 2023: Presenting at the INFORMS annual meeting on our paper "Debiasing Evaluations That Are Biased by Evaluations".
Sept 2023: Paper "Perceptual Adjustment Queries and an Inverted Measurement Paradigm for Low-Rank Metric Learning" accepted to NeurIPS 2023.
Aug 2023: Participating in the Introductory Workshop: Algorithms, Fairness, and Equity at SLMath in Berkeley.
July 2023: Presenting at EC 2023 on our paper "Modeling and Correcting Bias in Sequential Evaluation".

Research Overview

The goal of my research is to develop accurate, fair, reliable, and efficient evaluation systems. I draw inspirations from psychology to build human behavioral models, develop algorithms with theoretical guarantees, conduct crowdsourcing experiments, and implement policy changes that make real-world impacts. Specifically, I identify, analyze and improve three interrelated key components: algorithms, people, and design.

Algorithms: It's important to holistically think about different types of objectives, which induce intricate connections and tradeoffs underlying these evaluation systems. I develop algorithms that effectively aggregate evaluation data, integrating accuracy guarantees with the following considerations:
- Fairness: We consider a crowdsourcing model (Bradley-Terry model) that infers item quality from pairwise comparisons. We propose a modification to the widely-used maximum likelihood estimator, which is simple to implement and enjoys a win-win in the bias-accuracy tradeoff – that is, it significantly improves fairness while achieving optimal accuracy. [paper | talk]
- Reliability: In applications such as hiring and admissions, one important goal is to successfully recruit a pre-defined number of candidates. We formulate a Knapsack-style combinatorial optimization problem, and present theoretical guarantees for various algorithms that simultaneously consider quality and the uncertainty involved. [paper]
People: Algorithms do not operate in isolation; it is crucial to keep in mind that the input data come from human evaluators. People are not perfectly rational agents, and have biases even if we consciously try to avoid them. My work identifies different sources of human biases:
- Miscalibration: People have different numerical scales when reporting ratings. We design algorithms that take into account the miscalibration of people – for example, some people tend to be lenient and give higher scores, and some others tend to be strict. Our algorithms are provably more robust to people’s miscalibration, and hence make the evaluation more fair. [paper | blog post]
- Personal experience: Evaluators are influenced by their experiences. For example, in teaching evaluation, students often give higher ratings to an instructor if they receive higher grades in this course. We capture such tendencies by a natural monotonic constraint, and propose an estimator that reduces the bias in a data-dependent fashion, without prior knowledge on the amount of the bias. [paper | talk]
- Ordering effect: Evaluations are made sequentially in applications such as sports/arts competitions and hiring. We provide a model to capture the nature of such sequential evaluation, supported by theory and crowdsourcing experiments. We propose a ranking estimator that is both efficient and optimal (in certain notions), whereas using the ranking naively induced by the scores is provably not optimal. [paper]
Design: People engage with algorithms, and algorithms influence human behaviors. I study design and policy decisions that underpin the interactions between people and algorithms:
- Task allocation: In large-scale evaluation tasks, we examine allocation schemes that assign the task to reviewers in a distributed fashion, and provide a comprehensive comparison between these allocation schemes, using a mixture of theoretical and experimental approaches. [paper]
- Data elicitation: Cardinal and ordinal queries have their pros and cons. We propose a new data elicitation scheme that achieves the best of both worlds. Taking preference learning as an example, it asks questions of the form "to reach the same level of satisfaction with a 2-dollar bus ride, how much do you want to spend on taxi?", which combines ordinal reasoning and cardinal reporting. [paper]

I extend my insights from research to real policy improvements in practice. For example, we compile the data and evaluate the gender distribution in award-winning papers in 16 top computer science conferences in the past 10 years, which shows prominent differences across conferences. [blog post] We also consider the biases caused by the alphabetical ordering practice in scientific publication. Taking cognizance of this bias arising from alphabetical ordering, the Machine Learning Department at CMU has randomized the ordering of students and faculty on its webpages, along with many other institutions. [blog post] The debiasing algorithm that I have developed is currently deployed in practice with grant review agencies.

Preprints

What Can Natural Language Processing Do for Peer Review?

Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K. Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, Sheng Lu, Mausam, Margot Mieskes, Aurélie Névéol, Danish Pruthi, Lizhen Qu, Roy Schwartz, Noah A. Smith, Thamar Solorio, Jingyan Wang, Xiaodan Zhu, Anna Rogers, Nihar B. Shah, Iryna Gurevych
Available on arXiv, 2024
Producers Equilibria and Dynamics in Engagement-Driven Recommender Systems

Krishna Acharya, Varun Vangala, Jingyan Wang, Juba Ziani
Available on arXiv, 2024

[code]
Equilibria of Data Marketplaces with Privacy-Aware Sellers under Endogenous Privacy Costs

Diptangshu Sen, Jingyan Wang, Juba Ziani
Available on arXiv, 2024

Journal Publications and Under Review

Modeling and Correcting Bias in Sequential Evaluation

Jingyan Wang, Ashwin Pananjady
Under review
Debiasing Evaluations That Are Biased by Evaluations

Jingyan Wang, Ivan Stelmakh, Yuting Wei, Nihar B. Shah
JMLR, 2024
3D differential phase-contrast microscopy with computational illumination using an LED array

Lei Tian, Jingyan Wang, Laura Waller
Optics Letters, 2014

Peer-Reviewed Conference Publications

Learning the Eye of the Beholder: Statistical Modeling and Estimation for Personalized Color Perception

Xuanzhou Chen, Austin Xu, Jingyan Wang, Ashwin Pananjady
Allerton, 2024
Perceptual Adjustment Queries and an Inverted Measurement Paradigm for Low-Rank Metric Learning

Austin Xu, Andrew D. McRae, Jingyan Wang, Mark A. Davenport, Ashwin Pananjady
NeurIPS, 2023

[code]
Modeling and Correcting Bias in Sequential Evaluation

Jingyan Wang, Ashwin Pananjady
EC, 2023

[data and code]
Recruitment Strategies That Take a Chance

Gregory Kehne, Ariel Procaccia, Jingyan Wang
NeurIPS, 2022

[code]
Allocation Schemes in Analytic Evaluation: Applicant-Centric Holistic or Attribute-Centric Segmented?

Jingyan Wang, Carmel Baharav, Nihar B. Shah, Anita Williams Woolley, R Ravi
HCOMP, 2022

[data and code] [slides]
A Heuristic for Statistical Seriation

Komal Dhull, Jingyan Wang, Nihar B. Shah, Yuanzhi Li, R. Ravi
UAI, 2021

[talk]
Debiasing Evaluations That Are Biased by Evaluations

Jingyan Wang, Ivan Stelmakh, Yuting Wei, Nihar B. Shah
AAAI, 2021

[talk] [code] [slides]
Stretching the Effectiveness of MLE from Accuracy to Bias for Pairwise Comparisons

Jingyan Wang, Nihar B. Shah, R. Ravi
AISTATS, 2020

[talk]
Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings

Jingyan Wang, Nihar B. Shah
AAMAS, 2019

Best student paper award at AAMAS 2019
Nomination for best paper award at AAMAS 2019

Also appeared as "Ranking and Rating Rankings and Ratings" at AAAI 2020 Sister Conference Track

[slides] [blog post]
The more you look, the more you see: towards general object understanding through recursive refinement

Jingyan Wang, Olga Russakovsky, Deva Ramanan
WACV, 2018

[code] [supplementary]
Having Your Cake and Eating It Too: Jointly Optimal Codes for I/O, Storage and Network-bandwidth in Distributed Storage Systems

K. V. Rashmi, Preetum Nakkiran, Jingyan Wang, Nihar B. Shah, Kannan Ramchandran
USENIX FAST, 2015

Picked as the best paper of USENIX FAST 2015 by StorageMojo
Iterative Hard Thresholding for Keyword Extraction from Large Text Corpora. ICMLA 2014

Steve Yadlowsky, Preetum Nakkiran, Jingyan Wang, Rishi Sharma, Laurent El Ghaoui
ICMLA, 2014

Teaching

Co-instructor
ISYE 8813 (Algorithmic Foundations of Ethical Machine Learning), Georgia Tech, Fall 2023

Guest lecturer

IST402 (Crowdsourcing and Crowd-AI Systems), Penn State, Spring 2023

PIC 16B (Python with Applications II), UCLA, Winter 2023

ISYE 6740 (Computational Data Analysis), Georgia Tech, Fall 2022

TA
16-720 (Computer Vision), CMU, Fall 2017

Lab Assistant
EE 20N (Signals and Systems), UC Berkeley, Fall 2013

Misc

I play the violin with the Chicago Metropolitan Symphony Orchestra.

Next concert:
November 23, 2024 (Saturday at 7:30pm)
Gannon Concert Hall, Holtschneider Performance Center, DePaul University (2330 N Halsted St, Chicago, IL 60614)

During my time in Atlanta, I played with the Atlanta Community Symphony Orchestra. We performed free concerts in the Atlanta metro area.