In the late 1980s, in the city of Sichuan, China, Fei Fei Li’s middle school teacher accused Fei-Fei of spending too much time reading random books like The Unbearable Lightness of Being and magazines about marine life and UFOs, rather than preparing for high school entrance exams. Fei-Fei needed to learn to set aside her personal interests and study what was useful, or she’d never succeed.
Fei-Fei’s mom intervened. “Is this what Fei-Fei wants? Is this what I want for Fei-Fei?” she asked defiantly.
Over and over again, even as they moved to America, working minimum wage jobs to scrape by, Fei-Fei’s mom set aside conventional wisdom and societal pressures and insisted that Fei-Fei follow her intellectual curiosity rather than accept a lucrative job that she didn’t love.
By following her tremendous curiosity, Fei-Fei has become a tenured professor at Stanford University, a chief scientist at Google, and is known as the Godmother of AI for her scientific contributions to the technology that’s brought us ChatGPT, self-driving cars, and more, and she was even called to testify before the US Senate about the societal impact of AI.
In today’s episode, I’m going to tell you my key takeaways from Fei-Fei’s recent memoir, The Worlds I See. As a disclaimer, this episode is slightly less in-depth than previous ones because I have a lot less information to draw on — Fei-Fei has shared very little about her personal life with the public, and no one’s written a biography of her — so my only source of information about her personal life comes from her own memoir.
That said, I personally find her story really compelling. Obviously, everyone these days is talking about how AI is changing the world. Fei-Fei is one of extremely few women at the very top of the AI field. And, despite the memoir being short on details about her personal life, what she did share made me tear up multiple times.
On a more personal note, I actually took Fei-Fei’s AI class at Stanford and really liked her, and my parents, who recommended this book to me, immigrated from China to the US around the same time that Fei-Fei did.
So let’s get into it.
Childhood
Why was Fei-Fei reading Milan Kundera, the Bronte sisters, and all these science magazines as a middle schooler? To understand Fei-Fei’s intellectual curiosity and independence, you need to understand her mother.
Fei-Fei’s mother came from a highly intellectual family who was repressed during Mao’s Communist regime in China. Fei-Fei’s mom was very intelligent and curious, but because of her family’s political associations, she was consistently discriminated against and prevented from reaching her full potential. This made her scorn her peers’ obsession with conformity and social status, it made her a contrarian, and it made her very determined to ensure that her daughter’s intellectual potential could fully blossom.
While Fei-Fei’s mom insisted on Fei-Fei pursuing her intellectual interests, it was Fei-Fei’s dad who inspired her deep curiosity about physics in particular.
Her dad always had this infectious joy and curiosity around the natural world around him. The day Fei-Fei was born, her dad arrived extremely late to the hospital — not because of any traffic issues, but because he had completely lost track of time while birdwatching in a local park. He took great joy in showing her his love of the natural world, and in engineering numerous gadgets to use around the house.
When Fei-Fei was 11, her parents made the decision to immigrate to the US. Her father left first, to find a job and establish a home for them in the US, while Fei-Fei stayed in China with her mom for the time being. Fei-Fei missed him terribly. But, she says, quote:
“The more I grieved [my father’s] absence, the more I realized the things I missed about him were the things physics was trying to teach me. The way he naturally saw the world in terms of light, speed, torque, force, weight, and tension. The way he improvised devices of gears and pulleys to solve problems around the house... Physics had always been the hidden foundation of my father’s mind, and it was only now, when I missed him most, that I realized it.“
So in middle and high school, Fei-Fei fell in love with physics. She also developed a chip on her shoulder about her academic performance: at the very end of elementary school, she had overheard her female teacher giving the boys in their class a lecture about how, as boys, they were biologically smarter than girls, especially in math and science, and that the boys therefore ought to be ashamed that their average scores on the recent exam had been lower than the girls’. Fei-Fei recalls feeling a “heavy, jagged” new feeling. “I didn’t feel discouraged, or even offended. I was angry. It was an anger I wasn’t familiar with—a quiet heat, an indignation.” She cut her hair short, refused to wear dresses, and grew determined to succeed in school. Though she doesn’t say so explicitly, I imagine it must’ve contributed to her later ambition to become a great scientist, too.
When Fei-Fei was 15, she and her mom got visas to go to the US, and they boarded a plane to join her father in New Jersey.
Immigration
Life in New Jersey was tough. She described that every class was like an English class — she had previously excelled at math, but now in the US, she couldn’t understand the word problems. In those pre-Google Translate days, Fei-Fei recalls that “even the simplest homework assignments took hours, as nearly every step was saddled with a discouraging appeal to one of two enormous dictionaries, one for translating Chinese to English and the other for the reverse.” Still though, her academic ambition remained; she recalls that the “threshold between As and all other grades had taken on religious gravity for [her].”
And then of course, there was the poverty. Fei-Fei and her parents shared a tiny one-bedroom apartment, where Fei-Fei slept in a narrow gap between the kitchen and the dining table. To pay rent, her parents worked long hours as a cashier and a camera repairman. Fei-Fei herself waited tables at a Chinese restaurant on evenings and weekends. Sometimes during a break, she would sit down in the back and start reading a book, but her boss would tell her to stop wasting time that could be better used to clean the toilets.
Then there was the racism and bullying — one day, a fellow immigrant classmate accidentally bumped arms with a white student on his way out of the library. Fei-Fei could do nothing but watch in horror as a group of white students beat up the immigrant kid and blood poured from his face onto the pavement.
But not all her interactions with Americans were negative.
Partly as a result of her newfound fear of the library where the kid got beat up, Fei-Fei started spending more time in her math teacher Mr. Sabella’s office, where she asked Mr. Sabella for extra math help. Because there were so many English words and concepts she didn’t know, even a simple question would lead them to a longer back and forth conversation.
One day she asked Mr. Sabella another simple question: “Can you recommend some books to me, to help me improve my English?” Mr. Sabella’s eyes lit up. Turns out, Mr. Sabella loved reading, just like Fei-Fei. Fei-Fei eagerly shared the titles of the western classics she’d read Chinese translations of back in China: Dickens, Hemingway, etc. She recalls: “Mr. Sabella fell back into his chair, chuckling in pure, stunned delight. I believe it was the first moment an American had ever seen me as more than a Chinese-speaking immigrant.” Over time, Mr. Sabella became a guidance counselor to Fei-Fei—he saw her potential and wanted to foster it.
Their high school didn’t have an advanced calculus class, so he came up with a curriculum and taught Fei-Fei during lunch breaks. He invited her over for dinners with his family, where Fei-Fei tasted homemade American foods like brownies for the first time. Over time, the Sabella family became a second family to Fei-Fei — she considers Mr. Sabella her third parent, he’s referred to Fei-Fei as his daughter, and they had weekly phone calls for decades.
Thanks to her hard work and Mr. Sabella’s mentorship, Fei-Fei was accepted to Princeton with a full scholarship in 1995.
At Princeton, she continued to love physics. But in her first semester, her mom got terribly sick and needed heart surgery. Even after the surgery, her mom’s heart was extremely fragile; the doctor said that she could never work again — even part time work could put her life in serious danger. Which was quite the dilemma, because the family simply could not pay the bills without her mom’s income.
So at this point, with her mother’s life literally on the line, Fei-Fei considered dropping out to get a job, or at least switching her major from physics to something with more earning potential. But her mom insisted that Fei-Fei continue to study what she loved, and went back to full time work just two weeks after her heart surgery.
To try to increase their income, they started a dry cleaning business. Mr. Sabella kindly lent them the money to open the business, and every weekend Fei-Fei went home to help with the dry cleaning and translate questions from American customers. Fei-Fei didn’t go to a single party in college, and certainly didn’t join any of Princeton’s famous eating clubs. Slowly they started to make money from the business, so that Fei-Fei could continue to study what she loved.
Becoming a Scientist
But what was it exactly that she loved? Around sophomore year, she started wondering [quote] “if it was really physics per se that so inspired her, or simply the spirit that motivated physics—the courage that spurred some of history’s brightest minds to ask such brazen questions about our world.” [end quote] This part is really interesting to me — I think a lot of people are generally curious about the world, but this passage starts to highlight where Fei-Fei’s curiosity takes on a greater scale and magnitude than those of other people. She’s no longer content to just ask questions that have known answers; she says she “wanted desperately to follow in [the footsteps of these great scientists]—to help reveal some unknown truth.” She started studying the biographies of great scientists like Einstein, Feynman, and Bohr, presumably because she had the ambition to become a great scientist herself.
She noticed a common pattern in the lives of great physicists: later in their careers, they all seemed to develop an interest in “the mystery of life itself,” and often even formally shifted to studying biology.
So she started wondering what it was about biology that so fascinated these great minds, and she herself became particularly fascinated with how the mind works, how to “make sense of intelligence itself”, as she put it. She was drawn to the “humanistic thread” of this line of questioning — how it was applying the rigor of the scientific method to understanding how humans think and perceive the world — which makes sense, because she had a scientist’s brain but also loved classic works of literature that try to make sense of the human experience — and she also felt that the question of how intelligence worked was one of the biggest, most audacious questions that she could be asking at that time.
So Fei-Fei interned at a neuroscience research lab that summer to explore how the brain works. They basically hooked electrodes to the brain of a cat, then showed the cat an image, and then, using the electric signals from the cat’s brain, tried to recreate the image that the cat was seeing. Pretty cool right? She was hooked — despite the grueling hours, she felt so energized and content every single day — “it triggered the same feeling I got as a child exploring the mountains... with my father, when we’d spot a butterfly we’d never seen before, or happen upon a new variety of stick insect.”
By her senior year, Fei-Fei knew that she wanted to be a scientist and keep making these exciting discoveries. But what about the money? Academic researchers don’t get paid much, and her mom was still endangering her health every day working long hours at the family laundromat. So Fei-Fei contemplated a career in finance, which could put an end to all her family’s financial insecurities. She interviewed with a few Wall Street firms and told her mom about the high salaries and perks.
And her mom replied immediately: “Fei-Fei, is it what you want?”
Fei-Fei said: “You know what I want, Mom. I want to be a scientist.”
Her mom replied: “So what are we even talking about?”
So Fei-Fei stopped talking to the finance firms and did what she really wanted, which was grad school. But in what subject?
Computer Vision & Big Data
After her summer researching the cat brain, she realized that she wanted to study how intelligence worked, and that the first step was to understand how vision worked. How is it that, as light enters our eyes and fills them with blobs of color, we are able to translate these blobs instantly into understanding and knowledge about the world? Like, when I look around right now — how exactly is my brain so instantaneously labeling and making sense of the shapes around me— that’s a chair, that’s a lamp, that’s a golden retriever walking by and it looks friendly? Fei-Fei wanted to understand how the brain processes all this.
Ok so how do you go about understanding that? One way is to keep putting test subjects into various scenarios, measure their brain waves or whatever, and try to deduce what might be going on inside their brain.
The other method is to essentially try to recreate an artificial brain, and in the process, you might get to some understanding of how the real human brain works. And increasingly Fei-Fei got sucked into this approach, of trying to write a computer system that is able to see and recognize shapes and images the way that the human brain does.
So she ended up doing a dual PhD in neuroscience and computer vision at Caltech. Today obviously computer vision is super advanced, and we have Tesla cars basically driving themselves on autopilot out on the roads, but Fei-Fei started her PhD in the year 2000, when the field was really new and not very popular.
The core problem she focused on was, ok, let’s say you give the computer a picture of a cat, how do you make the computer capable of recognizing that it was a cat?
And at the time, the few researchers in the field were mostly focused on developing complex sets of rules or algorithms to explain to the computer what a cat looks like, like if there are two triangles on top of a circle, it might be a cat. But Fei-Fei was like, ok well how does a human child learn what a cat looks like? The human baby looks around the world and sees a ton of things, and its parent points to the things and tells them “that’s a cat, that’s a dog.” Over time, after seeing enough different examples of these different objects, somehow the baby’s brain is able to connect the dots and recognize cats and dogs and stuff.
So Fei-Fei’s like, what if I use a machine learning approach—which had been invented by then but was not very popular—where I compile a set of images, I label them with what objects are in the image, and then I try to get a computer system to learn to recognize the objects in the way that a human child learns?
Fei-Fei set about compiling a set of images that a computer could use to learn from in the way that a human child learns. Which requires a lot of manual labor, because you need to not only compile a diverse set of images, but also label them with the names of the objects in them. But the early results were really promising - Fei-Fei’s system set the world standard for image recognition. Which is super exciting — imagine that of everyone in the world to date, you as a twenty-something researcher have built the computer system that is most successful at recognizing images. So she decided to assemble bigger and bigger data sets for the computers to learn from, to further improve their performance.
This problem of assembling a bigger and bigger database to train a powerful computer consumed the next couple years of her life.
But a few years into it, her mom had to go through another terrible heart surgery. In the wake of the surgery, Fei-Fei contemplated abandoning her research career. She interviewed and got an offer to work as a management consultant at McKinsey, which would set her up for a lucrative, prestigious career and allow her mom to retire.
But again, her mom forbade her from taking the McKinsey job, and forced her to stay in grad school to keep pursuing her passion. To accommodate her mom’s health, her parents moved into her dorm room at Caltech and subsisted off her grad student stipend while Fei-Fei continued her research.
Around the same time, she met Silvio, a fellow PhD student from Italy. “My God, what a nerd this guy is,” she thought to herself. “But he’s the same kind of nerd as me.” They bonded over their intense and similar curiosity about the world, fell in love, and got married. She’s described Silvio as “a confidant without equal,” “my best friend, my fellow AI scientist, [and] my soulmate.”
A few years later, Fei-Fei got her PhD, became a professor at Princeton, and continued working on building a giant data set of images to use to train computers. The early years were pretty rough - nowadays “big data” is such a cliche term and everyone recognizes its value, but back then, Fei-Fei was sort of alone in believing in it — everyone thought she was wasting her time over-fixating on gathering training data rather than improving the underlying models that would learn from this data. As she put it, “algorithms were the center of our universe in 2006, and data just wasn’t a particularly interesting topic.”
After repeated no’s, she finally attracted one PhD student who was willing to work with her on the project. They tried many approaches to assembling the data set, but the estimate was that it would take 19 years to assemble.
Getting funding for the project was so tough that Fei-Fei considered going back into doing laundry to raise money for it.
In the end, by crowdsourcing the work of labeling images via Amazon Mechanical Turk, Fei-Fei’s team was able to compile by far the largest dataset of images in the world, with 3.2 million labeled images that a computer could use to learn how to see and recognize objects in the world.
Fei-Fei’s team released this huge data set to the public and tried really hard to market it to the AI research community, to get people to train their computers with their data. And it revealed to the world some really counterintuitive insights about what kinds of machine learning models and techniques worked. In particular, it turns out that a model called a neural network, which is structured like the neurons in an actual human brain, was extraordinarily good at learning from Fei-Fei’s giant image set.
Which is cool first because it tied back to Fei-Fei’s neuroscience roots, but also because it ended up being super powerful, and generalizable to all sorts of tasks beyond just labeling images. This approach that Fei-Fei and her collaborators had used, of big datasets, a neural network, and high-speed GPU machines to perform the computations—proved to be a generalizable formula with formidable and world-changing results — it came to be known as deep learning, and it now powers everything from ChatGPT to self-driving cars, and several of Fei-Fei’s grad students actually became founding and core members of the teams that built these things.
Growing her scope
As the AI revolution spread, Fei-Fei grew in her power and scope.
A turning point for her was one day in 2014 — she and her students believed they had built the very first system to auto-generate sentences to describe images, and had submitted their findings to an upcoming conference, when she got a call from the NY Times: another team had produced the exact same findings. And the team had come from Google. Her research was officially so impactful and commercially useful that Google was sponsoring teams to solve the exact same problems that she was.
Later, in 2016, she took a sabbatical from Stanford to spend two years as the chief scientist of AI at Google Cloud. This was not being a corporate sellout — AI had grown so commercially valuable that tech companies had actually become the best place to do AI research, because they had the money and the data to support the most ambitious projects. The day she arrived at Google, she was given a fifteen-person team of brilliant PhDs, and within eighteen months, her team had grown to twenty times that size.
And the more that AI continues to progress, the more recognition she’s gotten. Wired has called her “one of a tiny group of scientists―a group perhaps small enough to fit around a kitchen table―who are responsible for AI’s recent remarkable advances.”
In recent years, Fei-Fei’s increasingly focused her efforts on studying the societal impact of AI. At Google, she coauthored a pledge to use AI for social good. Back at Stanford, she cofounded the Institute for Human-Centered AI to research and influence AI’s impact on how people work and live, she kicked off a research project on using AI to advance healthcare, and she cofounded a nonprofit, AI4All to increase diversity in AI.
On top of all this, she now has two kids.
Takeaways
Ok so what can we learn from her? One obvious takeaway is to follow your curiosity and passion, instead of optimizing for short-term accolades and starting salaries.
And I think that’s fair, but it’s also slightly more nuanced in that Fei-Fei’s curiosity was no ordinary curiosity; her curiosity was on a much larger scale and magnitude than most people’s. She was always wondering — what is the most audacious question that I can be asking, that most people are too afraid to ask in earnest because we as a human race had not yet found a complete answer? Over and over, she talks about her quest to find a “North Star” question that felt central to advancing the frontier of human knowledge, that she could spend years of her life trying obsessively to answer, that felt historic in nature. She says she studied the lives of the greatest physicists with as much zeal as she studied their work, presumably to learn how to achieve greatness herself. I don’t think it’s an accident that the combination of her grand curiosity and ambition led her to the forefront of the field of AI, a field that’s so young and changes so rapidly and has such an outsized impact on the world.
So, yes, in order to succeed, you do need a long-term commitment to curiosity and passion — but equally important, I think, is the drive to do the biggest, most audacious version of whatever it is that you’re curious or passionate about, and I really admire the combination of these traits in Dr. Fei-Fei Li.
I hope you enjoyed this episode. Again, it’s a bit lighter on analysis than my usual full episodes, because it’s basically just based on this one memoir — but, if you wanna check out the memoir, it’s called The Worlds I See, and I’d especially recommend it if you want to learn more about the AI stuff I touched on in an exciting, accessible way. Thanks for listening and don’t forget to subscribe and rate the show!