Young Chinese American Henry Lee meets Keiko Okabe, a Japanese American girl, in the early 1940s. They both attend an (otherwise) white prep school as "scholarship students" in Seattle. Henry's father is adamantly anti-Japanese, as is the increasingly hostile general population of Seattle following Pearl Harbor. In spite of this, Henry and Keiko begin an intense friendship until Keiko is taken to an internment camp with her family.

Henry finds her, first at local Camp Harmony. After failing to make his feelings known at Camp Harmony, he follows her with his friend, a local Jazz musician named Sheldon, to Minidoka, Idaho. Upon finding her there, he promises to wait for her. They decide to write each other letters until the war is over, and Henry returns to Seattle.

He religiously mails Keiko letters, but receives very few in return. His father is intent on sending him to China, now that the Japanese are being pushed back, to finish his education traditionally. Henry arrives home one day to find a ticket to China in his name. He agrees to go on the condition that his father (as part of an association of elders) saves the Panama Hotel from being sold. The Panama Hotel is where Keiko's family stored the larger part of their belongings when they were shipped to the concentration camps. Many families stored their possessions in the basement of the Hotel.

Two years later, families are being released from the camps. Henry writes Keiko one last letter, since he has not heard from her in half a year. He tells her that he is going to China to finish school, and to meet him in front of the hotel. She does not appear, but the young Chinese clerk, Ethel, from the post office does. She has come to give him the letter he sent, returned to sender. By the time the war ends, she and Henry are dating. When news of Japan's surrender is heard, Henry impetuously proposes to Ethel. As she announces this to the crowd, Henry thinks he sees Keiko in the crowd. He hurries home to find his father dying. On his deathbed, Henry's father whispers to him, "I did it for you." The implication is that he used his ties in the Chinese community to stop the correspondence between Henry and Keiko.

Decades later, Sheldon is in the hospital. Ethel has died. She and Henry have a son, Marty, who tells his father that he is engaged. At the same time, the Panama Hotel is going to be refurbished, and the belongings in the basement are waiting to be reclaimed. Henry goes looking for possessions belonging to the Okabes, especially a significant record for him and Keiko, with Sheldon on saxophone. He finds it, broken, and tells his son and his daughter-in-law-to-be about Keiko. Marty and his fiancee manage to locate Keiko, across the country, and she sends back her whole version of the record to comfort Sheldon on his deathbed. Henry is once again gifted a ticket to travel, to fix what is broken.