This post may contain affiliate links. We may earn money or products from the highlighted keywords or companies or banners mentioned in this post.

Michelle Obama’s June 2016 words about diversity during her last commencement address as First Lady – and again at the July 2016 Democratic National Convention – sent a chill down my spine: “I wake up every morning in a house that was built by slaves.”

Obama was speaking broadly and philosophically about the US’ large 18th- and 19th-century labour force of enslaved persons, and, by extension, about the resources used in the construction of early American government buildings. As a writer based in the state of Virginia, I knew where the sandstone for the White House and Capitol – as well as a few other historically important US government buildings – had come from.

“Government Island”, as it has long been known, is now a Stafford County public park, thanks largely to the preservation efforts of local author, Jane Hollenbeck Conner. Conner explained to me that the quarry was originally established on Wigginton's Island by local businessman George Brent after 1694 to provide stone for tombstones and to houses and churches in northern Virginia. Acting on the government's behalf, Pierre Charles L’Enfant, the French-born American architect who designed the layout of Washington DC, purchased the Wigginton's Island quarry in 1791, after which it became known as Government Island. The quarry became derelict after the US Civil War (1861-1865); Stafford County acquired the property from the US government in 1963, and it was finally opened to the public in 2010.

I knew I had to go to see the quarry – located just 88 miles north of Richmond, Virginia, where I live – from which slaves had quarried the sandstone. When I pulled off at the exit and reached the “island”, I discovered it was really a peninsula surrounded on three sides by Aquia Creek. Interpretive signs informed me of its use by Native Americans for millennia – and that the federal government purchased it in 1791, the year the US Bill of Rights was added to the Constitution, a bill that did nothing to limit the institution of slavery.

Situated around 30 miles from both Gunston Hall, home of George Mason, author of the US Bill of Rights; and Mount Vernon Estate, home of first US president George Washington – both of whom incorporated Aquia sandstone quarried by slave labour into their homes – the park helps connect the dots in a significant part of colonial history.

George Washington had used the bright grey sandstone, or “freestone”, for the steps and walkways of his wooden home, and had sent the federal contractor to the Aquia quarry for stone to build Cape Henry Lighthouse in Virginia Beach – the very first federal work project. He knew the Aquia area well: he was born 54 miles southeast of here at what is now the George Washington Birthplace National Monument; grew up at Ferry Farm near Fredericksburg, 20 miles to the south; and later lived at Mount Vernon.

I was grateful that there were few other visitors around, as strolling through the 17-acre county park was an emotional experience. I viewed up-close – and touched – the abundant marks left on the sandstone rocks left by slaves and free blacks more than two centuries ago. I could envision, even without the interpretive signs, the men swinging axes and dislodging huge blocks of stone, putting them on rough-hewn logs to roll down to the creek, where barges waited to transport the stone up the Potomac River to construction sites in the new capital city.

If a site of forced labour sounds like a depressing hike, walking the 1.7-mile loop around the island eased me into a wonderful mood. I watched as a heron, poised at the edge of a fish-filled tidal creek, made his catch, while a couple of osprey flew overhead and squirrels played around nearby trees. A fellow hiker told me he had seen bald eagles overhead many times, especially as a nearby Virginia State Park, Caledon, is the summer home to one of the largest concentrations of them on the East Coast. Spring’s rhododendron and dogwood blossoms were fading, and by late autumn, the dominant oak and maple trees would be shedding their leaves, making it easier to see the wildlife as well as the remaining pockmarked outcroppings of stone. I resolved to come back again at a different time of year.

If I had a chance to chat with Obama as she departs the White House, I would tell her to come to this site. I would tell her that a visit to the Aquia quarry vividly brings to life the story behind her “house built by slaves”.

If you liked this story, sign up for the weekly bbc.com features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Future, Earth, Culture, Capital, Travel and Autos, delivered to your inbox every Friday.