Although he never set foot in America, Glassford's company did a lot of business; so much so that his company was, during the mid to late 18th century, the largest tobacco trading company in the Chesapeake. And his base of operations was in Port Tobacco. More specifically, we have found his company's name in the land records as having bought lot #46 in town from Daniel of St. Thomas Jenifer in 1762.
When the painting was first made in 1767, there was one of John Glassford's personal slaves in the area where the red circle is now. It was deliberately obscured during the 19th century anti-slavery movement, thus painting out Glasgow merchants' involvement with slavery. The painting has been in repair and cleaning in the Glasgow museum to uncover the missing slave.
John Glassford was just one of the 'tobacco lords' of the Chesapeake and not the only one with connections to Port Tobacco. I will see what I can dig up about the others and post about them as well.