The White House Historical Association tells us that although John Adams was the first president to occupy the executive mansion, it was Thomas Jefferson who established the traditions of a July 4th celebration at the White House or President’s House as it was called in his time. Jefferson opened the house and greeted the people along with diplomats, civil and military officers, and Cherokee chiefs in the center of the oval saloon under Gilbert Stuart’s famous portrait of George Washington. Jefferson also added music to the celebration. The Marine Band, already "The President’s Own," played in the Entrance Hall performing "The President’s March" and other "patriotic airs."
The north grounds of the President’s Park—the "common"—came alive at daybreak with the raising of tents and booths, soon followed by crowds of people. A festival took place just for the day. Food and drink and cottage goods of all types were sold. There were horse races and cockfights and parades of the Washington Militia and other military companies. A bare headed Jefferson with his "grey locks waving in the air" watched from the steps of the White House. Then he invited everyone in to partake of his hospitality and his thanksgiving for the preservation of independence.
An Account of July Fourth at the President’s House, 1801, from a letter from Mrs. Smith to her sister Mary Ann Smith: "About 12 o'clock yesterday, the citizens of Washington and Geo. Town waited upon the President to make their devoirs. I accompanied Mr. Sumpter (?). We found about 20 persons present in a room where sat Mr. J. surrounded by the five Cherokee chiefs. After a conversation of a few minutes, he invited his company into the usual dining room, whose four large sideboards were covered with refreshments, such as cakes of various kinds, wine, punch, &c. Every citizen was invited to partake, as his taste dictated, of them, and the invitation was most cheerfully accepted, and the consequent duties discharged with alacrity. The company soon increased to near a hundred, including all the public officers and most of the respectable citizens, and strangers of distinction. Martial music soon announced the approach of the marine corps of Capt. Burrows, who in due military form saluted the President, accompanied by the President's March played by an excellent hand attached to the corps. After undergoing various military evolutions, the company returned to the dining room, and the hand from an adjacent room played a succession of fine patriotic airs. All appeared to be cheerful, all happy. Mr. Jefferson mingled promiscuously with the citizens."
Source: Margaret Bayard Smith, The First Forty Years of Washington Society, ed. Galliard Hunt (New York: Scribner’s, 1906), 30.