I have always understood that just 'having' a home in another state is not good enough. Your home is where your primary residence is/where you live most of the time and where you have your drivers license, ccp, etc.Just to cut to the chase, the language is such that a trade need not require the involvement of an FFL provided that both parties undertake the transfer of ownership in the state in which they both have residency.
What constitutes residency in a state? | Bureau of Alcohol, Tobacco, Firearms and Explosives
For Gun Control Act (GCA) purposes, a person is a resident of a state in which he or she is present with the intention of making a home in that state. The state of residence for a corporation or other business entity is the state where it maintains a place of business. A member of the Armed...www.atf.gov
So, the seller just visiting the buyer's state (unless he has a second residence in that state too) would be a no.