Using linked servers is used when we want the link to last for the entire session. This was the only method long time back.
OPENROWSET can be used as part of the SQL statement to connect to a table on a different server and this can be any server (non mssql also) that can be connected though an OLEDB connection. Going through the documentation would help. This is the oft-used way.