A .264 is a raw video file. Usually, the .264 video is in some kind of container that the player will recognize like avi or mkv. You should be able to play these raw video files, but you may need to change the extension. Rename the file extension to . h 264 and try to play in VLC. That should work.
Looks like Handbrake has CUDA support for H.264, just not [strike]HEVC. I don't think any free programs do yet. Honestly using Adobe Encoder with a watch folder setup is amazing. I had two 1080Ti chugging away at 2 years of archives converting from MPEG2 to HEVC. Only took a week, mostly due to it being 6TB and 1Gbps line to and from.
The different H.264 codecs are different programmers' implementations of the H.264 standard. The standard defines how the video stream must be represented in order that any compliant decoder can play it back - how the I-frames, motion vectors, etc. are to be stored and what they mean.
No, I think I am using only H.264 in mp4 and mkv container when transcoding (from m2ts source). AVI (non H.264 divx/xvid codec from the old days, not avi container with H.264) is pretty much dead to me. I think people doing transcoding is using more and more mp4 and mkv to avoid confusing the viewers (mkv/mp4 usually associated with H.254) vs. (avi, usually divx/xvid). Many people don't know ...
2 - For those DVD movies you'd love to enjoy on your smart TV or tablets for car trips, you're suggested to convert them to MP4 (H.264) video files. WinX the ripper enables you to control the quality and size of it when choosing target format.