A V100 GPU gets 900GB/s of memory bandwidth. I am less of a CPU expert but afaict you'll be lucky to get much more than 10% of that out of a CPU. This is going to make a huge difference that Intel can't make up with bigger execution units.
bfloat helps with this because the data is half as large. But of course if you're doing Nvidia you're probably already doing (IEEE) fp16.
Correct, GPUs can do SGD iterations much much faster once the data is in GPU RAM. But if you have a ton of data relative to compute, you might be better off with a shallower model and/or fewer iterations. Here’s one start-up serving these sorts of projects: https://www.memverge.com/
When GPUs kickstarted deep learning research in 2012, people had already studied shallow models on mapreduce for a decade or so. Once NVME / Optane and modern CPUs get 1-10TB of useful “memory” in the hands of grad students, there should be another wave of new research. To date, my experience has been that 1TB of “memory” is only commonly available in industry.
Is that intel ARK published spec sheet bandwidth, or actual usable bandwidth? There is a difference.
I've found I get about 75-80% of the advertised bandwidth both from my real app (TLS crypto) and a toy memory copy benchmark using AVX256 instructions. The toy memory copy benchmark is how I realized that my bottleneck was actually memory bandwidth and not CPU horsepower on Broadwell based servers.
I haven't a clue. I just got it off the link quoted. It's a good question and when there's a difference, you know what marketing will say.
To make a stab, I suppose it might depend on whether all requests are coming from a single memory bank or spread evenly across all memory banks, assuming fully populated (again from the link "Octa-channel (up from hexa-channel)")
Intel is dominated by how many cores/threads are accessing simultaneously. So with many scientific libraries you can get about 80% of the throughput. And the V100 will not give you 900GBps. That's the theoretical, but nominally it's about 750GBps.
bfloat helps with this because the data is half as large. But of course if you're doing Nvidia you're probably already doing (IEEE) fp16.