The usual slowdown factor involved by Homomorphic Encryption is roughly speaking 1.000-1.000.000x nowadays but the good news is that there are several way to minimize this slowdown thanks to hardware accelerators like GPU, FPGA or even custom board.
The second good news is that we are already working on hardware accelerations, and Concrete Numpy is designed to support these accelerators natively. You could hope in a near future to have state of the art performances for your program built with Concrete Numpy!