According to docker's documentation, "Because the Docker daemon uses Linux-specific kernel features, you can’t run Docker natively in OS X.Instead, you must use docker-machine to create and attach to a virtual machine (VM). This machine is a Linux VM that hosts Docker for you on your Mac."
If you have a fileshare set up between your host and guest, you may experience performance slowness. In the case of drupal, if your project files are on your host machine, and you mounted it using "Docker's -v host/folder:container:/folder ".
The site you spin up can be significant slower due to vboxsf. It is said that NFS can be faster, but it seems there's an overhead for set this up in virtualbox.
Until there's a better and easy set up solution, to improve the site performance, one simple thing could do is to serve those files locally on the guest machine.
There have been some talks about this issues in the docker community. Some solutions are suggested: