Well actually I realized I was updating the light value every frame which is costly since I check all neighbors of the voxel. After I did that I was able to get a lot more on screen and animated properly.
But any ideas are welcome as well to what I could do to speed this up.