Use the old soil, put it loosely in the hole and ram compact it with the end of a 2×4. This allows the soil to be compacted through the hole with an inexpensive tool. The end of the 2×4 can really pack the soil. If you have difficulty ramming the 2×4, use a sledge hammer to hit the end of the 2×4 board.
After you are finished compacting the soil, then soak water into the hole to wet the new dirt. Using gravel in the hole will create a sump and collect water in the gravel cavity below the patio. Dirt is the better answer.