In a large bowl, using a hand mixer or stand mixer, beat butter and sugars on medium speed until creamy (about 3 minutes).
On low speed, add the eggs one at a time until combined, then beat on medium until light and fluffy (about 1 minute).
Add the vanilla extract and beat for 10-20 seconds, scraping down the side of the bowl as necessary to combine.
In a separate medium bowl, whisk the flours, baking powder, baking soda, cardamom, cinnamon, and salt together.
Add half the flour mixture to the egg mixture, mixing on low speed or using a spatula to mix until just combined, being careful not to over-mix. Add the milk and combine the same way. Repeat with the rest of the flour.
Scrape the batter over the figs in the prepared pan. Gently spread the batter to evenly cover the pan.
Bake for 40-45 minutes, or until a toothpick comes out clean. The cake will look dark brown in color. Cool for 10 minutes in the pan, then carefully run a knife around the inner edges of the cake pan. Using oven mitts, swiftly invert onto a plate for serving.