BEAT pudding mix and milk in medium bowl with whisk 2 min. Stir in COOL WHIP and zest.
ARRANGE half the ladyfingers, rounded-sides down, in 8-or 9-inch square baking dish; drizzle with 2 Tbsp. orange juice. Cover with half the pudding mixture. Repeat. Refrigerate 2 hours.
TOP with oranges, nuts and nutmeg just before serving.