Hilary Duff explains her friendship with ex Joel Madden and his wife Nicole Richie: ‘We actually hang out all the time’