A General Inverse Problem for Aggregation