The ability to gain muscle while losing fat is dependent on the relationship between your fatness and muscularity. An overly-fat and under-trained person will be able to achieve simultaneous fat loss and muscle gain. A very lean person near his/her limit for muscle growth will not. As one moves away from the overly-fat, under-trained status towards a leaner, more muscular body this ability is diminished. At some point, the vast majority of people will see better/quicker results by choosing to do one or the other (gain muscle, lose fat – often referred to as ‘bulking’ and ‘cutting’, respectively) at a time. For a highly general rule of thumb: if you have been training effectively for a year or more, you’re better off with a bulk/cut cycle. See Lyle McDonald’s Adding Muscle While Losing Fat – Q&A for more discussion.
A 2011 paper suggests a weekly rate of body-weight loss of 0.7% can permit muscle gain in both men and women while losing fat mass. Women were also able to increase muscle while losing fat at a weekly rate of body-weight loss of 1.0-1.4%. Men, however, lost muscle mass at this increased rate.
In real terms, 0.7% body-weight loss was achieved via a daily deficit of 3.2 calories per lb of total bodyweight (or 7 calories per kg of total bodyweight). At this rate, your goal calorie intake = TDEE – [BWlb x 3.2kcal] (or TDEE – [BWkg x 7kcal]). This is a good place to start. The 1.0-1.4% range works out to a daily deficit of 4.5-6.4 calories per lb of total bodyweight (or 10-14 calories per kg of total bodyweight).