Let's assume good reciprocating and rotational balance, mechanical issues not subject to inflight control. The two smoothness factors which may be influenced by the pilot are cycle-to-cycle variation and resonant vibration.
Not every cylinder charge lights and burns at the same rate. There are multiple factors at work here. The mixed fuel and air presented to the spark plug at the desired moment of ignition may not be homogeneous, being either too rich or too lean in the immediate vicinity of the spark, so initial inflammation may be slow. The overall mixture may be very lean, both more difficult to light and slower to burn. The spark duration lasts for a finite time, through some range of crankshaft rotation, and may achieve ignition at some point later than the planned initial spark discharge. And of course sometimes a cylinder fails to light at all, a misfire. Regardless of root cause, peak pressure and the point of peak pressure vary from optimum, and cylinders make an uneven contribution to torque. We call it roughness, and fix it with the mixture knob.
Resonant vibration happens when a periodic driving force is applied to an object, and the period frequency is at or near the object's natural frequency. When the driving frequency is within a range of (caution, rough example) 0.9 to 1.1 times the natural frequency, the object vibrates powerfully, with the peak amplitude happening at Fd = Fn. In theory, amplitude can reach infinity, and doesn't only because the world is full of damping factors.
Probably the most significant pairing in the context of "engine sweet spot" is one in which the prop blades are the resonating victim. It's the basis of the "Don't Operate Here" RPM and MP combinations.
Less appreciated is the fact that our airplanes are full of structures, all of which have a natural frequency. Any of them with a natural frequency which happens to match one of the multiple driving frequencies emanating from the flailing and whirling parts up front may vibrate, some with considerable amplitude. Here's the takeaway; it is possible to have a very smooth running engine, and still have an irritating vibration notable in the cockpit. Although the powerplant is vibrating at low amplitude, the magic of resonant amplification can result in an airframe component with high amplitude vibes.
The pilot's perception, based on what he feels, can be misleading. He may make a power change (notably RPM in this case) until the vibe subsides, when the real cure may be to add some damping to an airframe component, or change its natural frequency.
Steve, Dayton has dual P-mags that have been tuned with his EIC on his O-360. Same setup as I have, only he has a CS prop and I have a FP prop. The P-mags will allow the O-360 to run LoP very smoothly.
Two points please.
Tom and Steve mentioned Deakin articles. Advanced timing for LOP is not within that school of operating thought.
In the fixed timing approach, peak pressure is purposely moved further from TDC by slowing combustion speed with lean mixture, then an inch of MP is added to compensate. Easy to do in your turbocharged whatever.
With variable timing, we add advance, moving peak pressure back to where it was at best power mixture, ballpark being 15 ATDC.
With the former, peak pressure is lower near TDC and a bit higher near exhaust valve opening. With the latter, peak pressure is high near TDC and lower at valve opening. Nothing wrong with either approach, but they are quite different. You can see it on the gauges. Fixed timing slow burn is low CHT and higher EGT. Advanced timing to compensate for slow burn is high CHT and lower EGT.
Second, timing advance is not required to run smoothly when LOP. The ability to light a lean mixture is a function of healthy spark across a wide plug gap, not timing. It can be done with mags and narrow gaps, but in our world, it means an electronic ignition. However, we should not conflate the typical EI's two entirely separate features, (a) high spark energy and (b) spark timing.
Break
Dayton mentioned 8500 alt, 2500 rpm 20" MP and 9.5gph. That should be around peak, rather than significantly LOP. Nothing wrong with that either.