I suggest you decide according to how dirty the ductwork interior appears, how fast your filters fill with dust, or how dirty your house is from a dust standpoint.
I suggest that a relatively clean house, as your sounds, have the ductwork cleaned about every 5 years. If you have pets, dusty children or grand children, or like to have your windows open a lot for ventilation, then the ducts may be better cleaned every 2 or 3 years to maintain and less dusty environment.
This decision is also affected by residents of the home with asthma or respiratory ailments that may require maintaining a very clean house.