Destiny is a fixed path to travel. That is from point A to B.How we travel makes for free will if you use your mind.But you will reach any which way from A to B.The destination is not your choice, that is point B. But how to reach and the path taken is your choice hence its called free will. This only reduces your prarabdha karma. But since you used your mind you have accumulated new or Agami karma.This gets accumulated in the cache of Sanchit. Destiny does not allow you to mostly follow your swadharma if done with too many other things. Swadharma is not a choice but an underlined fixed path which literally all sway away from.
In executing the destiny, you normally take a lot of calls or use the mind which takes you away from the fixed path.
Going away from fixed path means you have used your mind.
Lets take an example.
You have to reach Dadar from Thane and to come to my place you have to follow a fixed path.
But you use your mind and say I shall go to Ghatkopar and finish this work, then go and pick up a laptop at Bandra, then have lunch at Mahim and finally at latest 3 I shall reach.
Now see how the utilitarian mind has got used to do this simple job, cramming in lots of other things.
In your world you call it multi tasking but see how a sage will do it?
He is focused, works slowly, steadily, single pointedly, focuses only in reaching and gets there without much rajasik action. You see in your own world how you work.
Have you not done too many other things which have accumulated to many newer karmas.
Hence you sway away from your true swadharma.
Lets see if the swadharma of a person is to be a teacher in school, then he takes tutions, writes books of accounts for part time job, takes up social duties, etc etc…and this makes him sway away from being true to his swadharma.
Take the case of Brahmanas of olden times.
They would only do poojas, marriages, etc of people and earn from that but look at the same person today?
For them the action of being a priest is not important. He joins a bank or takes government job and does all other things and says there is no money in his pooja path! And does poojas etc part time!
When a person does his swadharma he does not accumulate newer karma.
He gets rid of past karmas.
He goes in the right path.
He meets his Guru and then he takes him further in spiritual.

Photo credit: morguefile.com