If a spellcaster is riding a flying creature with flyby (peryton, for example), can the caster deliver a touch spell to a target along the flyby path and then get out without provoking an opportunity attack? Can I use my action in the middle of my mounts flyby action, or does the mount's movement have to be finished before I use my action?

Can you attack in the middle of your mount's movement? No.

You and your mount have separate turns on the same initiative, so you decide which one goes first, but you can't just freely act in the middle of your mount's turn.
It isn't your turn. This was clarified by Jeremy Crawford in his twitter feed a couple of years ago.

You could ready an action to make your attack when you get close enough, but that adds some limitations:

• Triggering a ready action is a reaction, so you can't do this and then, say, cast shield when the guy you just hit decides to retaliate, or use an opportunity attack of your own.




• Readying a spell requires concentration, even if the spell you're holding normally wouldn't, which puts a pretty significant extra limitation on you as a spellcaster: You can't ready a touch spell attack while maintaining a concentration spell at the same time.




• You can't use a bonus action while you're near the target, because your bonus action has to happen on your turn rather than your mount's. This is less of an issue for a caster using a touch spell, but some classes/items/feats/etc. (and dual-wielding) give various 'attack as a bonus action' abilities that couldn't be used during a "ride-by stabbing" like this. A sorcerer's quickened spell metamagic pops to mind as a relevant example.




• If your class lets you have multiple attacks as part of an attack action, afoot you can break up your movement between individual attack rolls (Player's Handbook p.190), but you can't do that if your attack action is a ready action, because your action interrupts the mount's movement rather than happening concurrently. (Or to put it another way, because it isn't your turn, you don't have any movement to break up.)

Does Flyby protect you from opportunity attacks? Yes.

Here are the relevant rules concepts:

1. 
   

   When you are mounted, your mount moves and you are taken with it; you aren't using your own movement.

   
   
   

   
   

   [Your mount] moves as you direct it, and it has only three action options; Dash, Disengage, and Dodge. A controlled mount can move and act even on the turn that you mount it. - Player's Handbook, p.198

   
   

   
   


2. 
   

   You only draw opportunity attacks when you use your own movement to leave an enemy's reach.

   
   
   

   
   

   You can make an opportunity attack when a hostile creature that you can see moves out of your reach... You also don't provoke an opportunity attack ... when someone or something moves you without using your movement... - Player's Handbook, p.195

   
   

   
   

Based on rules 1 and 2, you do not draw opportunity attacks while mounted; only your mount does. But...

3. A special rule relating to Mounted Combat says they can try to hit you anyway.
   
   

   
   

   If the mount provokes an opportunity attack while you're on it, the attacker can target you or the mount. - Player's Handbook, p.198

   
   

   
   

So, therefore, if your mount has an ability (such as flyby) or uses an action (such as Disengage) that prevents opportunity attacks against it, it also inherently prevents opportunity attacks against you, because you don't provoke on your own, and it isn't provoking any attacks that could be redirected to you.

Any mount can use Disengage, but Flyby is better because it allows you to use Dodge or Dash at the same time as not provoking attacks.

Yes it can, even without the Flyby feature

Can I use my action in the middle of my mounts flyby action

First and foremost, there is no specific "flyby action" the peryton has to take, it just doesn't provoke opportunity attacks. However, the mount itself acts on its own turn (more about it down below).

Now, let's see Monster Manual, page 251:

Flyby. The peryton doesn't provoke an opportunity attack when it flies out of an enemy's reach.

A peryton doesn't provoke opportunity attacks, neither does its rider. See Do I draw opportunity attacks when my mount uses flyby? for more details.

Considering casting a spell in-between — you allowed to do that, but only with your own movement (thanks to Darth Pseudonym for clarifying this):

You can break up your movement on your turn, using
some of your speed before and after your action. For
example, if you have a speed of 30 feet, you can move
10 feet, take your action, and then move 20 feel.
(PHB 190, "Breaking Up Your Move")

The mount acts on its own turn, so you probably want to Ready a spell beforehand to make this work.

The fun thing is — your mount doesn't even have to have the Flyby feature in order to evade opportunity attacks. Any controlled mount can take the Disengage action:

It moves as you direct it, and it has only three action options: Dash, Disengage, and Dodge.

Therefore, when it isn't busy dashing or dodging, it can disengage and be safe from opportunity attacks for the whole turn:

If you take the Disengage action, your movement doesn't provoke opportunity attacks for the rest of the turn.

That is one of the advantages of being mounted in combat.

First things first.

If your mount doesn't provoke Opportunity Attacks, neither do you

The mount is using its own movement, not yours, so irrespective of whether the mount is being directly controlled, or if it is being independently controlled, opportunity attacks against you are contingent on whether your mount can be targeted.

So there's nothing wrong with you flying down, delivering a touch spell, and then flying away...

... Unless you're letting your mount act independently. In that case, it's still possible, but you need to modify your Actions.

In this case, what you instead need to do is Ready the touch spell in question, and set the trigger condition to "When my mount brings me adjacent to the creature". Then your mount flies down to them during its own turn, and when it gets adjacent, you use your Reaction to deliver the spell. Then the mount flies away, and because it doesn't provoke Opportunity Attacks, you're not at risk either.