To simplify things, let's say that a DC motor is being used as a generator.
In order to compute some physical quantities (like braking power), you'll need to create a mathematical model of this system. You don't need to create a perfect model. A rough approximation should suffice.
The description of the model:
- The speed of the rollercoaster is known.
- The system has no mechanical frictions (otherwise, you cannot compute everything without computational methods or advanced mathematics).
- A DC motor is coupled to the roller coaster's main shaft.
- The internal resistance of the DC motor is zero.
- The inductance of the DC motor is neglected.
- A DC motor's terminals are connected with an electrical resistance so that the roller coaster's kinetic energy is converted into heat caused by the current generated by the generator. If you don't connect anything to the DC motor, it won't generate braking.
Mathematics of the model:
Find a relationship between the roller coasters' speed ($v$) and the DC motor's rotation speed ($\omega$):
$$
v = N\omega
$$
Assume N is a constant number (it depends on your mechanical design).
The rotation speed of a DC motor is always linearly proportional to the voltage difference between its terminals ($V$):
$$
V = K_V \omega
$$
where $K_V$ is a constant that depends on the DC motor.
The braking torque generated by a DC motor ($T$) is always linearly proportional to the current flow through the DC motor ($i$):
$$
T=K_Ti
$$
where $K_T$ is a constant that depends on the DC motor.
Since the DC motor's terminals are connected with a resistance ($R$):
$$
V=iR
$$
Then, the braking power exerted by the motor is:
$$
\begin{align}
\text{Braking Power} &= T \omega \\
&= K_T i \omega \\
&=K_T \frac{V}{R} \omega \\
&=K_T \frac{K_V \omega}{R} \omega \\
& =\frac{K_T K_V v^2}{R N^2}
\end{align}
$$
So the braking power depends on the DC motor selection ($K_T$, $K_V$), the speed of the rollercoaster ($v$), the mechanical design to couple the DC motor to the rollercoaster's shaft ($N$), the electrical resistance connected to the DC motor ($R$).
With no mechanical friction assumption, you will find that 100% of the kinetic energy of the roller coaster can be recovered. So, calculating the recovered energy is straightforward (the difference between the kinetic energies of the roller coaster at two different times). Accounting for friction in these types of models is an undergraduate-level topic, and I don't think you should dive into this.
For references to all the equations above, check out this post: sinaatalay.com/posts/design-and-construction-of-a-dynamometer/