If you consider that for a signal
s(t) = sin(phi(t))
the frequency is
f(t) = 1/(2pi) d/dt phi(t)
then given a desired frequency function f(t),
you construct first a phase function
phi(t) = 2pi \int_0 ^t f(x) dx
which you use in your phasor: sin(x), cos(x), exp(ix)
Example:
f(t) = f0 + (f1-f0) * t # linear FM
phi(t) = 2pi * (f0 * t +(f1-f0) * t^2) = 2pi * (f0+(f1-f0)/2 * t) * t)
Note: the integration explains the factor 2 in (f1-f0)/2
Edit: I did a small Matlab script to visualize better what the answer is about.
%signal generation
%
% sketch frequency
u=[0.1,1000
0.2,4000
0.3,1500
0.4,5000];
%
% interpolate
fs=24000;
T=(0:1/fs:0.5)';
v=interp1(u(:,1),u(:,2),T,'pchip');
%
% truncate if necessary
isel=(u(1,1)<=T) & (T<= u(end,1));
v=v(isel);
t=T(isel);
%
% integrate to obtain phase
om=2*pi*cumtrapz(v)/fs;
%
% generate amplitude function (here attenuate initial/final
% transient
aa=ones(size(t));
i1=(1:0.1*fs)';
aa(i1)=aa(i1).*exp(-1/2*((i1-i1(end))/(0.03*fs)).^2);
i2=length(t)+1-fliplr(1:0.1*fs)';
aa(i2)=aa(i2).*exp(-1/2*((i2-i2(1))/(0.03*fs)).^2);
%
% generate signal
s=aa.*sin(om);
%
% for visualisation add leading and trailing zeroes
s=[zeros(0.1*fs,1);s;zeros(0.1*fs,1)];
%
% for visualization add noise
s=s+0.01*randn(size(s));
%%
%visualize
figure(1)
subplot(211)
plot(u(:,1),u(:,2),'o',t,v)
subplot(212)
spectrogram(s,hann(256),128,1024,fs,'yaxis');
line(t*1000,v/1000,'color','k')
which generates

To address one of Dan Stowell concerns I added a amplitude weighting function that removes initial and final transients.
Edit: to understand if simulated signal shows unwanted effects, it is best to inspect spectrogram. I did this in example to detect and remove initial and final transients. Readers with access to Matlab are invited to experiment with script.