Understanding the errors
It is hard to come up with an algorithm to emulate "vintage" photos (if "vintage" is defined as "old mistreated negatives/photos" :-) ) as the result is multilayered and caused by various physical and chemical factors.
Understanding these factors will help you build the layers that you need in order to emulate the "vintage" look.
Light leak
On the old camera it happen from time to time that the back lid got open causing light to leak onto the negative. As the lid was opened from the side the typical leak appeared on and around the edges or as stripes. As the light exposure was relative long compare to typical intended exposure, the stripes often appeared white with a red blurred halo (on the positive).
Where the exact over-exposed stripes would appear would be random and depending on camera and ambient light conditions.
Additionally, if light leaked onto a film it will often result in a foggy looking image. This because the light already exposed the film and will create a gray color. This is them mixed with the intended exposure on top.
Film grain
Analog film has silver grains in them. The higher ISO the bigger the size of the grain - in fact ISO and grain size (filtered through a mask) are directly related. Typical ISO values for consumer films would be 200 to 400 ISO. In addition, chemical quality in the three steps that are used in laboratory development also affected the quality, not only when it came to grains but to contrast and color balance as well.
Also the 135-film was not the only typical size in the past. You had smaller film sizes and due to this the developed positives appeared more grainy at the same ISO value (grain size).
Chemicals
The chemicals are important in development, quality (non-saturated) the temperature and the time. If a laboratory forgot to replenish the chemicals the result would be poor contrast and color hue (in commercial labs time and temperature is automatically taken care of, but chemicals needs to be manually replaced).
Sometimes people manipulated temperature and chemicals to deliberately create certain effects such as bleaching, solarizing, adjusting tones (saturation, color balance, hue) etc.
Exposure
For really old cameras/film uneven exposure was a noticeable factor. You could have a "gradient" of various exposure on the film resulting it looking lighter at the top and darker at the bottom, or vica versa.
On the more modern cameras, where most settings where manual, if was easy to over- or under-expose a film. In development of positives the same could happen if the lab operator made a wrong adjustment or the chemicals where not working any more.
Optics
Optics plays a role especially with vignetting where you have a lighter center and darker corners. Poor optics and housing was more prone to produce vignette (personally I am a fan of vignette and often adds it to my own photos).
On many older automatic cameras the lens where fixed to "infinite" meaning anything closer than about 60 cm would appear out-of-focus - anything else as sharp as the lens allowed (plastic lenses became more or more usual at this time and had lower quality than the glass based lenses).
Photo paper
Another factor is the quality of the photo paper itself as well as the chemical process used to develop a positive.
Depending on how papers where constructed, grain size and what chemicals would be used you would get various results from raster-ish rough-grained images to super-sharp glossy ones. If both the positive (paper) and the negative (or dias film) used high ISO values the result would often be blurry and grainy - much like when you enlarge a low-resolution digital image to a much higher size.
Analog versus digital
It is also important to understand how analog film works versus digital.
In essence silver grains are unevenly distributed contrary to digital images where pixels are strictly ordered.
Over- and under-exposed films can still contain details whereas with digital these would be removed. This is related to dynamic range of the film versus the AD converter for digital.
Color representation
Colors are also different depending on film/paper emulsion as well as the chemical process. There is also specialized processes such as bleeching which can affect the end colors.
In the digital domain the equivalent would be LUT (Look-Up Tables) which can use a wider source range using 1D or 3D LUT to modify the color to give a certain look ("film", "cinematic", emulating bleech process and so forth).
Misc
And of course, there are other factors but I'll focus on these as they are typically considered part of the "vintage" style (vintage is not a correct term IMO as it refers to old style - not so much errors that was typical with analog cameras :-), but I'll use it here).
Putting it all together
So how can we use this knowledge? We can separate the different steps into layers of errors.
For example - for light leaks we now know why they appear and where they will appear. We might not need an algorithm per se for this as we can use overlays to create the same effect. As light leak technically is part of a multi-exposure we can mix a "blank" light leak onto an image:
For example - if we have two empty frames that is exposed to light this way we could use images such as these to mix onto our main image:
To produce these your self just find an old camera open the back lid, close, spool to next frame, repeat and then get it developed (or scanned). This will produce the most natural looking result.
Film grain can be produced by introducing blur and noise such as Gaussian. This will however not really look like grains but it is a simple way of doing it.
Alternatively also here you can use real grainy looking frames that you mix onto the image. For the best result you would mix in "grain clusters" (take the color of the current grain to map area of that grain, mix the colors of the image in the same area to an average color and fill the grain area with it). This is however a bit more complex and computation heavy.
The following examples comes with dust and hair as well which are also typical when dealing with analog films. They are not related to grains however, but you could combine them in this layer of the process:
Chemical process can be emulated by de-saturation and provide hue bias (or simply adjust color balance).
For color balance it's an advantage to understand color theory for additive colors (CMYK colors, ie. magenta versus green, yellow versus blue, red versus cyan etc.).
But we could convert RGB to HSL/HSV color space instead of going by CMYK, and change the hue degree and adjust saturation value, then convert back to RGB - or simply just try to adjust the color balance directly in RGB as if we where dealing directly with the negative film.
Under color adjustment comes also contrast adjustment. The more foggy is achieved by reducing the contrast.
Assembling
To assemble all these layers you could probably get away from iterating some layers by simply adjusting alpha level when you draw the overlays - for example for light leak, replace black with transparency and just mix it on top of the image. For grains however you would need to use multiply mode with adjustable bias for either the grains or the original image.
And of course for altering colors you would need to iterate the pixels.
You could make a compromise and combine the following layers into a single layer:
- Film grain
- Vignette
- Light leak
If you do notice that the grain layer would appear as a gray overlay instead of mixing well with the pixels. This can work in case you want to make the image appear more foggy, but you probably need to increase lightness afterwards.
Order is important and must be considered as well:
- First do the color adjustment
- Reduce contrast
- Blur (consider exaggerated edge blur to simulate "infinite" focus)
- Add the layers (or the combined grains, vignette, light leak)
- Consider adjusting lightness
Conclusion
Although, so far only theory.
I did not provide a working example as this would actually result in a rather complete solution which is not the purpose of SO of course.
I also tried to limit the answer to the basics as it could quickly become very complex if all where to be generated using algorithms-only as you would need several of them, and therefor being too broad to answer.
In any case, here are some resources that can help you with the various steps:
Resources
